diff options
| author | Leitwolf11 | 2026-11-01 14:23:09 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-11-01 14:23:09 +0300 |
| commit | 1db203d56b7c71db3d80b477da1ad49e9ae46468 (patch) | |
| tree | 48cd538e23e40ebdaae034a864082de30f68364f /docs | |
| parent | 1559307a6ac7731ffd6a8e28147552b3987dbd93 (diff) | |
| download | catch3-1db203d56b7c71db3d80b477da1ad49e9ae46468.tar.gz | |
Import the results of other test frameworks
Tests Catch3 does not run had no history, flaky detection or quarantine.
--import=FILE reads a report instead of running tests and sends it through
the analysis, history and reports of a run: JUnit with Surefire's reruns,
xUnit.net v2, TRX, TAP 12 to 14, CTRF, Catch3's own --json report and
catch3-results, a small JSON Lines format for tools with none of these.
Results that share an identifier are attempts at one test, so a report of a
rerun gives a flaky test. --import-revision says what was tested: a verdict
that changes on one revision also counts as flaky. --import-name keeps the
results in a history of their own, which --analytics reads too. A TAP stream
that ends early fails instead of reading as a run that passed.
The amalgamation script now accepts a relative include of the third-party
wrappers, because pugixml is included from a second directory.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/progmasoft-roadmap.md | 2 | ||||
| -rw-r--r-- | docs/progmasoft-runner.md | 64 | ||||
| -rw-r--r-- | docs/release-notes.md | 11 |
3 files changed, 77 insertions, 0 deletions
diff --git a/docs/progmasoft-roadmap.md b/docs/progmasoft-roadmap.md index f376e9a0..5ab04264 100644 --- a/docs/progmasoft-roadmap.md +++ b/docs/progmasoft-roadmap.md @@ -48,6 +48,8 @@ Catch3 runner replaced Catch2's in every test binary: - every failure gets a category, a signature and an analysis; - reports are console, JUnit XML with flaky and rerun elements, JSON and a trace; the discovery listing has a fixed, versioned format; +- reports of other frameworks are imported, so their tests get the same + history, flaky detection, quarantine and analytics; - every test gets a temporary directory of its own; - its command line is parsed by a hand-written, typed parser, like the other Progmasoft tools. diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md index 300c541e..486c719f 100644 --- a/docs/progmasoft-runner.md +++ b/docs/progmasoft-runner.md @@ -5,6 +5,7 @@ [How a run works](#how-a-run-works)<br> [Retries, flaky tests and order dependence](#retries-flaky-tests-and-order-dependence)<br> [History, quarantine and analytics](#history-quarantine-and-analytics)<br> +[Importing the results of other frameworks](#importing-the-results-of-other-frameworks)<br> [Choosing and ordering tests](#choosing-and-ordering-tests)<br> [Tags the runner reads](#tags-the-runner-reads)<br> [Time limits](#time-limits)<br> @@ -133,6 +134,67 @@ Under `bazel test` there is no history unless `CATCH3_HISTORY` names a directory that outlives the sandbox. +## Importing the results of other frameworks + +`--import=FILE` reads the report of another test framework instead of running +tests, so that tests Catch3 does not run get the same history, flaky detection, +quarantine and analytics. Any Catch3 test binary can do it, and the option can +be repeated: + +```sh +./tests --import=results/pytest.xml --import=results/vitest.ctrf.json \ + --import-name=web --junit=merged.xml +``` + +| `--import-format=` | Reads | +| --- | --- | +| `auto` | whichever the contents show (the default) | +| `catch3` | a `--json` report of Catch3, or [`catch3-results`](#the-catch3-results-format) | +| `junit` | JUnit XML, with Surefire's reruns; Gradle, pytest, Jest, GoogleTest and most others write it | +| `xunit` | xUnit.net v2 XML | +| `trx` | Visual Studio's TRX, with data-driven rows | +| `tap` | TAP 12, 13 and 14, with subtests and YAML diagnostics | +| `ctrf` | [CTRF](https://ctrf.io) JSON, with retries | + +The results go into a history of their own, named by `--import-name` and +`imported` by default; `--analytics --import-name=NAME` reads it. The console, +`--junit`, `--json`, `--trace`, `--quarantine`, `--history` and the exit code +work as they do for a run. Reports that were read count as one run. + +A test is identified by its class and its name. Results with the same +identifier, in one report or in several, are attempts at one test: a test that +both failed and passed is flaky, passes and is remembered as flaky, which is how +a report of a rerun is imported. TAP has no retries, so a name that comes back +in one stream is another test and gets a number. + +A verdict that changes on one revision also counts as flaky. +`--import-revision=REVISION` names what the reports tested; it defaults to the +Git commit of the working directory. + +Nothing is run, so an imported failure has no replay command, and only what the +report says is known: it has no order, seed or sections. A TAP stream that bails +out or reports fewer tests than it planned adds a failed test called +`the TAP stream`, so that it does not read as a run that passed. + +### The catch3-results format + +For a framework without a report of these kinds, `catch3-results` is the +smallest thing to write: JSON Lines with one test per line, after a line that +names the format. + +```json +{"format": "catch3-results", "version": 1} +{"name": "parses a date", "className": "Dates", "outcome": "passed", "durationMicroseconds": 1200} +{"name": "rounds down", "outcome": "failed", "message": "1.5 != 2", "details": "at Dates.rounds:12"} +``` + +`name` or `id`, and `outcome`, are required; `outcome` is a name from the JSON +report, such as `passed`, `failed`, `skipped`, `crashed` or `timed-out`. The other +members are `id`, `className`, `file`, `line`, `tags`, `category`, `message`, +`details`, `standardOutput` and `standardError`. A line can give `attempts` +instead, as the JSON report does. + + ## Choosing and ordering tests Test specs work as in Catch2: names with `*` as a wildcard, `[tags]`, `~` @@ -265,6 +327,8 @@ version. `quarantined`, `resources`, `dependsOn`, `timeoutSeconds` and `retries`. CMake's `catch3_discover_tests` reads it. * **Run report**, `--json`: `"format": "catch3-run"`, version 1. +* **Imported results**, `--import`: `"format": "catch3-results"`, version 1; + see [its format](#the-catch3-results-format). * **Analytics**, `--analytics=json`: `"format": "catch3-analytics"`, version 1. * **History**: `"format": "catch3-history"`, version 1. It is the runner's diff --git a/docs/release-notes.md b/docs/release-notes.md index f44d1369..e1567d97 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -7,11 +7,22 @@ based on Catch2 3.16.1; Catch2's own history is in [its release notes](https://github.com/catchorg/Catch2/blob/devel/docs/release-notes.md). **Contents**<br> +[Unreleased](#unreleased)<br> [0.3.0](#030)<br> [0.2.0](#020)<br> [0.1.0](#010)<br> +## Unreleased + +### The Catch3 runner + +* `--import=FILE` reads the reports of other test frameworks instead of running + tests: JUnit, xUnit, TRX, TAP, CTRF, Catch3's own, and a small JSON Lines + format for anything else. Their tests get the history, flaky detection, + quarantine, analytics and reports of a run. + + ## 0.3.0 Released 2026-11-01. |