Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
path: root/docs
diff options
context:
space:
mode:
authorLeitwolf112026-11-01 14:23:09 +0300
committerLeitwolf112026-11-01 14:23:09 +0300
commit1db203d56b7c71db3d80b477da1ad49e9ae46468 (patch)
tree48cd538e23e40ebdaae034a864082de30f68364f /docs
parent1559307a6ac7731ffd6a8e28147552b3987dbd93 (diff)
downloadcatch3-1db203d56b7c71db3d80b477da1ad49e9ae46468.tar.gz
Import the results of other test frameworks
Tests Catch3 does not run had no history, flaky detection or quarantine. --import=FILE reads a report instead of running tests and sends it through the analysis, history and reports of a run: JUnit with Surefire's reruns, xUnit.net v2, TRX, TAP 12 to 14, CTRF, Catch3's own --json report and catch3-results, a small JSON Lines format for tools with none of these. Results that share an identifier are attempts at one test, so a report of a rerun gives a flaky test. --import-revision says what was tested: a verdict that changes on one revision also counts as flaky. --import-name keeps the results in a history of their own, which --analytics reads too. A TAP stream that ends early fails instead of reading as a run that passed. The amalgamation script now accepts a relative include of the third-party wrappers, because pugixml is included from a second directory.
Diffstat (limited to 'docs')
-rw-r--r--docs/progmasoft-roadmap.md2
-rw-r--r--docs/progmasoft-runner.md64
-rw-r--r--docs/release-notes.md11
3 files changed, 77 insertions, 0 deletions
diff --git a/docs/progmasoft-roadmap.md b/docs/progmasoft-roadmap.md
index f376e9a0..5ab04264 100644
--- a/docs/progmasoft-roadmap.md
+++ b/docs/progmasoft-roadmap.md
@@ -48,6 +48,8 @@ Catch3 runner replaced Catch2's in every test binary:
- every failure gets a category, a signature and an analysis;
- reports are console, JUnit XML with flaky and rerun elements, JSON and a
trace; the discovery listing has a fixed, versioned format;
+- reports of other frameworks are imported, so their tests get the same
+ history, flaky detection, quarantine and analytics;
- every test gets a temporary directory of its own;
- its command line is parsed by a hand-written, typed parser, like the other
Progmasoft tools.
diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md
index 300c541e..486c719f 100644
--- a/docs/progmasoft-runner.md
+++ b/docs/progmasoft-runner.md
@@ -5,6 +5,7 @@
[How a run works](#how-a-run-works)<br>
[Retries, flaky tests and order dependence](#retries-flaky-tests-and-order-dependence)<br>
[History, quarantine and analytics](#history-quarantine-and-analytics)<br>
+[Importing the results of other frameworks](#importing-the-results-of-other-frameworks)<br>
[Choosing and ordering tests](#choosing-and-ordering-tests)<br>
[Tags the runner reads](#tags-the-runner-reads)<br>
[Time limits](#time-limits)<br>
@@ -133,6 +134,67 @@ Under `bazel test` there is no history unless `CATCH3_HISTORY` names a
directory that outlives the sandbox.
+## Importing the results of other frameworks
+
+`--import=FILE` reads the report of another test framework instead of running
+tests, so that tests Catch3 does not run get the same history, flaky detection,
+quarantine and analytics. Any Catch3 test binary can do it, and the option can
+be repeated:
+
+```sh
+./tests --import=results/pytest.xml --import=results/vitest.ctrf.json \
+ --import-name=web --junit=merged.xml
+```
+
+| `--import-format=` | Reads |
+| --- | --- |
+| `auto` | whichever the contents show (the default) |
+| `catch3` | a `--json` report of Catch3, or [`catch3-results`](#the-catch3-results-format) |
+| `junit` | JUnit XML, with Surefire's reruns; Gradle, pytest, Jest, GoogleTest and most others write it |
+| `xunit` | xUnit.net v2 XML |
+| `trx` | Visual Studio's TRX, with data-driven rows |
+| `tap` | TAP 12, 13 and 14, with subtests and YAML diagnostics |
+| `ctrf` | [CTRF](https://ctrf.io) JSON, with retries |
+
+The results go into a history of their own, named by `--import-name` and
+`imported` by default; `--analytics --import-name=NAME` reads it. The console,
+`--junit`, `--json`, `--trace`, `--quarantine`, `--history` and the exit code
+work as they do for a run. Reports that were read count as one run.
+
+A test is identified by its class and its name. Results with the same
+identifier, in one report or in several, are attempts at one test: a test that
+both failed and passed is flaky, passes and is remembered as flaky, which is how
+a report of a rerun is imported. TAP has no retries, so a name that comes back
+in one stream is another test and gets a number.
+
+A verdict that changes on one revision also counts as flaky.
+`--import-revision=REVISION` names what the reports tested; it defaults to the
+Git commit of the working directory.
+
+Nothing is run, so an imported failure has no replay command, and only what the
+report says is known: it has no order, seed or sections. A TAP stream that bails
+out or reports fewer tests than it planned adds a failed test called
+`the TAP stream`, so that it does not read as a run that passed.
+
+### The catch3-results format
+
+For a framework without a report of these kinds, `catch3-results` is the
+smallest thing to write: JSON Lines with one test per line, after a line that
+names the format.
+
+```json
+{"format": "catch3-results", "version": 1}
+{"name": "parses a date", "className": "Dates", "outcome": "passed", "durationMicroseconds": 1200}
+{"name": "rounds down", "outcome": "failed", "message": "1.5 != 2", "details": "at Dates.rounds:12"}
+```
+
+`name` or `id`, and `outcome`, are required; `outcome` is a name from the JSON
+report, such as `passed`, `failed`, `skipped`, `crashed` or `timed-out`. The other
+members are `id`, `className`, `file`, `line`, `tags`, `category`, `message`,
+`details`, `standardOutput` and `standardError`. A line can give `attempts`
+instead, as the JSON report does.
+
+
## Choosing and ordering tests
Test specs work as in Catch2: names with `*` as a wildcard, `[tags]`, `~`
@@ -265,6 +327,8 @@ version.
`quarantined`, `resources`, `dependsOn`, `timeoutSeconds` and `retries`.
CMake's `catch3_discover_tests` reads it.
* **Run report**, `--json`: `"format": "catch3-run"`, version 1.
+* **Imported results**, `--import`: `"format": "catch3-results"`, version 1;
+ see [its format](#the-catch3-results-format).
* **Analytics**, `--analytics=json`: `"format": "catch3-analytics"`,
version 1.
* **History**: `"format": "catch3-history"`, version 1. It is the runner's
diff --git a/docs/release-notes.md b/docs/release-notes.md
index f44d1369..e1567d97 100644
--- a/docs/release-notes.md
+++ b/docs/release-notes.md
@@ -7,11 +7,22 @@ based on Catch2 3.16.1; Catch2's own history is in
[its release notes](https://github.com/catchorg/Catch2/blob/devel/docs/release-notes.md).
**Contents**<br>
+[Unreleased](#unreleased)<br>
[0.3.0](#030)<br>
[0.2.0](#020)<br>
[0.1.0](#010)<br>
+## Unreleased
+
+### The Catch3 runner
+
+* `--import=FILE` reads the reports of other test frameworks instead of running
+ tests: JUnit, xUnit, TRX, TAP, CTRF, Catch3's own, and a small JSON Lines
+ format for anything else. Their tests get the history, flaky detection,
+ quarantine, analytics and reports of a run.
+
+
## 0.3.0
Released 2026-11-01.