Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
path: root/docs
diff options
context:
space:
mode:
authorLeitwolf112026-11-01 14:48:55 +0300
committerLeitwolf112026-11-01 14:48:55 +0300
commit569620dad14df7e13bb9fd31d256e9fd873ef6ec (patch)
treef933300c29658c3eb89d126d74b6c30fdff9dd04 /docs
parent1db203d56b7c71db3d80b477da1ad49e9ae46468 (diff)
downloadcatch3-feature/result-import.tar.gz
Keep real failures visible when importingfeature/result-import
A review of the import found ways for a failure to disappear or for the import to reach too far. - CTRF's retryAttempts ends before the final attempt, which was dropped when its outcome matched the one before it. - TAP names tests only within a stream, so a failure in one file and a pass under the same name in another merged into a flaky pass. The path given to --import is now the class of a stream's tests. - A failure followed by a skip with the same identifier read as skipped. - Directives are whole words: TODOING is not TODO. SKIPPED and SKIPPING stay directives, as harnesses treat them. - Verdicts that alternate count as flaky only under an explicit --import-revision. HEAD says nothing about a dirty checkout or a report from another one, and two such events could quarantine a real failure. - --import-name no longer redirects the history of a normal test run. - Importing does not depend on the tests registered in the binary.
Diffstat (limited to 'docs')
-rw-r--r--docs/progmasoft-runner.md18
1 files changed, 12 insertions, 6 deletions
diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md
index 486c719f..8134ae18 100644
--- a/docs/progmasoft-runner.md
+++ b/docs/progmasoft-runner.md
@@ -164,12 +164,18 @@ work as they do for a run. Reports that were read count as one run.
A test is identified by its class and its name. Results with the same
identifier, in one report or in several, are attempts at one test: a test that
both failed and passed is flaky, passes and is remembered as flaky, which is how
-a report of a rerun is imported. TAP has no retries, so a name that comes back
-in one stream is another test and gets a number.
-
-A verdict that changes on one revision also counts as flaky.
-`--import-revision=REVISION` names what the reports tested; it defaults to the
-Git commit of the working directory.
+a report of a rerun is imported. A failure stays a failure when a later report
+skips the test. Reports of the same tests on different platforms are not reruns
+of each other: import each with an `--import-name` of its own, or a failure on one
+reads as flakiness.
+
+TAP names tests only within one stream, so the path given to `--import` is the
+class of its tests: keep it the same from run to run for the history to follow
+them. A name that comes back in one stream is another test and gets a number.
+
+`--import-revision=REVISION` names what the reports tested, and a verdict that
+changes on one revision then counts as flaky too. Without it only retries in the
+reports do, because a working directory can have edits or be another checkout.
Nothing is run, so an imported failure has no replay command, and only what the
report says is known: it has no order, seed or sections. A TAP stream that bails