diff options
| author | Leitwolf11 | 2026-11-01 14:48:55 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-11-01 14:48:55 +0300 |
| commit | 569620dad14df7e13bb9fd31d256e9fd873ef6ec (patch) | |
| tree | f933300c29658c3eb89d126d74b6c30fdff9dd04 /docs | |
| parent | 1db203d56b7c71db3d80b477da1ad49e9ae46468 (diff) | |
| download | catch3-feature/result-import.tar.gz | |
Keep real failures visible when importingfeature/result-import
A review of the import found ways for a failure to disappear or for the
import to reach too far.
- CTRF's retryAttempts ends before the final attempt, which was dropped when
its outcome matched the one before it.
- TAP names tests only within a stream, so a failure in one file and a pass
under the same name in another merged into a flaky pass. The path given to
--import is now the class of a stream's tests.
- A failure followed by a skip with the same identifier read as skipped.
- Directives are whole words: TODOING is not TODO. SKIPPED and SKIPPING stay
directives, as harnesses treat them.
- Verdicts that alternate count as flaky only under an explicit
--import-revision. HEAD says nothing about a dirty checkout or a report from
another one, and two such events could quarantine a real failure.
- --import-name no longer redirects the history of a normal test run.
- Importing does not depend on the tests registered in the binary.
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/progmasoft-runner.md | 18 |
1 files changed, 12 insertions, 6 deletions
diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md index 486c719f..8134ae18 100644 --- a/docs/progmasoft-runner.md +++ b/docs/progmasoft-runner.md @@ -164,12 +164,18 @@ work as they do for a run. Reports that were read count as one run. A test is identified by its class and its name. Results with the same identifier, in one report or in several, are attempts at one test: a test that both failed and passed is flaky, passes and is remembered as flaky, which is how -a report of a rerun is imported. TAP has no retries, so a name that comes back -in one stream is another test and gets a number. - -A verdict that changes on one revision also counts as flaky. -`--import-revision=REVISION` names what the reports tested; it defaults to the -Git commit of the working directory. +a report of a rerun is imported. A failure stays a failure when a later report +skips the test. Reports of the same tests on different platforms are not reruns +of each other: import each with an `--import-name` of its own, or a failure on one +reads as flakiness. + +TAP names tests only within one stream, so the path given to `--import` is the +class of its tests: keep it the same from run to run for the history to follow +them. A name that comes back in one stream is another test and gets a number. + +`--import-revision=REVISION` names what the reports tested, and a verdict that +changes on one revision then counts as flaky too. Without it only retries in the +reports do, because a working directory can have edits or be another checkout. Nothing is run, so an imported failure has no replay command, and only what the report says is known: it has no order, seed or sections. A TAP stream that bails |