<feed xmlns='http://www.w3.org/2005/Atom'>
<title>catch3/tests/Progmasoft/RunnerImport.tests.cpp, branch feature/mocking</title>
<subtitle>Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.</subtitle>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/'/>
<entry>
<title>Keep real failures visible when importing</title>
<updated>2026-11-01T11:48:55+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-11-01T11:48:55+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=569620dad14df7e13bb9fd31d256e9fd873ef6ec'/>
<id>569620dad14df7e13bb9fd31d256e9fd873ef6ec</id>
<content type='text'>
A review of the import found ways for a failure to disappear or for the
import to reach too far.

- CTRF's retryAttempts ends before the final attempt, which was dropped when
  its outcome matched the one before it.
- TAP names tests only within a stream, so a failure in one file and a pass
  under the same name in another merged into a flaky pass. The path given to
  --import is now the class of a stream's tests.
- A failure followed by a skip with the same identifier read as skipped.
- Directives are whole words: TODOING is not TODO. SKIPPED and SKIPPING stay
  directives, as harnesses treat them.
- Verdicts that alternate count as flaky only under an explicit
  --import-revision. HEAD says nothing about a dirty checkout or a report from
  another one, and two such events could quarantine a real failure.
- --import-name no longer redirects the history of a normal test run.
- Importing does not depend on the tests registered in the binary.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A review of the import found ways for a failure to disappear or for the
import to reach too far.

- CTRF's retryAttempts ends before the final attempt, which was dropped when
  its outcome matched the one before it.
- TAP names tests only within a stream, so a failure in one file and a pass
  under the same name in another merged into a flaky pass. The path given to
  --import is now the class of a stream's tests.
- A failure followed by a skip with the same identifier read as skipped.
- Directives are whole words: TODOING is not TODO. SKIPPED and SKIPPING stay
  directives, as harnesses treat them.
- Verdicts that alternate count as flaky only under an explicit
  --import-revision. HEAD says nothing about a dirty checkout or a report from
  another one, and two such events could quarantine a real failure.
- --import-name no longer redirects the history of a normal test run.
- Importing does not depend on the tests registered in the binary.
</pre>
</div>
</content>
</entry>
<entry>
<title>Import the results of other test frameworks</title>
<updated>2026-11-01T11:23:09+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-11-01T11:23:09+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=1db203d56b7c71db3d80b477da1ad49e9ae46468'/>
<id>1db203d56b7c71db3d80b477da1ad49e9ae46468</id>
<content type='text'>
Tests Catch3 does not run had no history, flaky detection or quarantine.
--import=FILE reads a report instead of running tests and sends it through
the analysis, history and reports of a run: JUnit with Surefire's reruns,
xUnit.net v2, TRX, TAP 12 to 14, CTRF, Catch3's own --json report and
catch3-results, a small JSON Lines format for tools with none of these.

Results that share an identifier are attempts at one test, so a report of a
rerun gives a flaky test. --import-revision says what was tested: a verdict
that changes on one revision also counts as flaky. --import-name keeps the
results in a history of their own, which --analytics reads too. A TAP stream
that ends early fails instead of reading as a run that passed.

The amalgamation script now accepts a relative include of the third-party
wrappers, because pugixml is included from a second directory.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Tests Catch3 does not run had no history, flaky detection or quarantine.
--import=FILE reads a report instead of running tests and sends it through
the analysis, history and reports of a run: JUnit with Surefire's reruns,
xUnit.net v2, TRX, TAP 12 to 14, CTRF, Catch3's own --json report and
catch3-results, a small JSON Lines format for tools with none of these.

Results that share an identifier are attempts at one test, so a report of a
rerun gives a flaky test. --import-revision says what was tested: a verdict
that changes on one revision also counts as flaky. --import-name keeps the
results in a history of their own, which --analytics reads too. A TAP stream
that ends early fails instead of reading as a run that passed.

The amalgamation script now accepts a relative include of the third-party
wrappers, because pugixml is included from a second directory.
</pre>
</div>
</content>
</entry>
</feed>
