<feed xmlns='http://www.w3.org/2005/Atom'>
<title>catch3/tests/TestScripts/DiscoverTests/VerifyRegistration.py, branch feature/cpp-tools</title>
<subtitle>Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.</subtitle>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/'/>
<entry>
<title>Replace Catch2's runner with the Catch3 runner</title>
<updated>2026-10-23T20:25:41+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-23T20:25:41+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=93d4e7dbebab867e0a7f59fc25308460a7a76280'/>
<id>93d4e7dbebab867e0a7f59fc25308460a7a76280</id>
<content type='text'>
Catch3 keeps Catch2's API for writing tests (test cases, sections,
assertions, matchers, generators, listeners, benchmarks) and replaces its
session, command line and reporters with a runner of its own.

The runner starts the test binary again as worker processes that take the
next test when they are free. A crash, an exit or a hang ends one test, not
the run. On top of that it adds:

- retries that tell deterministic, flaky and order-dependent failures
  apart, and a bisection that names the test leaving state behind;
- a history of verdicts and durations, kept under a file lock, that drives
  automatic quarantine, adaptive time limits, predictive ordering,
  incremental runs, failure replay and analytics;
- selection by change, past failure or prediction, dependency closure,
  and sharding by a stable hash or by recorded durations;
- tags for serial and isolated tests, shared resources, dependencies,
  time limits and retries;
- a failure category, a signature, an analysis and a replay command for
  every failure;
- console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome
  trace reports, and a versioned discovery listing (--list=json);
- a temporary directory per test (TMPDIR, TMP, TEMP and
  TestTemporaryDirectory());
- Bazel's test environment: XML output file, test filter, sharding, seed,
  premature exit file and undeclared outputs.

The command line is parsed by a hand-written, typed parser. Catch::Session
remains as a thin facade over the runner for custom main functions.

The result model in Progmasoft/Catch3/Results.hpp was rewritten around
tests and attempts, and the JUnit writer now writes it. The property
engine's example database is updated under a file lock with an atomic
rename, so parallel workers can write it at the same time.

Removed: Clara, the command-line and reporter-spec parsers, every reporter
and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width,
the JSON and XML writers, sharding in the test registry, approval tests and
their baselines, Conan packaging, and the unused third_party/clara.hpp.
Listeners keep working; reporters' list and skip events are gone.

The Catch2 library is now compiled as C++20; code that includes only
Catch2's headers still needs C++14. Duplicate test cases are still refused
at start-up. Version 0.3.0.

ExtraTests now turn exceptions off for clang-cl with MSVC-style options,
which -fno-exceptions did not do.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Catch3 keeps Catch2's API for writing tests (test cases, sections,
assertions, matchers, generators, listeners, benchmarks) and replaces its
session, command line and reporters with a runner of its own.

The runner starts the test binary again as worker processes that take the
next test when they are free. A crash, an exit or a hang ends one test, not
the run. On top of that it adds:

- retries that tell deterministic, flaky and order-dependent failures
  apart, and a bisection that names the test leaving state behind;
- a history of verdicts and durations, kept under a file lock, that drives
  automatic quarantine, adaptive time limits, predictive ordering,
  incremental runs, failure replay and analytics;
- selection by change, past failure or prediction, dependency closure,
  and sharding by a stable hash or by recorded durations;
- tags for serial and isolated tests, shared resources, dependencies,
  time limits and retries;
- a failure category, a signature, an analysis and a replay command for
  every failure;
- console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome
  trace reports, and a versioned discovery listing (--list=json);
- a temporary directory per test (TMPDIR, TMP, TEMP and
  TestTemporaryDirectory());
- Bazel's test environment: XML output file, test filter, sharding, seed,
  premature exit file and undeclared outputs.

The command line is parsed by a hand-written, typed parser. Catch::Session
remains as a thin facade over the runner for custom main functions.

The result model in Progmasoft/Catch3/Results.hpp was rewritten around
tests and attempts, and the JUnit writer now writes it. The property
engine's example database is updated under a file lock with an atomic
rename, so parallel workers can write it at the same time.

Removed: Clara, the command-line and reporter-spec parsers, every reporter
and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width,
the JSON and XML writers, sharding in the test registry, approval tests and
their baselines, Conan packaging, and the unused third_party/clara.hpp.
Listeners keep working; reporters' list and skip events are gone.

The Catch2 library is now compiled as C++20; code that includes only
Catch2's headers still needs C++14. Duplicate test cases are still refused
at start-up. Version 0.3.0.

ExtraTests now turn exceptions off for clang-cl with MSVC-style options,
which -fno-exceptions did not do.
</pre>
</div>
</content>
</entry>
<entry>
<title>catch_discover_tests registers tests with --order decl</title>
<updated>2026-09-11T11:13:15+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-09-11T11:13:12+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=6534dbc3ef3a162111830d9b5d8a9092d378336d'/>
<id>6534dbc3ef3a162111830d9b5d8a9092d378336d</id>
<content type='text'>
Catch2 defaults to random test execution order, which has to process
all tests present in the binary, even if they will be filtered down
to just 1 later. For the `SelfTest` binary, the overhead from this is
about 20% in Debug configuration and 5% in Release.

By registering the CTest tests with `--order decl`, we avoid this
overhead.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Catch2 defaults to random test execution order, which has to process
all tests present in the binary, even if they will be filtered down
to just 1 later. For the `SelfTest` binary, the overhead from this is
about 20% in Debug configuration and 5% in Release.

By registering the CTest tests with `--order decl`, we avoid this
overhead.
</pre>
</div>
</content>
</entry>
<entry>
<title>Support TEST_{PREFIX,SUFFIX} with leading/trailing whitespace</title>
<updated>2026-08-20T14:24:18+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-20T12:45:30+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=8582805f548e3e091d8d35188f4c3a690f989b0f'/>
<id>8582805f548e3e091d8d35188f4c3a690f989b0f</id>
<content type='text'>
Closes #2149
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Closes #2149
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix &lt;target&gt;_TESTS variable generated by `catch_discover_tests`</title>
<updated>2026-08-09T22:17:26+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-08T12:59:04+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=630840c5000ec9f5e169c8a2f5fd163d3b0c54ca'/>
<id>630840c5000ec9f5e169c8a2f5fd163d3b0c54ca</id>
<content type='text'>
There were 2 separate bugs, one recently added and one ancient
(since roughly the first version of the registration script).

1) The recent refactoring of how the JSON output from Catch2 is
   parsed caused shadowing between the variable containing the
   per-test JSON fragments and the accumulator of test names for
   &lt;target&gt;_TESTS variable. This led to the variable containing
   both the names and the JSON fragments, and thus being completely
   wrong.

2) The test name accumulation has never accounted for characters
   that need escaping to be present in a CMake list. This means
   that e.g. test names with semicolon in them would end up with
   two elements in the test list.

Both of these are now fixed, at the cost of extra complexity and
my sanity as I had to learn more about CMake escaping rules.

(CMake escaping rules are dumb)
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
There were 2 separate bugs, one recently added and one ancient
(since roughly the first version of the registration script).

1) The recent refactoring of how the JSON output from Catch2 is
   parsed caused shadowing between the variable containing the
   per-test JSON fragments and the accumulator of test names for
   &lt;target&gt;_TESTS variable. This led to the variable containing
   both the names and the JSON fragments, and thus being completely
   wrong.

2) The test name accumulation has never accounted for characters
   that need escaping to be present in a CMake list. This means
   that e.g. test names with semicolon in them would end up with
   two elements in the test list.

Both of these are now fixed, at the cost of extra complexity and
my sanity as I had to learn more about CMake escaping rules.

(CMake escaping rules are dumb)
</pre>
</div>
</content>
</entry>
<entry>
<title>JSONReporter's listTests only lists class-name if it is non-empty</title>
<updated>2026-08-07T21:07:46+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-07-29T08:05:39+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=0b4d7a5a1696dbdfb23fccad8c50db9dc9a417ea'/>
<id>0b4d7a5a1696dbdfb23fccad8c50db9dc9a417ea</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>catch_discover_tests uses tempfile to retrieve JSON from the binary</title>
<updated>2026-07-04T14:36:05+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-07-04T14:35:50+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=9ec44dd62b1b593e01feaf95eb485d26fcdd50ce'/>
<id>9ec44dd62b1b593e01feaf95eb485d26fcdd50ce</id>
<content type='text'>
This allows it to deal with badly behaved code, where 3rd party
dependencies write into stdout during global construction.

Closes #3162
Closes #3166
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This allows it to deal with badly behaved code, where 3rd party
dependencies write into stdout during global construction.

Closes #3162
Closes #3166
</pre>
</div>
</content>
</entry>
<entry>
<title>fix(python): fix invalid escape sequences</title>
<updated>2025-06-24T21:32:27+00:00</updated>
<author>
<name>Eisuke Kawashima</name>
</author>
<published>2025-06-09T01:10:19+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=ceed26842b3b3f93523998f7adb672672c7c2d8f'/>
<id>ceed26842b3b3f93523998f7adb672672c7c2d8f</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Support adding test tags as CTest labels in catch_discover_tests</title>
<updated>2025-01-05T19:02:00+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2025-01-03T09:30:47+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=7d7b2f89f24cbd277f6eb6912705e5e13c68a3c7'/>
<id>7d7b2f89f24cbd277f6eb6912705e5e13c68a3c7</id>
<content type='text'>
We also bump the minimum CMake version to 3.20 as per #2943
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
We also bump the minimum CMake version to 3.20 as per #2943
</pre>
</div>
</content>
</entry>
<entry>
<title>Add tests for multiple args to DL_PATHS in catch_discover_tests</title>
<updated>2024-03-26T17:04:17+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2024-03-26T16:49:55+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=3cd90c5c3bf845935ae0bc854aadf48fcb240088'/>
<id>3cd90c5c3bf845935ae0bc854aadf48fcb240088</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Escape Catch2 test names in catch_discover_tests tests</title>
<updated>2023-06-15T12:19:39+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2023-06-15T12:19:39+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=e4b16053a6763cd6f8b89aeaf59303a461ccf755'/>
<id>e4b16053a6763cd6f8b89aeaf59303a461ccf755</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
