<feed xmlns='http://www.w3.org/2005/Atom'>
<title>catch3/src/Progmasoft/Runner/Runner.cpp, branch feature/cpp-tools</title>
<subtitle>Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.</subtitle>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/'/>
<entry>
<title>Keep real failures visible when importing</title>
<updated>2026-11-01T11:48:55+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-11-01T11:48:55+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=569620dad14df7e13bb9fd31d256e9fd873ef6ec'/>
<id>569620dad14df7e13bb9fd31d256e9fd873ef6ec</id>
<content type='text'>
A review of the import found ways for a failure to disappear or for the
import to reach too far.

- CTRF's retryAttempts ends before the final attempt, which was dropped when
  its outcome matched the one before it.
- TAP names tests only within a stream, so a failure in one file and a pass
  under the same name in another merged into a flaky pass. The path given to
  --import is now the class of a stream's tests.
- A failure followed by a skip with the same identifier read as skipped.
- Directives are whole words: TODOING is not TODO. SKIPPED and SKIPPING stay
  directives, as harnesses treat them.
- Verdicts that alternate count as flaky only under an explicit
  --import-revision. HEAD says nothing about a dirty checkout or a report from
  another one, and two such events could quarantine a real failure.
- --import-name no longer redirects the history of a normal test run.
- Importing does not depend on the tests registered in the binary.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A review of the import found ways for a failure to disappear or for the
import to reach too far.

- CTRF's retryAttempts ends before the final attempt, which was dropped when
  its outcome matched the one before it.
- TAP names tests only within a stream, so a failure in one file and a pass
  under the same name in another merged into a flaky pass. The path given to
  --import is now the class of a stream's tests.
- A failure followed by a skip with the same identifier read as skipped.
- Directives are whole words: TODOING is not TODO. SKIPPED and SKIPPING stay
  directives, as harnesses treat them.
- Verdicts that alternate count as flaky only under an explicit
  --import-revision. HEAD says nothing about a dirty checkout or a report from
  another one, and two such events could quarantine a real failure.
- --import-name no longer redirects the history of a normal test run.
- Importing does not depend on the tests registered in the binary.
</pre>
</div>
</content>
</entry>
<entry>
<title>Import the results of other test frameworks</title>
<updated>2026-11-01T11:23:09+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-11-01T11:23:09+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=1db203d56b7c71db3d80b477da1ad49e9ae46468'/>
<id>1db203d56b7c71db3d80b477da1ad49e9ae46468</id>
<content type='text'>
Tests Catch3 does not run had no history, flaky detection or quarantine.
--import=FILE reads a report instead of running tests and sends it through
the analysis, history and reports of a run: JUnit with Surefire's reruns,
xUnit.net v2, TRX, TAP 12 to 14, CTRF, Catch3's own --json report and
catch3-results, a small JSON Lines format for tools with none of these.

Results that share an identifier are attempts at one test, so a report of a
rerun gives a flaky test. --import-revision says what was tested: a verdict
that changes on one revision also counts as flaky. --import-name keeps the
results in a history of their own, which --analytics reads too. A TAP stream
that ends early fails instead of reading as a run that passed.

The amalgamation script now accepts a relative include of the third-party
wrappers, because pugixml is included from a second directory.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Tests Catch3 does not run had no history, flaky detection or quarantine.
--import=FILE reads a report instead of running tests and sends it through
the analysis, history and reports of a run: JUnit with Surefire's reruns,
xUnit.net v2, TRX, TAP 12 to 14, CTRF, Catch3's own --json report and
catch3-results, a small JSON Lines format for tools with none of these.

Results that share an identifier are attempts at one test, so a report of a
rerun gives a flaky test. --import-revision says what was tested: a verdict
that changes on one revision also counts as flaky. --import-name keeps the
results in a history of their own, which --analytics reads too. A TAP stream
that ends early fails instead of reading as a run that passed.

The amalgamation script now accepts a relative include of the third-party
wrappers, because pugixml is included from a second directory.
</pre>
</div>
</content>
</entry>
<entry>
<title>Move to the Progmasoft Linking Exception, Version 2.0</title>
<updated>2026-10-31T10:42:35+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-31T10:42:35+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=31486e3217b8424809bcf129c0ef24ef5639e919'/>
<id>31486e3217b8424809bcf129c0ef24ef5639e919</id>
<content type='text'>
Version 1.1 left programs built with Progmasoft tools under MPL-2.0 duties
for the code the tools place in them, and left other cases unclear.
Version 2.0 adds a generated code permission, covers runtime and standard
libraries, compiler transformations, interfaces and examples, and settles
how a file is identified and how to tell recipients about source.

Every project SPDX expression moves to AdditionRef-Progmasoft-Exception-2.0.
The older texts stay, because released versions name them.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Version 1.1 left programs built with Progmasoft tools under MPL-2.0 duties
for the code the tools place in them, and left other cases unclear.
Version 2.0 adds a generated code permission, covers runtime and standard
libraries, compiler transformations, interfaces and examples, and settles
how a file is identified and how to tell recipients about source.

Every project SPDX expression moves to AdditionRef-Progmasoft-Exception-2.0.
The older texts stay, because released versions name them.
</pre>
</div>
</content>
</entry>
<entry>
<title>Read the options of a project from catch3.json</title>
<updated>2026-10-29T16:22:21+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-29T16:22:21+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=f41a9c3c01b4df1100e24a63987716cb4dbf28f2'/>
<id>f41a9c3c01b4df1100e24a63987716cb4dbf28f2</id>
<content type='text'>
A project can now keep the options it always wants in a catch3.json
instead of repeating them in every CTest, Bazel or CI command line.

Where it is found: the runner looks in the working directory and in each
directory above it, up to the root of the Git repository that holds the
working directory, and reads the nearest file. Outside a repository it
looks only in the working directory, so a stray file in a home directory
never applies. --config=FILE names a file instead (relative to the
working directory) and --no-config reads none; the last of the two wins.
Workers, --help and --version read no file, so a broken file never stands
in the way of help, and workers keep taking everything from their runner.

What it says: strict JSON with "version": 1 required and "$schema"
allowed for editors. The keys are the long options of the command line,
and each value becomes the arguments it stands for and goes through the
same parser, so there is one set of checks and one set of messages; the
file and the key are named in front of them. Values are typed: switches
take true (false for the negated adaptive-timeout and bisect), counts
take whole numbers, timeout and the confidence interval take numbers,
retries and seed take a number or a word, tests and warn take arrays,
verbosity takes quiet, normal or verbose, and history takes a path or
false. Paths are relative to the directory of the file. An unknown key is
an error with a suggestion when it is close to a known one; a key that
names an option for one run (shard, section, list, replay, ...) is told
that it belongs on the command line. Repeated keys are refused.

Precedence: the file's options are applied first and the command line on
top, so the command line wins; test specs on the command line replace the
file's tests instead of widening them with "or". --verbose names the file
that was read.

Options gains --config and --no-config, a ParseOptions overload that
starts from given options, and a public EditDistance for the suggestions.
The new ProjectConfig module holds the search, the key table and
ParseCommandLine, which Runner now calls in place of ParseOptions.

Tests cover the search (nearest file, the repository boundary, outside a
repository), every key, words and defaults, relative and absolute paths,
the messages for wrong files, precedence, --config, --no-config and the
modes that read no file. A fuzzer reads arbitrary file text and checks
that the worker arguments of the result parse again. The command line
reference, the runner guide and the release notes describe the file; the
amalgamated sources are regenerated.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A project can now keep the options it always wants in a catch3.json
instead of repeating them in every CTest, Bazel or CI command line.

Where it is found: the runner looks in the working directory and in each
directory above it, up to the root of the Git repository that holds the
working directory, and reads the nearest file. Outside a repository it
looks only in the working directory, so a stray file in a home directory
never applies. --config=FILE names a file instead (relative to the
working directory) and --no-config reads none; the last of the two wins.
Workers, --help and --version read no file, so a broken file never stands
in the way of help, and workers keep taking everything from their runner.

What it says: strict JSON with "version": 1 required and "$schema"
allowed for editors. The keys are the long options of the command line,
and each value becomes the arguments it stands for and goes through the
same parser, so there is one set of checks and one set of messages; the
file and the key are named in front of them. Values are typed: switches
take true (false for the negated adaptive-timeout and bisect), counts
take whole numbers, timeout and the confidence interval take numbers,
retries and seed take a number or a word, tests and warn take arrays,
verbosity takes quiet, normal or verbose, and history takes a path or
false. Paths are relative to the directory of the file. An unknown key is
an error with a suggestion when it is close to a known one; a key that
names an option for one run (shard, section, list, replay, ...) is told
that it belongs on the command line. Repeated keys are refused.

Precedence: the file's options are applied first and the command line on
top, so the command line wins; test specs on the command line replace the
file's tests instead of widening them with "or". --verbose names the file
that was read.

Options gains --config and --no-config, a ParseOptions overload that
starts from given options, and a public EditDistance for the suggestions.
The new ProjectConfig module holds the search, the key table and
ParseCommandLine, which Runner now calls in place of ParseOptions.

Tests cover the search (nearest file, the repository boundary, outside a
repository), every key, words and defaults, relative and absolute paths,
the messages for wrong files, precedence, --config, --no-config and the
modes that read no file. A fuzzer reads arbitrary file text and checks
that the worker arguments of the result parse again. The command line
reference, the runner guide and the release notes describe the file; the
amalgamated sources are regenerated.
</pre>
</div>
</content>
</entry>
<entry>
<title>Give Catch3 its own build names and headers</title>
<updated>2026-10-24T23:37:35+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-24T23:37:35+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=126dc333564cb72b5115d9a9242d66694694b6d0'/>
<id>126dc333564cb72b5115d9a9242d66694694b6d0</id>
<content type='text'>
Catch2 compatibility means the API tests are written in: the macros, such
as TEST_CASE and REQUIRE, and the Catch:: namespace. It does not mean
Catch2's build names or header paths. find_package(Catch2) found Catch3,
and a project using both frameworks met two packages, two sets of targets,
two Catch.cmake modules and two catch2/ header trees with the same names.

Build names:
- CMake: project and package Catch3, targets Catch3::Catch3 and
  Catch3::Catch3WithMain, libraries Catch3 and Catch3Main, options
  CATCH3_*, lib/cmake/Catch3 with Catch3Config, and pkg-config catch3 and
  catch3-with-main. The version is Catch3's, 0.3.0, compatible within a
  minor version.
- The CMake scripts are Catch3.cmake (catch3_discover_tests),
  Catch3AddTests.cmake and Catch3ShardTests.cmake
  (catch3_add_sharded_tests); the helpers they define at ctest time have a
  catch3_ prefix. ParseAndAddCatchTests.cmake, deprecated upstream, is gone.
- Bazel //:catch3 and //:catch3_main, Meson catch3 and catch3-with-main
  (with override_dependency), Xmake catch3 and catch3_main, Premake Catch3
  and Catch3WithMain. Visual Studio projects get GUIDs of their own.
- The Progmasoft::Catch3 target and package are gone: property testing
  and snapshots are part of Catch3::Catch3. Their sources now build with
  the library's warning flags, so property failures are built field by
  field (no missing initializers) and the default property name is a
  thread-local pointer to the name its scope owns (no global destructor).

Headers: the engine moved from src/catch2 to src/Progmasoft/Catch3 with
Progmasoft names, so &lt;catch2/catch_test_macros.hpp&gt; is
&lt;Progmasoft/Catch3/TestMacros.hpp&gt;, &lt;catch2/matchers/catch_matchers_string.hpp&gt;
is &lt;Progmasoft/Catch3/Matchers/String.hpp&gt;, and interface headers are named
after their interface, such as Interfaces/IConfig.hpp. Include guards
became #pragma once. The installed headers go to include/Progmasoft.
src/Progmasoft/Context.cpp is StrategyContext.cpp, so no two sources share
a file name.

tools/scripts/upstreamFiles.txt lists the files that come from Catch2.
The license check expects BSL-1.0 on them and Progmasoft's header on the
rest of src/Progmasoft; the convenience-header check, the amalgamation
script and the surrogate tests take the engine's headers from it. The
amalgamated files are regenerated. The header-guard workflow checks for
#pragma once instead of checkguard.

Documents, the README and the release notes describe the new names.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Catch2 compatibility means the API tests are written in: the macros, such
as TEST_CASE and REQUIRE, and the Catch:: namespace. It does not mean
Catch2's build names or header paths. find_package(Catch2) found Catch3,
and a project using both frameworks met two packages, two sets of targets,
two Catch.cmake modules and two catch2/ header trees with the same names.

Build names:
- CMake: project and package Catch3, targets Catch3::Catch3 and
  Catch3::Catch3WithMain, libraries Catch3 and Catch3Main, options
  CATCH3_*, lib/cmake/Catch3 with Catch3Config, and pkg-config catch3 and
  catch3-with-main. The version is Catch3's, 0.3.0, compatible within a
  minor version.
- The CMake scripts are Catch3.cmake (catch3_discover_tests),
  Catch3AddTests.cmake and Catch3ShardTests.cmake
  (catch3_add_sharded_tests); the helpers they define at ctest time have a
  catch3_ prefix. ParseAndAddCatchTests.cmake, deprecated upstream, is gone.
- Bazel //:catch3 and //:catch3_main, Meson catch3 and catch3-with-main
  (with override_dependency), Xmake catch3 and catch3_main, Premake Catch3
  and Catch3WithMain. Visual Studio projects get GUIDs of their own.
- The Progmasoft::Catch3 target and package are gone: property testing
  and snapshots are part of Catch3::Catch3. Their sources now build with
  the library's warning flags, so property failures are built field by
  field (no missing initializers) and the default property name is a
  thread-local pointer to the name its scope owns (no global destructor).

Headers: the engine moved from src/catch2 to src/Progmasoft/Catch3 with
Progmasoft names, so &lt;catch2/catch_test_macros.hpp&gt; is
&lt;Progmasoft/Catch3/TestMacros.hpp&gt;, &lt;catch2/matchers/catch_matchers_string.hpp&gt;
is &lt;Progmasoft/Catch3/Matchers/String.hpp&gt;, and interface headers are named
after their interface, such as Interfaces/IConfig.hpp. Include guards
became #pragma once. The installed headers go to include/Progmasoft.
src/Progmasoft/Context.cpp is StrategyContext.cpp, so no two sources share
a file name.

tools/scripts/upstreamFiles.txt lists the files that come from Catch2.
The license check expects BSL-1.0 on them and Progmasoft's header on the
rest of src/Progmasoft; the convenience-header check, the amalgamation
script and the surrogate tests take the engine's headers from it. The
amalgamated files are regenerated. The header-guard workflow checks for
#pragma once instead of checkguard.

Documents, the README and the release notes describe the new names.
</pre>
</div>
</content>
</entry>
<entry>
<title>End the output capture when a test exits, fail on missing reports</title>
<updated>2026-10-24T09:41:08+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-24T09:41:08+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=e18f19b7dd70c99fbcf2b3fe812ee18dabc92663'/>
<id>e18f19b7dd70c99fbcf2b3fe812ee18dabc92663</id>
<content type='text'>
A test that called exit() crashed its worker with SIGSEGV on Linux and
macOS. The exit hook read Catch2's captured output through a scoped
deactivation, which switched the capture back on; libstdc++ then flushed
std::cout at exit into a capture buffer that static destruction had
already freed. RunContext::endCapture() now ends the capture for good and
returns what it held, so the standard streams are their own again.

A JUnit, JSON or trace report that could not be written was only
reported on stderr, and a run whose tests passed still exited with 0, so a
build server could miss it. Such a run now exits with 1; failed tests
still exit with 42.

Integration tests of the runner's output move their helpers into
RunnerFixtureClient.hpp and cover --version, --help, the listing modes, a
bad option, exit codes 1 to 4, -v, -q, --slowest, colours, --analytics
in text and JSON, and Bazel's test environment.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A test that called exit() crashed its worker with SIGSEGV on Linux and
macOS. The exit hook read Catch2's captured output through a scoped
deactivation, which switched the capture back on; libstdc++ then flushed
std::cout at exit into a capture buffer that static destruction had
already freed. RunContext::endCapture() now ends the capture for good and
returns what it held, so the standard streams are their own again.

A JUnit, JSON or trace report that could not be written was only
reported on stderr, and a run whose tests passed still exited with 0, so a
build server could miss it. Such a run now exits with 1; failed tests
still exit with 42.

Integration tests of the runner's output move their helpers into
RunnerFixtureClient.hpp and cover --version, --help, the listing modes, a
bad option, exit codes 1 to 4, -v, -q, --slowest, colours, --analytics
in text and JSON, and Bazel's test environment.
</pre>
</div>
</content>
</entry>
<entry>
<title>Replace Catch2's runner with the Catch3 runner</title>
<updated>2026-10-23T20:25:41+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-23T20:25:41+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=93d4e7dbebab867e0a7f59fc25308460a7a76280'/>
<id>93d4e7dbebab867e0a7f59fc25308460a7a76280</id>
<content type='text'>
Catch3 keeps Catch2's API for writing tests (test cases, sections,
assertions, matchers, generators, listeners, benchmarks) and replaces its
session, command line and reporters with a runner of its own.

The runner starts the test binary again as worker processes that take the
next test when they are free. A crash, an exit or a hang ends one test, not
the run. On top of that it adds:

- retries that tell deterministic, flaky and order-dependent failures
  apart, and a bisection that names the test leaving state behind;
- a history of verdicts and durations, kept under a file lock, that drives
  automatic quarantine, adaptive time limits, predictive ordering,
  incremental runs, failure replay and analytics;
- selection by change, past failure or prediction, dependency closure,
  and sharding by a stable hash or by recorded durations;
- tags for serial and isolated tests, shared resources, dependencies,
  time limits and retries;
- a failure category, a signature, an analysis and a replay command for
  every failure;
- console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome
  trace reports, and a versioned discovery listing (--list=json);
- a temporary directory per test (TMPDIR, TMP, TEMP and
  TestTemporaryDirectory());
- Bazel's test environment: XML output file, test filter, sharding, seed,
  premature exit file and undeclared outputs.

The command line is parsed by a hand-written, typed parser. Catch::Session
remains as a thin facade over the runner for custom main functions.

The result model in Progmasoft/Catch3/Results.hpp was rewritten around
tests and attempts, and the JUnit writer now writes it. The property
engine's example database is updated under a file lock with an atomic
rename, so parallel workers can write it at the same time.

Removed: Clara, the command-line and reporter-spec parsers, every reporter
and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width,
the JSON and XML writers, sharding in the test registry, approval tests and
their baselines, Conan packaging, and the unused third_party/clara.hpp.
Listeners keep working; reporters' list and skip events are gone.

The Catch2 library is now compiled as C++20; code that includes only
Catch2's headers still needs C++14. Duplicate test cases are still refused
at start-up. Version 0.3.0.

ExtraTests now turn exceptions off for clang-cl with MSVC-style options,
which -fno-exceptions did not do.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Catch3 keeps Catch2's API for writing tests (test cases, sections,
assertions, matchers, generators, listeners, benchmarks) and replaces its
session, command line and reporters with a runner of its own.

The runner starts the test binary again as worker processes that take the
next test when they are free. A crash, an exit or a hang ends one test, not
the run. On top of that it adds:

- retries that tell deterministic, flaky and order-dependent failures
  apart, and a bisection that names the test leaving state behind;
- a history of verdicts and durations, kept under a file lock, that drives
  automatic quarantine, adaptive time limits, predictive ordering,
  incremental runs, failure replay and analytics;
- selection by change, past failure or prediction, dependency closure,
  and sharding by a stable hash or by recorded durations;
- tags for serial and isolated tests, shared resources, dependencies,
  time limits and retries;
- a failure category, a signature, an analysis and a replay command for
  every failure;
- console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome
  trace reports, and a versioned discovery listing (--list=json);
- a temporary directory per test (TMPDIR, TMP, TEMP and
  TestTemporaryDirectory());
- Bazel's test environment: XML output file, test filter, sharding, seed,
  premature exit file and undeclared outputs.

The command line is parsed by a hand-written, typed parser. Catch::Session
remains as a thin facade over the runner for custom main functions.

The result model in Progmasoft/Catch3/Results.hpp was rewritten around
tests and attempts, and the JUnit writer now writes it. The property
engine's example database is updated under a file lock with an atomic
rename, so parallel workers can write it at the same time.

Removed: Clara, the command-line and reporter-spec parsers, every reporter
and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width,
the JSON and XML writers, sharding in the test registry, approval tests and
their baselines, Conan packaging, and the unused third_party/clara.hpp.
Listeners keep working; reporters' list and skip events are gone.

The Catch2 library is now compiled as C++20; code that includes only
Catch2's headers still needs C++14. Duplicate test cases are still refused
at start-up. Version 0.3.0.

ExtraTests now turn exceptions off for clang-cl with MSVC-style options,
which -fno-exceptions did not do.
</pre>
</div>
</content>
</entry>
</feed>
