catch3/tests/ExtraTests, branch devel Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results. Carry mocks in the amalgamated files 2026-11-01T19:05:27+00:00 Leitwolf11 2026-11-01T19:05:27+00:00 241b5567d09d59e82bff18287c32ada480deb8c0 Mock.hpp goes into catch_amalgamated.hpp, since tests use it directly, and Mock.cpp into catch_amalgamated.cpp. The test of the amalgamated files now uses a mock, so a file left out fails it. 
Mock.hpp goes into catch_amalgamated.hpp, since tests use it directly, and
Mock.cpp into catch_amalgamated.cpp. The test of the amalgamated files
now uses a mock, so a file left out fails it.
Say C++17 everywhere the project describes itself 2026-10-31T21:25:59+00:00 Leitwolf11 2026-10-31T21:25:59+00:00 ff1da91edffa9ee887e9fe737fcd87ec540dfa60 The move to C++17 left a trail of older claims. The Runner's own .clang-format still said c++20; the result is byte for byte the same under either value. Premake and Xmake still compiled the library as C++20 and the main library as C++14, which meant their CI jobs were not testing the language the project promises. Both now use C++17 for every target. The rest is text: the Meson pkg-config description claimed C++14, comments in the CMake and Meson files explained requirements that no longer exist, .clang-tidy gave C++14 as the reason for two exclusions, issue templates asked for C++20 samples, and the example headers named -std=c++14 and catch2-with-main. The release notes and the migration guide state the C++17 minimum. Doxygen's brief called the project "Catch2's test API", which presents it as Catch2. Meson declared only BSL-1.0; it now names both licenses of the tree. The amalgamated files are regenerated. 
The move to C++17 left a trail of older claims. The Runner's own
.clang-format still said c++20; the result is byte for byte the same under
either value. Premake and Xmake still compiled the library as C++20 and
the main library as C++14, which meant their CI jobs were not testing the
language the project promises. Both now use C++17 for every target.

The rest is text: the Meson pkg-config description claimed C++14, comments
in the CMake and Meson files explained requirements that no longer exist,
.clang-tidy gave C++14 as the reason for two exclusions, issue templates
asked for C++20 samples, and the example headers named -std=c++14 and
catch2-with-main. The release notes and the migration guide state the
C++17 minimum. Doxygen's brief called the project "Catch2's test API",
which presents it as Catch2. Meson declared only BSL-1.0; it now names both
licenses of the tree. The amalgamated files are regenerated.
Build Catch3 as C++17 2026-10-31T10:00:24+00:00 Leitwolf11 2026-10-31T10:00:24+00:00 8814bd0b796b2e291fa5b4a1033f0313c8a26065 C++17 is still the widest common target, and a library built on it stays usable from C++17 and every later standard, while C++20 would have left out many users. Concepts become enable_if on named traits in Detail/TypeTraits.hpp, which also replaces std::remove_cvref_t and std::identity. std::span becomes std::vector references, std::bit_cast becomes static_cast and memcpy, template lambdas become helper templates, and std::erase_if, contains, starts_with and ends_with have their C++17 forms. Random::Between no longer uses a lambda over a constexpr local, which MSVC rejects outside /permissive-, so the headers work under its default mode too. Tests give up designated initializers, which MSVC refuses in C++17. CI builds the matrix in C++17 and C++20, since the headers promise both. 
C++17 is still the widest common target, and a library built on it stays
usable from C++17 and every later standard, while C++20 would have left
out many users.

Concepts become enable_if on named traits in Detail/TypeTraits.hpp, which
also replaces std::remove_cvref_t and std::identity. std::span becomes
std::vector references, std::bit_cast becomes static_cast and memcpy,
template lambdas become helper templates, and std::erase_if, contains,
starts_with and ends_with have their C++17 forms. Random::Between no
longer uses a lambda over a constexpr local, which MSVC rejects outside
/permissive-, so the headers work under its default mode too.

Tests give up designated initializers, which MSVC refuses in C++17. CI
builds the matrix in C++17 and C++20, since the headers promise both.
Read and write JSON with yyjson and JUnit XML with pugixml 2026-10-25T10:57:03+00:00 Leitwolf11 2026-10-25T10:57:03+00:00 45f8c539e86099cb80ecf29b0bf2b3340e141001 The runner's hand-written JSON parser and writer, and its XML escaping, are replaced by two established MIT libraries, vendored in third_party/: yyjson 0.13.0 and pugixml 1.16. JsonValue and the JUnit model stay as they are. Both are compiled into Catch3's own translation units, so a project that links its own yyjson or pugixml is not affected: - src/Progmasoft/Runner/Yyjson.inl includes yyjson.c into Json.cpp with yyjson_api defined as static, so no yyjson symbol leaves that file. - src/Progmasoft/PugiXml.inl includes pugixml header-only into XmlWriter.cpp with its namespace renamed to Progmasoft_Catch3_Pugi. - Both are read as system headers, outside the warnings Catch3 builds with; they are .inl files, which the header install rules leave out. JSON keeps its behaviour: strings are bytes and come back as written (invalid UTF-8 is allowed both ways), non-finite numbers are written as null, documents nested deeper than 256 levels are refused while they are converted, and whole numbers are written as integers. Fractions are now written as the shortest text that reads back as the same double. JUnit XML stays equivalent; pugixml writes ' and > in attributes as they are, U+FFFD as the character itself and empty elements as <x />. The tests follow. third_party/{yyjson,pugixml} each have BUILD.bazel, CMakeLists.txt, meson.build, xmake.lua and premake5.lua, which only provide the include directory to the library; CMake links them through BUILD_INTERFACE. tools/scripts/updateThirdParty.py pins the version and sha256 of each archive; --check, now a step of the header and license workflow, fails when a vendored file differs from its archive, and --update writes them. Renovate proposes new versions through a regex manager on that table. The amalgamation now has a third file, catch_amalgamated_third_party.hpp, with both wrappers and the libraries inlined; catch_amalgamated.cpp includes it once. The documents and the release notes say so. 
The runner's hand-written JSON parser and writer, and its XML escaping,
are replaced by two established MIT libraries, vendored in third_party/:
yyjson 0.13.0 and pugixml 1.16. JsonValue and the JUnit model stay as they
are.

Both are compiled into Catch3's own translation units, so a project that
links its own yyjson or pugixml is not affected:
- src/Progmasoft/Runner/Yyjson.inl includes yyjson.c into Json.cpp with
  yyjson_api defined as static, so no yyjson symbol leaves that file.
- src/Progmasoft/PugiXml.inl includes pugixml header-only into
  XmlWriter.cpp with its namespace renamed to Progmasoft_Catch3_Pugi.
- Both are read as system headers, outside the warnings Catch3 builds
  with; they are .inl files, which the header install rules leave out.

JSON keeps its behaviour: strings are bytes and come back as written
(invalid UTF-8 is allowed both ways), non-finite numbers are written as
null, documents nested deeper than 256 levels are refused while they are
converted, and whole numbers are written as integers. Fractions are now
written as the shortest text that reads back as the same double.

JUnit XML stays equivalent; pugixml writes ' and > in attributes as they
are, U+FFFD as the character itself and empty elements as <x />. The
tests follow.

third_party/{yyjson,pugixml} each have BUILD.bazel, CMakeLists.txt,
meson.build, xmake.lua and premake5.lua, which only provide the include
directory to the library; CMake links them through BUILD_INTERFACE.

tools/scripts/updateThirdParty.py pins the version and sha256 of each
archive; --check, now a step of the header and license workflow, fails
when a vendored file differs from its archive, and --update writes them.
Renovate proposes new versions through a regex manager on that table.

The amalgamation now has a third file, catch_amalgamated_third_party.hpp,
with both wrappers and the libraries inlined; catch_amalgamated.cpp
includes it once. The documents and the release notes say so.
Give Catch3 its own build names and headers 2026-10-24T23:37:35+00:00 Leitwolf11 2026-10-24T23:37:35+00:00 126dc333564cb72b5115d9a9242d66694694b6d0 Catch2 compatibility means the API tests are written in: the macros, such as TEST_CASE and REQUIRE, and the Catch:: namespace. It does not mean Catch2's build names or header paths. find_package(Catch2) found Catch3, and a project using both frameworks met two packages, two sets of targets, two Catch.cmake modules and two catch2/ header trees with the same names. Build names: - CMake: project and package Catch3, targets Catch3::Catch3 and Catch3::Catch3WithMain, libraries Catch3 and Catch3Main, options CATCH3_*, lib/cmake/Catch3 with Catch3Config, and pkg-config catch3 and catch3-with-main. The version is Catch3's, 0.3.0, compatible within a minor version. - The CMake scripts are Catch3.cmake (catch3_discover_tests), Catch3AddTests.cmake and Catch3ShardTests.cmake (catch3_add_sharded_tests); the helpers they define at ctest time have a catch3_ prefix. ParseAndAddCatchTests.cmake, deprecated upstream, is gone. - Bazel //:catch3 and //:catch3_main, Meson catch3 and catch3-with-main (with override_dependency), Xmake catch3 and catch3_main, Premake Catch3 and Catch3WithMain. Visual Studio projects get GUIDs of their own. - The Progmasoft::Catch3 target and package are gone: property testing and snapshots are part of Catch3::Catch3. Their sources now build with the library's warning flags, so property failures are built field by field (no missing initializers) and the default property name is a thread-local pointer to the name its scope owns (no global destructor). Headers: the engine moved from src/catch2 to src/Progmasoft/Catch3 with Progmasoft names, so <catch2/catch_test_macros.hpp> is <Progmasoft/Catch3/TestMacros.hpp>, <catch2/matchers/catch_matchers_string.hpp> is <Progmasoft/Catch3/Matchers/String.hpp>, and interface headers are named after their interface, such as Interfaces/IConfig.hpp. Include guards became #pragma once. The installed headers go to include/Progmasoft. src/Progmasoft/Context.cpp is StrategyContext.cpp, so no two sources share a file name. tools/scripts/upstreamFiles.txt lists the files that come from Catch2. The license check expects BSL-1.0 on them and Progmasoft's header on the rest of src/Progmasoft; the convenience-header check, the amalgamation script and the surrogate tests take the engine's headers from it. The amalgamated files are regenerated. The header-guard workflow checks for #pragma once instead of checkguard. Documents, the README and the release notes describe the new names. 
Catch2 compatibility means the API tests are written in: the macros, such
as TEST_CASE and REQUIRE, and the Catch:: namespace. It does not mean
Catch2's build names or header paths. find_package(Catch2) found Catch3,
and a project using both frameworks met two packages, two sets of targets,
two Catch.cmake modules and two catch2/ header trees with the same names.

Build names:
- CMake: project and package Catch3, targets Catch3::Catch3 and
  Catch3::Catch3WithMain, libraries Catch3 and Catch3Main, options
  CATCH3_*, lib/cmake/Catch3 with Catch3Config, and pkg-config catch3 and
  catch3-with-main. The version is Catch3's, 0.3.0, compatible within a
  minor version.
- The CMake scripts are Catch3.cmake (catch3_discover_tests),
  Catch3AddTests.cmake and Catch3ShardTests.cmake
  (catch3_add_sharded_tests); the helpers they define at ctest time have a
  catch3_ prefix. ParseAndAddCatchTests.cmake, deprecated upstream, is gone.
- Bazel //:catch3 and //:catch3_main, Meson catch3 and catch3-with-main
  (with override_dependency), Xmake catch3 and catch3_main, Premake Catch3
  and Catch3WithMain. Visual Studio projects get GUIDs of their own.
- The Progmasoft::Catch3 target and package are gone: property testing
  and snapshots are part of Catch3::Catch3. Their sources now build with
  the library's warning flags, so property failures are built field by
  field (no missing initializers) and the default property name is a
  thread-local pointer to the name its scope owns (no global destructor).

Headers: the engine moved from src/catch2 to src/Progmasoft/Catch3 with
Progmasoft names, so <catch2/catch_test_macros.hpp> is
<Progmasoft/Catch3/TestMacros.hpp>, <catch2/matchers/catch_matchers_string.hpp>
is <Progmasoft/Catch3/Matchers/String.hpp>, and interface headers are named
after their interface, such as Interfaces/IConfig.hpp. Include guards
became #pragma once. The installed headers go to include/Progmasoft.
src/Progmasoft/Context.cpp is StrategyContext.cpp, so no two sources share
a file name.

tools/scripts/upstreamFiles.txt lists the files that come from Catch2.
The license check expects BSL-1.0 on them and Progmasoft's header on the
rest of src/Progmasoft; the convenience-header check, the amalgamation
script and the surrogate tests take the engine's headers from it. The
amalgamated files are regenerated. The header-guard workflow checks for
#pragma once instead of checkguard.

Documents, the README and the release notes describe the new names.
Fix the CI failures of the runner and test its planner, options and analysis 2026-10-24T07:06:25+00:00 Leitwolf11 2026-10-24T07:06:25+00:00 e104b694513964d5a84e77f751203cda970d741b ExtraTests X21 and X27 defined their listeners at namespace scope, which clang rejects under -Wweak-vtables with -Werror on Linux and macOS; they are now in anonymous namespaces. After a fatal signal Catch2 closes the open sections itself, so on POSIX a crash was reported outside the section it happened in. The scheduler now ignores section ends that follow a fatal condition. New unit tests cover the planner (every selection, dependency closure and cycles, hash and duration shards, incremental runs, learned time limits, ordering, retries, quarantine and replay), every command-line option and its errors, and the failure analysis (history classes, signature groups, crash and hang locations, changed files, replay commands). Catalog::FromEntries builds a catalog without a test registry so the planner can be tested on its own. 
ExtraTests X21 and X27 defined their listeners at namespace scope, which
clang rejects under -Wweak-vtables with -Werror on Linux and macOS; they
are now in anonymous namespaces.

After a fatal signal Catch2 closes the open sections itself, so on POSIX a
crash was reported outside the section it happened in. The scheduler now
ignores section ends that follow a fatal condition.

New unit tests cover the planner (every selection, dependency closure and
cycles, hash and duration shards, incremental runs, learned time limits,
ordering, retries, quarantine and replay), every command-line option and
its errors, and the failure analysis (history classes, signature groups,
crash and hang locations, changed files, replay commands).
Catalog::FromEntries builds a catalog without a test registry so the
planner can be tested on its own.
Replace Catch2's runner with the Catch3 runner 2026-10-23T20:25:41+00:00 Leitwolf11 2026-10-23T20:25:41+00:00 93d4e7dbebab867e0a7f59fc25308460a7a76280 Catch3 keeps Catch2's API for writing tests (test cases, sections, assertions, matchers, generators, listeners, benchmarks) and replaces its session, command line and reporters with a runner of its own. The runner starts the test binary again as worker processes that take the next test when they are free. A crash, an exit or a hang ends one test, not the run. On top of that it adds: - retries that tell deterministic, flaky and order-dependent failures apart, and a bisection that names the test leaving state behind; - a history of verdicts and durations, kept under a file lock, that drives automatic quarantine, adaptive time limits, predictive ordering, incremental runs, failure replay and analytics; - selection by change, past failure or prediction, dependency closure, and sharding by a stable hash or by recorded durations; - tags for serial and isolated tests, shared resources, dependencies, time limits and retries; - a failure category, a signature, an analysis and a replay command for every failure; - console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome trace reports, and a versioned discovery listing (--list=json); - a temporary directory per test (TMPDIR, TMP, TEMP and TestTemporaryDirectory()); - Bazel's test environment: XML output file, test filter, sharding, seed, premature exit file and undeclared outputs. The command line is parsed by a hand-written, typed parser. Catch::Session remains as a thin facade over the runner for custom main functions. The result model in Progmasoft/Catch3/Results.hpp was rewritten around tests and attempts, and the JUnit writer now writes it. The property engine's example database is updated under a file lock with an atomic rename, so parallel workers can write it at the same time. Removed: Clara, the command-line and reporter-spec parsers, every reporter and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width, the JSON and XML writers, sharding in the test registry, approval tests and their baselines, Conan packaging, and the unused third_party/clara.hpp. Listeners keep working; reporters' list and skip events are gone. The Catch2 library is now compiled as C++20; code that includes only Catch2's headers still needs C++14. Duplicate test cases are still refused at start-up. Version 0.3.0. ExtraTests now turn exceptions off for clang-cl with MSVC-style options, which -fno-exceptions did not do. 
Catch3 keeps Catch2's API for writing tests (test cases, sections,
assertions, matchers, generators, listeners, benchmarks) and replaces its
session, command line and reporters with a runner of its own.

The runner starts the test binary again as worker processes that take the
next test when they are free. A crash, an exit or a hang ends one test, not
the run. On top of that it adds:

- retries that tell deterministic, flaky and order-dependent failures
  apart, and a bisection that names the test leaving state behind;
- a history of verdicts and durations, kept under a file lock, that drives
  automatic quarantine, adaptive time limits, predictive ordering,
  incremental runs, failure replay and analytics;
- selection by change, past failure or prediction, dependency closure,
  and sharding by a stable hash or by recorded durations;
- tags for serial and isolated tests, shared resources, dependencies,
  time limits and retries;
- a failure category, a signature, an analysis and a replay command for
  every failure;
- console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome
  trace reports, and a versioned discovery listing (--list=json);
- a temporary directory per test (TMPDIR, TMP, TEMP and
  TestTemporaryDirectory());
- Bazel's test environment: XML output file, test filter, sharding, seed,
  premature exit file and undeclared outputs.

The command line is parsed by a hand-written, typed parser. Catch::Session
remains as a thin facade over the runner for custom main functions.

The result model in Progmasoft/Catch3/Results.hpp was rewritten around
tests and attempts, and the JUnit writer now writes it. The property
engine's example database is updated under a file lock with an atomic
rename, so parallel workers can write it at the same time.

Removed: Clara, the command-line and reporter-spec parsers, every reporter
and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width,
the JSON and XML writers, sharding in the test registry, approval tests and
their baselines, Conan packaging, and the unused third_party/clara.hpp.
Listeners keep working; reporters' list and skip events are gone.

The Catch2 library is now compiled as C++20; code that includes only
Catch2's headers still needs C++14. Duplicate test cases are still refused
at start-up. Version 0.3.0.

ExtraTests now turn exceptions off for clang-cl with MSVC-style options,
which -fno-exceptions did not do.
Backport Catch2 3.16.1 2026-10-10T23:56:25+00:00 Leitwolf11 2026-10-10T23:56:25+00:00 1f78d8903c2fb18afea8c3055f8df83d8c60d9b5 Review Catch2 devel through 08092139210881f01a34e60c5f2fd6afa0ea4024, the commit of 3.16.1, and apply the eleven changes since the last reviewed revision in upstream order: d7683484 better message when a reporter cannot be initialized ea5020f0 custom argument validation in every first-party reporter aa01559e throwing-reporter test also throws in assertionEnded 45cd3987 noexcept on handleUnexpectedInflightException 6adbd539 MSVC optimizations enabled again after the benchmark headers e87f2882 no quadratic scaling for NTTP test registration 73d546cd REQUIRE expands its arguments fewer times under GCC and Clang b710039a decomposer operators as plain functions f899467b type-erased streaming of reconstructed expressions 8f0e70ca retired macos-14 runner dropped 08092139 version 3.16.1 and its release notes All of them applied to the sources without a conflict. The amalgamated distribution under extras/ conflicted because it is generated; it is generated again from the sources of this repository, which also brings it up to date with fork changes it had not picked up: the list-based stream pool, the Clang diagnostic boundary and the cassert include. No file under src/Progmasoft changes. The fork-owned texts that name the engine version say 3.16.1, and docs/upstream-status.md records the review. Upstream files keep their names, licenses and compiler options. Built with Clang 22 on Windows and tested: 148 of 149 tests pass. The one that fails, Reporters::CrashInJunitReporter, fails the same way on devel without this change with that compiler. 
Review Catch2 devel through 08092139210881f01a34e60c5f2fd6afa0ea4024,
the commit of 3.16.1, and apply the eleven changes since the last
reviewed revision in upstream order:

  d7683484  better message when a reporter cannot be initialized
  ea5020f0  custom argument validation in every first-party reporter
  aa01559e  throwing-reporter test also throws in assertionEnded
  45cd3987  noexcept on handleUnexpectedInflightException
  6adbd539  MSVC optimizations enabled again after the benchmark headers
  e87f2882  no quadratic scaling for NTTP test registration
  73d546cd  REQUIRE expands its arguments fewer times under GCC and Clang
  b710039a  decomposer operators as plain functions
  f899467b  type-erased streaming of reconstructed expressions
  8f0e70ca  retired macos-14 runner dropped
  08092139  version 3.16.1 and its release notes

All of them applied to the sources without a conflict. The amalgamated
distribution under extras/ conflicted because it is generated; it is
generated again from the sources of this repository, which also brings
it up to date with fork changes it had not picked up: the list-based
stream pool, the Clang diagnostic boundary and the cassert include.

No file under src/Progmasoft changes. The fork-owned texts that name
the engine version say 3.16.1, and docs/upstream-status.md records the
review. Upstream files keep their names, licenses and compiler options.

Built with Clang 22 on Windows and tested: 148 of 149 tests pass. The
one that fails, Reporters::CrashInJunitReporter, fails the same way on
devel without this change with that compiler.
Add tests for tag validation during test registration 2026-09-11T09:27:27+00:00 Martin Hořeňovský 2026-09-10T21:33:38+00:00 1767d9fad0bc28e2b80744daa93c3430d0c6d238 






Emit warning when using --shard-count (> 1) with --order rand
2026-08-10T22:08:23+00:00

Matt Rasa

2026-07-31T13:37:21+00:00

0aeb818520028ca05cc2539ed242402ef7a7ae41