<feed xmlns='http://www.w3.org/2005/Atom'>
<title>catch3/tests/CMakeLists.txt, branch feature/native-property-engine</title>
<subtitle>Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.</subtitle>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/'/>
<entry>
<title>Add a native property-testing engine with choice-sequence shrinking</title>
<updated>2026-10-22T14:39:46+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-22T14:39:46+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=afac14ac5a62d8d2e7efd2184ad34e16593eddfd'/>
<id>afac14ac5a62d8d2e7efd2184ad34e16593eddfd</id>
<content type='text'>
Catch3's own property engine, independent of RapidCheck, next to the
existing CheckProperty API, which stays unchanged.

Strategies (Progmasoft/Catch3/Strategy/): a Strategy&lt;T&gt; builds a value
from a ChoiceSource, a stream of integer choices where 0 is always the
simplest decision. Gen:: provides Integers, Booleans, Floats (ranges,
NaN and infinities on request, edge values), Characters, Strings,
Utf8Strings, Vectors, UniqueVectors, Sets, Maps, Optionals, Tuples,
Construct, Permutations, Just, SampledFrom, OneOf, Frequency, Lazy and
Recursive; every strategy has Map, Filter and FlatMap.

Shrinking: the engine records the choices of a failing trial and
shrinks them, not the value, with deletion and zeroing of blocks,
binary search per choice with a neighbour step for non-monotone
conditions, sorting and moving value between neighbours. Replaying the
shrunk choices through the same strategies means mapped, filtered,
dependent and stateful inputs all shrink, and a shrunk input still
satisfies its filters.

ForAll([options,] strategy..., body) runs the trials (size rising over
the first half), then a targeted search when the body calls Target,
then checks Cover requirements. A failure report carries the
minimized input, the notes and a replay token (PropertyOptions::Replay)
that runs exactly that input. An example database (DatabasePath or
CATCH3_PROPERTY_DATABASE) replays stored failures first and drops them
once they pass. Bodies return bool, or void with CATCH3_PROPERTY_ASSERT
and CATCH3_PROPERTY_ASSUME, which return instead of throwing; Note,
Classify, Cover and Target record into a per-thread trial record.
CATCH3_CHECK_FOR_ALL and CATCH3_REQUIRE_FOR_ALL report through Catch2
and name the property after the test case for the database.

CheckStateful runs model-based command sequences against a model and
the system under test; preconditions and state-dependent arguments are
replayed during shrinking. Relations.hpp provides Equivalent
(differential), Metamorphic, RoundTrip and Idempotent bodies.

PropertyOptions, PropertyStatus, PropertyFailure and PropertyResult
gain fields and statuses at their ends (kInsufficientCoverage,
kGaveUp); existing designated initializers and reports are unchanged.
The engine's untemplated parts are compiled in ChoiceShrinker.cpp,
PropertyEngine.cpp and Context.cpp, and every build system (Bazel,
CMake, Meson, Xmake, Premake) lists the new files and tests.

Docs: progmasoft-properties.md describes the engine,
progmasoft-roadmap.md the steps that follow (the standalone catch3
runner, mocking, per-test coverage, change-impact selection and
prioritization, mutation testing, guided test generation). The
.clang-format comment names Progmasoft C++ Style instead of Visual X#.

Verified with clang-cl 23 through CMake: ProgmasoftSelfTest passes, 57
test cases and 75,461 assertions, including the shrinking results
(1001 for x &lt;= 1000, { 10 }, { 0, 0, 0 } through FlatMap, 102 through
a filter, { 0, 100 } for two arguments, four push(0) steps for a
queue that drops its fourth element, { 0, 0 } for a sort that loses
duplicates) and the database, replay, coverage and discard behaviour.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Catch3's own property engine, independent of RapidCheck, next to the
existing CheckProperty API, which stays unchanged.

Strategies (Progmasoft/Catch3/Strategy/): a Strategy&lt;T&gt; builds a value
from a ChoiceSource, a stream of integer choices where 0 is always the
simplest decision. Gen:: provides Integers, Booleans, Floats (ranges,
NaN and infinities on request, edge values), Characters, Strings,
Utf8Strings, Vectors, UniqueVectors, Sets, Maps, Optionals, Tuples,
Construct, Permutations, Just, SampledFrom, OneOf, Frequency, Lazy and
Recursive; every strategy has Map, Filter and FlatMap.

Shrinking: the engine records the choices of a failing trial and
shrinks them, not the value, with deletion and zeroing of blocks,
binary search per choice with a neighbour step for non-monotone
conditions, sorting and moving value between neighbours. Replaying the
shrunk choices through the same strategies means mapped, filtered,
dependent and stateful inputs all shrink, and a shrunk input still
satisfies its filters.

ForAll([options,] strategy..., body) runs the trials (size rising over
the first half), then a targeted search when the body calls Target,
then checks Cover requirements. A failure report carries the
minimized input, the notes and a replay token (PropertyOptions::Replay)
that runs exactly that input. An example database (DatabasePath or
CATCH3_PROPERTY_DATABASE) replays stored failures first and drops them
once they pass. Bodies return bool, or void with CATCH3_PROPERTY_ASSERT
and CATCH3_PROPERTY_ASSUME, which return instead of throwing; Note,
Classify, Cover and Target record into a per-thread trial record.
CATCH3_CHECK_FOR_ALL and CATCH3_REQUIRE_FOR_ALL report through Catch2
and name the property after the test case for the database.

CheckStateful runs model-based command sequences against a model and
the system under test; preconditions and state-dependent arguments are
replayed during shrinking. Relations.hpp provides Equivalent
(differential), Metamorphic, RoundTrip and Idempotent bodies.

PropertyOptions, PropertyStatus, PropertyFailure and PropertyResult
gain fields and statuses at their ends (kInsufficientCoverage,
kGaveUp); existing designated initializers and reports are unchanged.
The engine's untemplated parts are compiled in ChoiceShrinker.cpp,
PropertyEngine.cpp and Context.cpp, and every build system (Bazel,
CMake, Meson, Xmake, Premake) lists the new files and tests.

Docs: progmasoft-properties.md describes the engine,
progmasoft-roadmap.md the steps that follow (the standalone catch3
runner, mocking, per-test coverage, change-impact selection and
prioritization, mutation testing, guided test generation). The
.clang-format comment names Progmasoft C++ Style instead of Visual X#.

Verified with clang-cl 23 through CMake: ProgmasoftSelfTest passes, 57
test cases and 75,461 assertions, including the shrinking results
(1001 for x &lt;= 1000, { 10 }, { 0, 0, 0 } through FlatMap, 102 through
a filter, { 0, 100 } for two arguments, four push(0) steps for a
queue that drops its fourth element, { 0, 0 } for a sort that loses
duplicates) and the database, replay, coverage and discard behaviour.
</pre>
</div>
</content>
</entry>
<entry>
<title>Add composite property generators, single-trial replay and stable counterexample printing</title>
<updated>2026-10-02T00:27:05+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-02T00:27:05+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=5d55a3226360fae1eaa5ef91908b0fa7d7d07234'/>
<id>5d55a3226360fae1eaa5ef91908b0fa7d7d07234</id>
<content type='text'>
Property tests could only generate one integer or Boolean. This adds the
composite strategies the documentation listed as missing, without changing
any Catch2 file or the existing Progmasoft API.

Generators:
- Tuple(generators...) draws components left to right and shrinks one
  component at a time, which is how a property receives several arguments.
- Vector(element, minimumSize, maximumSize) and Text(alphabet,
  minimumLength, maximumLength) share one ordered shrink: shortest allowed
  prefix, the two halves, single removals, then single element
  simplifications. A shrunk value never falls below its minimum, each step
  offers at most 256 candidates, and a size limit above 1,048,576 is
  rejected.
- Element(values) picks from a fixed list and shrinks toward earlier
  entries.

Runner:
- PropertyOptions::ReplayTrialSeed runs exactly one trial with a reported
  replay seed, including its shrinking, so a failure from a log can be
  reproduced without rerunning earlier trials. The report now ends with the
  line that names this option and the seed.
- Counterexamples of tuples, pairs and vectors are printed element by
  element instead of relying on Catch2's optional string makers, so the
  report does not depend on configuration macros or include order.

The headers still compile for clients built without language exceptions.
Thirteen new test cases cover bounds, determinism, shrink order, candidate
limits, minimized counterexamples, replay and invalid configurations, and
every build description (Bazel, CMake, Meson, Xmake, Premake) lists the new
headers and test file.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Property tests could only generate one integer or Boolean. This adds the
composite strategies the documentation listed as missing, without changing
any Catch2 file or the existing Progmasoft API.

Generators:
- Tuple(generators...) draws components left to right and shrinks one
  component at a time, which is how a property receives several arguments.
- Vector(element, minimumSize, maximumSize) and Text(alphabet,
  minimumLength, maximumLength) share one ordered shrink: shortest allowed
  prefix, the two halves, single removals, then single element
  simplifications. A shrunk value never falls below its minimum, each step
  offers at most 256 candidates, and a size limit above 1,048,576 is
  rejected.
- Element(values) picks from a fixed list and shrinks toward earlier
  entries.

Runner:
- PropertyOptions::ReplayTrialSeed runs exactly one trial with a reported
  replay seed, including its shrinking, so a failure from a log can be
  reproduced without rerunning earlier trials. The report now ends with the
  line that names this option and the seed.
- Counterexamples of tuples, pairs and vectors are printed element by
  element instead of relying on Catch2's optional string makers, so the
  report does not depend on configuration macros or include order.

The headers still compile for clients built without language exceptions.
Thirteen new test cases cover bounds, determinism, shrink order, candidate
limits, minimized counterexamples, replay and invalid configurations, and
every build description (Bazel, CMake, Meson, Xmake, Premake) lists the new
headers and test file.
</pre>
</div>
</content>
</entry>
<entry>
<title>Add snapshots and JUnit result serialization</title>
<updated>2026-09-24T20:05:47+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-09-24T20:05:47+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=d79b294903e4d0f4b24fdf1cc50be4b0fec7f745'/>
<id>d79b294903e4d0f4b24fdf1cc50be4b0fec7f745</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Add Progmasoft Catch3 property testing and build support</title>
<updated>2026-09-24T19:07:03+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-09-24T19:07:03+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=9ad804cedf4cc4a85ef89fe097f97a3b44654360'/>
<id>9ad804cedf4cc4a85ef89fe097f97a3b44654360</id>
<content type='text'>
Introduce deterministic property execution, built-in generators, shrinking, replayable seeds, and a separate Progmasoft-owned header API while preserving Catch2 compatibility targets and the C++14 engine baseline.

Add Bazel, Meson, Xmake, and Premake integration and tests alongside the existing CMake build. Replace Linux-only workflow coverage with Windows and macOS jobs and document the Catch2/Catch3 distinction.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Introduce deterministic property execution, built-in generators, shrinking, replayable seeds, and a separate Progmasoft-owned header API while preserving Catch2 compatibility targets and the C++14 engine baseline.

Add Bazel, Meson, Xmake, and Premake integration and tests alongside the existing CMake build. Replace Linux-only workflow coverage with Windows and macOS jobs and document the Catch2/Catch3 distinction.
</pre>
</div>
</content>
</entry>
<entry>
<title>Pass the parent toolchain to CMake configuration tests</title>
<updated>2026-09-14T12:08:28+00:00</updated>
<author>
<name>宁岑</name>
</author>
<published>2026-09-08T09:37:45+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=02f8b5593a15cdec8558256e6f40a0c7fb5862d6'/>
<id>02f8b5593a15cdec8558256e6f40a0c7fb5862d6</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Add optimizer barrier to benchmark calls of void-returning functions</title>
<updated>2026-08-22T18:43:13+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-22T13:19:33+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=45f9ea1e798c2e128dcba53602ccbddfe5676f6f'/>
<id>45f9ea1e798c2e128dcba53602ccbddfe5676f6f</id>
<content type='text'>
The goal is to do the equivalent of clobbering memory, which forces
the compiler to keep side effects that are external to the benchmarked
function, while allowing it to optimize inside the benchmarked function.

E.g. this cannot be optimized away:
```cpp
BENCHMARK("foo") {
    global_count += 1;
};
```

while this can:
```cpp
BENCHMARK("bar") {
    size_t local_count = 0;
    for (size_t i = 0; i &lt; 100; ++i) {
        local_count += 1;
    }
};
```
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The goal is to do the equivalent of clobbering memory, which forces
the compiler to keep side effects that are external to the benchmarked
function, while allowing it to optimize inside the benchmarked function.

E.g. this cannot be optimized away:
```cpp
BENCHMARK("foo") {
    global_count += 1;
};
```

while this can:
```cpp
BENCHMARK("bar") {
    size_t local_count = 0;
    for (size_t i = 0; i &lt; 100; ++i) {
        local_count += 1;
    }
};
```
</pre>
</div>
</content>
</entry>
<entry>
<title>catch_discover_tests: Escape test-invariant parts of CTest script only once</title>
<updated>2026-08-10T09:53:22+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-10T09:45:08+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=97ec4e8e2e4a86719701746d1cb2426f16292ff8'/>
<id>97ec4e8e2e4a86719701746d1cb2426f16292ff8</id>
<content type='text'>
Previously, the `catch_discover_tests` would prepare the entire CTest
command (e.g. `add_test(...)` or `set_tests_properties(...)`) first, and
then escape it when finished. However, this caused lot of the command args
to be escaped over and over again (e.g. executable name or Catch2's reporter
args), for no reason, as they were always the same, and thus their escaping
was always the same.

Until recently, the performance overhead didn't matter as there were many
spots which had quadratic runtime in number of tests. However, the recent
refactorings fixed these, and this commit now improves the throughput by
10-20%.

Also extended the benchmarked COUNTS in `benchmark_discovery.py`,
because the performance is now good enough that it is reasonable
to benchmark 16k tests.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Previously, the `catch_discover_tests` would prepare the entire CTest
command (e.g. `add_test(...)` or `set_tests_properties(...)`) first, and
then escape it when finished. However, this caused lot of the command args
to be escaped over and over again (e.g. executable name or Catch2's reporter
args), for no reason, as they were always the same, and thus their escaping
was always the same.

Until recently, the performance overhead didn't matter as there were many
spots which had quadratic runtime in number of tests. However, the recent
refactorings fixed these, and this commit now improves the throughput by
10-20%.

Also extended the benchmarked COUNTS in `benchmark_discovery.py`,
because the performance is now good enough that it is reasonable
to benchmark 16k tests.
</pre>
</div>
</content>
</entry>
<entry>
<title>JSONReporter considers verbosity when listing tests</title>
<updated>2026-08-07T22:05:18+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-07-28T20:08:55+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=4cde12851773b32fc5a0171f262e4281c8e28b01'/>
<id>4cde12851773b32fc5a0171f262e4281c8e28b01</id>
<content type='text'>
* Quiet verbosity provides just the test names
* Normal verbosity adds tags
* High verbosity add the source location of the tests

Listing tests with the quiet verbosity results in about 1/4 of the
previous output, which leads to measurably faster execution of
`catch_discover_tests` when it does not need tags for labels.
(If it does need tags, the output is about 1/2 of previous).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* Quiet verbosity provides just the test names
* Normal verbosity adds tags
* High verbosity add the source location of the tests

Listing tests with the quiet verbosity results in about 1/4 of the
previous output, which leads to measurably faster execution of
`catch_discover_tests` when it does not need tags for labels.
(If it does need tags, the output is about 1/2 of previous).
</pre>
</div>
</content>
</entry>
<entry>
<title>Make verbosity per-reporter</title>
<updated>2026-08-07T18:57:42+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-01T14:52:08+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=a0eba50e9d56c743aff1de7a50d02a6e01e73cdc'/>
<id>a0eba50e9d56c743aff1de7a50d02a6e01e73cdc</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Avoid quadratic JSON array parse behaviour in `catch_discover_tests`</title>
<updated>2026-07-26T17:51:58+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-07-26T11:15:10+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=1079da4c5f5fea2a31d795bba95eacaa2f622849'/>
<id>1079da4c5f5fea2a31d795bba95eacaa2f622849</id>
<content type='text'>
Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985

This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.

This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}&lt;ws&gt;*,&lt;ws&gt;*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.

This is all around a hilariously terrible idea, however:

1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
   with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985

This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.

This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}&lt;ws&gt;*,&lt;ws&gt;*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.

This is all around a hilariously terrible idea, however:

1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
   with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
</pre>
</div>
</content>
</entry>
</feed>
