<feed xmlns='http://www.w3.org/2005/Atom'>
<title>catch3/tests, branch v3.16.0</title>
<subtitle>Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.</subtitle>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/'/>
<entry>
<title>Improve String matcher handling of case sensitivity</title>
<updated>2026-08-24T13:11:48+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-24T13:11:44+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=fc5bc1fa8b57326b5bc8eaa189b3534b99913d07'/>
<id>fc5bc1fa8b57326b5bc8eaa189b3534b99913d07</id>
<content type='text'>
The old implementation of String matchers worked by making a, potentially
lower-cased, copy of both the constructor arg, and the arg to `match` calls.

The new implementation does not make copies, and instead does per-character
lowering if needed. This is a win in **most** cases, with two exceptions:

1) Case-insensitive `Equals` between long strings. The old implementation
   is easy to autovectorize by the compiler, which wins over the extra copy
   with sufficiently long inputs.
2) Case-insensitive `ContainsSubstring` with inputs that trigger the worst
   case complexity of O(m*n) of the naive implementation.

The latter is fixable with some elbow grease by implementing a proper
string search algorithm, but the former is a cost we have to live with.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The old implementation of String matchers worked by making a, potentially
lower-cased, copy of both the constructor arg, and the arg to `match` calls.

The new implementation does not make copies, and instead does per-character
lowering if needed. This is a win in **most** cases, with two exceptions:

1) Case-insensitive `Equals` between long strings. The old implementation
   is easy to autovectorize by the compiler, which wins over the extra copy
   with sufficiently long inputs.
2) Case-insensitive `ContainsSubstring` with inputs that trigger the worst
   case complexity of O(m*n) of the naive implementation.

The latter is fixable with some elbow grease by implementing a proper
string search algorithm, but the former is a cost we have to live with.
</pre>
</div>
</content>
</entry>
<entry>
<title>Add optimizer barrier to benchmark calls of void-returning functions</title>
<updated>2026-08-22T18:43:13+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-22T13:19:33+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=45f9ea1e798c2e128dcba53602ccbddfe5676f6f'/>
<id>45f9ea1e798c2e128dcba53602ccbddfe5676f6f</id>
<content type='text'>
The goal is to do the equivalent of clobbering memory, which forces
the compiler to keep side effects that are external to the benchmarked
function, while allowing it to optimize inside the benchmarked function.

E.g. this cannot be optimized away:
```cpp
BENCHMARK("foo") {
    global_count += 1;
};
```

while this can:
```cpp
BENCHMARK("bar") {
    size_t local_count = 0;
    for (size_t i = 0; i &lt; 100; ++i) {
        local_count += 1;
    }
};
```
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The goal is to do the equivalent of clobbering memory, which forces
the compiler to keep side effects that are external to the benchmarked
function, while allowing it to optimize inside the benchmarked function.

E.g. this cannot be optimized away:
```cpp
BENCHMARK("foo") {
    global_count += 1;
};
```

while this can:
```cpp
BENCHMARK("bar") {
    size_t local_count = 0;
    for (size_t i = 0; i &lt; 100; ++i) {
        local_count += 1;
    }
};
```
</pre>
</div>
</content>
</entry>
<entry>
<title>Support TEST_{PREFIX,SUFFIX} with leading/trailing whitespace</title>
<updated>2026-08-20T14:24:18+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-20T12:45:30+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=8582805f548e3e091d8d35188f4c3a690f989b0f'/>
<id>8582805f548e3e091d8d35188f4c3a690f989b0f</id>
<content type='text'>
Closes #2149
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Closes #2149
</pre>
</div>
</content>
</entry>
<entry>
<title>Emit warning when using --shard-count (&gt; 1) with --order rand</title>
<updated>2026-08-10T22:08:23+00:00</updated>
<author>
<name>Matt Rasa</name>
</author>
<published>2026-07-31T13:37:21+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=0aeb818520028ca05cc2539ed242402ef7a7ae41'/>
<id>0aeb818520028ca05cc2539ed242402ef7a7ae41</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>catch_discover_tests: Escape test-invariant parts of CTest script only once</title>
<updated>2026-08-10T09:53:22+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-10T09:45:08+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=97ec4e8e2e4a86719701746d1cb2426f16292ff8'/>
<id>97ec4e8e2e4a86719701746d1cb2426f16292ff8</id>
<content type='text'>
Previously, the `catch_discover_tests` would prepare the entire CTest
command (e.g. `add_test(...)` or `set_tests_properties(...)`) first, and
then escape it when finished. However, this caused lot of the command args
to be escaped over and over again (e.g. executable name or Catch2's reporter
args), for no reason, as they were always the same, and thus their escaping
was always the same.

Until recently, the performance overhead didn't matter as there were many
spots which had quadratic runtime in number of tests. However, the recent
refactorings fixed these, and this commit now improves the throughput by
10-20%.

Also extended the benchmarked COUNTS in `benchmark_discovery.py`,
because the performance is now good enough that it is reasonable
to benchmark 16k tests.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Previously, the `catch_discover_tests` would prepare the entire CTest
command (e.g. `add_test(...)` or `set_tests_properties(...)`) first, and
then escape it when finished. However, this caused lot of the command args
to be escaped over and over again (e.g. executable name or Catch2's reporter
args), for no reason, as they were always the same, and thus their escaping
was always the same.

Until recently, the performance overhead didn't matter as there were many
spots which had quadratic runtime in number of tests. However, the recent
refactorings fixed these, and this commit now improves the throughput by
10-20%.

Also extended the benchmarked COUNTS in `benchmark_discovery.py`,
because the performance is now good enough that it is reasonable
to benchmark 16k tests.
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix &lt;target&gt;_TESTS variable generated by `catch_discover_tests`</title>
<updated>2026-08-09T22:17:26+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-08T12:59:04+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=630840c5000ec9f5e169c8a2f5fd163d3b0c54ca'/>
<id>630840c5000ec9f5e169c8a2f5fd163d3b0c54ca</id>
<content type='text'>
There were 2 separate bugs, one recently added and one ancient
(since roughly the first version of the registration script).

1) The recent refactoring of how the JSON output from Catch2 is
   parsed caused shadowing between the variable containing the
   per-test JSON fragments and the accumulator of test names for
   &lt;target&gt;_TESTS variable. This led to the variable containing
   both the names and the JSON fragments, and thus being completely
   wrong.

2) The test name accumulation has never accounted for characters
   that need escaping to be present in a CMake list. This means
   that e.g. test names with semicolon in them would end up with
   two elements in the test list.

Both of these are now fixed, at the cost of extra complexity and
my sanity as I had to learn more about CMake escaping rules.

(CMake escaping rules are dumb)
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
There were 2 separate bugs, one recently added and one ancient
(since roughly the first version of the registration script).

1) The recent refactoring of how the JSON output from Catch2 is
   parsed caused shadowing between the variable containing the
   per-test JSON fragments and the accumulator of test names for
   &lt;target&gt;_TESTS variable. This led to the variable containing
   both the names and the JSON fragments, and thus being completely
   wrong.

2) The test name accumulation has never accounted for characters
   that need escaping to be present in a CMake list. This means
   that e.g. test names with semicolon in them would end up with
   two elements in the test list.

Both of these are now fixed, at the cost of extra complexity and
my sanity as I had to learn more about CMake escaping rules.

(CMake escaping rules are dumb)
</pre>
</div>
</content>
</entry>
<entry>
<title>JSONReporter considers verbosity when listing tests</title>
<updated>2026-08-07T22:05:18+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-07-28T20:08:55+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=4cde12851773b32fc5a0171f262e4281c8e28b01'/>
<id>4cde12851773b32fc5a0171f262e4281c8e28b01</id>
<content type='text'>
* Quiet verbosity provides just the test names
* Normal verbosity adds tags
* High verbosity add the source location of the tests

Listing tests with the quiet verbosity results in about 1/4 of the
previous output, which leads to measurably faster execution of
`catch_discover_tests` when it does not need tags for labels.
(If it does need tags, the output is about 1/2 of previous).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* Quiet verbosity provides just the test names
* Normal verbosity adds tags
* High verbosity add the source location of the tests

Listing tests with the quiet verbosity results in about 1/4 of the
previous output, which leads to measurably faster execution of
`catch_discover_tests` when it does not need tags for labels.
(If it does need tags, the output is about 1/2 of previous).
</pre>
</div>
</content>
</entry>
<entry>
<title>JSONReporter's listTests only lists class-name if it is non-empty</title>
<updated>2026-08-07T21:07:46+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-07-29T08:05:39+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=0b4d7a5a1696dbdfb23fccad8c50db9dc9a417ea'/>
<id>0b4d7a5a1696dbdfb23fccad8c50db9dc9a417ea</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Make verbosity per-reporter</title>
<updated>2026-08-07T18:57:42+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-01T14:52:08+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=a0eba50e9d56c743aff1de7a50d02a6e01e73cdc'/>
<id>a0eba50e9d56c743aff1de7a50d02a6e01e73cdc</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Avoid quadratic JSON array parse behaviour in `catch_discover_tests`</title>
<updated>2026-07-26T17:51:58+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-07-26T11:15:10+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=1079da4c5f5fea2a31d795bba95eacaa2f622849'/>
<id>1079da4c5f5fea2a31d795bba95eacaa2f622849</id>
<content type='text'>
Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985

This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.

This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}&lt;ws&gt;*,&lt;ws&gt;*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.

This is all around a hilariously terrible idea, however:

1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
   with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Using CMake's `string(JSON` to parse JSON array leads to quadratic
running time in number of tests, see https://gitlab.kitware.com/cmake/cmake/-/work_items/27985

This leads to _terrible_ runtime for `catch_discover_tests` when called
on binaries with lot of tests (1k+). To get reasonable runtimes, we have
to avoid using `string(JSON` to parse out the individual test objects
from the array with all tests.

This commit replaces the sane approach of using real JSON parser with
a set of terrible hacks, where we use CMake's string APIs to split the
JSON array on what looks like object boundary (`}&lt;ws&gt;*,&lt;ws&gt;*{`), and then
checking whether the resulting thing can be parsed as JSON object. If not,
we append the next piece and check again. And again, and again, until we
get a proper JSON object.

This is all around a hilariously terrible idea, however:

1) It works in practice for all tested inputs.
2) It improves the time it takes to run `catch_discover_tests` on binary
   with 1k tests from 4.2s to 1.1s and 2k tests from 16s to 3.9s.
</pre>
</div>
</content>
</entry>
</feed>
