Review Catch2 devel through 08092139210881f01a34e60c5f2fd6afa0ea4024, the commit of 3.16.1, and apply the eleven changes since the last reviewed revision in upstream order: d7683484 better message when a reporter cannot be initialized ea5020f0 custom argument validation in every first-party reporter aa01559e throwing-reporter test also throws in assertionEnded 45cd3987 noexcept on handleUnexpectedInflightException 6adbd539 MSVC optimizations enabled again after the benchmark headers e87f2882 no quadratic scaling for NTTP test registration 73d546cd REQUIRE expands its arguments fewer times under GCC and Clang b710039a decomposer operators as plain functions f899467b type-erased streaming of reconstructed expressions 8f0e70ca retired macos-14 runner dropped 08092139 version 3.16.1 and its release notes All of them applied to the sources without a conflict. The amalgamated distribution under extras/ conflicted because it is generated; it is generated again from the sources of this repository, which also brings it up to date with fork changes it had not picked up: the list-based stream pool, the Clang diagnostic boundary and the cassert include. No file under src/Progmasoft changes. The fork-owned texts that name the engine version say 3.16.1, and docs/upstream-status.md records the review. Upstream files keep their names, licenses and compiler options. Built with Clang 22 on Windows and tested: 148 of 149 tests pass. The one that fails, Reporters::CrashInJunitReporter, fails the same way on devel without this change with that compiler.
Catch2 defaults to random test execution order, which has to process all tests present in the binary, even if they will be filtered down to just 1 later. For the `SelfTest` binary, the overhead from this is about 20% in Debug configuration and 5% in Release. By registering the CTest tests with `--order decl`, we avoid this overhead.
Closes #2149
Previously, the `catch_discover_tests` would prepare the entire CTest command (e.g. `add_test(...)` or `set_tests_properties(...)`) first, and then escape it when finished. However, this caused lot of the command args to be escaped over and over again (e.g. executable name or Catch2's reporter args), for no reason, as they were always the same, and thus their escaping was always the same. Until recently, the performance overhead didn't matter as there were many spots which had quadratic runtime in number of tests. However, the recent refactorings fixed these, and this commit now improves the throughput by 10-20%. Also extended the benchmarked COUNTS in `benchmark_discovery.py`, because the performance is now good enough that it is reasonable to benchmark 16k tests.
This was another place where the script triggered the quadratic runtime from calling `string(APPEND` (or `list(APPEND`) repeatedly. As in 60c8b87, we avoid this by flushing the test list into a file every 50kB of text input. As the string with the test list never grew quite as much as the string that contains all the test definitions, this only provides significant savings for high number of tests. It starts being properly measurable around 4k tests at ~100ms, but grows to ~4s at 32k tests.
There were 2 separate bugs, one recently added and one ancient (since roughly the first version of the registration script). 1) The recent refactoring of how the JSON output from Catch2 is parsed caused shadowing between the variable containing the per-test JSON fragments and the accumulator of test names for <target>_TESTS variable. This led to the variable containing both the names and the JSON fragments, and thus being completely wrong. 2) The test name accumulation has never accounted for characters that need escaping to be present in a CMake list. This means that e.g. test names with semicolon in them would end up with two elements in the test list. Both of these are now fixed, at the cost of extra complexity and my sanity as I had to learn more about CMake escaping rules. (CMake escaping rules are dumb)
* Quiet verbosity provides just the test names * Normal verbosity adds tags * High verbosity add the source location of the tests Listing tests with the quiet verbosity results in about 1/4 of the previous output, which leads to measurably faster execution of `catch_discover_tests` when it does not need tags for labels. (If it does need tags, the output is about 1/2 of previous).