Review Catch2 devel through 08092139210881f01a34e60c5f2fd6afa0ea4024, the commit of 3.16.1, and apply the eleven changes since the last reviewed revision in upstream order: d7683484 better message when a reporter cannot be initialized ea5020f0 custom argument validation in every first-party reporter aa01559e throwing-reporter test also throws in assertionEnded 45cd3987 noexcept on handleUnexpectedInflightException 6adbd539 MSVC optimizations enabled again after the benchmark headers e87f2882 no quadratic scaling for NTTP test registration 73d546cd REQUIRE expands its arguments fewer times under GCC and Clang b710039a decomposer operators as plain functions f899467b type-erased streaming of reconstructed expressions 8f0e70ca retired macos-14 runner dropped 08092139 version 3.16.1 and its release notes All of them applied to the sources without a conflict. The amalgamated distribution under extras/ conflicted because it is generated; it is generated again from the sources of this repository, which also brings it up to date with fork changes it had not picked up: the list-based stream pool, the Clang diagnostic boundary and the cassert include. No file under src/Progmasoft changes. The fork-owned texts that name the engine version say 3.16.1, and docs/upstream-status.md records the review. Upstream files keep their names, licenses and compiler options. Built with Clang 22 on Windows and tested: 148 of 149 tests pass. The one that fails, Reporters::CrashInJunitReporter, fails the same way on devel without this change with that compiler.
Ideally JUnit tools would handle multiple failures per test case, but they don't, so here we are. Closes #1919
The fast path allows `RunContext` to skip disabling output redirect, and notifying the reporters, turning `RunContext::notifyAssertionStarted` into a no-op. This improves the overall performance of assertion handling significantly, and also prepares ground for future changes around assertion handling and thread safety. For simple 10M assertion run, this improves the running time by ~30% in Debug build and ~40% in Release build. For backwards-compatibility reasons, the fast path is disabled by default. However, none of the first party reporters use the `assertionStarting` event, so all first party reporters are opted-in.
This is done by no longer requiring all assertions to be seen
by the JUnit reporter. Since the JUnit reporter never outputs
all assertions, even with `-s`, the only difference from storing
passing assertions in the `CumulativeReporterBase` was some
bookkeeping in deciding which sections are relevant to the output.
Given `TEST_CASE` that just runs many passing assertions, e.g.
```
TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) {
for (size_t i = 0; i < 1000'000'000; ++i) {
REQUIRE( i == i );
}
}
```
the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM.
The old JUnit reporter would finish in 0:30, due to bad_alloc
after using up 14.5 GB of RAM (the system has 16 GB total).
If the total number of assertions is lowered to 10M, the old
implementation uses up ~7.1 GB of RAM and finishes in 12 minutes.
The new implementation still needs only ~7.7 MB of RAM, and finishes
in 4 minutes.
There is a slight downside in that the output is slightly different;
the new implementation will include the `TEST_CASE` level section
in output, even if it does not have its own assertion. In other words,
given a `TEST_CASE` like this
```
TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) {
std::stringstream stream;
SECTION( "Newly constructed JsonWriter does nothing" ) {
Catch::JsonValueWriter writer{ stream };
REQUIRE( stream.str() == "" );
}
SECTION( "Calling writeObject will create an empty pair of braces" ) {
{ auto writer = Catch::JsonValueWriter{ stream }.writeObject(); }
REQUIRE( stream.str() == "{\n}" );
}
}
```
the new implementation will output 3 `testcase` tags, 2 for the explicit
`SECTION`s with tests, and 1 for the top level section.
However, this can be worked-around if required, and the performance
improvement is such that it is worth changing the current output,
even if it needs to be fixed in the future.
Closes #2897
* bugprone-branch-clone * bugprone-copy-constructor-init * bugprone-empty-catch * bugprone-sizeof-expression * bugprone-switch-missing-default-case * bugprone-unused-local-non-trivial-variable * clang-analyzer-core.uninitialized.Assign * clang-analyzer-cplusplus.Move * clang-analyzer-optin.cplusplus.VirtualCall * modernize-loop-convert * modernize-raw-string-literal * modernize-use-equals-default * modernize-use-override * modernize-use-using * performance-avoid-endl * performance-inefficient-string-concatenation * performance-inefficient-vector-operation * performance-noexcept-move-constructor * performance-unnecessary-value-param (and improve generator example) * readability-duplicate-include * readability-inconsistent-declaration-parameter-name * readability-non-const-parameter * readability-redundant-casting * readability-redundant-member-init * readability-redundant-smartptr-get * readability-static-accessed-through-instance * unused variable in amalgamted tests
* Add new SKIP macro for skipping tests at runtime This adds a new `SKIP` macro for dynamically skipping tests at runtime. The "skipped" status of a test case is treated as a first-class citizen, like "succeeded" or "failed", and is reported with a new color on the console. * Don't show "skipped assertions" in console/compact reporters Also extend skip tests to cover a few more use cases. * Return exit code 4 if all test cases are skipped * Use LightGrey for the skip colour This isn't great, but is better than the deep blue that was borderline invisible on dark backgrounds. The fix is to redo the colouring a bit, including introducing light-blue that is actually visible. * Add support for explicit skips in all reporters * --allow-running-no-tests also allows all tests to be skipped * Add docs for SKIP macro, deprecate IEventListener::skipTest Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>