Catch3 keeps Catch2's API for writing tests (test cases, sections, assertions, matchers, generators, listeners, benchmarks) and replaces its session, command line and reporters with a runner of its own. The runner starts the test binary again as worker processes that take the next test when they are free. A crash, an exit or a hang ends one test, not the run. On top of that it adds: - retries that tell deterministic, flaky and order-dependent failures apart, and a bisection that names the test leaving state behind; - a history of verdicts and durations, kept under a file lock, that drives automatic quarantine, adaptive time limits, predictive ordering, incremental runs, failure replay and analytics; - selection by change, past failure or prediction, dependency closure, and sharding by a stable hash or by recorded durations; - tags for serial and isolated tests, shared resources, dependencies, time limits and retries; - a failure category, a signature, an analysis and a replay command for every failure; - console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome trace reports, and a versioned discovery listing (--list=json); - a temporary directory per test (TMPDIR, TMP, TEMP and TestTemporaryDirectory()); - Bazel's test environment: XML output file, test filter, sharding, seed, premature exit file and undeclared outputs. The command line is parsed by a hand-written, typed parser. Catch::Session remains as a thin facade over the runner for custom main functions. The result model in Progmasoft/Catch3/Results.hpp was rewritten around tests and attempts, and the JUnit writer now writes it. The property engine's example database is updated under a file lock with an atomic rename, so parallel workers can write it at the same time. Removed: Clara, the command-line and reporter-spec parsers, every reporter and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width, the JSON and XML writers, sharding in the test registry, approval tests and their baselines, Conan packaging, and the unused third_party/clara.hpp. Listeners keep working; reporters' list and skip events are gone. The Catch2 library is now compiled as C++20; code that includes only Catch2's headers still needs C++14. Duplicate test cases are still refused at start-up. Version 0.3.0. ExtraTests now turn exceptions off for clang-cl with MSVC-style options, which -fno-exceptions did not do.
Review Catch2 devel through 08092139210881f01a34e60c5f2fd6afa0ea4024, the commit of 3.16.1, and apply the eleven changes since the last reviewed revision in upstream order: d7683484 better message when a reporter cannot be initialized ea5020f0 custom argument validation in every first-party reporter aa01559e throwing-reporter test also throws in assertionEnded 45cd3987 noexcept on handleUnexpectedInflightException 6adbd539 MSVC optimizations enabled again after the benchmark headers e87f2882 no quadratic scaling for NTTP test registration 73d546cd REQUIRE expands its arguments fewer times under GCC and Clang b710039a decomposer operators as plain functions f899467b type-erased streaming of reconstructed expressions 8f0e70ca retired macos-14 runner dropped 08092139 version 3.16.1 and its release notes All of them applied to the sources without a conflict. The amalgamated distribution under extras/ conflicted because it is generated; it is generated again from the sources of this repository, which also brings it up to date with fork changes it had not picked up: the list-based stream pool, the Clang diagnostic boundary and the cassert include. No file under src/Progmasoft changes. The fork-owned texts that name the engine version say 3.16.1, and docs/upstream-status.md records the review. Upstream files keep their names, licenses and compiler options. Built with Clang 22 on Windows and tested: 148 of 149 tests pass. The one that fails, Reporters::CrashInJunitReporter, fails the same way on devel without this change with that compiler.
The old internals reached into the global hub to stash the allocation(s) for enum value -> string value there, then kept around a potentially invalid (in case the hub was cleaned up) reference into it, going through whole bunch of virtual dispatch in the process. The new internals just store the data in a static variable inside the `StringMaker` specialization. This avoids potential lifetime issues, avoids all virtual dispatch and (almost) reduces the include bloat in the main header path. The reason for (almost) there is that for full include correctness, `EnumInfo` needs `<utility>` include for `std::pair`. However, this brings in things like `std::relops`, because the std headers in C++ are dumb. As this was not included before, and instead we relied on `std::pair` existing in an internal stdlib header that we transitively included, the full include size ends up bigger than before.
* bugprone-branch-clone * bugprone-copy-constructor-init * bugprone-empty-catch * bugprone-sizeof-expression * bugprone-switch-missing-default-case * bugprone-unused-local-non-trivial-variable * clang-analyzer-core.uninitialized.Assign * clang-analyzer-cplusplus.Move * clang-analyzer-optin.cplusplus.VirtualCall * modernize-loop-convert * modernize-raw-string-literal * modernize-use-equals-default * modernize-use-override * modernize-use-using * performance-avoid-endl * performance-inefficient-string-concatenation * performance-inefficient-vector-operation * performance-noexcept-move-constructor * performance-unnecessary-value-param (and improve generator example) * readability-duplicate-include * readability-inconsistent-declaration-parameter-name * readability-non-const-parameter * readability-redundant-casting * readability-redundant-member-init * readability-redundant-smartptr-get * readability-static-accessed-through-instance * unused variable in amalgamted tests
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
To keep the compilation firewall effect, the implementations are hidden behind a PIMPL. In this case it is probably not worth it, but we can inline it later if needed.
The license file was renamed with 6a502cc2f5fedd59b3495b58708f0d6d987ed9e1