Catch3 keeps Catch2's API for writing tests (test cases, sections, assertions, matchers, generators, listeners, benchmarks) and replaces its session, command line and reporters with a runner of its own. The runner starts the test binary again as worker processes that take the next test when they are free. A crash, an exit or a hang ends one test, not the run. On top of that it adds: - retries that tell deterministic, flaky and order-dependent failures apart, and a bisection that names the test leaving state behind; - a history of verdicts and durations, kept under a file lock, that drives automatic quarantine, adaptive time limits, predictive ordering, incremental runs, failure replay and analytics; - selection by change, past failure or prediction, dependency closure, and sharding by a stable hash or by recorded durations; - tags for serial and isolated tests, shared resources, dependencies, time limits and retries; - a failure category, a signature, an analysis and a replay command for every failure; - console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome trace reports, and a versioned discovery listing (--list=json); - a temporary directory per test (TMPDIR, TMP, TEMP and TestTemporaryDirectory()); - Bazel's test environment: XML output file, test filter, sharding, seed, premature exit file and undeclared outputs. The command line is parsed by a hand-written, typed parser. Catch::Session remains as a thin facade over the runner for custom main functions. The result model in Progmasoft/Catch3/Results.hpp was rewritten around tests and attempts, and the JUnit writer now writes it. The property engine's example database is updated under a file lock with an atomic rename, so parallel workers can write it at the same time. Removed: Clara, the command-line and reporter-spec parsers, every reporter and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width, the JSON and XML writers, sharding in the test registry, approval tests and their baselines, Conan packaging, and the unused third_party/clara.hpp. Listeners keep working; reporters' list and skip events are gone. The Catch2 library is now compiled as C++20; code that includes only Catch2's headers still needs C++14. Duplicate test cases are still refused at start-up. Version 0.3.0. ExtraTests now turn exceptions off for clang-cl with MSVC-style options, which -fno-exceptions did not do.
The new implementation keeps track of found escape codes in a separate vector, which has two advantages: 1) The handling of inputs without any escape codes is much faster. As this will be the case for majority of inputs, this should provide nice speedup all around. 2) We no longer mangle unrelated 0xFFs in the user-provided input. Closes #3199
We generally do not even try to pretend to support more than plain ASCII, because doing unicode in old C++ standards is a pain and a half. For `toLower`, we used to wrap `std::tolower` with multiple casts to avoid UB, but calling into it is relatively slow, and it tries to handle different locales, which could cause unexpected results. In addition, the new implementation is significantly faster, which helps speed up the initial process bring up (as comparing tags, and thus test cases is now faster).
The old implementation of String matchers worked by making a, potentially lower-cased, copy of both the constructor arg, and the arg to `match` calls. The new implementation does not make copies, and instead does per-character lowering if needed. This is a win in **most** cases, with two exceptions: 1) Case-insensitive `Equals` between long strings. The old implementation is easy to autovectorize by the compiler, which wins over the extra copy with sufficiently long inputs. 2) Case-insensitive `ContainsSubstring` with inputs that trigger the worst case complexity of O(m*n) of the naive implementation. The latter is fixable with some elbow grease by implementing a proper string search algorithm, but the former is a cost we have to live with.
* Quiet verbosity provides just the test names * Normal verbosity adds tags * High verbosity add the source location of the tests Listing tests with the quiet verbosity results in about 1/4 of the previous output, which leads to measurably faster execution of `catch_discover_tests` when it does not need tags for labels. (If it does need tags, the output is about 1/2 of previous).
As the types and values sent into the writer are determined by Catch2, I do not expect to actually need this support, but it is better to have it and not be surprised in the future.
Closes #3176
The old internals reached into the global hub to stash the allocation(s) for enum value -> string value there, then kept around a potentially invalid (in case the hub was cleaned up) reference into it, going through whole bunch of virtual dispatch in the process. The new internals just store the data in a static variable inside the `StringMaker` specialization. This avoids potential lifetime issues, avoids all virtual dispatch and (almost) reduces the include bloat in the main header path. The reason for (almost) there is that for full include correctness, `EnumInfo` needs `<utility>` include for `std::pair`. However, this brings in things like `std::relops`, because the std headers in C++ are dumb. As this was not included before, and instead we relied on `std::pair` existing in an internal stdlib header that we transitively included, the full include size ends up bigger than before.