--coverage=DIR runs every attempt in a process of its own and keeps the coverage of each test, for binaries built with LLVM source-based coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov (GCC: --coverage). How it works: - The scheduler gives each attempt a fresh worker, as --isolate does, and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the process writes at exit is then the coverage of that one test, in DIR/profiles/t<index>-a<attempt>.profraw or a .gcov directory. Each runtime reads its own variable, so no symbol of either runtime is looked up: weak references would not pull them out of their static archives, and Mach-O does not allow undefined weak symbols. - A process that runs other tests first (bisection) writes beside the profiles, never into one. The attempt records its profile, also in the JSON results. - The profiles of an earlier run are removed when the run starts, because gcov adds to the .gcda files it finds. - After the run, CoverageReport reads the last profile of every test: llvm-profdata and llvm-cov (PATH, xcrun on macOS, or CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is DIR/coverage.json, the files, lines and branches each test reached, and DIR/coverage.lcov for the whole run. Without the tools the profiles stay and the run says so. - --coverage with --in-process or --break is refused. Tests: option parsing, LCOV reading and writing, gcov JSON, and an integration test on CoverageFixture, a binary CMake builds with the coverage flags of Clang or GCC: two tests reach different lines, each map holds only its own, the LCOV report holds both, and a stale profile of an earlier run is gone. Locally all 84 CTest tests pass with Clang, and the coverage tests pass with GCC (built with CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as errors, before and after this change). CI: the Linux jobs install LLVM's coverage tools, a new job checks the gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map into a failure on Linux and macOS instead of a skip. MSVC has no source-based coverage, so its builds skip those tests; the roadmap lists step 4 as in progress until Windows with clang-cl is tested.
EnvironmentBlock replaced every non-ASCII character of an inherited name with '?' before asking whether to withhold it, so --pass-env=MY_?_TOKEN kept an unrelated MY_é_TOKEN. Names are now converted with Narrow, as everything else the runner compares. A block whose variables were all withheld, with no additions, ended in one terminator; CreateProcessW needs two for an empty block. The amalgamated sources are regenerated for the runner changes.
Workers and the other processes the runner starts copied its whole environment, tokens and passwords included. ChildProcess::Start now leaves out every variable whose name contains, in any case, TOKEN, SECRET, PASSWORD, PASSWD, CREDENTIAL, API_KEY, APIKEY, PRIVATE_KEY or ACCESS_KEY (IsSecretVariable), unless ChildOptions::keptEnvironment names it; names compare without case on Windows. --pass-env=NAME, repeatable, and "pass-env" in catch3.json keep a variable for the tests. The runner itself still starts with the whole environment, so code that runs before the first worker sees it; docs/progmasoft-runner.md says so. A runner test starts the shell's environment listing with a secret and a plain variable set, sees only the plain one, and sees the secret once it is kept.
A report no longer throws, so a noexcept mock survives an aborting run. A call that has nothing to return still leaves the test case, which ends the process in a noexcept function; the documentation says so. RunContext clears the result capture when it goes, so a mock of static lifetime finds none and writes to standard error. Decision owns its source line, since Verify can free the expectation while a call is on its way. Moving a result out takes the lock. Vector matchers refer to their vector, so they are rejected at compile time like composed ones. A null text expectation accepts only null, and integers compare by value, so 256 no longer accepts an unsigned char.
A static mutex needs an exit-time destructor, which Clang's warnings refuse, and a mock of static lifetime can outlive it. The test seam is held the same way, and a test covers a mock method with nothing after its parameters, which needs an empty variadic argument to stay valid under -Wpedantic.
Step 3 of the roadmap. MockFunction checks the calls of one function against its expectations; CATCH3_MOCK_METHOD declares one for a virtual member, and CATCH3_MOCK_FUNCTION and ScopedSeam serve free functions behind a seam. Expectations take values, Catch matchers, Any or Where; count calls with Times, Between, AtLeast, AtMost, AnyTimes and Never; answer with Returns, Does or Throws; and order calls across mocks with Sequence. A failure goes through AssertionHandler at the line of the expectation, so every report of the runner shows it. Met expectations count as passed assertions. A test case that is already being left does not get unmet expectations on top of its own failure. Composed matchers refer to temporaries, so they are rejected at compile time rather than left dangling. A call that fails returns a value-initialized result, or leaves the test case when no such result exists. The failure paths are tested through the runner fixture, reading the messages from its JSON report.
Versions have four numbers from this release on.
A review of the import found ways for a failure to disappear or for the import to reach too far. - CTRF's retryAttempts ends before the final attempt, which was dropped when its outcome matched the one before it. - TAP names tests only within a stream, so a failure in one file and a pass under the same name in another merged into a flaky pass. The path given to --import is now the class of a stream's tests. - A failure followed by a skip with the same identifier read as skipped. - Directives are whole words: TODOING is not TODO. SKIPPED and SKIPPING stay directives, as harnesses treat them. - Verdicts that alternate count as flaky only under an explicit --import-revision. HEAD says nothing about a dirty checkout or a report from another one, and two such events could quarantine a real failure. - --import-name no longer redirects the history of a normal test run. - Importing does not depend on the tests registered in the binary.
Tests Catch3 does not run had no history, flaky detection or quarantine. --import=FILE reads a report instead of running tests and sends it through the analysis, history and reports of a run: JUnit with Surefire's reruns, xUnit.net v2, TRX, TAP 12 to 14, CTRF, Catch3's own --json report and catch3-results, a small JSON Lines format for tools with none of these. Results that share an identifier are attempts at one test, so a report of a rerun gives a flaky test. --import-revision says what was tested: a verdict that changes on one revision also counts as flaky. --import-name keeps the results in a history of their own, which --analytics reads too. A TAP stream that ends early fails instead of reading as a run that passed. The amalgamation script now accepts a relative include of the third-party wrappers, because pugixml is included from a second directory.