Keep per-test coverage with --coverage (roadmap step 4)feature/per-test-coverage

--coverage=DIR runs every attempt in a process of its own and keeps the
coverage of each test, for binaries built with LLVM source-based
coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov
(GCC: --coverage).

How it works:

- The scheduler gives each attempt a fresh worker, as --isolate does,
  and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the
  process writes at exit is then the coverage of that one test, in
  DIR/profiles/t<index>-a<attempt>.profraw or a .gcov directory. Each
  runtime reads its own variable, so no symbol of either runtime is
  looked up: weak references would not pull them out of their static
  archives, and Mach-O does not allow undefined weak symbols.
- A process that runs other tests first (bisection) writes beside the
  profiles, never into one. The attempt records its profile, also in
  the JSON results.
- The profiles of an earlier run are removed when the run starts,
  because gcov adds to the .gcda files it finds.
- After the run, CoverageReport reads the last profile of every test:
  llvm-profdata and llvm-cov (PATH, xcrun on macOS, or
  CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or
  CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is
  DIR/coverage.json, the files, lines and branches each test reached,
  and DIR/coverage.lcov for the whole run. Without the tools the
  profiles stay and the run says so.
- --coverage with --in-process or --break is refused.

Tests: option parsing, LCOV reading and writing, gcov JSON, and an
integration test on CoverageFixture, a binary CMake builds with the
coverage flags of Clang or GCC: two tests reach different lines, each
map holds only its own, the LCOV report holds both, and a stale profile
of an earlier run is gone. Locally all 84 CTest tests pass with Clang,
and the coverage tests pass with GCC (built with
CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as
errors, before and after this change).

CI: the Linux jobs install LLVM's coverage tools, a new job checks the
gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map
into a failure on Linux and macOS instead of a skip. MSVC has no
source-based coverage, so its builds skip those tests; the roadmap
lists step 4 as in progress until Windows with clang-cl is tested.

1 files changed, 21 insertions, 1 deletions

diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index b6b9b52c..32d1ce57 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -166,6 +166,7 @@ add_executable(ProgmasoftSelfTest
   Progmasoft/RunnerConfig.tests.cpp
   Progmasoft/RunnerAnalysis.tests.cpp
   Progmasoft/RunnerImport.tests.cpp
+  Progmasoft/RunnerCoverage.tests.cpp
   Progmasoft/RunnerOutput.tests.cpp
   Progmasoft/Snapshot.tests.cpp
   Progmasoft/Stateful.tests.cpp
@@ -180,9 +181,28 @@ target_link_libraries(RunnerFixture PRIVATE Catch3WithMain)
 target_compile_features(RunnerFixture PRIVATE cxx_std_17)
 add_dependencies(ProgmasoftSelfTest RunnerFixture)
 
+# Tests built with coverage instrumentation, for the integration tests of
+# --coverage: LLVM source-based coverage with Clang, gcov with GCC. MSVC has
+# neither, so its builds skip those tests.
+set(_catch3_coverage_environment "")
+if(CMAKE_CXX_COMPILER_ID MATCHES "Clang" AND NOT CMAKE_CXX_SIMULATE_ID STREQUAL "MSVC")
+  set(_catch3_coverage_flags -fprofile-instr-generate -fcoverage-mapping)
+elseif(CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
+  set(_catch3_coverage_flags --coverage)
+endif()
+if(_catch3_coverage_flags)
+  add_executable(CoverageFixture Progmasoft/CoverageFixture.cpp)
+  target_link_libraries(CoverageFixture PRIVATE Catch3WithMain)
+  target_compile_features(CoverageFixture PRIVATE cxx_std_17)
+  target_compile_options(CoverageFixture PRIVATE ${_catch3_coverage_flags})
+  target_link_options(CoverageFixture PRIVATE ${_catch3_coverage_flags})
+  add_dependencies(ProgmasoftSelfTest CoverageFixture)
+  set(_catch3_coverage_environment "CATCH3_COVERAGE_FIXTURE=$<TARGET_FILE:CoverageFixture>")
+endif()
+
 add_test(NAME Progmasoft::PropertyTests COMMAND $<TARGET_FILE:ProgmasoftSelfTest> --no-history)
 set_tests_properties(Progmasoft::PropertyTests PROPERTIES
-  ENVIRONMENT "CATCH3_RUNNER_FIXTURE=$<TARGET_FILE:RunnerFixture>"
+  ENVIRONMENT "CATCH3_RUNNER_FIXTURE=$<TARGET_FILE:RunnerFixture>;${_catch3_coverage_environment}"
 )
 if(BUILD_SHARED_LIBS AND WIN32)
   add_custom_command(TARGET SelfTest PRE_LINK