Keep per-test coverage with --coverage (roadmap step 4)feature/per-test-coverage

--coverage=DIR runs every attempt in a process of its own and keeps the
coverage of each test, for binaries built with LLVM source-based
coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov
(GCC: --coverage).

How it works:

- The scheduler gives each attempt a fresh worker, as --isolate does,
  and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the
  process writes at exit is then the coverage of that one test, in
  DIR/profiles/t<index>-a<attempt>.profraw or a .gcov directory. Each
  runtime reads its own variable, so no symbol of either runtime is
  looked up: weak references would not pull them out of their static
  archives, and Mach-O does not allow undefined weak symbols.
- A process that runs other tests first (bisection) writes beside the
  profiles, never into one. The attempt records its profile, also in
  the JSON results.
- The profiles of an earlier run are removed when the run starts,
  because gcov adds to the .gcda files it finds.
- After the run, CoverageReport reads the last profile of every test:
  llvm-profdata and llvm-cov (PATH, xcrun on macOS, or
  CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or
  CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is
  DIR/coverage.json, the files, lines and branches each test reached,
  and DIR/coverage.lcov for the whole run. Without the tools the
  profiles stay and the run says so.
- --coverage with --in-process or --break is refused.

Tests: option parsing, LCOV reading and writing, gcov JSON, and an
integration test on CoverageFixture, a binary CMake builds with the
coverage flags of Clang or GCC: two tests reach different lines, each
map holds only its own, the LCOV report holds both, and a stale profile
of an earlier run is gone. Locally all 84 CTest tests pass with Clang,
and the coverage tests pass with GCC (built with
CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as
errors, before and after this change).

CI: the Linux jobs install LLVM's coverage tools, a new job checks the
gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map
into a failure on Linux and macOS instead of a skip. MSVC has no
source-based coverage, so its builds skip those tests; the roadmap
lists step 4 as in progress until Windows with clang-cl is tested.

1 files changed, 58 insertions, 3 deletions

diff --git a/src/Progmasoft/Runner/Scheduler.cpp b/src/Progmasoft/Runner/Scheduler.cpp
index 56a2aa36..263257f7 100644
--- a/src/Progmasoft/Runner/Scheduler.cpp
+++ b/src/Progmasoft/Runner/Scheduler.cpp
@@ -346,6 +346,10 @@ namespace Progmasoft::Catch3::Runner
                 }
                 attempt.temporaryDirectory
                     = TemporaryFor(catalogIndex, attempt.number);
+                if (target)
+                {
+                    attempt.coverageProfile = worker.coverageProfile;
+                }
                 worker.running = std::move(running);
                 if (options_.inProcess)
                 {
@@ -973,8 +977,33 @@ namespace Progmasoft::Catch3::Runner
 
             // Worker processes.
 
+            /// Where the coverage profile of the next attempt of `job` goes,
+            /// without its extension. Only an attempt that runs alone in its
+            /// process gets one: a job that runs other tests first would
+            /// count their lines as its own.
+            std::string
+            CoverageProfileFor(const Job &job) const
+            {
+                if (options_.coverageDirectory.empty()
+                    || job.kind != Job::Kind::kAttempt || !job.before.empty())
+                {
+                    return {};
+                }
+                const std::size_t catalogIndex
+                    = plan_.tests[job.test].entry->index;
+                const std::size_t attempt
+                    = states_[job.test].result.attempts.size() + 1;
+                return (std::filesystem::path{ options_.coverageDirectory }
+                        / "profiles"
+                        / ("t" + std::to_string(catalogIndex) + "-a"
+                           + std::to_string(attempt)))
+                    .string();
+            }
+
             Worker *
-            SpawnWorker(bool dedicated, std::optional<std::uint32_t> seed)
+            SpawnWorker(bool dedicated,
+                        std::optional<std::uint32_t> seed,
+                        std::string coverageProfile = {})
             {
                 if (infrastructureFailed_)
                 {
@@ -994,6 +1023,26 @@ namespace Progmasoft::Catch3::Runner
                 const std::size_t id = ++workersStarted_;
                 child.environment.emplace_back("CATCH3_WORKER",
                                                std::to_string(id));
+                if (!options_.coverageDirectory.empty())
+                {
+                    // Each runtime reads its own variable: LLVM's profile
+                    // runtime the file it writes at exit, gcov the directory
+                    // its .gcda files go below. A process that is not one
+                    // attempt alone writes beside the profiles, never into
+                    // one of them.
+                    const std::string base
+                        = !coverageProfile.empty()
+                              ? coverageProfile
+                              : (std::filesystem::path{
+                                     options_.coverageDirectory }
+                                 / "profiles"
+                                 / ("other-w" + std::to_string(id)))
+                                    .string();
+                    child.environment.emplace_back("LLVM_PROFILE_FILE",
+                                                   base + ".profraw");
+                    child.environment.emplace_back("GCOV_PREFIX",
+                                                   base + ".gcov");
+                }
                 child.keptEnvironment = options_.passedEnvironment;
                 auto process = ChildProcess::Start(child);
                 if (!process)
@@ -1010,6 +1059,7 @@ namespace Progmasoft::Catch3::Runner
                 worker->id = id;
                 worker->dedicated = dedicated;
                 worker->seed = seed;
+                worker->coverageProfile = std::move(coverageProfile);
                 worker->started = Clock::now();
                 worker->process = std::move(*process);
                 Worker *raw = worker.get();
@@ -1099,8 +1149,12 @@ namespace Progmasoft::Catch3::Runner
                         return;
                     }
                     const PlannedTest &planned = plan_.tests[job->test];
+                    // A coverage run gives every attempt a process of its
+                    // own: the profile a process writes at exit is then the
+                    // coverage of that one test.
                     const bool fresh = job->fresh || options_.isolate
-                                       || planned.entry->isolated;
+                                       || planned.entry->isolated
+                                       || !options_.coverageDirectory.empty();
                     if (!fresh && idle != nullptr)
                     {
                         Assign(*idle, std::move(*job));
@@ -1116,7 +1170,8 @@ namespace Progmasoft::Catch3::Runner
                         queue_.push_front(std::move(*job));
                         return;
                     }
-                    Worker *worker = SpawnWorker(fresh, job->seed);
+                    Worker *worker
+                        = SpawnWorker(fresh, job->seed, CoverageProfileFor(*job));
                     if (worker == nullptr)
                     {
                         queue_.push_front(std::move(*job));