Keep per-test coverage with --coverage (roadmap step 4)feature/per-test-coverage

--coverage=DIR runs every attempt in a process of its own and keeps the
coverage of each test, for binaries built with LLVM source-based
coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov
(GCC: --coverage).

How it works:

- The scheduler gives each attempt a fresh worker, as --isolate does,
  and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the
  process writes at exit is then the coverage of that one test, in
  DIR/profiles/t<index>-a<attempt>.profraw or a .gcov directory. Each
  runtime reads its own variable, so no symbol of either runtime is
  looked up: weak references would not pull them out of their static
  archives, and Mach-O does not allow undefined weak symbols.
- A process that runs other tests first (bisection) writes beside the
  profiles, never into one. The attempt records its profile, also in
  the JSON results.
- The profiles of an earlier run are removed when the run starts,
  because gcov adds to the .gcda files it finds.
- After the run, CoverageReport reads the last profile of every test:
  llvm-profdata and llvm-cov (PATH, xcrun on macOS, or
  CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or
  CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is
  DIR/coverage.json, the files, lines and branches each test reached,
  and DIR/coverage.lcov for the whole run. Without the tools the
  profiles stay and the run says so.
- --coverage with --in-process or --break is refused.

Tests: option parsing, LCOV reading and writing, gcov JSON, and an
integration test on CoverageFixture, a binary CMake builds with the
coverage flags of Clang or GCC: two tests reach different lines, each
map holds only its own, the LCOV report holds both, and a stale profile
of an earlier run is gone. Locally all 84 CTest tests pass with Clang,
and the coverage tests pass with GCC (built with
CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as
errors, before and after this change).

CI: the Linux jobs install LLVM's coverage tools, a new job checks the
gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map
into a failure on Linux and macOS instead of a skip. MSVC has no
source-based coverage, so its builds skip those tests; the roadmap
lists step 4 as in progress until Windows with clang-cl is tested.

3 files changed, 62 insertions, 1 deletions

diff --git a/docs/command-line.md b/docs/command-line.md
index 236128e1..ccca35ff 100644
--- a/docs/command-line.md
+++ b/docs/command-line.md
@@ -120,6 +120,7 @@ Without a spec, every test runs except hidden ones, whose tags start with
 | `--junit=FILE` | write a JUnit XML report |
 | `--json=FILE` | write a JSON report with every attempt and its analysis |
 | `--trace=FILE` | write a trace for `chrome://tracing` or Perfetto |
+| `--coverage=DIR` | run every test in a process of its own and keep its coverage in DIR, with a map of what each test reached; see [Per-test coverage](progmasoft-runner.md#per-test-coverage) |
 | `--slowest[=N]` | list the N slowest tests (default 10) |
 | `--premature-exit-guard-file=FILE` | create FILE, private to the user, when the run starts and remove it when the run ends normally |
 
diff --git a/docs/progmasoft-roadmap.md b/docs/progmasoft-roadmap.md
index 81a8956f..1f61be79 100644
--- a/docs/progmasoft-roadmap.md
+++ b/docs/progmasoft-roadmap.md
@@ -22,7 +22,7 @@ The steps are ordered by what each one builds on.
 | 1 | Native property testing: choice-sequence shrinking, combinators, state machines, statistics, targeted search, example database, differential and metamorphic oracles | done; see [Native property testing](progmasoft-properties.md#top) |
 | 2 | The Catch3 runner | done; see [The Catch3 runner](progmasoft-runner.md#top) |
 | 3 | Mocking | done; see [Mocking](progmasoft-mocking.md#top) |
-| 4 | Per-test code coverage | planned |
+| 4 | Per-test code coverage | in progress: Linux and macOS, with Clang and GCC; see [Per-test coverage](progmasoft-runner.md#per-test-coverage). Windows with clang-cl is next |
 | 5 | Change-impact test selection and test prioritization | planned |
 | 6 | Mutation testing | planned |
 | 7 | Coverage-, mutation- and history-guided test generation | planned |
diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md
index 19d1e06d..cb5f29c5 100644
--- a/docs/progmasoft-runner.md
+++ b/docs/progmasoft-runner.md
@@ -12,6 +12,7 @@
 [Temporary files](#temporary-files)<br>
 [What a failure report says](#what-a-failure-report-says)<br>
 [Reports and traces](#reports-and-traces)<br>
+[Per-test coverage](#per-test-coverage)<br>
 [Formats other tools read](#formats-other-tools-read)<br>
 [Project configuration](#project-configuration)<br>
 [Bazel and CTest](#bazel-and-ctest)<br>
@@ -332,6 +333,65 @@ replay line runs the failure again as it happened.
   attempt and section.
 
 
+## Per-test coverage
+
+`--coverage=DIR` runs every test in a process of its own and keeps what each
+one reached. Build the tests with coverage instrumentation:
+
+* Clang: `-fprofile-instr-generate -fcoverage-mapping`, LLVM source-based
+  coverage;
+* GCC: `--coverage`, gcov's format.
+
+```text
+$ ./tests --coverage=coverage
+Catch3: 120 test(s), up to 8 worker(s), seed 2153251184
+note: --coverage: 120 test profile(s); what each test reached is in coverage/coverage.json, the run combined in coverage/coverage.lcov
+
+120 test(s): 120 passed
+```
+
+Each attempt gets a process of its own, so the profile its process writes at
+exit is the coverage of that one test: `DIR/profiles/t<index>-a<attempt>`
+with `.profraw` for LLVM, or a `.gcov` directory below which gcov writes its
+`.gcda` files under their absolute paths. The runner sets `LLVM_PROFILE_FILE`
+and `GCOV_PREFIX` for the process; each runtime reads its own. A process
+that runs other tests first, to find a test that leaves state behind, writes
+beside the profiles, never into one. The profiles of an earlier run in
+`DIR/profiles` are removed when the run starts, because gcov adds to the
+data it finds.
+
+After the run, the runner reads the profile of the last attempt of every
+test:
+
+* `DIR/coverage.json` holds, for each test, its outcome, its profile and the
+  files it reached, with the lines it ran and, per line, how many of its
+  branches it took:
+
+  ```json
+  { "version": 1, "format": "llvm", "binary": "./tests",
+    "tests": [ { "name": "Adds one", "outcome": "passed",
+                 "profile": "coverage/profiles/t0-a1.profraw",
+                 "files": [ { "file": "/src/calc.cpp", "lines": [ 12, 13 ],
+                              "branches": [ { "line": 12, "taken": 1, "total": 2 } ] } ] } ] }
+  ```
+
+* `DIR/coverage.lcov` is the coverage of the whole run as an LCOV tracefile,
+  which `genhtml`, Codecov and editors read.
+
+LLVM profiles are read with `llvm-profdata` and `llvm-cov`, gcov data with
+`gcov`. They are looked up on `PATH`, through `xcrun` on macOS, or named by
+`CATCH3_LLVM_PROFDATA`, `CATCH3_LLVM_COV` and `CATCH3_GCOV`. Without them the
+profiles stay in `DIR/profiles` and the run says so. A test that crashes or
+is stopped at its time limit leaves no profile. The runner's own process is
+instrumented too and writes its profile where any instrumented program does,
+`default.profraw` in the working directory unless `LLVM_PROFILE_FILE` says
+otherwise; it holds no test.
+
+`--coverage` cannot be used with `--in-process` or `--break`, which run the
+tests in one process. Per-test coverage is tested on Linux and macOS; Windows
+builds with clang-cl are next.
+
+
 ## Formats other tools read
 
 These formats carry a `format` and a `version`. Within a version they only