Merge pull request #39 from Progmasoft/feature/per-test-coverage

Keep per-test coverage with --coverage (roadmap step 4)

3 files changed, 67 insertions, 1 deletions

diff --git a/docs/command-line.md b/docs/command-line.md
index 236128e1..ccca35ff 100644
--- a/docs/command-line.md
+++ b/docs/command-line.md
@@ -120,6 +120,7 @@ Without a spec, every test runs except hidden ones, whose tags start with
 | `--junit=FILE` | write a JUnit XML report |
 | `--json=FILE` | write a JSON report with every attempt and its analysis |
 | `--trace=FILE` | write a trace for `chrome://tracing` or Perfetto |
+| `--coverage=DIR` | run every test in a process of its own and keep its coverage in DIR, with a map of what each test reached; see [Per-test coverage](progmasoft-runner.md#per-test-coverage) |
 | `--slowest[=N]` | list the N slowest tests (default 10) |
 | `--premature-exit-guard-file=FILE` | create FILE, private to the user, when the run starts and remove it when the run ends normally |
 
diff --git a/docs/progmasoft-roadmap.md b/docs/progmasoft-roadmap.md
index 81a8956f..1f61be79 100644
--- a/docs/progmasoft-roadmap.md
+++ b/docs/progmasoft-roadmap.md
@@ -22,7 +22,7 @@ The steps are ordered by what each one builds on.
 | 1 | Native property testing: choice-sequence shrinking, combinators, state machines, statistics, targeted search, example database, differential and metamorphic oracles | done; see [Native property testing](progmasoft-properties.md#top) |
 | 2 | The Catch3 runner | done; see [The Catch3 runner](progmasoft-runner.md#top) |
 | 3 | Mocking | done; see [Mocking](progmasoft-mocking.md#top) |
-| 4 | Per-test code coverage | planned |
+| 4 | Per-test code coverage | in progress: Linux and macOS, with Clang and GCC; see [Per-test coverage](progmasoft-runner.md#per-test-coverage). Windows with clang-cl is next |
 | 5 | Change-impact test selection and test prioritization | planned |
 | 6 | Mutation testing | planned |
 | 7 | Coverage-, mutation- and history-guided test generation | planned |
diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md
index 19d1e06d..d61ebadd 100644
--- a/docs/progmasoft-runner.md
+++ b/docs/progmasoft-runner.md
@@ -12,6 +12,7 @@
 [Temporary files](#temporary-files)<br>
 [What a failure report says](#what-a-failure-report-says)<br>
 [Reports and traces](#reports-and-traces)<br>
+[Per-test coverage](#per-test-coverage)<br>
 [Formats other tools read](#formats-other-tools-read)<br>
 [Project configuration](#project-configuration)<br>
 [Bazel and CTest](#bazel-and-ctest)<br>
@@ -332,6 +333,70 @@ replay line runs the failure again as it happened.
   attempt and section.
 
 
+## Per-test coverage
+
+`--coverage=DIR` runs every test in a process of its own and keeps what each
+one reached. Build the tests with coverage instrumentation:
+
+* Clang: `-fprofile-instr-generate -fcoverage-mapping`, LLVM source-based
+  coverage;
+* GCC: `--coverage`, gcov's format.
+
+```text
+$ ./tests --coverage=coverage
+Catch3: 120 test(s), up to 8 worker(s), seed 2153251184
+note: --coverage: 120 test profile(s); what each test reached is in coverage/coverage.json, the run combined in coverage/coverage.lcov
+
+120 test(s): 120 passed
+```
+
+Each attempt gets a process of its own, so the profile its process writes at
+exit is the coverage of that one test: `DIR/profiles/t<index>-a<attempt>`
+with `.profraw` for LLVM, or a `.gcov` directory below which gcov writes its
+`.gcda` files under their absolute paths. The runner sets `LLVM_PROFILE_FILE`
+and `GCOV_PREFIX` for the process; each runtime reads its own. A process
+that runs other tests first, to find a test that leaves state behind, writes
+beside the profiles, never into one. The profiles and reports of an earlier
+run in `DIR` are removed when the run starts: gcov adds to the data it finds,
+and a run that writes no reports leaves none to be read as its own.
+
+After the run, the runner reads, on as many threads as the run had workers,
+the profile of the last attempt of every test that left one; when a retry or
+repeat crashed before writing its profile, the attempt before it counts:
+
+* `DIR/coverage.json` holds, for each test, its outcome, its profile and the
+  files it reached, with the lines it ran and, per line, how many of its
+  branches it took:
+
+  ```json
+  { "version": 1, "format": "llvm", "binary": "./tests",
+    "tests": [ { "name": "Adds one", "outcome": "passed",
+                 "profile": "coverage/profiles/t0-a1.profraw",
+                 "files": [ { "file": "/src/calc.cpp", "lines": [ 12, 13 ],
+                              "branches": [ { "line": 12, "taken": 1, "total": 2 } ] } ] } ] }
+  ```
+
+* `DIR/coverage.lcov` is the coverage of the whole run as an LCOV tracefile,
+  which `genhtml`, Codecov and editors read.
+
+LLVM profiles are read with `llvm-profdata` and `llvm-cov`, gcov data with
+`gcov`. They are looked up on `PATH`, through `xcrun` on macOS, or named by
+`CATCH3_LLVM_PROFDATA`, `CATCH3_LLVM_COV` and `CATCH3_GCOV`. Without them the
+profiles stay in `DIR/profiles` and the run says so, as it does when the
+tests wrote no profiles; neither fails the run. A profile the tools cannot
+read, or a report that cannot be written, fails a run whose tests passed,
+like any report the run was asked for. A test that crashes or is stopped at
+its time limit leaves no profile. gcov is run with `--branch-probabilities`,
+without which its JSON has no branches. The runner's own process is
+instrumented too and writes its profile where any instrumented program does,
+`default.profraw` in the working directory unless `LLVM_PROFILE_FILE` says
+otherwise; it holds no test.
+
+`--coverage` cannot be used with `--in-process` or `--break`, which run the
+tests in one process. Per-test coverage is tested on Linux and macOS; Windows
+builds with clang-cl are next.
+
+
 ## Formats other tools read
 
 These formats carry a `format` and a `version`. Within a version they only