diff options
| author | Leitwolf11 | 2026-11-13 07:46:08 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-11-13 07:46:08 +0300 |
| commit | 17248837a78462596809f902dd47cf113a39670b (patch) | |
| tree | 666d16ddc0a69acb805a5f32f17c8502208349e7 | |
| parent | 43dabc9adb45b808ae24173ff4633774aa71b2ee (diff) | |
| download | catch3-feature/per-test-coverage.tar.gz | |
Keep per-test coverage with --coverage (roadmap step 4)feature/per-test-coverage
--coverage=DIR runs every attempt in a process of its own and keeps the
coverage of each test, for binaries built with LLVM source-based
coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov
(GCC: --coverage).
How it works:
- The scheduler gives each attempt a fresh worker, as --isolate does,
and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the
process writes at exit is then the coverage of that one test, in
DIR/profiles/t<index>-a<attempt>.profraw or a .gcov directory. Each
runtime reads its own variable, so no symbol of either runtime is
looked up: weak references would not pull them out of their static
archives, and Mach-O does not allow undefined weak symbols.
- A process that runs other tests first (bisection) writes beside the
profiles, never into one. The attempt records its profile, also in
the JSON results.
- The profiles of an earlier run are removed when the run starts,
because gcov adds to the .gcda files it finds.
- After the run, CoverageReport reads the last profile of every test:
llvm-profdata and llvm-cov (PATH, xcrun on macOS, or
CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or
CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is
DIR/coverage.json, the files, lines and branches each test reached,
and DIR/coverage.lcov for the whole run. Without the tools the
profiles stay and the run says so.
- --coverage with --in-process or --break is refused.
Tests: option parsing, LCOV reading and writing, gcov JSON, and an
integration test on CoverageFixture, a binary CMake builds with the
coverage flags of Clang or GCC: two tests reach different lines, each
map holds only its own, the LCOV report holds both, and a stale profile
of an earlier run is gone. Locally all 84 CTest tests pass with Clang,
and the coverage tests pass with GCC (built with
CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as
errors, before and after this change).
CI: the Linux jobs install LLVM's coverage tools, a new job checks the
gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map
into a failure on Linux and macOS instead of a skip. MSVC has no
source-based coverage, so its builds skip those tests; the roadmap
lists step 4 as in progress until Windows with clang-cl is tested.
| -rw-r--r-- | .github/workflows/linux-builds.yml | 52 | ||||
| -rw-r--r-- | .github/workflows/mac-builds.yml | 2 | ||||
| -rw-r--r-- | docs/command-line.md | 1 | ||||
| -rw-r--r-- | docs/progmasoft-roadmap.md | 2 | ||||
| -rw-r--r-- | docs/progmasoft-runner.md | 60 | ||||
| -rw-r--r-- | premake5.lua | 1 | ||||
| -rw-r--r-- | src/CMakeLists.txt | 2 | ||||
| -rw-r--r-- | src/Progmasoft/Catch3/Results.hpp | 3 | ||||
| -rw-r--r-- | src/Progmasoft/Catch3/meson.build | 1 | ||||
| -rw-r--r-- | src/Progmasoft/Runner/CoverageReport.cpp | 594 | ||||
| -rw-r--r-- | src/Progmasoft/Runner/CoverageReport.hpp | 58 | ||||
| -rw-r--r-- | src/Progmasoft/Runner/Options.cpp | 20 | ||||
| -rw-r--r-- | src/Progmasoft/Runner/Options.hpp | 4 | ||||
| -rw-r--r-- | src/Progmasoft/Runner/ResultsJson.cpp | 2 | ||||
| -rw-r--r-- | src/Progmasoft/Runner/Runner.cpp | 20 | ||||
| -rw-r--r-- | src/Progmasoft/Runner/Scheduler.cpp | 61 | ||||
| -rw-r--r-- | src/Progmasoft/Runner/SchedulerModel.hpp | 4 | ||||
| -rw-r--r-- | tests/CMakeLists.txt | 22 | ||||
| -rw-r--r-- | tests/Progmasoft/BUILD.bazel | 1 | ||||
| -rw-r--r-- | tests/Progmasoft/CoverageFixture.cpp | 33 | ||||
| -rw-r--r-- | tests/Progmasoft/RunnerCoverage.tests.cpp | 226 | ||||
| -rw-r--r-- | tests/meson.build | 1 | ||||
| -rw-r--r-- | xmake.lua | 1 |
23 files changed, 1160 insertions, 11 deletions
diff --git a/.github/workflows/linux-builds.yml b/.github/workflows/linux-builds.yml index a9c38c66..02ee8d67 100644 --- a/.github/workflows/linux-builds.yml +++ b/.github/workflows/linux-builds.yml @@ -32,10 +32,10 @@ jobs: with: persist-credentials: false - - name: Install Clang, CMake and Ninja + - name: Install Clang, LLVM's coverage tools, CMake and Ninja run: | sudo apt-get update - sudo apt-get install -y clang cmake ninja-build python3 + sudo apt-get install -y clang llvm cmake ninja-build python3 - name: Configure run: | @@ -51,6 +51,8 @@ jobs: - name: Test run: ctest --test-dir build -j + env: + CATCH3_COVERAGE_REQUIRE_MAP: 1 - name: Install, and find the installed package run: | @@ -75,10 +77,10 @@ jobs: with: persist-credentials: false - - name: Install Clang, CMake and Ninja + - name: Install Clang, LLVM's coverage tools, CMake and Ninja run: | sudo apt-get update - sudo apt-get install -y clang cmake ninja-build python3 + sudo apt-get install -y clang llvm cmake ninja-build python3 - name: Configure with the sanitizers env: @@ -111,8 +113,8 @@ jobs: steps: # Git comes first: without it the checkout falls back to an archive of # the repository, which is not a checkout the tests can ask questions of. - - name: Install Git, Clang, CMake and Ninja - run: dnf install -y git clang gcc-c++ cmake ninja-build python3 + - name: Install Git, Clang, LLVM's coverage tools, CMake and Ninja + run: dnf install -y git clang llvm gcc-c++ cmake ninja-build python3 - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 with: @@ -132,3 +134,41 @@ jobs: - name: Test run: ctest --test-dir build -j + env: + CATCH3_COVERAGE_REQUIRE_MAP: 1 + + # Catch3 is built with Clang everywhere; GCC's stricter warnings are not + # errors here. The job checks per-test coverage in gcov's format, which + # only GCC writes. + gcc-coverage: + name: Ubuntu 26.04, GCC, per-test coverage with gcov + runs-on: ubuntu-26.04 + timeout-minutes: 30 + + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0 + with: + persist-credentials: false + + - name: Install GCC, CMake and Ninja + run: | + sudo apt-get update + sudo apt-get install -y g++ cmake ninja-build python3 + + - name: Configure + run: | + cmake --preset basic-tests -GNinja \ + -DCMAKE_BUILD_TYPE=Debug \ + -DCMAKE_CXX_COMPILER=g++ \ + -DCMAKE_CXX_STANDARD=17 \ + -DCATCH3_ENABLE_WERROR=OFF + + - name: Build + run: cmake --build build --target ProgmasoftSelfTest + + - name: Test per-test coverage + env: + CATCH3_RUNNER_FIXTURE: build/tests/RunnerFixture + CATCH3_COVERAGE_FIXTURE: build/tests/CoverageFixture + CATCH3_COVERAGE_REQUIRE_MAP: 1 + run: build/tests/ProgmasoftSelfTest --no-history "[coverage]" diff --git a/.github/workflows/mac-builds.yml b/.github/workflows/mac-builds.yml index c115bf00..4044db9b 100644 --- a/.github/workflows/mac-builds.yml +++ b/.github/workflows/mac-builds.yml @@ -44,3 +44,5 @@ jobs: - name: Test run: ctest --test-dir build -j + env: + CATCH3_COVERAGE_REQUIRE_MAP: 1 diff --git a/docs/command-line.md b/docs/command-line.md index 236128e1..ccca35ff 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -120,6 +120,7 @@ Without a spec, every test runs except hidden ones, whose tags start with | `--junit=FILE` | write a JUnit XML report | | `--json=FILE` | write a JSON report with every attempt and its analysis | | `--trace=FILE` | write a trace for `chrome://tracing` or Perfetto | +| `--coverage=DIR` | run every test in a process of its own and keep its coverage in DIR, with a map of what each test reached; see [Per-test coverage](progmasoft-runner.md#per-test-coverage) | | `--slowest[=N]` | list the N slowest tests (default 10) | | `--premature-exit-guard-file=FILE` | create FILE, private to the user, when the run starts and remove it when the run ends normally | diff --git a/docs/progmasoft-roadmap.md b/docs/progmasoft-roadmap.md index 81a8956f..1f61be79 100644 --- a/docs/progmasoft-roadmap.md +++ b/docs/progmasoft-roadmap.md @@ -22,7 +22,7 @@ The steps are ordered by what each one builds on. | 1 | Native property testing: choice-sequence shrinking, combinators, state machines, statistics, targeted search, example database, differential and metamorphic oracles | done; see [Native property testing](progmasoft-properties.md#top) | | 2 | The Catch3 runner | done; see [The Catch3 runner](progmasoft-runner.md#top) | | 3 | Mocking | done; see [Mocking](progmasoft-mocking.md#top) | -| 4 | Per-test code coverage | planned | +| 4 | Per-test code coverage | in progress: Linux and macOS, with Clang and GCC; see [Per-test coverage](progmasoft-runner.md#per-test-coverage). Windows with clang-cl is next | | 5 | Change-impact test selection and test prioritization | planned | | 6 | Mutation testing | planned | | 7 | Coverage-, mutation- and history-guided test generation | planned | diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md index 19d1e06d..cb5f29c5 100644 --- a/docs/progmasoft-runner.md +++ b/docs/progmasoft-runner.md @@ -12,6 +12,7 @@ [Temporary files](#temporary-files)<br> [What a failure report says](#what-a-failure-report-says)<br> [Reports and traces](#reports-and-traces)<br> +[Per-test coverage](#per-test-coverage)<br> [Formats other tools read](#formats-other-tools-read)<br> [Project configuration](#project-configuration)<br> [Bazel and CTest](#bazel-and-ctest)<br> @@ -332,6 +333,65 @@ replay line runs the failure again as it happened. attempt and section. +## Per-test coverage + +`--coverage=DIR` runs every test in a process of its own and keeps what each +one reached. Build the tests with coverage instrumentation: + +* Clang: `-fprofile-instr-generate -fcoverage-mapping`, LLVM source-based + coverage; +* GCC: `--coverage`, gcov's format. + +```text +$ ./tests --coverage=coverage +Catch3: 120 test(s), up to 8 worker(s), seed 2153251184 +note: --coverage: 120 test profile(s); what each test reached is in coverage/coverage.json, the run combined in coverage/coverage.lcov + +120 test(s): 120 passed +``` + +Each attempt gets a process of its own, so the profile its process writes at +exit is the coverage of that one test: `DIR/profiles/t<index>-a<attempt>` +with `.profraw` for LLVM, or a `.gcov` directory below which gcov writes its +`.gcda` files under their absolute paths. The runner sets `LLVM_PROFILE_FILE` +and `GCOV_PREFIX` for the process; each runtime reads its own. A process +that runs other tests first, to find a test that leaves state behind, writes +beside the profiles, never into one. The profiles of an earlier run in +`DIR/profiles` are removed when the run starts, because gcov adds to the +data it finds. + +After the run, the runner reads the profile of the last attempt of every +test: + +* `DIR/coverage.json` holds, for each test, its outcome, its profile and the + files it reached, with the lines it ran and, per line, how many of its + branches it took: + + ```json + { "version": 1, "format": "llvm", "binary": "./tests", + "tests": [ { "name": "Adds one", "outcome": "passed", + "profile": "coverage/profiles/t0-a1.profraw", + "files": [ { "file": "/src/calc.cpp", "lines": [ 12, 13 ], + "branches": [ { "line": 12, "taken": 1, "total": 2 } ] } ] } ] } + ``` + +* `DIR/coverage.lcov` is the coverage of the whole run as an LCOV tracefile, + which `genhtml`, Codecov and editors read. + +LLVM profiles are read with `llvm-profdata` and `llvm-cov`, gcov data with +`gcov`. They are looked up on `PATH`, through `xcrun` on macOS, or named by +`CATCH3_LLVM_PROFDATA`, `CATCH3_LLVM_COV` and `CATCH3_GCOV`. Without them the +profiles stay in `DIR/profiles` and the run says so. A test that crashes or +is stopped at its time limit leaves no profile. The runner's own process is +instrumented too and writes its profile where any instrumented program does, +`default.profraw` in the working directory unless `LLVM_PROFILE_FILE` says +otherwise; it holds no test. + +`--coverage` cannot be used with `--in-process` or `--break`, which run the +tests in one process. Per-test coverage is tested on Linux and macOS; Windows +builds with clang-cl are next. + + ## Formats other tools read These formats carry a `format` and a `version`. Within a version they only diff --git a/premake5.lua b/premake5.lua index d4c55c9e..def0463c 100644 --- a/premake5.lua +++ b/premake5.lua @@ -92,6 +92,7 @@ project "ProgmasoftPropertyTests" "tests/Progmasoft/RunnerConfig.tests.cpp", "tests/Progmasoft/RunnerAnalysis.tests.cpp", "tests/Progmasoft/RunnerImport.tests.cpp", + "tests/Progmasoft/RunnerCoverage.tests.cpp", "tests/Progmasoft/RunnerOutput.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp", "tests/Progmasoft/Stateful.tests.cpp", diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 43754fff..fac2fb62 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -301,6 +301,7 @@ set(PROGMASOFT_RUNNER_FILES "${CATCH_DIR}/src/Progmasoft/Runner/Execution.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Expected.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Files.hpp" + "${CATCH_DIR}/src/Progmasoft/Runner/CoverageReport.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Git.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Hash.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/History.hpp" @@ -320,6 +321,7 @@ set(PROGMASOFT_RUNNER_FILES "${CATCH_DIR}/src/Progmasoft/Runner/Console.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Execution.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Files.cpp" + "${CATCH_DIR}/src/Progmasoft/Runner/CoverageReport.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Git.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Hash.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/History.cpp" diff --git a/src/Progmasoft/Catch3/Results.hpp b/src/Progmasoft/Catch3/Results.hpp index 4c6bb1ce..bff2a72e 100644 --- a/src/Progmasoft/Catch3/Results.hpp +++ b/src/Progmasoft/Catch3/Results.hpp @@ -153,6 +153,9 @@ namespace Progmasoft::Catch3 /// The directory the test had for its temporary files; kept when the /// attempt failed, so its contents can be inspected. std::string temporaryDirectory; + /// In a coverage run, where the profile of this attempt went, + /// without its extension (`.profraw` or `.gcov`). + std::string coverageProfile; }; /// One test of the run and everything the runner learned about it. diff --git a/src/Progmasoft/Catch3/meson.build b/src/Progmasoft/Catch3/meson.build index 8420003b..4bbab28e 100644 --- a/src/Progmasoft/Catch3/meson.build +++ b/src/Progmasoft/Catch3/meson.build @@ -288,6 +288,7 @@ runner_sources = files( '../Runner/Console.cpp', '../Runner/Execution.cpp', '../Runner/Files.cpp', + '../Runner/CoverageReport.cpp', '../Runner/Git.cpp', '../Runner/Hash.cpp', '../Runner/History.cpp', diff --git a/src/Progmasoft/Runner/CoverageReport.cpp b/src/Progmasoft/Runner/CoverageReport.cpp new file mode 100644 index 00000000..c443c158 --- /dev/null +++ b/src/Progmasoft/Runner/CoverageReport.cpp @@ -0,0 +1,594 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 +#include <algorithm> +#include <cstdlib> +#include <optional> +#include <string_view> +#include <system_error> + +#include "CoverageReport.hpp" +#include "Files.hpp" +#include "Json.hpp" +#include "Process.hpp" + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + /// Reads a non-negative decimal number; nothing for anything else, + /// such as the `-` LCOV writes for a branch never reached. + std::optional<std::uint64_t> + Number(std::string_view text) + { + if (text.empty() || text.size() > 19) + { + return std::nullopt; + } + std::uint64_t value = 0; + for (const char character : text) + { + if (character < '0' || character > '9') + { + return std::nullopt; + } + value + = value * 10 + static_cast<std::uint64_t>(character - '0'); + } + return value; + } + + std::vector<std::string_view> + Split(std::string_view text, char separator) + { + std::vector<std::string_view> parts; + std::size_t start = 0; + while (true) + { + const auto end = text.find(separator, start); + parts.push_back(text.substr(start, end - start)); + if (end == std::string_view::npos) + { + return parts; + } + start = end + 1; + } + } + + /// A tool of a coverage format: the program, or the command that + /// runs it, such as `xcrun llvm-cov` on macOS. + using Tool = std::vector<std::string>; + + std::optional<Tool> + FindTool(const char *variable, const std::string &name) + { + std::vector<Tool> candidates; + if (auto configured = ReadEnvironment(variable); + !configured.empty()) + { + candidates.push_back({ std::move(configured) }); + } + candidates.push_back({ name }); +#if defined(__APPLE__) + candidates.push_back({ "xcrun", name }); +#endif + for (auto &candidate : candidates) + { + ChildOptions options; + options.arguments = candidate; + options.arguments.emplace_back("--version"); + options.searchPath = true; + if (auto run = RunAndCapture(options); + run && run->status.Succeeded()) + { + return std::move(candidate); + } + } + return std::nullopt; + } + + /// Runs `tool` with `arguments`; its output, or why it failed. + Expected<std::string> + RunTool(const Tool &tool, + const std::vector<std::string> &arguments, + const std::filesystem::path &directory = {}) + { + ChildOptions options; + options.arguments = tool; + options.arguments.insert(options.arguments.end(), + arguments.begin(), + arguments.end()); + options.searchPath = true; + options.workingDirectory = directory; + auto run = RunAndCapture(options); + if (!run) + { + return run.GetError(); + } + if (!run->status.Succeeded()) + { + std::string message = run->errors; + while (!message.empty() + && (message.back() == '\n' || message.back() == '\r')) + { + message.pop_back(); + } + return Error{ tool.back() + " failed: " + message }; + } + return std::move(run->output); + } + + void + Add(CoverageMap &total, const CoverageMap &part) + { + for (const auto &[file, coverage] : part) + { + FileCoverage &into = total[file]; + for (const auto &[line, count] : coverage.lines) + { + into.lines[line] += count; + } + for (const auto &[branch, count] : coverage.branches) + { + into.branches[branch] += count; + } + } + } + + /// What one test reached: the lines it ran and, per line, how many + /// of its branches it took. + JsonValue + ReachedJson(const CoverageMap &map) + { + JsonValue files = JsonValue::Array(); + for (const auto &[file, coverage] : map) + { + JsonValue lines = JsonValue::Array(); + for (const auto &[line, count] : coverage.lines) + { + if (count > 0) + { + lines.Append(line); + } + } + JsonValue branches = JsonValue::Array(); + std::map<std::size_t, std::pair<std::size_t, std::size_t>> + perLine; + for (const auto &[branch, count] : coverage.branches) + { + auto &[taken, total] = perLine[branch.first]; + ++total; + if (count > 0) + { + ++taken; + } + } + for (const auto &[line, counts] : perLine) + { + if (counts.first == 0) + { + continue; + } + branches.Append( + JsonValue::Object({ { "line", line }, + { "taken", counts.first }, + { "total", counts.second } })); + } + if (lines.Items().empty()) + { + continue; + } + files.Append( + JsonValue::Object({ { "file", file }, + { "lines", std::move(lines) }, + { "branches", std::move(branches) } })); + } + return files; + } + + /// The final attempt of a test that left a profile. + const Attempt * + ProfiledAttempt(const TestResult &test) + { + for (auto attempt = test.attempts.rbegin(); + attempt != test.attempts.rend(); + ++attempt) + { + if (!attempt->coverageProfile.empty()) + { + return &*attempt; + } + } + return nullptr; + } + + /// The .gcda files gcov wrote below `prefix`, each with the path the + /// object was built at, where its .gcno is. + std::vector<std::pair<std::filesystem::path, std::filesystem::path>> + GcdaFiles(const std::filesystem::path &prefix) + { + std::vector<std::pair<std::filesystem::path, std::filesystem::path>> + files; + std::error_code error; + for (auto entry + = std::filesystem::recursive_directory_iterator(prefix, error); + !error + && entry != std::filesystem::recursive_directory_iterator(); + entry.increment(error)) + { + if (!entry->is_regular_file(error) + || entry->path().extension() != ".gcda") + { + continue; + } + // GCOV_PREFIX is put before the absolute path of the object. + const auto relative + = std::filesystem::relative(entry->path(), prefix, error); + std::filesystem::path original + = std::filesystem::path{ "/" } / relative; + files.emplace_back(entry->path(), original); + } + std::sort(files.begin(), files.end()); + return files; + } + + Expected<CoverageMap> + GcovCoverage(const Tool &gcov, const std::filesystem::path &prefix) + { + CoverageMap map; + for (const auto &[gcda, original] : GcdaFiles(prefix)) + { + auto gcno = original; + gcno.replace_extension(".gcno"); + auto beside = gcda; + beside.replace_extension(".gcno"); + std::error_code error; + std::filesystem::copy_file( + gcno, + beside, + std::filesystem::copy_options::overwrite_existing, + error); + if (error) + { + return Error{ "the notes file " + gcno.string() + + " could not be read: " + error.message() }; + } + auto json = RunTool( + gcov, + { "--json-format", "--stdout", gcda.filename().string() }, + gcda.parent_path()); + if (!json) + { + return json.GetError(); + } + // One JSON document per data file, one per line. + for (const auto line : Split(*json, '\n')) + { + if (!line.empty()) + { + AddGcovJson(std::string{ line }, map); + } + } + } + return map; + } + } // namespace + + CoverageMap + ParseLcov(const std::string &text) + { + CoverageMap map; + FileCoverage *current = nullptr; + for (auto line : Split(text, '\n')) + { + if (!line.empty() && line.back() == '\r') + { + line.remove_suffix(1); + } + if (line.substr(0, 3) == "SF:") + { + current = &map[std::string{ line.substr(3) }]; + } + else if (line == "end_of_record") + { + current = nullptr; + } + else if (current != nullptr && line.substr(0, 3) == "DA:") + { + const auto fields = Split(line.substr(3), ','); + const auto number = Number(fields[0]); + const auto count + = fields.size() > 1 ? Number(fields[1]) : std::nullopt; + if (number && count) + { + current->lines[*number] += *count; + } + } + else if (current != nullptr && line.substr(0, 5) == "BRDA:") + { + const auto fields = Split(line.substr(5), ','); + if (fields.size() != 4) + { + continue; + } + const auto number = Number(fields[0]); + if (!number) + { + continue; + } + // The branches of a line are numbered in the order they come; + // `-` counts as never taken. + std::size_t index = 0; + for (auto it = current->branches.lower_bound({ *number, 0 }); + it != current->branches.end() + && it->first.first == *number; + ++it) + { + ++index; + } + current->branches[{ *number, index }] + += Number(fields[3]).value_or(0); + } + } + return map; + } + + void + AddGcovJson(const std::string &text, CoverageMap &map) + { + auto document = ParseJson(text); + if (!document) + { + return; + } + for (const auto &file : (*document)["files"].Items()) + { + FileCoverage &coverage = map[file["file"].AsString()]; + for (const auto &line : file["lines"].Items()) + { + const auto number = static_cast<std::size_t>( + line["line_number"].AsUnsigned()); + coverage.lines[number] += line["count"].AsUnsigned(); + std::size_t index = 0; + for (const auto &branch : line["branches"].Items()) + { + coverage.branches[{ number, index++ }] + += branch["count"].AsUnsigned(); + } + } + } + } + + std::string + ToLcov(const CoverageMap &map) + { + std::string text; + for (const auto &[file, coverage] : map) + { + text += "SF:" + file + "\n"; + std::size_t linesHit = 0; + for (const auto &[line, count] : coverage.lines) + { + text += "DA:" + std::to_string(line) + "," + + std::to_string(count) + "\n"; + linesHit += count > 0 ? 1 : 0; + } + std::size_t branchesHit = 0; + for (const auto &[branch, count] : coverage.branches) + { + text += "BRDA:" + std::to_string(branch.first) + ",0," + + std::to_string(branch.second) + "," + + (count > 0 ? std::to_string(count) : "-") + "\n"; + branchesHit += count > 0 ? 1 : 0; + } + text += "LF:" + std::to_string(coverage.lines.size()) + "\n"; + text += "LH:" + std::to_string(linesHit) + "\n"; + text += "BRF:" + std::to_string(coverage.branches.size()) + "\n"; + text += "BRH:" + std::to_string(branchesHit) + "\n"; + text += "end_of_record\n"; + } + return text; + } + + Status + PrepareCoverageDirectory(const std::filesystem::path &directory) + { + std::error_code error; + std::filesystem::remove_all(directory / "profiles", error); + if (error) + { + return Error{ "the profiles of an earlier run in " + + directory.string() + + " could not be removed: " + error.message() }; + } + std::filesystem::create_directories(directory / "profiles", error); + if (error) + { + return Error{ "the coverage directory " + directory.string() + + " could not be created: " + error.message() }; + } + return Status::Ok(); + } + + std::vector<std::string> + WriteCoverageReports(const std::filesystem::path &directory, + const RunResult &run, + const std::string &program) + { + std::vector<std::string> notes; + std::size_t llvmProfiles = 0; + std::size_t gcovProfiles = 0; + for (const auto &test : run.tests) + { + if (const Attempt *attempt = ProfiledAttempt(test)) + { + std::error_code error; + llvmProfiles += std::filesystem::exists(attempt->coverageProfile + + ".profraw", + error) + ? 1 + : 0; + gcovProfiles += std::filesystem::exists(attempt->coverageProfile + + ".gcov", + |