diff options
| author | Leitwolf11 | 2026-11-13 07:46:08 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-11-13 07:46:08 +0300 |
| commit | 17248837a78462596809f902dd47cf113a39670b (patch) | |
| tree | 666d16ddc0a69acb805a5f32f17c8502208349e7 /src/Progmasoft/Runner/CoverageReport.hpp | |
| parent | 43dabc9adb45b808ae24173ff4633774aa71b2ee (diff) | |
| download | catch3-feature/per-test-coverage.tar.gz | |
Keep per-test coverage with --coverage (roadmap step 4)feature/per-test-coverage
--coverage=DIR runs every attempt in a process of its own and keeps the
coverage of each test, for binaries built with LLVM source-based
coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov
(GCC: --coverage).
How it works:
- The scheduler gives each attempt a fresh worker, as --isolate does,
and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the
process writes at exit is then the coverage of that one test, in
DIR/profiles/t<index>-a<attempt>.profraw or a .gcov directory. Each
runtime reads its own variable, so no symbol of either runtime is
looked up: weak references would not pull them out of their static
archives, and Mach-O does not allow undefined weak symbols.
- A process that runs other tests first (bisection) writes beside the
profiles, never into one. The attempt records its profile, also in
the JSON results.
- The profiles of an earlier run are removed when the run starts,
because gcov adds to the .gcda files it finds.
- After the run, CoverageReport reads the last profile of every test:
llvm-profdata and llvm-cov (PATH, xcrun on macOS, or
CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or
CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is
DIR/coverage.json, the files, lines and branches each test reached,
and DIR/coverage.lcov for the whole run. Without the tools the
profiles stay and the run says so.
- --coverage with --in-process or --break is refused.
Tests: option parsing, LCOV reading and writing, gcov JSON, and an
integration test on CoverageFixture, a binary CMake builds with the
coverage flags of Clang or GCC: two tests reach different lines, each
map holds only its own, the LCOV report holds both, and a stale profile
of an earlier run is gone. Locally all 84 CTest tests pass with Clang,
and the coverage tests pass with GCC (built with
CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as
errors, before and after this change).
CI: the Linux jobs install LLVM's coverage tools, a new job checks the
gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map
into a failure on Linux and macOS instead of a skip. MSVC has no
source-based coverage, so its builds skip those tests; the roadmap
lists step 4 as in progress until Windows with clang-cl is tested.
Diffstat (limited to 'src/Progmasoft/Runner/CoverageReport.hpp')
| -rw-r--r-- | src/Progmasoft/Runner/CoverageReport.hpp | 58 |
1 files changed, 58 insertions, 0 deletions
diff --git a/src/Progmasoft/Runner/CoverageReport.hpp b/src/Progmasoft/Runner/CoverageReport.hpp new file mode 100644 index 00000000..35f9dc75 --- /dev/null +++ b/src/Progmasoft/Runner/CoverageReport.hpp @@ -0,0 +1,58 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 +#pragma once + +#include <Progmasoft/Catch3/Results.hpp> +#include <cstdint> +#include <filesystem> +#include <map> +#include <string> +#include <utility> +#include <vector> + +#include "Expected.hpp" + +namespace Progmasoft::Catch3::Runner +{ + /// What one test, or the whole run, reached in one source file. + struct FileCoverage final + { + /// Execution count of each instrumented line. + std::map<std::size_t, std::uint64_t> lines; + /// Count of each branch, by line and by its position on the line. + std::map<std::pair<std::size_t, std::size_t>, std::uint64_t> branches; + }; + + /// Coverage by source file. + using CoverageMap = std::map<std::string, FileCoverage>; + + /// Reads an LCOV tracefile: `SF`, `DA` and `BRDA` records. + [[nodiscard]] CoverageMap + ParseLcov(const std::string &text); + + /// Reads the JSON that `gcov --json-format` writes, adding to `map`. + void + AddGcovJson(const std::string &text, CoverageMap &map); + + /// Writes `map` as an LCOV tracefile. + [[nodiscard]] std::string + ToLcov(const CoverageMap &map); + + /// Prepares the directory of a coverage run: the profiles of an earlier + /// run below it are removed, because gcov adds to the .gcda files it + /// finds instead of replacing them. + Status + PrepareCoverageDirectory(const std::filesystem::path &directory); + + /// Turns the profiles a coverage run left in `directory` into + /// `directory/coverage.json`, the lines and branches each test reached, + /// and `directory/coverage.lcov`, the run combined. LLVM profiles are + /// read with `llvm-profdata` and `llvm-cov`, gcov data with `gcov`, + /// found on PATH or named by CATCH3_LLVM_PROFDATA, CATCH3_LLVM_COV and + /// CATCH3_GCOV. Returns what the run should be told: how many profiles + /// there were and where the reports are, or why there are none. + [[nodiscard]] std::vector<std::string> + WriteCoverageReports(const std::filesystem::path &directory, + const RunResult &run, + const std::string &program); +} // namespace Progmasoft::Catch3::Runner |