Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
path: root/src/Progmasoft/Runner
diff options
context:
space:
mode:
authorLeitwolf112026-11-13 07:46:08 +0300
committerLeitwolf112026-11-13 07:46:08 +0300
commit17248837a78462596809f902dd47cf113a39670b (patch)
tree666d16ddc0a69acb805a5f32f17c8502208349e7 /src/Progmasoft/Runner
parent43dabc9adb45b808ae24173ff4633774aa71b2ee (diff)
downloadcatch3-feature/per-test-coverage.tar.gz
Keep per-test coverage with --coverage (roadmap step 4)feature/per-test-coverage
--coverage=DIR runs every attempt in a process of its own and keeps the coverage of each test, for binaries built with LLVM source-based coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov (GCC: --coverage). How it works: - The scheduler gives each attempt a fresh worker, as --isolate does, and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the process writes at exit is then the coverage of that one test, in DIR/profiles/t<index>-a<attempt>.profraw or a .gcov directory. Each runtime reads its own variable, so no symbol of either runtime is looked up: weak references would not pull them out of their static archives, and Mach-O does not allow undefined weak symbols. - A process that runs other tests first (bisection) writes beside the profiles, never into one. The attempt records its profile, also in the JSON results. - The profiles of an earlier run are removed when the run starts, because gcov adds to the .gcda files it finds. - After the run, CoverageReport reads the last profile of every test: llvm-profdata and llvm-cov (PATH, xcrun on macOS, or CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is DIR/coverage.json, the files, lines and branches each test reached, and DIR/coverage.lcov for the whole run. Without the tools the profiles stay and the run says so. - --coverage with --in-process or --break is refused. Tests: option parsing, LCOV reading and writing, gcov JSON, and an integration test on CoverageFixture, a binary CMake builds with the coverage flags of Clang or GCC: two tests reach different lines, each map holds only its own, the LCOV report holds both, and a stale profile of an earlier run is gone. Locally all 84 CTest tests pass with Clang, and the coverage tests pass with GCC (built with CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as errors, before and after this change). CI: the Linux jobs install LLVM's coverage tools, a new job checks the gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map into a failure on Linux and macOS instead of a skip. MSVC has no source-based coverage, so its builds skip those tests; the roadmap lists step 4 as in progress until Windows with clang-cl is tested.
Diffstat (limited to 'src/Progmasoft/Runner')
-rw-r--r--src/Progmasoft/Runner/CoverageReport.cpp594
-rw-r--r--src/Progmasoft/Runner/CoverageReport.hpp58
-rw-r--r--src/Progmasoft/Runner/Options.cpp20
-rw-r--r--src/Progmasoft/Runner/Options.hpp4
-rw-r--r--src/Progmasoft/Runner/ResultsJson.cpp2
-rw-r--r--src/Progmasoft/Runner/Runner.cpp20
-rw-r--r--src/Progmasoft/Runner/Scheduler.cpp61
-rw-r--r--src/Progmasoft/Runner/SchedulerModel.hpp4
8 files changed, 760 insertions, 3 deletions
diff --git a/src/Progmasoft/Runner/CoverageReport.cpp b/src/Progmasoft/Runner/CoverageReport.cpp
new file mode 100644
index 00000000..c443c158
--- /dev/null
+++ b/src/Progmasoft/Runner/CoverageReport.cpp
@@ -0,0 +1,594 @@
+// Copyright (c) 2026 Progmasoft.
+// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0
+#include <algorithm>
+#include <cstdlib>
+#include <optional>
+#include <string_view>
+#include <system_error>
+
+#include "CoverageReport.hpp"
+#include "Files.hpp"
+#include "Json.hpp"
+#include "Process.hpp"
+
+namespace Progmasoft::Catch3::Runner
+{
+ namespace
+ {
+ /// Reads a non-negative decimal number; nothing for anything else,
+ /// such as the `-` LCOV writes for a branch never reached.
+ std::optional<std::uint64_t>
+ Number(std::string_view text)
+ {
+ if (text.empty() || text.size() > 19)
+ {
+ return std::nullopt;
+ }
+ std::uint64_t value = 0;
+ for (const char character : text)
+ {
+ if (character < '0' || character > '9')
+ {
+ return std::nullopt;
+ }
+ value
+ = value * 10 + static_cast<std::uint64_t>(character - '0');
+ }
+ return value;
+ }
+
+ std::vector<std::string_view>
+ Split(std::string_view text, char separator)
+ {
+ std::vector<std::string_view> parts;
+ std::size_t start = 0;
+ while (true)
+ {
+ const auto end = text.find(separator, start);
+ parts.push_back(text.substr(start, end - start));
+ if (end == std::string_view::npos)
+ {
+ return parts;
+ }
+ start = end + 1;
+ }
+ }
+
+ /// A tool of a coverage format: the program, or the command that
+ /// runs it, such as `xcrun llvm-cov` on macOS.
+ using Tool = std::vector<std::string>;
+
+ std::optional<Tool>
+ FindTool(const char *variable, const std::string &name)
+ {
+ std::vector<Tool> candidates;
+ if (auto configured = ReadEnvironment(variable);
+ !configured.empty())
+ {
+ candidates.push_back({ std::move(configured) });
+ }
+ candidates.push_back({ name });
+#if defined(__APPLE__)
+ candidates.push_back({ "xcrun", name });
+#endif
+ for (auto &candidate : candidates)
+ {
+ ChildOptions options;
+ options.arguments = candidate;
+ options.arguments.emplace_back("--version");
+ options.searchPath = true;
+ if (auto run = RunAndCapture(options);
+ run && run->status.Succeeded())
+ {
+ return std::move(candidate);
+ }
+ }
+ return std::nullopt;
+ }
+
+ /// Runs `tool` with `arguments`; its output, or why it failed.
+ Expected<std::string>
+ RunTool(const Tool &tool,
+ const std::vector<std::string> &arguments,
+ const std::filesystem::path &directory = {})
+ {
+ ChildOptions options;
+ options.arguments = tool;
+ options.arguments.insert(options.arguments.end(),
+ arguments.begin(),
+ arguments.end());
+ options.searchPath = true;
+ options.workingDirectory = directory;
+ auto run = RunAndCapture(options);
+ if (!run)
+ {
+ return run.GetError();
+ }
+ if (!run->status.Succeeded())
+ {
+ std::string message = run->errors;
+ while (!message.empty()
+ && (message.back() == '\n' || message.back() == '\r'))
+ {
+ message.pop_back();
+ }
+ return Error{ tool.back() + " failed: " + message };
+ }
+ return std::move(run->output);
+ }
+
+ void
+ Add(CoverageMap &total, const CoverageMap &part)
+ {
+ for (const auto &[file, coverage] : part)
+ {
+ FileCoverage &into = total[file];
+ for (const auto &[line, count] : coverage.lines)
+ {
+ into.lines[line] += count;
+ }
+ for (const auto &[branch, count] : coverage.branches)
+ {
+ into.branches[branch] += count;
+ }
+ }
+ }
+
+ /// What one test reached: the lines it ran and, per line, how many
+ /// of its branches it took.
+ JsonValue
+ ReachedJson(const CoverageMap &map)
+ {
+ JsonValue files = JsonValue::Array();
+ for (const auto &[file, coverage] : map)
+ {
+ JsonValue lines = JsonValue::Array();
+ for (const auto &[line, count] : coverage.lines)
+ {
+ if (count > 0)
+ {
+ lines.Append(line);
+ }
+ }
+ JsonValue branches = JsonValue::Array();
+ std::map<std::size_t, std::pair<std::size_t, std::size_t>>
+ perLine;
+ for (const auto &[branch, count] : coverage.branches)
+ {
+ auto &[taken, total] = perLine[branch.first];
+ ++total;
+ if (count > 0)
+ {
+ ++taken;
+ }
+ }
+ for (const auto &[line, counts] : perLine)
+ {
+ if (counts.first == 0)
+ {
+ continue;
+ }
+ branches.Append(
+ JsonValue::Object({ { "line", line },
+ { "taken", counts.first },
+ { "total", counts.second } }));
+ }
+ if (lines.Items().empty())
+ {
+ continue;
+ }
+ files.Append(
+ JsonValue::Object({ { "file", file },
+ { "lines", std::move(lines) },
+ { "branches", std::move(branches) } }));
+ }
+ return files;
+ }
+
+ /// The final attempt of a test that left a profile.
+ const Attempt *
+ ProfiledAttempt(const TestResult &test)
+ {
+ for (auto attempt = test.attempts.rbegin();
+ attempt != test.attempts.rend();
+ ++attempt)
+ {
+ if (!attempt->coverageProfile.empty())
+ {
+ return &*attempt;
+ }
+ }
+ return nullptr;
+ }
+
+ /// The .gcda files gcov wrote below `prefix`, each with the path the
+ /// object was built at, where its .gcno is.
+ std::vector<std::pair<std::filesystem::path, std::filesystem::path>>
+ GcdaFiles(const std::filesystem::path &prefix)
+ {
+ std::vector<std::pair<std::filesystem::path, std::filesystem::path>>
+ files;
+ std::error_code error;
+ for (auto entry
+ = std::filesystem::recursive_directory_iterator(prefix, error);
+ !error
+ && entry != std::filesystem::recursive_directory_iterator();
+ entry.increment(error))
+ {
+ if (!entry->is_regular_file(error)
+ || entry->path().extension() != ".gcda")
+ {
+ continue;
+ }
+ // GCOV_PREFIX is put before the absolute path of the object.
+ const auto relative
+ = std::filesystem::relative(entry->path(), prefix, error);
+ std::filesystem::path original
+ = std::filesystem::path{ "/" } / relative;
+ files.emplace_back(entry->path(), original);
+ }
+ std::sort(files.begin(), files.end());
+ return files;
+ }
+
+ Expected<CoverageMap>
+ GcovCoverage(const Tool &gcov, const std::filesystem::path &prefix)
+ {
+ CoverageMap map;
+ for (const auto &[gcda, original] : GcdaFiles(prefix))
+ {
+ auto gcno = original;
+ gcno.replace_extension(".gcno");
+ auto beside = gcda;
+ beside.replace_extension(".gcno");
+ std::error_code error;
+ std::filesystem::copy_file(
+ gcno,
+ beside,
+ std::filesystem::copy_options::overwrite_existing,
+ error);
+ if (error)
+ {
+ return Error{ "the notes file " + gcno.string()
+ + " could not be read: " + error.message() };
+ }
+ auto json = RunTool(
+ gcov,
+ { "--json-format", "--stdout", gcda.filename().string() },
+ gcda.parent_path());
+ if (!json)
+ {
+ return json.GetError();
+ }
+ // One JSON document per data file, one per line.
+ for (const auto line : Split(*json, '\n'))
+ {
+ if (!line.empty())
+ {
+ AddGcovJson(std::string{ line }, map);
+ }
+ }
+ }
+ return map;
+ }
+ } // namespace
+
+ CoverageMap
+ ParseLcov(const std::string &text)
+ {
+ CoverageMap map;
+ FileCoverage *current = nullptr;
+ for (auto line : Split(text, '\n'))
+ {
+ if (!line.empty() && line.back() == '\r')
+ {
+ line.remove_suffix(1);
+ }
+ if (line.substr(0, 3) == "SF:")
+ {
+ current = &map[std::string{ line.substr(3) }];
+ }
+ else if (line == "end_of_record")
+ {
+ current = nullptr;
+ }
+ else if (current != nullptr && line.substr(0, 3) == "DA:")
+ {
+ const auto fields = Split(line.substr(3), ',');
+ const auto number = Number(fields[0]);
+ const auto count
+ = fields.size() > 1 ? Number(fields[1]) : std::nullopt;
+ if (number && count)
+ {
+ current->lines[*number] += *count;
+ }
+ }
+ else if (current != nullptr && line.substr(0, 5) == "BRDA:")
+ {
+ const auto fields = Split(line.substr(5), ',');
+ if (fields.size() != 4)
+ {
+ continue;
+ }
+ const auto number = Number(fields[0]);
+ if (!number)
+ {
+ continue;
+ }
+ // The branches of a line are numbered in the order they come;
+ // `-` counts as never taken.
+ std::size_t index = 0;
+ for (auto it = current->branches.lower_bound({ *number, 0 });
+ it != current->branches.end()
+ && it->first.first == *number;
+ ++it)
+ {
+ ++index;
+ }
+ current->branches[{ *number, index }]
+ += Number(fields[3]).value_or(0);
+ }
+ }
+ return map;
+ }
+
+ void
+ AddGcovJson(const std::string &text, CoverageMap &map)
+ {
+ auto document = ParseJson(text);
+ if (!document)
+ {
+ return;
+ }
+ for (const auto &file : (*document)["files"].Items())
+ {
+ FileCoverage &coverage = map[file["file"].AsString()];
+ for (const auto &line : file["lines"].Items())
+ {
+ const auto number = static_cast<std::size_t>(
+ line["line_number"].AsUnsigned());
+ coverage.lines[number] += line["count"].AsUnsigned();
+ std::size_t index = 0;
+ for (const auto &branch : line["branches"].Items())
+ {
+ coverage.branches[{ number, index++ }]
+ += branch["count"].AsUnsigned();
+ }
+ }
+ }
+ }
+
+ std::string
+ ToLcov(const CoverageMap &map)
+ {
+ std::string text;
+ for (const auto &[file, coverage] : map)
+ {
+ text += "SF:" + file + "\n";
+ std::size_t linesHit = 0;
+ for (const auto &[line, count] : coverage.lines)
+ {
+ text += "DA:" + std::to_string(line) + ","
+ + std::to_string(count) + "\n";
+ linesHit += count > 0 ? 1 : 0;
+ }
+ std::size_t branchesHit = 0;
+ for (const auto &[branch, count] : coverage.branches)
+ {
+ text += "BRDA:" + std::to_string(branch.first) + ",0,"
+ + std::to_string(branch.second) + ","
+ + (count > 0 ? std::to_string(count) : "-") + "\n";
+ branchesHit += count > 0 ? 1 : 0;
+ }
+ text += "LF:" + std::to_string(coverage.lines.size()) + "\n";
+ text += "LH:" + std::to_string(linesHit) + "\n";
+ text += "BRF:" + std::to_string(coverage.branches.size()) + "\n";
+ text += "BRH:" + std::to_string(branchesHit) + "\n";
+ text += "end_of_record\n";
+ }
+ return text;
+ }
+
+ Status
+ PrepareCoverageDirectory(const std::filesystem::path &directory)
+ {
+ std::error_code error;
+ std::filesystem::remove_all(directory / "profiles", error);
+ if (error)
+ {
+ return Error{ "the profiles of an earlier run in "
+ + directory.string()
+ + " could not be removed: " + error.message() };
+ }
+ std::filesystem::create_directories(directory / "profiles", error);
+ if (error)
+ {
+ return Error{ "the coverage directory " + directory.string()
+ + " could not be created: " + error.message() };
+ }
+ return Status::Ok();
+ }
+
+ std::vector<std::string>
+ WriteCoverageReports(const std::filesystem::path &directory,
+ const RunResult &run,
+ const std::string &program)
+ {
+ std::vector<std::string> notes;
+ std::size_t llvmProfiles = 0;
+ std::size_t gcovProfiles = 0;
+ for (const auto &test : run.tests)
+ {
+ if (const Attempt *attempt = ProfiledAttempt(test))
+ {
+ std::error_code error;
+ llvmProfiles += std::filesystem::exists(attempt->coverageProfile
+ + ".profraw",
+ error)
+ ? 1
+ : 0;
+ gcovProfiles += std::filesystem::exists(attempt->coverageProfile
+ + ".gcov",
+ error)
+ ? 1
+ : 0;
+ }
+ }
+ if (llvmProfiles == 0 && gcovProfiles == 0)
+ {
+ notes.emplace_back(
+ "--coverage: the tests wrote no coverage; build them with "
+ "-fprofile-instr-generate -fcoverage-mapping (Clang) or "
+ "--coverage (GCC)");
+ return notes;
+ }
+ const bool llvm = llvmProfiles >= gcovProfiles;
+ std::optional<Tool> profdata;
+ std::optional<Tool> cov;
+ std::optional<Tool> gcov;
+ if (llvm)
+ {
+ profdata = FindTool("CATCH3_LLVM_PROFDATA", "llvm-profdata");
+ cov = FindTool("CATCH3_LLVM_COV", "llvm-cov");
+ }
+ else
+ {
+ gcov = FindTool("CATCH3_GCOV", "gcov");
+ }
+ if (llvm ? (!profdata || !cov) : !gcov)
+ {
+ notes.push_back(
+ "--coverage: "
+ + std::to_string(llvm ? llvmProfiles : gcovProfiles)
+ + " profile(s) in " + (directory / "profiles").string() + "; "
+ + (llvm ? "llvm-profdata and llvm-cov were" : "gcov was")
+ + " not found, so there is no map of what each test reached");
+ return notes;
+ }
+
+ CoverageMap combined;
+ std::vector<std::string> raw;
+ JsonValue tests = JsonValue::Array();
+ std::vector<std::string> failures;
+ for (const auto &test : run.tests)
+ {
+ const Attempt *attempt = ProfiledAttempt(test);
+ if (attempt == nullptr)
+ {
+ continue;
+ }
+ const std::string base = attempt->coverageProfile;
+ Expected<CoverageMap> reached = CoverageMap{};
+ std::string profile;
+ if (llvm)
+ {
+ profile = base + ".profraw";
+ std::error_code error;
+ if (!std::filesystem::exists(profile, error))
+ {
+ continue;
+ }
+ raw.push_back(profile);
+ auto merged = RunTool(
+ *profdata,
+ { "merge", "-sparse", profile, "-o", base + ".profdata" });
+ if (!merged)
+ {
+ failures.push_back(test.name + ": "
+ + merged.GetError().Message());
+ continue;
+ }
+ auto lcov = RunTool(*cov,
+ { "export",
+ "-format=lcov",
+ "-instr-profile=" + base + ".profdata",
+ program });
+ if (!lcov)
+ {
+ failures.push_back(test.name + ": "
+ + lcov.GetError().Message());
+ continue;
+ }
+ reached = ParseLcov(*lcov);
+ }
+ else
+ {
+ profile = base + ".gcov";
+ std::error_code error;
+ if (!std::filesystem::exists(profile, error))
+ {
+ continue;
+ }
+ reached = GcovCoverage(*gcov, profile);
+ if (!reached)
+ {
+ failures.push_back(test.name + ": "
+ + reached.GetError().Message());
+ continue;
+ }
+ Add(combined, *reached);
+ }
+ tests.Append(
+ JsonValue::Object({ { "name", test.name },
+ { "outcome", OutcomeName(test.outcome) },
+ { "profile", profile },
+ { "files", ReachedJson(*reached) } }));
+ }
+
+ std::string lcovText;
+ if (llvm && !raw.empty())
+ {
+ std::vector<std::string> arguments{ "merge", "-sparse" };
+ arguments.insert(arguments.end(), raw.begin(), raw.end());
+ const std::string all = (directory / "coverage.profdata").string();
+ arguments.emplace_back("-o");
+ arguments.push_back(all);
+ auto merged = RunTool(*profdata, arguments);
+ auto exported = merged ? RunTool(*cov,
+ { "export",
+ "-format=lcov",
+ "-instr-profile=" + all,
+ program })
+ : Expected<std::string>{ merged.GetError() };
+ if (exported)
+ {
+ lcovText = *exported;
+ }
+ else
+ {
+ failures.push_back("the combined report: "
+ + exported.GetError().Message());
+ }
+ }
+ else
+ {
+ lcovText = ToLcov(combined);
+ }
+
+ JsonValue report
+ = JsonValue::Object({ { "version", 1 },
+ { "format", llvm ? "llvm" : "gcov" },
+ { "binary", program },
+ { "tests", std::move(tests) } });
+ const auto json = directory / "coverage.json";
+ const auto lcov = directory / "coverage.lcov";
+ if (auto status = WriteFileAtomically(json, report.ToPrettyString());
+ !status)
+ {
+ failures.push_back(status.GetError().Message());
+ }
+ if (auto status = WriteFileAtomically(lcov, lcovText); !status)
+ {
+ failures.push_back(status.GetError().Message());
+ }
+ notes.push_back(
+ "--coverage: " + std::to_string(llvm ? llvmProfiles : gcovProfiles)
+ + " test profile(s); what each test reached is in " + json.string()
+ + ", the run combined in " + lcov.string());
+ for (const auto &failure : failures)
+ {
+ notes.push_back("--coverage: " + failure);
+ }
+ return notes;
+ }
+} // namespace Progmasoft::Catch3::Runner
diff --git a/src/Progmasoft/Runner/CoverageReport.hpp b/src/Progmasoft/Runner/CoverageReport.hpp
new file mode 100644
index 00000000..35f9dc75
--- /dev/null
+++ b/src/Progmasoft/Runner/CoverageReport.hpp
@@ -0,0 +1,58 @@
+// Copyright (c) 2026 Progmasoft.
+// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0
+#pragma once
+
+#include <Progmasoft/Catch3/Results.hpp>
+#include <cstdint>
+#include <filesystem>
+#include <map>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "Expected.hpp"
+
+namespace Progmasoft::Catch3::Runner
+{
+ /// What one test, or the whole run, reached in one source file.
+ struct FileCoverage final
+ {
+ /// Execution count of each instrumented line.
+ std::map<std::size_t, std::uint64_t> lines;
+ /// Count of each branch, by line and by its position on the line.
+ std::map<std::pair<std::size_t, std::size_t>, std::uint64_t> branches;
+ };
+
+ /// Coverage by source file.
+ using CoverageMap = std::map<std::string, FileCoverage>;
+
+ /// Reads an LCOV tracefile: `SF`, `DA` and `BRDA` records.
+ [[nodiscard]] CoverageMap
+ ParseLcov(const std::string &text);
+
+ /// Reads the JSON that `gcov --json-format` writes, adding to `map`.
+ void
+ AddGcovJson(const std::string &text, CoverageMap &map);
+
+ /// Writes `map` as an LCOV tracefile.
+ [[nodiscard]] std::string
+ ToLcov(const CoverageMap &map);
+
+ /// Prepares the directory of a coverage run: the profiles of an earlier
+ /// run below it are removed, because gcov adds to the .gcda files it
+ /// finds instead of replacing them.
+ Status
+ PrepareCoverageDirectory(const std::filesystem::path &directory);
+
+ /// Turns the profiles a coverage run left in `directory` into
+ /// `directory/coverage.json`, the lines and branches each test reached,
+ /// and `directory/coverage.lcov`, the run combined. LLVM profiles are
+ /// read with `llvm-profdata` and `llvm-cov`, gcov data with `gcov`,
+ /// found on PATH or named by CATCH3_LLVM_PROFDATA, CATCH3_LLVM_COV and
+ /// CATCH3_GCOV. Returns what the run should be told: how many profiles
+ /// there were and where the reports are, or why there are none.
+ [[nodiscard]] std::vector<std::string>
+ WriteCoverageReports(const std::filesystem::path &directory,
+ const RunResult &run,
+ const std::string &program);
+} // namespace Progmasoft::Catch3::Runner
diff --git a/src/Progmasoft/Runner/Options.cpp b/src/Progmasoft/Runner/Options.cpp
index 13cf4be7..aef0df6c 100644
--- a/src/Progmasoft/Runner/Options.cpp
+++ b/src/Progmasoft/Runner/Options.cpp
@@ -830,6 +830,21 @@ namespace Progmasoft::Catch3::Runner
options.tracePath = *value;
return Status::Ok();
}},
+ {"coverage",
+ '\0',
+ Takes::kValue,
+ "DIR",
+ "Output",
+ "run every test in a process of its own and keep its "
+ "coverage in DIR, with a map of what each test reached",
+ [](Options &options, const std::optional<std::string> &value) {
+ if (value->empty())
+ {
+ return Status{ Error{ "--coverage needs a directory" } };
+ }
+ options.coverageDirectory = *value;
+ return Status::Ok();
+ }},
{"premature-exit-guard-file",
'\0',
Takes::kValue,
@@ -1105,6 +1120,11 @@ namespace Progmasoft::Catch3::Runner
return status.GetError();
}
}
+ if (!options.coverageDirectory.empty() && options.inProcess)
+ {
+ return Error{ "--coverage runs every test in a process of its own, "
+ "so it cannot be used with --in-process or --break" };
+ }
return options;
}
diff --git a/src/Progmasoft/Runner/Options.hpp b/src/Progmasoft/Runner/Options.hpp
index 256f5ac7..3d53eb42 100644
--- a/src/Progmasoft/Runner/Options.hpp
+++ b/src/Progmasoft/Runner/Options.hpp
@@ -190,6 +190,10 @@ namespace Progmasoft::Catch3::Runner
std::string junitPath;
std::string jsonPath;
std::string tracePath;
+ /// Per-test coverage: every attempt runs in a process of its own and
+ /// leaves its profile below this directory, with the map of what
+ /// each test reached and the combined report; see CoverageReport.
+ std::string coverageDirectory;
unsigned slowest = 0;
/// Created when the run starts and removed when it ends normally, so
/// a file left behind shows that the run ended early.
diff --git a/src/Progmasoft/Runner/ResultsJson.cpp b/src/Progmasoft/Runner/ResultsJson.cpp
index ab892cf7..c82475ec 100644
--- a/src/Progmasoft/Runner/ResultsJson.cpp
+++ b/src/Progmasoft/Runner/ResultsJson.cpp
@@ -203,6 +203,7 @@ namespace Progmasoft::Catch3::Runner
SetText(value, "exitDescription", attempt.exitDescription);
SetText(value, "lastLocation", attempt.lastLocation);
SetText(value, "temporaryDirectory", attempt.temporaryDirectory);
+ SetText(value, "coverageProfile", attempt.coverageProfile);
return value;
}
@@ -242,6 +243,7 @@ namespace Progmasoft::Catch3::Runner
attempt.exitDescription = value["exitDescription"].AsString();
attempt.lastLocation = value["lastLocation"].AsString();
attempt.temporaryDirectory = value["temporaryDirectory"].AsString();
+ attempt.coverageProfile = value["coverageProfile"].AsString();
return attempt;
}
diff --git a/src/Progmasoft/Runner/Runner.cpp b/src/Progmasoft/Runner/Runner.cpp
index 77011322..61f8f2b7 100644
--- a/src/Progmasoft/Runner/Runner.cpp
+++ b/src/Progmasoft/Runner/Runner.cpp
@@ -22,6 +22,7 @@
#include <random>
#include <system_error>
+#include "CoverageReport.hpp"
#include "Analysis.hpp"
#include "Catalog.hpp"
#include "Console.hpp"
@@ -949,6 +950,17 @@ namespace Progmasoft::Catch3
}
}
+ if (!options.coverageDirectory.empty())
+ {
+ if (auto status = Runner::PrepareCoverageDirectory(
+ options.coverageDirectory);
+ !status)
+ {
+ PrintError(status.GetError().Message());
+ return Catch::UnspecifiedErrorExitCode;
+ }
+ }
+
const auto started = std::chrono::steady_clock::now();
RunResult run;
run.binary = programName;
@@ -983,6 +995,14 @@ namespace Progmasoft::Catch3
run.tests.push_back(std::move(kept));
}
run.notes = std::move(outcome.notes);
+ if (!options.coverageDirectory.empty())
+ {
+ for (auto &note : Runner::WriteCoverageReports(
+ options.coverageDirectory, run, program.string()))
+ {
+ run.notes.push_back(std::move(note));
+ }
+ }
const bool anyFailure
= std::any_of(run.tests.begin(),
diff --git a/src/Progmasoft/Runner/Scheduler.cpp b/src/Progmasoft/Runner/Scheduler.cpp
index 56a2aa36..263257f7 100644
--- a/src/Progmasoft/Runner/Scheduler.cpp
+++ b/src/Progmasoft/Runner/Scheduler.cpp
@@ -346,6 +346,10 @@ namespace Progmasoft::Catch3::Runner
}
attempt.temporaryDirectory
= TemporaryFor(catalogIndex, attempt.number);
+ if (target)
+ {
+ attempt.coverageProfile = worker.coverageProfile;
+ }
worker.running = std::move(running);
if (options_.inProcess)
{
@@ -973,8 +977,33 @@ namespace Progmasoft::Catch3::Runner
// Worker processes.
+ /// Where the coverage profile of the next attempt of `job` goes,
+ /// without its extension. Only an attempt that runs alone in its
+ /// process gets one: a job that runs other tests first would
+ //