catch3
Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.
Clonegit clone https://git.progmasoft.com/catch3
Fix the coverage review findings: branches, stale reports, exit status
- Commit
454cca9b63991416f4d9bd1ca350f1ae14e278e7- Author
- Leitwolf11 ·
- Parents
55dab34729c1
Changes
docs/progmasoft-runner.md | 19 +- extras/catch_amalgamated.cpp | 391 ++++++++++++++++++++---------- src/Progmasoft/Runner/CoverageReport.cpp | 337 +++++++++++++++---------- src/Progmasoft/Runner/CoverageReport.hpp | 35 ++- src/Progmasoft/Runner/Runner.cpp | 19 +- tests/Progmasoft/CoverageFixture.cpp | 11 + tests/Progmasoft/RunnerCoverage.tests.cpp | 67 +++++ 7 files changed, 604 insertions(+), 275 deletions(-) diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md index cb5f29c5..d61ebadd 100644 --- a/docs/progmasoft-runner.md +++ b/docs/progmasoft-runner.md @@ -356,12 +356,13 @@ with `.profraw` for LLVM, or a `.gcov` directory below which gcov writes its `.gcda` files under their absolute paths. The runner sets `LLVM_PROFILE_FILE` and `GCOV_PREFIX` for the process; each runtime reads its own. A process that runs other tests first, to find a test that leaves state behind, writes -beside the profiles, never into one. The profiles of an earlier run in -`DIR/profiles` are removed when the run starts, because gcov adds to the -data it finds. +beside the profiles, never into one. The profiles and reports of an earlier +run in `DIR` are removed when the run starts: gcov adds to the data it finds, +and a run that writes no reports leaves none to be read as its own. -After the run, the runner reads the profile of the last attempt of every -test: +After the run, the runner reads, on as many threads as the run had workers, +the profile of the last attempt of every test that left one; when a retry or +repeat crashed before writing its profile, the attempt before it counts: * `DIR/coverage.json` holds, for each test, its outcome, its profile and the files it reached, with the lines it ran and, per line, how many of its @@ -381,8 +382,12 @@ test: LLVM profiles are read with `llvm-profdata` and `llvm-cov`, gcov data with `gcov`. They are looked up on `PATH`, through `xcrun` on macOS, or named by `CATCH3_LLVM_PROFDATA`, `CATCH3_LLVM_COV` and `CATCH3_GCOV`. Without them the -profiles stay in `DIR/profiles` and the run says so. A test that crashes or -is stopped at its time limit leaves no profile. The runner's own process is +profiles stay in `DIR/profiles` and the run says so, as it does when the +tests wrote no profiles; neither fails the run. A profile the tools cannot +read, or a report that cannot be written, fails a run whose tests passed, +like any report the run was asked for. A test that crashes or is stopped at +its time limit leaves no profile. gcov is run with `--branch-probabilities`, +without which its JSON has no branches. The runner's own process is instrumented too and writes its profile where any instrumented program does, `default.profraw` in the working directory unless `LLVM_PROFILE_FILE` says otherwise; it holds no test. diff --git a/extras/catch_amalgamated.cpp b/extras/catch_amalgamated.cpp index 76fab4f2..cc4612f2 100644 --- a/extras/catch_amalgamated.cpp +++ b/extras/catch_amalgamated.cpp @@ -1378,23 +1378,44 @@ namespace Progmasoft::Catch3::Runner [[nodiscard]] std::string ToLcov(const CoverageMap &map); - /// Prepares the directory of a coverage run: the profiles of an earlier - /// run below it are removed, because gcov adds to the .gcda files it - /// finds instead of replacing them. + /// Prepares the directory of a coverage run: the profiles and reports of + /// an earlier run below it are removed, because gcov adds to the .gcda + /// files it finds instead of replacing them, and because a run that + /// writes no reports must not leave the earlier ones to be read as its + /// own. Status PrepareCoverageDirectory(const std::filesystem::path &directory); + /// The last attempt of `test` whose profile, its base name followed by + /// `suffix` (`.profraw` or `.gcov`), exists: an attempt that crashed or + /// timed out before writing one is passed over for an earlier one. + [[nodiscard]] const Attempt * + ProfiledAttempt(const TestResult &test, const std::string &suffix); + + /// What writing the coverage reports gave. + struct CoverageReports final + { + /// What the run should be told: how many profiles there were and + /// where the reports are, or why there are none. + std::vector<std::string> notes; + /// What failed: a profile that could not be read, a report that + /// could not be written. A run with failures is missing a report. + std::vector<std::string> failures; + }; + /// Turns the profiles a coverage run left in `directory` into /// `directory/coverage.json`, the lines and branches each test reached, /// and `directory/coverage.lcov`, the run combined. LLVM profiles are /// read with `llvm-profdata` and `llvm-cov`, gcov data with `gcov`, /// found on PATH or named by CATCH3_LLVM_PROFDATA, CATCH3_LLVM_COV and - /// CATCH3_GCOV. Returns what the run should be told: how many profiles - /// there were and where the reports are, or why there are none. - [[nodiscard]] std::vector<std::string> + /// CATCH3_GCOV; the profiles of the tests are read up to `jobs` at a + /// time. A run whose tests wrote no profiles, or whose tools are + /// missing, gets a note and no reports, not a failure. + [[nodiscard]] CoverageReports WriteCoverageReports(const std::filesystem::path &directory, const RunResult &run, - const std::string &program); + const std::string &program, + std::size_t jobs); } // namespace Progmasoft::Catch3::Runner // Copyright (c) 2026 Progmasoft. // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 @@ -10931,10 +10952,13 @@ namespace Progmasoft::Catch3::Runner // Copyright (c) 2026 Progmasoft. // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 #include <algorithm> +#include <atomic> #include <cstdlib> +#include <initializer_list> #include <optional> #include <string_view> #include <system_error> +#include <thread> namespace Progmasoft::Catch3::Runner @@ -11111,22 +11135,6 @@ namespace Progmasoft::Catch3::Runner return files; } - /// The final attempt of a test that left a profile. - const Attempt * - ProfiledAttempt(const TestResult &test) - { - for (auto attempt = test.attempts.rbegin(); - attempt != test.attempts.rend(); - ++attempt) - { - if (!attempt->coverageProfile.empty()) - { - return &*attempt; - } - } - return nullptr; - } - /// The .gcda files gcov wrote below `prefix`, each with the path the /// object was built at, where its .gcno is. std::vector<std::pair<std::filesystem::path, std::filesystem::path>> @@ -11180,7 +11188,11 @@ namespace Progmasoft::Catch3::Runner } auto json = RunTool( gcov, - { "--json-format", "--stdout", gcda.filename().string() }, + // Without --branch-probabilities the JSON has no branches. + { "--json-format", + "--branch-probabilities", + "--stdout", + gcda.filename().string() }, gcda.parent_path()); if (!json) { @@ -11327,6 +11339,17 @@ namespace Progmasoft::Catch3::Runner + directory.string() + " could not be removed: " + error.message() }; } + for (const char *report : + { "coverage.json", "coverage.lcov", "coverage.profdata" }) + { + std::filesystem::remove(directory / report, error); + if (error) + { + return Error{ "the report " + (directory / report).string() + + " of an earlier run could not be removed: " + + error.message() }; + } + } std::filesystem::create_directories(directory / "profiles", error); if (error) { @@ -11336,142 +11359,215 @@ namespace Progmasoft::Catch3::Runner return Status::Ok(); } - std::vector<std::string> + const Attempt * + ProfiledAttempt(const TestResult &test, const std::string &suffix) + { + for (auto attempt = test.attempts.rbegin(); + attempt != test.attempts.rend(); + ++attempt) + { + std::error_code error; + if (!attempt->coverageProfile.empty() + && std::filesystem::exists(attempt->coverageProfile + suffix, + error)) + { + return &*attempt; + } + } + return nullptr; + } + + namespace + { + /// The coverage tools of one profile format. + struct CoverageTools final + { + bool llvm = false; + std::optional<Tool> profdata; + std::optional<Tool> cov; + std::optional<Tool> gcov; + }; + + /// What the profile of one test gave. + struct TestCoverage final + { + const TestResult *test = nullptr; + std::string profile; + std::optional<CoverageMap> reached; + std::string failure; + }; + + TestCoverage + ReadTestCoverage(const TestResult &test, + const Attempt &attempt, + const CoverageTools &tools, + const std::string &program) + { + TestCoverage result; + result.test = &test; + const std::string base = attempt.coverageProfile; + if (tools.llvm) + { + result.profile = base + ".profraw"; + auto merged = RunTool(*tools.profdata, + { "merge", + "-sparse", + result.profile, + "-o", + base + ".profdata" }); + if (!merged) + { + result.failure = merged.GetError().Message(); + return result; + } + auto lcov = RunTool(*tools.cov, + { "export", + "-format=lcov", + "-instr-profile=" + base + ".profdata", + program }); + if (!lcov) + { + result.failure = lcov.GetError().Message(); + return result; + } + result.reached = ParseLcov(*lcov); + return result; + } + result.profile = base + ".gcov"; + auto reached = GcovCoverage(*tools.gcov, result.profile); + if (!reached) + { + result.failure = reached.GetError().Message(); + return result; + } + result.reached = std::move(*reached); + return result; + } + + /// Reads the profile of each of `tests`, up to `jobs` at a time; + /// the results keep the order of `tests`. + std::vector<TestCoverage> + ReadAll( + const std::vector<std::pair<const TestResult *, const Attempt *>> + &tests, + const CoverageTools &tools, + const std::string &program, + std::size_t jobs) + { + std::vector<TestCoverage> results(tests.size()); + std::atomic<std::size_t> next{ 0 }; + const auto work = [&] { + for (std::size_t index = next++; index < tests.size(); + index = next++) + { + results[index] = ReadTestCoverage(*tests[index].first, + *tests[index].second, + tools, + program); + } + }; + const std::size_t threads + = std::min(std::max<std::size_t>(jobs, 1), tests.size()); + std::vector<std::thread> pool; + for (std::size_t thread = 1; thread < threads; ++thread) + { + pool.emplace_back(work); + } + work(); + for (auto &thread : pool) + { + thread.join(); + } + return results; + } + } // namespace + + CoverageReports WriteCoverageReports(const std::filesystem::path &directory, const RunResult &run, - const std::string &program) + const std::string &program, + std::size_t jobs) { - std::vector<std::string> notes; - std::size_t llvmProfiles = 0; - std::size_t gcovProfiles = 0; + CoverageReports reports; + std::vector<std::pair<const TestResult *, const Attempt *>> llvmTests; + std::vector<std::pair<const TestResult *, const Attempt *>> gcovTests; for (const auto &test : run.tests) { - if (const Attempt *attempt = ProfiledAttempt(test)) + if (const Attempt *attempt = ProfiledAttempt(test, ".profraw")) { - std::error_code error; - llvmProfiles += std::filesystem::exists(attempt->coverageProfile - + ".profraw", - error) - ? 1 - : 0; - gcovProfiles += std::filesystem::exists(attempt->coverageProfile - + ".gcov", - error) - ? 1 - : 0; + llvmTests.emplace_back(&test, attempt); + } + if (const Attempt *attempt = ProfiledAttempt(test, ".gcov")) + { + gcovTests.emplace_back(&test, attempt); } } - if (llvmProfiles == 0 && gcovProfiles == 0) + if (llvmTests.empty() && gcovTests.empty()) { - notes.emplace_back( + reports.notes.emplace_back( "--coverage: the tests wrote no coverage; build them with " "-fprofile-instr-generate -fcoverage-mapping (Clang) or " "--coverage (GCC)"); - return notes; + return reports; } - const bool llvm = llvmProfiles >= gcovProfiles; - std::optional<Tool> profdata; - std::optional<Tool> cov; - std::optional<Tool> gcov; - if (llvm) + CoverageTools tools; + tools.llvm = llvmTests.size() >= gcovTests.size(); + const auto &profiled = tools.llvm ? llvmTests : gcovTests; + if (tools.llvm) { - profdata = FindTool("CATCH3_LLVM_PROFDATA", "llvm-profdata"); - cov = FindTool("CATCH3_LLVM_COV", "llvm-cov"); + tools.profdata = FindTool("CATCH3_LLVM_PROFDATA", "llvm-profdata"); + tools.cov = FindTool("CATCH3_LLVM_COV", "llvm-cov"); } else { - gcov = FindTool("CATCH3_GCOV", "gcov"); + tools.gcov = FindTool("CATCH3_GCOV", "gcov"); } - if (llvm ? (!profdata || !cov) : !gcov) + if (tools.llvm ? (!tools.profdata || !tools.cov) : !tools.gcov) { - notes.push_back( - "--coverage: " - + std::to_string(llvm ? llvmProfiles : gcovProfiles) + reports.notes.push_back( + "--coverage: " + std::to_string(profiled.size()) + " profile(s) in " + (directory / "profiles").string() + "; " - + (llvm ? "llvm-profdata and llvm-cov were" : "gcov was") + + (tools.llvm ? "llvm-profdata and llvm-cov were" : "gcov was") + " not found, so there is no map of what each test reached"); - return notes; + return reports; } CoverageMap combined; std::vector<std::string> raw; JsonValue tests = JsonValue::Array(); - std::vector<std::string> failures; - for (const auto &test : run.tests) + for (auto &read : ReadAll(profiled, tools, program, jobs)) { - const Attempt *attempt = ProfiledAttempt(test); - if (attempt == nullptr) + if (!read.reached) { + reports.failures.push_back(read.test->name + ": " + + read.failure); continue; } - const std::string base = attempt->coverageProfile; - Expected<CoverageMap> reached = CoverageMap{}; - std::string profile; - if (llvm) + if (tools.llvm) { - profile = base + ".profraw"; - std::error_code error; - if (!std::filesystem::exists(profile, error)) - { - continue; - } - raw.push_back(profile); - auto merged = RunTool( - *profdata, - { "merge", "-sparse", profile, "-o", base + ".profdata" }); - if (!merged) - { - failures.push_back(test.name + ": " - + merged.GetError().Message()); - continue; - } - auto lcov = RunTool(*cov, - { "export", - "-format=lcov", - "-instr-profile=" + base + ".profdata", - program }); - if (!lcov) - { - failures.push_back(test.name + ": " - + lcov.GetError().Message()); - continue; - } - reached = ParseLcov(*lcov); + raw.push_back(read.profile); } else { - profile = base + ".gcov"; - std::error_code error; - if (!std::filesystem::exists(profile, error)) - { - continue; - } - reached = GcovCoverage(*gcov, profile); - if (!reached) - { - failures.push_back(test.name + ": " - + reached.GetError().Message()); - continue; - } - Add(combined, *reached); + Add(combined, *read.reached); } - tests.Append( - JsonValue::Object({ { "name", test.name }, - { "outcome", OutcomeName(test.outcome) }, - { "profile", profile }, - { "files", ReachedJson(*reached) } })); + tests.Append(JsonValue::Object( + { { "name", read.test->name }, + { "outcome", OutcomeName(read.test->outcome) }, + { "profile", read.profile }, + { "files", ReachedJson(*read.reached) } })); } std::string lcovText; - if (llvm && !raw.empty()) + if (tools.llvm && !raw.empty()) { std::vector<std::string> arguments{ "merge", "-sparse" }; arguments.insert(arguments.end(), raw.begin(), raw.end()); const std::string all = (directory / "coverage.profdata").string(); arguments.emplace_back("-o"); arguments.push_back(all); - auto merged = RunTool(*profdata, arguments); - auto exported = merged ? RunTool(*cov, + auto merged = RunTool(*tools.profdata, arguments); + auto exported = merged ? RunTool(*tools.cov, { "export", "-format=lcov", "-instr-profile=" + all, @@ -11483,8 +11579,8 @@ namespace Progmasoft::Catch3::Runner } else { - failures.push_back("the combined report: " - + exported.GetError().Message()); + reports.failures.push_back("the combined report: " + + exported.GetError().Message()); } } else @@ -11494,29 +11590,43 @@ namespace Progmasoft::Catch3::Runner JsonValue report = JsonValue::Object({ { "version", 1 }, - { "format", llvm ? "llvm" : "gcov" }, + { "format", tools.llvm ? "llvm" : "gcov" }, { "binary", program }, { "tests", std::move(tests) } }); const auto json = directory / "coverage.json"; const auto lcov = directory / "coverage.lcov"; - if (auto status = WriteFileAtomically(json, report.ToPrettyString()); - !status) + const auto jsonWritten + = WriteFileAtomically(json, report.ToPrettyString()); + if (!jsonWritten) + { + reports.failures.push_back(jsonWritten.GetError().Message()); + } + const auto lcovWritten = WriteFileAtomically(lcov, lcovText); + if (!lcovWritten) { - failures.push_back(status.GetError().Message()); + reports.failures.push_back(lcovWritten.GetError().Message()); } - if (auto status = WriteFileAtomically(lcov, lcovText); !status) + std::string written; + if (jsonWritten && lcovWritten) { - failures.push_back(status.GetError().Message()); + written = "; what each test reached is in " + json.string() + + ", the run combined in " + lcov.string(); } - notes.push_back( - "--coverage: " + std::to_string(llvm ? llvmProfiles : gcovProfiles) - + " test profile(s); what each test reached is in " + json.string() - + ", the run combined in " + lcov.string()); - for (const auto &failure : failures) + else if (jsonWritten) { - notes.push_back("--coverage: " + failure); + written = "; what each test reached is in " + json.string(); } - return notes; + else if (lcovWritten) + { + written = "; the run combined is in " + lcov.string(); + } + reports.notes.push_back("--coverage: " + std::to_string(profiled.size()) + + " test profile(s)" + written); + for (auto &failure : reports.failures) + { + failure = "--coverage: " + failure; + } + return reports; } } // namespace Progmasoft::Catch3::Runner // Copyright (c) 2026 Progmasoft. @@ -20349,13 +20459,25 @@ namespace Progmasoft::Catch3 run.tests.push_back(std::move(kept)); } run.notes = std::move(outcome.notes); + bool coverageFailed = false; if (!options.coverageDirectory.empty()) { - for (auto ¬e : Runner::WriteCoverageReports( - options.coverageDirectory, run, program.string())) + auto coverage + = Runner::WriteCoverageReports(options.coverageDirectory, + run, + program.string(), + run.workers); + for (auto ¬e : coverage.notes) { run.notes.push_back(std::move(note)); } + // Like a report that could not be written, a coverage + // failure makes the run fail when its tests passed. + coverageFailed = !coverage.failures.empty(); + for (auto &failure : coverage.failures) + { + run.notes.push_back(std::move(failure)); + } } const bool anyFailure @@ -20377,7 +20499,8 @@ namespace Progmasoft::Catch3 Runner::Analyze(run, analysis); console.RunEnded(run); - const bool reportMissing = !WriteReports(options, run); + const bool reportMissing + = !WriteReports(options, run) || coverageFailed; if (options.historyEnabled) { SaveHistory(historyFile, diff --git a/src/Progmasoft/Runner/CoverageReport.cpp b/src/Progmasoft/Runner/CoverageReport.cpp index 68d6e5ee..fb2760c3 100644 --- a/src/Progmasoft/Runner/CoverageReport.cpp +++ b/src/Progmasoft/Runner/CoverageReport.cpp @@ -1,10 +1,13 @@ // Copyright (c) 2026 Progmasoft. // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 #include <algorithm> +#include <atomic> #include <cstdlib> +#include <initializer_list> #include <optional> #include <string_view> #include <system_error> +#include <thread> #include "CoverageReport.hpp" #include "Files.hpp" @@ -185,22 +188,6 @@ namespace Progmasoft::Catch3::Runner return files; } - /// The final attempt of a test that left a profile. - const Attempt * - ProfiledAttempt(const TestResult &test) - { - for (auto attempt = test.attempts.rbegin(); - attempt != test.attempts.rend(); - ++attempt) - { - if (!attempt->coverageProfile.empty()) - { - return &*attempt; - } - } - return nullptr; - } - /// The .gcda files gcov wrote below `prefix`, each with the path the /// object was built at, where its .gcno is. std::vector<std::pair<std::filesystem::path, std::filesystem::path>> @@ -254,7 +241,11 @@ namespace Progmasoft::Catch3::Runner } auto json = RunTool( gcov, - { "--json-format", "--stdout", gcda.filename().string() }, + // Without --branch-probabilities the JSON has no branches. + { "--json-format", + "--branch-probabilities", + "--stdout", + gcda.filename().string() }, gcda.parent_path()); if (!json) { @@ -401,6 +392,17 @@ namespace Progmasoft::Catch3::Runner + directory.string() + " could not be removed: " + error.message() }; } + for (const char *report : + { "coverage.json", "coverage.lcov", "coverage.profdata" }) + { + std::filesystem::remove(directory / report, error); + if (error) + { + return Error{ "the report " + (directory / report).string() + + " of an earlier run could not be removed: " + + error.message() }; + } + } std::filesystem::create_directories(directory / "profiles", error); if (error) { @@ -410,142 +412,215 @@ namespace Progmasoft::Catch3::Runner return Status::Ok(); } - std::vector<std::string> + const Attempt * + ProfiledAttempt(const TestResult &test, const std::string &suffix) + { + for (auto attempt = test.attempts.rbegin(); + attempt != test.attempts.rend(); + ++attempt) + { + std::error_code error; + if (!attempt->coverageProfile.empty() + && std::filesystem::exists(attempt->coverageProfile + suffix, + error)) + { + return &*attempt; + } + } + return nullptr; + } + + namespace + { + /// The coverage tools of one profile format. + struct CoverageTools final + { + bool llvm = false; + std::optional<Tool> profdata; + std::optional<Tool> cov; + std::optional<Tool> gcov; + }; + + /// What the profile of one test gave. + struct TestCoverage final + { + const TestResult *test = nullptr; + std::string profile; + std::optional<CoverageMap> reached; + std::string failure; + }; + + TestCoverage + ReadTestCoverage(const TestResult &test, + const Attempt &attempt, + const CoverageTools &tools, + const std::string &program) + { + TestCoverage result; + result.test = &test; + const std::string base = attempt.coverageProfile; + if (tools.llvm) + { + result.profile = base + ".profraw"; + auto merged = RunTool(*tools.profdata, + { "merge", + "-sparse", + result.profile, + "-o", + base + ".profdata" }); + if (!merged) + { + result.failure = merged.GetError().Message(); + return result; + } + auto lcov = RunTool(*tools.cov, + { "export", + "-format=lcov", + "-instr-profile=" + base + ".profdata", + program }); + if (!lcov) + { + result.failure = lcov.GetError().Message(); + return result; + } + result.reached = ParseLcov(*lcov); + return result; + } + result.profile = base + ".gcov"; + auto reached = GcovCoverage(*tools.gcov, result.profile); + if (!reached) + { + result.failure = reached.GetError().Message(); + return result; + } + result.reached = std::move(*reached); + return result; + } + + /// Reads the profile of each of `tests`, up to `jobs` at a time; + /// the results keep the order of `tests`. + std::vector<TestCoverage> + ReadAll( + const std::vector<std::pair<const TestResult *, const Attempt *>> + &tests, + const CoverageTools &tools, + const std::string &program, + std::size_t jobs) + { + std::vector<TestCoverage> results(tests.size()); + std::atomic<std::size_t> next{ 0 }; + const auto work = [&] { + for (std::size_t index = next++; index < tests.size(); + index = next++) + { + results[index] = ReadTestCoverage(*tests[index].first, + *tests[index].second, + tools, + program); + } + }; + const std::size_t threads + = std::min(std::max<std::size_t>(jobs, 1), tests.size()); + std::vector<std::thread> pool; + for (std::size_t thread = 1; thread < threads; ++thread) + { + pool.emplace_back(work); + } + work(); + for (auto &thread : pool) + { + thread.join(); + } + return results; + } + } // namespace + + CoverageReports WriteCoverageReports(const std::filesystem::path &directory, const RunResult &run, - const std::string &program) + const std::string &program, + std::size_t jobs) { - std::vector<std::string> notes; - std::size_t llvmProfiles = 0; - std::size_t gcovProfiles = 0; + CoverageReports reports; + std::vector<std::pair<const TestResult *, const Attempt *>> llvmTests; + std::vector<std::pair<const TestResult *, const Attempt *>> gcovTests; for (const auto &test : run.tests) { - if (const Attempt *attempt = ProfiledAttempt(test)) + if (const Attempt *attempt = ProfiledAttempt(test, ".profraw")) { - std::error_code error; - llvmProfiles += std::filesystem::exists(attempt->coverageProfile - + ".profraw", - error) - ? 1 - : 0; - gcovProfiles += std::filesystem::exists(attempt->coverageProfile - + ".gcov", - error) - ? 1 - : 0; + llvmTests.emplace_back(&test, attempt); + } + if (const Attempt *attempt = ProfiledAttempt(test, ".gcov")) + { + gcovTests.emplace_back(&test, attempt); } } - if (llvmProfiles == 0 && gcovProfiles == 0) + if (llvmTests.empty() && gcovTests.empty()) { - notes.emplace_back( + reports.notes.emplace_back( "--coverage: the tests wrote no coverage; build them with " "-fprofile-instr-generate -fcoverage-mapping (Clang) or " "--coverage (GCC)"); - return notes; + return reports; } - const bool llvm = llvmProfiles >= gcovProfiles; - std::optional<Tool> profdata; - std::optional<Tool> cov; - std::optional<Tool> gcov; - if (llvm) + CoverageTools tools; + tools.llvm = llvmTests.size() >= gcovTests.size(); + const auto &profiled = tools.llvm ? llvmTests : gcovTests; + if (tools.llvm) { - profdata = FindTool("CATCH3_LLVM_PROFDATA", "llvm-profdata"); - cov = FindTool("CATCH3_LLVM_COV", "llvm-cov"); + tools.profdata = FindTool("CATCH3_LLVM_PROFDATA", "llvm-profdata"); + tools.cov = FindTool("CATCH3_LLVM_COV", "llvm-cov"); } else { - gcov = FindTool("CATCH3_GCOV", "gcov"); + tools.gcov = FindTool("CATCH3_GCOV", "gcov"); } - if (llvm ? (!profdata || !cov) : !gcov) + if (tools.llvm ? (!tools.profdata || !tools.cov) : !tools.gcov) { - notes.push_back( - "--coverage: " - + std::to_string(llvm ? llvmProfiles : gcovProfiles) + reports.notes.push_back( + "--coverage: " + std::to_string(profiled.size()) + " profile(s) in " + (directory / "profiles").string() + "; " - + (llvm ? "llvm-profdata and llvm-cov were" : "gcov was") + + (tools.llvm ? "llvm-profdata and llvm-cov were" : "gcov was") + " not found, so there is no map of what each test reached"); - return notes; + return reports; } CoverageMap combined; std::vector<std::string> raw; JsonValue tests = JsonValue::Array(); - std::vector<std::string> failures; - for (const auto &test : run.tests) + for (auto &read : ReadAll(profiled, tools, program, jobs)) { - const Attempt *attempt = ProfiledAttempt(test); - if (attempt == nullptr) + if (!read.reached) { + reports.failures.push_back(read.test->name + ": " + + read.failure); continue; } - const std::string base = attempt->coverageProfile; - Expected<CoverageMap> reached = CoverageMap{}; - std::string profile; - if (llvm) + if (tools.llvm) { - profile = base + ".profraw"; - std::error_code error; - if (!std::filesystem::exists(profile, error)) - { - continue; - } - raw.push_back(profile); - auto merged = RunTool( - *profdata, - { "merge", "-sparse", profile, "-o", base + ".profdata" }); - if (!merged) - { - failures.push_back(test.name + ": " - + merged.GetError().Message()); - continue; - } - auto lcov = RunTool(*cov, - { "export", - "-format=lcov", - "-instr-profile=" + base + ".profdata", - program }); - if (!lcov) - { - failures.push_back(test.name + ": " - + lcov.GetError().Message()); - continue; - } - reached = ParseLcov(*lcov); + raw.push_back(read.profile); } else { - profile = base + ".gcov"; - std::error_code error; - if (!std::filesystem::exists(profile, error)) - { - continue; - } - reached = GcovCoverage(*gcov, profile); - if (!reached) - { - failures.push_back(test.name + ": " - + reached.GetError().Message()); - continue; - } - Add(combined, *reached); + Add(combined, *read.reached); } - tests.Append( - JsonValue::Object({ { "name", test.name }, - { "outcome", OutcomeName(test.outcome) }, - { "profile", profile }, - { "files", ReachedJson(*reached) } })); + tests.Append(JsonValue::Object( + { { "name", read.test->name }, + { "outcome", OutcomeName(read.test->outcome) }, + { "profile", read.profile }, + { "files", ReachedJson(*read.reached) } })); } std::string lcovText; - if (llvm && !raw.empty()) + if (tools.llvm && !raw.empty()) { std::vector<std::string> arguments{ "merge", "-sparse" }; arguments.insert(arguments.end(), raw.begin(), raw.end()); const std::string all = (directory / "coverage.profdata").string(); arguments.emplace_back("-o"); arguments.push_back(all); - auto merged = RunTool(*profdata, arguments); - auto exported = merged ? RunTool(*cov, + auto merged = RunTool(*tools.profdata, arguments); + auto exported = merged ? RunTool(*tools.cov, { "export", "-format=lcov", "-instr-profile=" + all, @@ -557,8 +632,8 @@ namespace Progmasoft::Catch3::Runner } else { - failures.push_back("the combined report: " - + exported.GetError().Message()); + reports.failures.push_back("the combined report: " + + exported.GetError().Message()); } } else @@ -568,28 +643,42 @@ namespace Progmasoft::Catch3::Runner JsonValue report = JsonValue::Object({ { "version", 1 }, - { "format", llvm ? "llvm" : "gcov" }, + { "format", tools.llvm ? "llvm" : "gcov" }, { "binary", program }, { "tests", std::move(tests) } }); const auto json = directory / "coverage.json"; const auto lcov = directory / "coverage.lcov"; - if (auto status = WriteFileAtomically(json, report.ToPrettyString()); - !status) + const auto jsonWritten + = WriteFileAtomically(json, report.ToPrettyString()); + if (!jsonWritten) + { + reports.failures.push_back(jsonWritten.GetError().Message()); + } + const auto lcovWritten = WriteFileAtomically(lcov, lcovText); + if (!lcovWritten) + { + reports.failures.push_back(lcovWritten.GetError().Message()); + } + std::string written; + if (jsonWritten && lcovWritten) + { + written = "; what each test reached is in " + json.string() + + ", the run combined in " + lcov.string(); + } + else if (jsonWritten) { - failures.push_back(status.GetError().Message()); + written = "; what each test reached is in " + json.string(); } - if (auto status = WriteFileAtomically(lcov, lcovText); !status) + else if (lcovWritten) { - failures.push_back(status.GetError().Message()); + written = "; the run combined is in " + lcov.string(); } - notes.push_back( - "--coverage: " + std::to_string(llvm ? llvmProfiles : gcovProfiles) - + " test profile(s); what each test reached is in " + json.string() - + ", the run combined in " + lcov.string()); - for (const auto &failure : failures) + reports.notes.push_back("--coverage: " + std::to_string(profiled.size()) + + " test profile(s)" + written); + for (auto &failure : reports.failures) { - notes.push_back("--coverage: " + failure); + failure = "--coverage: " + failure; } - return notes; + return reports; } } // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/CoverageReport.hpp b/src/Progmasoft/Runner/CoverageReport.hpp index 35f9dc75..35c3b1e5 100644 --- a/src/Progmasoft/Runner/CoverageReport.hpp +++ b/src/Progmasoft/Runner/CoverageReport.hpp @@ -38,21 +38,42 @@ namespace Progmasoft::Catch3::Runner [[nodiscard]] std::string ToLcov(const CoverageMap &map); - /// Prepares the directory of a coverage run: the profiles of an earlier - /// run below it are removed, because gcov adds to the .gcda files it - /// finds instead of replacing them. + /// Prepares the directory of a coverage run: the profiles and reports of + /// an earlier run below it are removed, because gcov adds to the .gcda + /// files it finds instead of replacing them, and because a run that + /// writes no reports must not leave the earlier ones to be read as its + /// own. Status PrepareCoverageDirectory(const std::filesystem::path &directory); + /// The last attempt of `test` whose profile, its base name followed by + /// `suffix` (`.profraw` or `.gcov`), exists: an attempt that crashed or + /// timed out before writing one is passed over for an earlier one. + [[nodiscard]] const Attempt * + ProfiledAttempt(const TestResult &test, const std::string &suffix); + + /// What writing the coverage reports gave. + struct CoverageReports final + { + /// What the run should be told: how many profiles there were and + /// where the reports are, or why there are none. + std::vector<std::string> notes; + /// What failed: a profile that could not be read, a report that + /// could not be written. A run with failures is missing a report. + std::vector<std::string> failures; + }; + /// Turns the profiles a coverage run left in `directory` into /// `directory/coverage.json`, the lines and branches each test reached, /// and `directory/coverage.lcov`, the run combined. LLVM profiles are /// read with `llvm-profdata` and `llvm-cov`, gcov data with `gcov`, /// found on PATH or named by CATCH3_LLVM_PROFDATA, CATCH3_LLVM_COV and - /// CATCH3_GCOV. Returns what the run should be told: how many profiles - /// there were and where the reports are, or why there are none. - [[nodiscard]] std::vector<std::string> + /// CATCH3_GCOV; the profiles of the tests are read up to `jobs` at a + /// time. A run whose tests wrote no profiles, or whose tools are + /// missing, gets a note and no reports, not a failure. + [[nodiscard]] CoverageReports WriteCoverageReports(const std::filesystem::path &directory, const RunResult &run, - const std::string &program); + const std::string &program, + std::size_t jobs); } // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/Runner.cpp b/src/Progmasoft/Runner/Runner.cpp index 61f8f2b7..5667f65e 100644 --- a/src/Progmasoft/Runner/Runner.cpp +++ b/src/Progmasoft/Runner/Runner.cpp @@ -995,13 +995,25 @@ namespace Progmasoft::Catch3 run.tests.push_back(std::move(kept)); } run.notes = std::move(outcome.notes); + bool coverageFailed = false; if (!options.coverageDirectory.empty()) { - for (auto ¬e : Runner::WriteCoverageReports( - options.coverageDirectory, run, program.string())) + auto coverage + = Runner::WriteCoverageReports(options.coverageDirectory, + run, + program.string(), + run.workers); + for (auto ¬e : coverage.notes) { run.notes.push_back(std::move(note)); } + // Like a report that could not be written, a coverage + // failure makes the run fail when its tests passed. + coverageFailed = !coverage.failures.empty(); + for (auto &failure : coverage.failures) + { + run.notes.push_back(std::move(failure)); + } } const bool anyFailure @@ -1023,7 +1035,8 @@ namespace Progmasoft::Catch3 Runner::Analyze(run, analysis); console.RunEnded(run); - const bool reportMissing = !WriteReports(options, run); + const bool reportMissing + = !WriteReports(options, run) || coverageFailed; if (options.historyEnabled) { SaveHistory(historyFile, diff --git a/tests/Progmasoft/CoverageFixture.cpp b/tests/Progmasoft/CoverageFixture.cpp index a046a296..e728bbca 100644 --- a/tests/Progmasoft/CoverageFixture.cpp +++ b/tests/Progmasoft/CoverageFixture.cpp @@ -20,11 +20,22 @@ namespace { return value - 1; // coverage: subtract-one } + + int + NonNegative(int value) + { + if (value < 0) // coverage: branch + { + return 0; + } + return value; + } } // namespace TEST_CASE("Adds one", "[coverage]") { CHECK(AddOne(1) == 2); + CHECK(NonNegative(AddOne(1)) == 2); } TEST_CASE("Subtracts one", "[coverage]") diff --git a/tests/Progmasoft/RunnerCoverage.tests.cpp b/tests/Progmasoft/RunnerCoverage.tests.cpp index 9dcca663..b3e10489 100644 --- a/tests/Progmasoft/RunnerCoverage.tests.cpp +++ b/tests/Progmasoft/RunnerCoverage.tests.cpp @@ -157,6 +157,58 @@ TEST_CASE("gcov JSON adds up across data files", CHECK(map.size() == 1); } +TEST_CASE("A coverage run starts without the reports of an earlier one", + "[progmasoft][runner][coverage]") +{ + const Scratch scratch; + const auto directory = scratch.Path() / "coverage"; + std::filesystem::create_directories(directory / "profiles"); + for (const char *name : + { "coverage.json", "coverage.lcov", "coverage.profdata" }) + { + REQUIRE(Runner::WriteFileAtomically(directory / name, "earlier")); + } + REQUIRE(Runner::WriteFileAtomically(directory / "profiles" / "t.profraw", + "earlier")); + + REQUIRE(Runner::PrepareCoverageDirectory(directory)); + CHECK_FALSE(std::filesystem::exists(directory / "coverage.json")); + CHECK_FALSE(std::filesystem::exists(directory / "coverage.lcov")); + CHECK_FALSE(std::filesystem::exists(directory / "coverage.profdata")); + CHECK_FALSE(std::filesystem::exists(directory / "profiles" / "t.profraw")); + CHECK(std::filesystem::is_directory(directory / "profiles")); + + // With no profiles there is a note, no report and no failure. + Progmasoft::Catch3::RunResult run; + run.tests.emplace_back(); + run.tests.back().name = "no profile"; + const auto reports + = Runner::WriteCoverageReports(directory, run, "binary", 2); + CHECK(reports.failures.empty()); + REQUIRE(reports.notes.size() == 1); + CHECK_THAT(reports.notes.front(), ContainsSubstring("wrote no coverage")); + CHECK_FALSE(std::filesystem::exists(directory / "coverage.json")); +} + +TEST_CASE("An attempt that left no profile gives way to an earlier one", + "[progmasoft][runner][coverage]") +{ + const Scratch scratch; + const auto base = (scratch.Path() / "test").string(); + Progmasoft::Catch3::TestResult test; + test.attempts.resize(3); + test.attempts[0].coverageProfile = base + "-1"; + test.attempts[1].coverageProfile = base + "-2"; + test.attempts[2].coverageProfile = base + "-3"; // crashed: no profile + REQUIRE(Runner::WriteFileAtomically(base + "-1.profraw", "1")); + REQUIRE(Runner::WriteFileAtomically(base + "-2.profraw", "2")); + + const auto *attempt = Runner::ProfiledAttempt(test, ".profraw"); + REQUIRE(attempt != nullptr); + CHECK(attempt == &test.attempts[1]); + CHECK(Runner::ProfiledAttempt(test, ".gcov") == nullptr); +} + TEST_CASE("Each test of a coverage run gets its own profile and map", "[progmasoft][runner][coverage]") { @@ -223,4 +275,19 @@ TEST_CASE("Each test of a coverage run gets its own profile and map", REQUIRE(file != combined.end()); CHECK(file->second.lines.at(addLine) > 0); CHECK(file->second.lines.at(subtractLine) > 0); + + // Both formats carry branches: the condition of the fixture went one + // way only, so one of its branches was taken and one was not. + const auto branchLine = MarkedLine(source, "branch"); + std::size_t taken = 0; + std::size_t notTaken = 0; + for (const auto &[position, count] : file->second.branches) + { + if (position.first == branchLine) + { + (count > 0 ? taken : notTaken) += 1; + } + } + CHECK(taken > 0); + CHECK(notTaken > 0); }