catch3
Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.
Clonegit clone https://git.progmasoft.com/catch3
Import the results of other test frameworks
- Commit
1db203d56b7c71db3d80b477da1ad49e9ae46468- Author
- Leitwolf11 ·
- Parents
1559307a6ac7
Changes
docs/progmasoft-roadmap.md | 2 + docs/progmasoft-runner.md | 64 + docs/release-notes.md | 11 + extras/catch_amalgamated.cpp | 2126 +++++++++++++++++++++++++--- extras/catch_amalgamated.hpp | 2 +- extras/catch_amalgamated_third_party.hpp | 6 +- premake5.lua | 1 + src/CMakeLists.txt | 5 + src/Progmasoft/Catch3/meson.build | 4 + src/Progmasoft/PugiXml.inl | 4 +- src/Progmasoft/Runner/Analysis.cpp | 5 + src/Progmasoft/Runner/Console.cpp | 15 + src/Progmasoft/Runner/Console.hpp | 4 + src/Progmasoft/Runner/Options.cpp | 52 + src/Progmasoft/Runner/Options.hpp | 25 + src/Progmasoft/Runner/ProjectConfig.cpp | 4 + src/Progmasoft/Runner/ResultImport.cpp | 305 ++++ src/Progmasoft/Runner/ResultImport.hpp | 89 ++ src/Progmasoft/Runner/ResultImportJson.cpp | 279 ++++ src/Progmasoft/Runner/ResultImportTap.cpp | 338 +++++ src/Progmasoft/Runner/ResultImportXml.cpp | 507 +++++++ src/Progmasoft/Runner/Runner.cpp | 318 +++-- tests/CMakeLists.txt | 1 + tests/Progmasoft/BUILD.bazel | 1 + tests/Progmasoft/RunnerImport.tests.cpp | 596 ++++++++ tests/Progmasoft/RunnerOptions.tests.cpp | 4 +- tests/meson.build | 1 + tools/scripts/generateAmalgamatedFiles.py | 2 +- xmake.lua | 1 + 29 files changed, 4476 insertions(+), 296 deletions(-) diff --git a/docs/progmasoft-roadmap.md b/docs/progmasoft-roadmap.md index f376e9a0..5ab04264 100644 --- a/docs/progmasoft-roadmap.md +++ b/docs/progmasoft-roadmap.md @@ -48,6 +48,8 @@ Catch3 runner replaced Catch2's in every test binary: - every failure gets a category, a signature and an analysis; - reports are console, JUnit XML with flaky and rerun elements, JSON and a trace; the discovery listing has a fixed, versioned format; +- reports of other frameworks are imported, so their tests get the same + history, flaky detection, quarantine and analytics; - every test gets a temporary directory of its own; - its command line is parsed by a hand-written, typed parser, like the other Progmasoft tools. diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md index 300c541e..486c719f 100644 --- a/docs/progmasoft-runner.md +++ b/docs/progmasoft-runner.md @@ -5,6 +5,7 @@ [How a run works](#how-a-run-works)<br> [Retries, flaky tests and order dependence](#retries-flaky-tests-and-order-dependence)<br> [History, quarantine and analytics](#history-quarantine-and-analytics)<br> +[Importing the results of other frameworks](#importing-the-results-of-other-frameworks)<br> [Choosing and ordering tests](#choosing-and-ordering-tests)<br> [Tags the runner reads](#tags-the-runner-reads)<br> [Time limits](#time-limits)<br> @@ -133,6 +134,67 @@ Under `bazel test` there is no history unless `CATCH3_HISTORY` names a directory that outlives the sandbox. +## Importing the results of other frameworks + +`--import=FILE` reads the report of another test framework instead of running +tests, so that tests Catch3 does not run get the same history, flaky detection, +quarantine and analytics. Any Catch3 test binary can do it, and the option can +be repeated: + +```sh +./tests --import=results/pytest.xml --import=results/vitest.ctrf.json \ + --import-name=web --junit=merged.xml +``` + +| `--import-format=` | Reads | +| --- | --- | +| `auto` | whichever the contents show (the default) | +| `catch3` | a `--json` report of Catch3, or [`catch3-results`](#the-catch3-results-format) | +| `junit` | JUnit XML, with Surefire's reruns; Gradle, pytest, Jest, GoogleTest and most others write it | +| `xunit` | xUnit.net v2 XML | +| `trx` | Visual Studio's TRX, with data-driven rows | +| `tap` | TAP 12, 13 and 14, with subtests and YAML diagnostics | +| `ctrf` | [CTRF](https://ctrf.io) JSON, with retries | + +The results go into a history of their own, named by `--import-name` and +`imported` by default; `--analytics --import-name=NAME` reads it. The console, +`--junit`, `--json`, `--trace`, `--quarantine`, `--history` and the exit code +work as they do for a run. Reports that were read count as one run. + +A test is identified by its class and its name. Results with the same +identifier, in one report or in several, are attempts at one test: a test that +both failed and passed is flaky, passes and is remembered as flaky, which is how +a report of a rerun is imported. TAP has no retries, so a name that comes back +in one stream is another test and gets a number. + +A verdict that changes on one revision also counts as flaky. +`--import-revision=REVISION` names what the reports tested; it defaults to the +Git commit of the working directory. + +Nothing is run, so an imported failure has no replay command, and only what the +report says is known: it has no order, seed or sections. A TAP stream that bails +out or reports fewer tests than it planned adds a failed test called +`the TAP stream`, so that it does not read as a run that passed. + +### The catch3-results format + +For a framework without a report of these kinds, `catch3-results` is the +smallest thing to write: JSON Lines with one test per line, after a line that +names the format. + +```json +{"format": "catch3-results", "version": 1} +{"name": "parses a date", "className": "Dates", "outcome": "passed", "durationMicroseconds": 1200} +{"name": "rounds down", "outcome": "failed", "message": "1.5 != 2", "details": "at Dates.rounds:12"} +``` + +`name` or `id`, and `outcome`, are required; `outcome` is a name from the JSON +report, such as `passed`, `failed`, `skipped`, `crashed` or `timed-out`. The other +members are `id`, `className`, `file`, `line`, `tags`, `category`, `message`, +`details`, `standardOutput` and `standardError`. A line can give `attempts` +instead, as the JSON report does. + + ## Choosing and ordering tests Test specs work as in Catch2: names with `*` as a wildcard, `[tags]`, `~` @@ -265,6 +327,8 @@ version. `quarantined`, `resources`, `dependsOn`, `timeoutSeconds` and `retries`. CMake's `catch3_discover_tests` reads it. * **Run report**, `--json`: `"format": "catch3-run"`, version 1. +* **Imported results**, `--import`: `"format": "catch3-results"`, version 1; + see [its format](#the-catch3-results-format). * **Analytics**, `--analytics=json`: `"format": "catch3-analytics"`, version 1. * **History**: `"format": "catch3-history"`, version 1. It is the runner's diff --git a/docs/release-notes.md b/docs/release-notes.md index f44d1369..e1567d97 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -7,11 +7,22 @@ based on Catch2 3.16.1; Catch2's own history is in [its release notes](https://github.com/catchorg/Catch2/blob/devel/docs/release-notes.md). **Contents**<br> +[Unreleased](#unreleased)<br> [0.3.0](#030)<br> [0.2.0](#020)<br> [0.1.0](#010)<br> +## Unreleased + +### The Catch3 runner + +* `--import=FILE` reads the reports of other test frameworks instead of running + tests: JUnit, xUnit, TRX, TAP, CTRF, Catch3's own, and a small JSON Lines + format for anything else. Their tests get the history, flaky detection, + quarantine, analytics and reports of a run. + + ## 0.3.0 Released 2026-11-01. diff --git a/extras/catch_amalgamated.cpp b/extras/catch_amalgamated.cpp index 7556181e..caee2a35 100644 --- a/extras/catch_amalgamated.cpp +++ b/extras/catch_amalgamated.cpp @@ -7,7 +7,7 @@ // SPDX-License-Identifier: BSL-1.0 // Catch3 v0.3.0, on the Catch2 v3.16.1 engine -// Generated: 2026-11-01 00:25:29.768099 +// Generated: 2026-11-01 14:22:22.249604 // ---------------------------------------------------------- // This file is an amalgamation of multiple different files. // You probably shouldn't edit it directly. @@ -965,6 +965,8 @@ namespace Progmasoft::Catch3::Runner kListTests, kListTags, kAnalytics, + /// Read the reports of other test frameworks. + kImport, kHelp, kVersion, /// Internal: run tests on behalf of a runner, over the protocol. @@ -1022,6 +1024,20 @@ namespace Progmasoft::Catch3::Runner kOff, }; + /// The reports `--import` reads. + enum class ImportFormat : std::uint8_t + { + /// Whichever the contents show. + kAuto, + /// Catch3's own: `--json` reports and catch3-results JSON Lines. + kCatch3, + kJUnit, + kXUnit, + kTrx, + kTap, + kCtrf, + }; + enum class ColorMode : std::uint8_t { kAuto, @@ -1101,6 +1117,15 @@ namespace Progmasoft::Catch3::Runner bool replayFailures = false; bool analyticsJson = false; + // Importing the reports of other frameworks. + std::vector<std::string> importFiles; + ImportFormat importFormat = ImportFormat::kAuto; + /// The history they go to; "imported" when empty. + std::string importName; + /// What was tested, so that a verdict that changes on one revision + /// counts as flaky; the Git commit here when empty. + std::string importRevision; + // Output. Verbosity verbosity = Verbosity::kNormal; ColorMode color = ColorMode::kAuto; @@ -1268,6 +1293,10 @@ namespace Progmasoft::Catch3::Runner std::uint32_t seed, const std::vector<std::string> ¬es); + /// The start of `--import`: results read from `files` reports. + void + ImportStarting(std::size_t tests, std::size_t files); + void TestFinished(const TestResult &test); @@ -1856,6 +1885,90 @@ namespace Progmasoft::Catch3::Runner // Copyright (c) 2026 Progmasoft. // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 +#include <cstdint> +#include <optional> +#include <string> +#include <string_view> +#include <vector> + + +namespace Progmasoft::Catch3::Runner +{ + /// The format of a report, from its contents; nothing when none fits. + [[nodiscard]] std::optional<ImportFormat> + DetectImportFormat(std::string_view text); + + /// The results a report holds, in its order. A test the report names + /// twice comes back twice; `MergeImported` makes one test of them. + [[nodiscard]] Expected<std::vector<TestResult>> + ParseImport(std::string_view text, ImportFormat format); + + /// One test per identifier: results that share one are attempts at that + /// test, in the order given, and the verdict is worked out over them. A + /// test that both failed and passed is flaky, as it is in a run of the + /// runner. + [[nodiscard]] std::vector<TestResult> + MergeImported(std::vector<TestResult> results); + + /// A result without attempts yet. + [[nodiscard]] TestResult + ImportedTest(std::string name, std::string className); + + /// One attempt as a report describes it. A failure gets an assertion + /// record with `message` and `details`, so the analysis treats it as it + /// treats a failure of Catch3's own; `category` defaults by outcome. + [[nodiscard]] Attempt + ImportedAttempt(Outcome outcome, + std::int64_t durationMicroseconds, + std::string message = {}, + std::string details = {}, + FailureCategory category = FailureCategory::kNone); + + /// Seconds as a report gives them; what makes no sense as a duration + /// counts as zero. + [[nodiscard]] std::int64_t + ImportedMicroseconds(double seconds); + + [[nodiscard]] std::string_view + ImportTrim(std::string_view text); + + [[nodiscard]] bool + ImportStartsWith(std::string_view text, std::string_view prefix); + + /// The lines of a text, without their line endings. + [[nodiscard]] std::vector<std::string_view> + ImportLines(std::string_view text); + + /// A format of Catch3's own or CTRF, from a parsed document. + [[nodiscard]] std::optional<ImportFormat> + ImportJsonFormat(const JsonValue &value); + + /// Whether a line reports a test in TAP. + [[nodiscard]] bool + IsTapTestLine(std::string_view line); + + /// A `--json` report or catch3-results lines. + [[nodiscard]] Expected<std::vector<TestResult>> + ParseCatch3Report(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseCtrf(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseTap(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseJUnit(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseXUnit(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseTrx(std::string_view text); +} // namespace Progmasoft::Catch3::Runner +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + namespace Progmasoft::Catch3::Runner @@ -9702,6 +9815,11 @@ namespace Progmasoft::Catch3::Runner test.analysis.insert( test.analysis.end(), findings.begin(), findings.end()); + // An imported failure ran somewhere else. + if (inputs.program.empty()) + { + continue; + } std::string replay = QuoteArgument(inputs.program) + " " + QuoteArgument(QuoteTestSpec(test.name)) + " --seed=" + std::to_string(failure->seed); @@ -10201,6 +10319,21 @@ namespace Progmasoft::Catch3::Runner std::fflush(output_); } + void + ConsoleReport::ImportStarting(std::size_t tests, std::size_t files) + { + total_ = tests; + if (options_.verbosity == Verbosity::kQuiet) + { + return; + } + Write(output_, + Paint("Catch3", kBold) + ": " + std::to_string(tests) + + " test(s) imported from " + std::to_string(files) + + " report(s)\n"); + std::fflush(output_); + } + void ConsoleReport::Retrying(const TestResult &test, const Attempt &attempt) { @@ -13375,6 +13508,58 @@ namespace Progmasoft::Catch3::Runner options.replayFailures = true; return Status::Ok(); }}, + {"import", + '\0', + Takes::kValue, + "FILE", + "Importing results", + "read a report of another framework instead of running tests " + "(repeatable)", + [](Options &options, const std::optional<std::string> &value) { + options.mode = Mode::kImport; + options.importFiles.push_back(*value); + return Status::Ok(); + }}, + {"import-format", + '\0', + Takes::kValue, + "FORMAT", + "Importing results", + "auto (default), catch3, junit, xunit, trx, tap or ctrf", + [](Options &options, const std::optional<std::string> &value) { + return ReadChoice<ImportFormat>( + value, + "import-format", + {{"auto", ImportFormat::kAuto}, + {"catch3", ImportFormat::kCatch3}, + {"junit", ImportFormat::kJUnit}, + {"xunit", ImportFormat::kXUnit}, + {"trx", ImportFormat::kTrx}, + {"tap", ImportFormat::kTap}, + {"ctrf", ImportFormat::kCtrf}}, + options.importFormat); + }}, + {"import-name", + '\0', + Takes::kValue, + "NAME", + "Importing results", + "the history imported results go to (default: imported); " + "--analytics reads it too", + [](Options &options, const std::optional<std::string> &value) { + options.importName = *value; + return Status::Ok(); + }}, + {"import-revision", + '\0', + Takes::kValue, + "REVISION", + "Importing results", + "what the reports tested (default: the Git commit here)", + [](Options &options, const std::optional<std::string> &value) { + options.importRevision = *value; + return Status::Ok(); + }}, {"analytics", '\0', Takes::kOptionalValue, @@ -15881,6 +16066,10 @@ namespace Progmasoft::Catch3::Runner "break", "replay", "analytics", + "import", + "import-format", + "import-name", + "import-revision", "verbose", "quiet", "premature-exit-guard-file", @@ -16379,142 +16568,1562 @@ namespace Progmasoft::Catch3::Runner } // namespace Progmasoft::Catch3::Runner // Copyright (c) 2026 Progmasoft. // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 +#include <algorithm> +#include <cctype> +#include <cmath> +#include <iterator> +#include <map> +#include <utility> namespace Progmasoft::Catch3::Runner { - namespace + std::string_view + ImportTrim(std::string_view text) { - JsonValue - Strings(const std::vector<std::string> &values) + const auto blank = [](char character) { + return std::isspace(static_cast<unsigned char>(character)) != 0; + }; + while (!text.empty() && blank(text.front())) { - JsonValue array = JsonValue::Array(); - for (const auto &value : values) - { - array.Append(value); - } - return array; + text.remove_prefix(1); + } + while (!text.empty() && blank(text.back())) + { + text.remove_suffix(1); } + return text; + } - std::vector<std::string> - StringsFrom(const JsonValue &value) + bool + ImportStartsWith(std::string_view text, std::string_view prefix) + { + return text.substr(0, prefix.size()) == prefix; + } + + std::vector<std::string_view> + ImportLines(std::string_view text) + { + std::vector<std::string_view> lines; + while (!text.empty()) { - std::vector<std::string> strings; - for (const auto &item : value.Items()) + const auto end = text.find('\n'); + auto line = text.substr(0, end); + if (!line.empty() && line.back() == '\r') { - strings.push_back(item.AsString()); + line.remove_suffix(1); } - return strings; + lines.push_back(line); + if (end == std::string_view::npos) + { + break; + } + text.remove_prefix(end + 1); } + return lines; + } - std::size_t - Size(const JsonValue &value) + namespace + { + namespace ImportText { - return static_cast<std::size_t>(value.AsUnsigned()); - } + /// Without a byte order mark and the white space before the + /// first character that counts. + std::string_view + Contents(std::string_view text) + { + if (ImportStartsWith(text, "\xEF\xBB\xBF")) + { + text.remove_prefix(3); + } + return ImportTrim(text); + } - /// Adds a member only when it says something, so reports and - /// protocol messages stay small. - void - SetText(JsonValue &object, std::string_view key, const std::string &text) - { - if (!text.empty()) + /// The name of the root element of an XML document. + std::string_view + RootElement(std::string_view text) { - object.Set(key, text); + while (!text.empty()) + { + const auto open = text.find('<'); + if (open == std::string_view::npos + || open + 1 >= text.size()) + { + return {}; + } + text.remove_prefix(open + 1); + if (text.front() == '?' || text.front() == '!') + { + const auto close = text.find('>'); + if (close == std::string_view::npos) + { + return {}; + } + text.remove_prefix(close + 1); + continue; + } + return text.substr(0, text.find_first_of(" \t\r\n/>")); + } + return {}; } - } + } // namespace ImportText } // namespace - JsonValue - ToJson(const AssertionRecord &assertion) + std::optional<ImportFormat> + DetectImportFormat(std::string_view text) { - JsonValue value = JsonValue::Object(); - value.Set("file", assertion.file); - value.Set("line", assertion.line); - SetText(value, "macro", assertion.macro); - SetText(value, "expression", assertion.expression); - SetText(value, "expansion", assertion.expansion); - SetText(value, "message", assertion.message); - if (!assertion.context.empty()) + const std::string_view contents = ImportText::Contents(text); + if (contents.empty()) { - value.Set("context", Strings(assertion.context)); + return std::nullopt; } - if (!assertion.sections.empty()) + if (contents.front() == '<') { - value.Set("sections", Strings(assertion.sections)); + const std::string_view root = ImportText::RootElement(contents); + if (root == "testsuites" || root == "testsuite") + { + return ImportFormat::kJUnit; + } + if (root == "assemblies" || root == "assembly") + { + return ImportFormat::kXUnit; + } + if (root == "TestRun") + { + return ImportFormat::kTrx; + } + return std::nullopt; } - value.Set("category", CategoryName(assertion.category)); - value.Set("passed", assertion.passed); - if (assertion.allowed) + if (contents.front() == '{') { - value.Set("allowed", true); + auto whole = ParseJson(contents); + if (!whole) + { + // JSON Lines: the first line says what the others are. + whole = ParseJson(contents.substr(0, contents.find('\n'))); + } + return whole ? ImportJsonFormat(*whole) : std::nullopt; } - return value; + for (const auto raw : ImportLines(contents)) + { + const std::string_view line = ImportTrim(raw); + if (line.empty() || line.front() == '#') + { + continue; + } + if (ImportStartsWith(line, "TAP version") + || ImportStartsWith(line, "1..") || IsTapTestLine(line)) + { + return ImportFormat::kTap; + } + break; + } + return std::nullopt; } - AssertionRecord - AssertionFromJson(const JsonValue &value) + Expected<std::vector<TestResult>> + ParseImport(std::string_view text, ImportFormat format) { - AssertionRecord assertion; - assertion.file = value["file"].AsString(); - assertion.line = Size(value["line"]); - assertion.macro = value["macro"].AsString(); - assertion.expression = value["expression"].AsString(); - assertion.expansion = value["expansion"].AsString(); - assertion.message = value["message"].AsString(); - assertion.context = StringsFrom(value["context"]); - assertion.sections = StringsFrom(value["sections"]); - assertion.category = CategoryFromName(value["category"].AsString()) - .value_or(FailureCategory::kAssertion); - assertion.passed = value["passed"].AsBoolean(); - assertion.allowed = value["allowed"].AsBoolean(); - return assertion; + const std::string_view contents = ImportText::Contents(text); + switch (format) + { + case ImportFormat::kAuto: + break; + case ImportFormat::kCatch3: + return ParseCatch3Report(contents); + case ImportFormat::kJUnit: + return ParseJUnit(contents); + case ImportFormat::kXUnit: + return ParseXUnit(contents); + case ImportFormat::kTrx: + return ParseTrx(contents); + case ImportFormat::kTap: + return ParseTap(contents); + case ImportFormat::kCtrf: + return ParseCtrf(contents); + } + const auto detected = DetectImportFormat(contents); + if (!detected) + { + return Error{ "its format is not one Catch3 knows; " + "--import-format names it" }; + } + return ParseImport(contents, *detected); } - JsonValue - ToJson(const BenchmarkRecord &benchmark) + std::vector<TestResult> + MergeImported(std::vector<TestResult> results) { - JsonValue value = JsonValue::Object(); - value.Set("name", benchmark.name); - value.Set("samples", benchmark.samples); - value.Set("iterations", benchmark.iterations); - value.Set("meanNanoseconds", benchmark.meanNanoseconds); - value.Set("lowMeanNanoseconds", benchmark.lowMeanNanoseconds); - value.Set("highMeanNanoseconds", benchmark.highMeanNanoseconds); - value.Set("standardDeviationNanoseconds", - benchmark.standardDeviationNanoseconds); - value.Set("outlierVariance", benchmark.outlierVariance); - return value; + std::vector<TestResult> tests; + std::map<std::string, std::size_t> positions; + for (auto &result : results) + { + if (result.id.empty()) + { + result.id = History::KeyOf(result); + } + const auto [position, added] + = positions.emplace(result.id, tests.size()); + if (added) + { + tests.push_back(std::move(result)); + continue; + } + TestResult &test = tests[position->second]; + test.flaky = test.flaky || result.flaky; + std::move(result.attempts.begin(), + result.attempts.end(), + std::back_inserter(test.attempts)); + } + for (auto &test : tests) + { + std::size_t passes = 0; + std::size_t failures = 0; + for (std::size_t index = 0; index < test.attempts.size(); ++index) + { + Attempt &attempt = test.attempts[index]; + attempt.number = index + 1; + passes += attempt.outcome == Outcome::kPassed ? 1 : 0; + failures += IsFailure(attempt.outcome) ? 1 : 0; + } + if (test.attempts.empty()) + { + continue; + } + test.outcome = test.attempts.back().outcome; + test.category = test.attempts.back().category; + if (passes > 0 && failures > 0) + { + test.outcome = Outcome::kPassed; + test.category = FailureCategory::kNone; + test.flaky = true; + test.analysis.push_back( + "flaky: failed " + std::to_string(failures) + " of " + + std::to_string(test.attempts.size()) + " attempts"); + } + } + return tests; } - BenchmarkRecord - BenchmarkFromJson(const JsonValue &value) + TestResult + ImportedTest(std::string name, std::string className) { - BenchmarkRecord benchmark; - benchmark.name = value["name"].AsString(); - benchmark.samples = Size(value["samples"]); - benchmark.iterations = Size(value["iterations"]); - benchmark.meanNanoseconds = value["meanNanoseconds"].AsNumber(); - benchmark.lowMeanNanoseconds = value["lowMeanNanoseconds"].AsNumber(); - benchmark.highMeanNanoseconds = value["highMeanNanoseconds"].AsNumber(); - benchmark.standardDeviationNanoseconds = - value["standardDeviationNanoseconds"].AsNumber(); - benchmark.outlierVariance = value["outlierVariance"].AsNumber(); - return benchmark; + TestResult test; + test.name = std::move(name); + test.className = std::move(className); + test.id = History::KeyOf(test); + return test; } - JsonValue - ToJson(const SectionRecord §ion) + Attempt + ImportedAttempt(Outcome outcome, + std::int64_t durationMicroseconds, + std::string message, + std::string details, + FailureCategory category) { - JsonValue value = JsonValue::Object(); - value.Set("name", section.name); - value.Set("file", section.file); - value.Set("line", section.line); - value.Set("depth", section.depth); - value.Set("startMicroseconds", section.startMicroseconds); - value.Set("durationMicroseconds", section.durationMicroseconds); - return value; - } - + Attempt attempt; + attempt.outcome = outcome; + attempt.durationMicroseconds + = std::max<std::int64_t>(durationMicroseconds, 0); + if (outcome == Outcome::kSkipped) + { + attempt.standardOutput = std::move(message); + return attempt; + } + if (outcome != Outcome::kFailedAsExpected && !IsFailure(outcome)) + { + return attempt; + } + if (category == FailureCategory::kNone) + { + category = outcome == Outcome::kCrashed ? FailureCategory::kCrash + : outcome == Outcome::kTimedOut + ? FailureCategory::kTimeout + : FailureCategory::kAssertion; + } + if (message.empty()) + { + message = details.substr(0, details.find('\n')); + } + attempt.category = category; + attempt.assertionsFailed = 1; + AssertionRecord assertion; + assertion.message = std::move(message); + assertion.expansion = std::move(details); + assertion.category = category; + assertion.allowed = outcome == Outcome::kFailedAsExpected; + attempt.assertions.push_back(std::move(assertion)); + return attempt; + } + + std::int64_t + ImportedMicroseconds(double seconds) + { + // About thirty years; a longer duration is not a duration. + constexpr double kLongest = 1e9; + if (!std::isfinite(seconds) || seconds <= 0.0 || seconds > kLongest) + { + return 0; + } + return static_cast<std::int64_t>(std::llround(seconds * 1e6)); + } +} // namespace Progmasoft::Catch3::Runner +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + +// The JSON reports: Catch3's own and CTRF. +#include <utility> + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + namespace ImportJson + { + /// A format of Catch3's own or CTRF, from the parsed document. + std::optional<ImportFormat> + FormatOf(const JsonValue &value) + { + if (value["format"].AsString() == "catch3-run" + || value["format"].AsString() == "catch3-results") + { + return ImportFormat::kCatch3; + } + if (value["results"]["tests"].IsArray()) + { + return ImportFormat::kCtrf; + } + return std::nullopt; + } + + /// What a test owes the verdict of an earlier run is worked out + /// again here, from this run's point of view. + void + Forget(TestResult &test) + { + test.quarantined = false; + test.history = FailureHistory::kUnknown; + test.selectionReason.clear(); + test.signature.clear(); + test.analysis.clear(); + test.replayCommand.clear(); + test.predictedDurationMicroseconds = 0; + test.predictedFailureProbability = 0.0; + } + + Expected<TestResult> + ReadLine(const JsonValue &value) + { + TestResult test = TestResultFromJson(value); + Forget(test); + if (test.name.empty()) + { + test.name = test.id; + } + if (test.name.empty()) + { + return Error{ "a test needs an 'id' or a 'name'" }; + } + const std::string &outcome = value["outcome"].AsString(); + if (!OutcomeFromName(outcome)) + { + return Error{ "'" + test.name + "' has the outcome '" + + outcome + "'" }; + } + if (test.attempts.empty()) + { + Attempt attempt = ImportedAttempt( + test.outcome, + value["durationMicroseconds"].AsInteger(), + value["message"].AsString(), + value["details"].AsString(), + test.category); + attempt.standardOutput = value["standardOutput"].AsString(); + attempt.standardError = value["standardError"].AsString(); + test.attempts.push_back(std::move(attempt)); + } + return test; + } + + Expected<std::vector<TestResult>> + ParseLines(std::string_view text) + { + std::vector<TestResult> tests; + std::size_t number = 0; + bool started = false; + for (const auto line : ImportLines(text)) + { + ++number; + if (ImportTrim(line).empty()) + { + continue; + } + const std::string where = "line " + std::to_string(number); + auto value = ParseJson(line); + if (!value) + { + return value.GetError().Context(where); + } + if (!started) + { + started = true; + if ((*value)["format"].AsString() != "catch3-results") + { + return Error{ "the first line must name the format " + "catch3-results" }; + } + if ((*value)["version"].AsInteger() + > kResultsFormatVersion) + { + return Error{ "the format is newer than this " + "version of Catch3 reads" }; + } + continue; + } + auto test = ReadLine(*value); + if (!test) + { + return test.GetError().Context(where); + } + tests.push_back(std::move(*test)); + } + return tests; + } + + Expected<std::vector<TestResult>> + ParseCatch3(std::string_view text) + { + auto whole = ParseJson(text); + if (!whole) + { + return ParseLines(text); + } + if ((*whole)["format"].AsString() != "catch3-run") + { + return ParseLines(text); + } + auto run = RunResultFromJson(*whole); + if (!run) + { + return run.GetError(); + } + for (auto &test : run->tests) + { + Forget(test); + } + return std::move(run->tests); + } + + std::string + TextOf(const JsonValue &value) + { + if (!value.IsArray()) + { + return value.AsString(); + } + std::string text; + for (const auto &item : value.Items()) + { + text += item.AsString() + "\n"; + } + return text; + } + + Outcome + CtrfOutcome(const std::string &status) + { + if (status == "passed") + { + return Outcome::kPassed; + } + if (status == "failed") + { + return Outcome::kFailed; + } + if (status == "skipped" || status == "pending") + { + return Outcome::kSkipped; + } + return Outcome::kNotRun; + } + + /// A test, or one retry of it: the two share these fields. + Attempt + CtrfAttempt(const JsonValue &value) + { + Attempt attempt = ImportedAttempt( + CtrfOutcome(value["status"].AsString()), + ImportedMicroseconds(value["duration"].AsNumber() / 1e3), + value["message"].AsString(), + value["trace"].AsString()); + attempt.standardOutput = TextOf(value["stdout"]); + attempt.standardError = TextOf(value["stderr"]); + return attempt; + } + + std::string + CtrfSuite(const JsonValue &suite) + { + if (!suite.IsArray()) + { + return suite.AsString(); + } + std::string text; + for (const auto &item : suite.Items()) + { + text += (text.empty() ? "" : " > ") + item.AsString(); + } + return text; + } + + Expected<std::vector<TestResult>> + ParseCtrf(std::string_view text) + { + auto document = ParseJson(text); + if (!document) + { + return document.GetError(); + } + const JsonValue &tests = (*document)["results"]["tests"]; + if (!tests.IsArray()) + { + return Error{ "a CTRF report has results.tests" }; + } + std::vector<TestResult> results; + for (const auto &item : tests.Items()) + { + const std::string &name = item["name"].AsString(); + if (name.empty()) + { + return Error{ "a CTRF test has no name" }; + } + TestResult test + = ImportedTest(name, CtrfSuite(item["suite"])); + test.file = item["filePath"].AsString(); + test.line + = static_cast<std::size_t>(item["line"].AsUnsigned()); + test.flaky = item["flaky"].AsBoolean(); + for (const auto &tag : item["tags"].Items()) + { + test.tags.push_back(tag.AsString()); + } + for (const auto &retry : item["retryAttempts"].Items()) + { + test.attempts.push_back(CtrfAttempt(retry)); + } + // The retries may or may not list the last attempt, which + // is the test itself. + Attempt last = CtrfAttempt(item); + if (test.attempts.empty() + || test.attempts.back().outcome != last.outcome) + { + test.attempts.push_back(std::move(last)); + } + results.push_back(std::move(test)); + } + return results; + } + } // namespace ImportJson + } // namespace + + std::optional<ImportFormat> + ImportJsonFormat(const JsonValue &value) + { + return ImportJson::FormatOf(value); + } + + Expected<std::vector<TestResult>> + ParseCatch3Report(std::string_view text) + { + return ImportJson::ParseCatch3(text); + } + + Expected<std::vector<TestResult>> + ParseCtrf(std::string_view text) + { + return ImportJson::ParseCtrf(text); + } +} // namespace Progmasoft::Catch3::Runner +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + +// The Test Anything Protocol, versions 12 to 14. +#include <algorithm> +#include <cctype> +#include <cstdlib> +#include <map> +#include <utility> + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + namespace ImportTap + { + std::string + Lowered(std::string_view text) + { + std::string lowered{ text }; + std::transform(lowered.begin(), + lowered.end(), + lowered.begin(), + [](unsigned char character) { + return static_cast<char>( + std::tolower(character)); + }); + return lowered; + } + + struct Line final + { + bool ok = false; + std::string description; + std::string directive; + std::string reason; + }; + + /// Where the `#` that starts a directive is: `\#` is part of the + /// description. + std::size_t + DirectiveStart(std::string_view text) + { + for (std::size_t index = 0; index < text.size(); ++index) + { + if (text[index] == '\\') + { + ++index; + } + else if (text[index] == '#') + { + return index; + } + } + return std::string_view::npos; + } + + std::string + Unescaped(std::string_view text) + { + std::string result; + for (std::size_t index = 0; index < text.size(); ++index) + { + if (text[index] == '\\' && index + 1 < text.size()) + { + ++index; + } + result.push_back(text[index]); + } + return result; + } + + /// Reads a test line; nothing when `text` is not one. + std::optional<Line> + ReadLine(std::string_view text) + { + Line line; + if (ImportStartsWith(text, "not ok")) + { + text.remove_prefix(6); + } + else if (ImportStartsWith(text, "ok")) + { + line.ok = true; + text.remove_prefix(2); + } + else + { + return std::nullopt; + } + if (!text.empty() && text.front() != ' ' + && text.front() != '\t') + { + return std::nullopt; + } + text = ImportTrim(text); + while (!text.empty() + && std::isdigit(static_cast<unsigned char>(text.front())) + != 0) + { + text.remove_prefix(1); + } + text = ImportTrim(text); + if (ImportStartsWith(text, "- ")) + { + text.remove_prefix(2); + } + const auto directive = DirectiveStart(text); + line.description + = Unescaped(ImportTrim(text.substr(0, directive))); + if (directive != std::string_view::npos) + { + const std::string_view rest + = ImportTrim(text.substr(directive + 1)); + const std::string word = Lowered(rest.substr(0, 4)); + if (word == "skip" || word == "todo") + { + line.directive = word; + const auto space = rest.find_first_of(" \t"); + line.reason = space == std::string_view::npos + ? std::string{} + : std::string{ ImportTrim( + rest.substr(space)) }; + } + } + return line; + } + + Outcome + OutcomeOf(const Line &line) + { + if (line.directive == "skip") + { + return Outcome::kSkipped; + } + if (line.ok) + { + return Outcome::kPassed; + } + return line.directive == "todo" ? Outcome::kFailedAsExpected + : Outcome::kFailed; + } + + std::string + Unquoted(std::string_view text) + { + text = ImportTrim(text); + if (text.size() >= 2 && text.front() == text.back() + && (text.front() == '"' || text.front() == '\'')) + { + text.remove_prefix(1); + text.remove_suffix(1); + } + return std::string{ text }; + } + + /// A YAML block after a test: its text goes into the failure, and + /// the two keys that tools agree on are read. + void + ApplyDiagnostics(TestResult &test, const std::string &block) + { + Attempt &attempt = test.attempts.back(); + for (const auto raw : ImportLines(block)) + { + const std::string_view line = ImportTrim(raw); + if (ImportStartsWith(line, "duration_ms:")) + { + const std::string value{ line.substr(12) }; + attempt.durationMicroseconds = ImportedMicroseconds( + std::strtod(value.c_str(), nullptr) / 1e3); + } + else if (ImportStartsWith(line, "message:") + && !attempt.assertions.empty()) + { + attempt.assertions.front().message + = Unquoted(line.substr(8)); + } + } + if (!attempt.assertions.empty()) + { + attempt.assertions.front().expansion += block; + } + } + + Expected<std::vector<TestResult>> + Parse(std::string_view text) + { + std::vector<TestResult> tests; + std::vector<std::string> subtests; + std::map<std::string, std::size_t> seen; + std::optional<std::size_t> planned; + std::size_t topLevel = 0; + std::string bailOut; + bool anything = false; + std::optional<std::string> block; + for (const auto raw : ImportLines(text)) + { + const std::string_view line = ImportTrim(raw); + if (block) + { + if (line == "...") + { + if (!tests.empty()) + { + ApplyDiagnostics(tests.back(), *block); + } + block.reset(); + } + else + { + block->append(raw).push_back('\n'); + } + continue; + } + if (line == "---") + { + block.emplace(); + continue; + } + std::size_t indent = 0; + for (const char character : raw) + { + if (character != ' ' && character != '\t') + { + break; + } + indent += character == '\t' ? 4 : 1; + } + const std::size_t depth = indent / 4; + if (ImportStartsWith(line, "# Subtest:")) + { + subtests.resize(depth); + subtests.emplace_back(ImportTrim(line.substr(10))); + continue; + } + if (ImportStartsWith(line, "Bail out!")) + { + anything = true; + bailOut = std::string{ line }; + break; + } + if (ImportStartsWith(line, "1..") && depth == 0) + { + anything = true; + const std::string count{ line.substr(3) }; + planned = static_cast<std::size_t>( + std::strtoull(count.c_str(), nullptr, 10)); + continue; + } + if (line.size() > 1 && line.front() == '#' && !tests.empty() + && !tests.back().attempts.back().assertions.empty()) + { + // Comments after a failure usually explain it. + auto &failure + = tests.back().attempts.back().assertions.front(); + failure.expansion + += std::string{ ImportTrim(line.substr(1)) } + "\n"; + continue; + } + const auto parsed = ReadLine(line); + if (!parsed) + { + continue; + } + anything = true; + topLevel += depth == 0 ? 1 : 0; + std::string parent; + for (std::size_t level = 0; + level < std::min(depth, subtests.size()); + ++level) + { + parent + += (parent.empty() ? "" : " > ") + subtests[level]; + } + std::string name = parsed->description; + if (name.empty()) + { + name = "test " + + std::to_string(++seen["unnamed\n" + parent]); + } + // TAP has no retries: a name that comes back in one + // stream is another test. + const std::size_t repeat = ++seen[parent + "\n" + name]; + if (repeat > 1) + { + name += " #" + std::to_string(repeat); + } + TestResult test + = ImportedTest(std::move(name), std::move(parent)); + test.attempts.push_back( + ImportedAttempt(OutcomeOf(*parsed), 0, parsed->reason)); + tests.push_back(std::move(test)); + } + if (!anything) + { + return Error{ "there are no TAP results in it" }; + } + // A stream that ends early must not read as a run that passed. + std::string incomplete; + if (!bailOut.empty()) + { + incomplete = bailOut; + } + else if (planned && *planned != topLevel) + { + incomplete = "planned " + std::to_string(*planned) + + " test(s), reported " + + std::to_string(topLevel); + } + if (!incomplete.empty()) + { + TestResult test = ImportedTest("the TAP stream", {}); + test.attempts.push_back( + ImportedAttempt(Outcome::kFailed, + 0, + std::move(incomplete), + {}, + FailureCategory::kInfrastructure)); + tests.push_back(std::move(test)); + } + return tests; + } + } // namespace ImportTap + } // namespace + + bool + IsTapTestLine(std::string_view line) + { + return ImportTap::ReadLine(line).has_value(); + } + + Expected<std::vector<TestResult>> + ParseTap(std::string_view text) + { + return ImportTap::Parse(text); + } +} // namespace Progmasoft::Catch3::Runner +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + +// The XML reports: JUnit, xUnit.net v2 and Visual Studio's TRX. +#include <cstdlib> +#include <map> +#include <string_view> +#include <utility> + + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + namespace ImportXml + { + Expected<bool> + Load(pugi::xml_document &document, std::string_view text) + { + const pugi::xml_parse_result parsed + = document.load_buffer(text.data(), text.size()); + if (!parsed) + { + return Error{ std::string{ "it is not valid XML: " } + + parsed.description() + " at byte " + + std::to_string(parsed.offset) }; + } + return true; + } + + double + NumberOf(const char *text) + { + std::string digits; + for (; *text != '\0'; ++text) + { + if (*text != ',') + { + digits.push_back(*text); + } + } + return std::strtod(digits.c_str(), nullptr); + } + + std::string + TextOf(pugi::xml_node node) + { + return node.text().get(); + } + + /// `error` and `failure` of JUnit and what Surefire writes for + /// retries, which have the same shape. + Attempt + JUnitFailure(pugi::xml_node node, FailureCategory category) + { + std::string details = TextOf(node); + if (const auto trace = node.child("stackTrace")) + { + details = TextOf(trace); + } + Attempt attempt + = ImportedAttempt(Outcome::kFailed, + ImportedMicroseconds(NumberOf( + node.attribute("time").value())), + node.attribute("message").value(), + std::move(details), + category); + attempt.assertions.front().macro + = node.attribute("type").value(); + attempt.standardOutput = TextOf(node.child("system-out")); + attempt.standardError = TextOf(node.child("system-err")); + return attempt; + } + + bool + NameIs(pugi::xml_node node, std::string_view name) + { + return name == node.name(); + } + + bool + NameEndsWithError(pugi::xml_node node) + { + return NameIs(node, "flakyError") || NameIs(node, "rerunError"); + } + + bool + NotRunStatus(pugi::xml_node testCase) + { + const std::string_view status + = testCase.attribute("status").value(); + const std::string_view result + = testCase.attribute("result").value(); + return status == "notrun" || status == "skipped" + || status == "disabled" || result == "skipped" + || result == "suppressed"; + } + + void + ReadJUnitCase(pugi::xml_node testCase, + pugi::xml_node suite, + std::vector<TestResult> &tests) + { + TestResult test + = ImportedTest(testCase.attribute("name").value(), + testCase.attribute("classname").value()); + test.file = testCase.attribute("file").value(); + for (auto outer = suite; test.file.empty() && outer; + outer = outer.parent()) + { + test.file = outer.attribute("file").value(); + } + test.line = testCase.attribute("line").as_ullong(); + + std::vector<Attempt> failures; + std::optional<Attempt> skipped; + for (const auto child : testCase.children()) + { + if (NameIs(child, "flakyFailure") + || NameIs(child, "rerunFailure") + || NameEndsWithError(child)) + { + test.attempts.push_back(JUnitFailure( + child, + NameEndsWithError(child) + ? FailureCategory::kUnexpectedException + : FailureCategory::kAssertion)); + } + else if (NameIs(child, "failure")) + { + failures.push_back( + JUnitFailure(child, FailureCategory::kAssertion)); + } + else if (NameIs(child, "error")) + { + failures.push_back(JUnitFailure( + child, + FailureCategory::kUnexpectedException)); + } + else if (NameIs(child, "skipped")) + { + std::string reason = child.attribute("message").value(); + skipped = ImportedAttempt( + Outcome::kSkipped, + 0, + reason.empty() ? TextOf(child) : std::move(reason)); + } + } + + const std::int64_t duration = ImportedMicroseconds( + NumberOf(testCase.attribute("time").value())); + Attempt verdict; + if (!failures.empty()) + { + // Several failures are one attempt with several findings. + verdict = std::move(failures.front()); + for (std::size_t index = 1; index < failures.size(); + ++index) + { + auto &record = failures[index].assertions.front(); + verdict.assertions.push_back(std::move(record)); + ++verdict.assertionsFailed; + } + verdict.durationMicroseconds = duration; + } + else if (skipped) + { + verdict = std::move(*skipped); + } + else if (NotRunStatus(testCase)) + { + verdict = ImportedAttempt(Outcome::kSkipped, 0); + } + else + { + verdict = ImportedAttempt(Outcome::kPassed, duration); + } + if (verdict.standardOutput.empty()) + { + verdict.standardOutput + = TextOf(testCase.child("system-out")); + } + if (verdict.standardError.empty()) + { + verdict.standardError + = TextOf(testCase.child("system-err")); + } + test.attempts.push_back(std::move(verdict)); + tests.push_back(std::move(test)); + } + + void + ReadJUnitSuite(pugi::xml_node suite, std::vector<TestResult> &tests) + { + for (const auto child : suite.children()) + { + if (NameIs(child, "testcase")) + { + ReadJUnitCase(child, suite, tests); + } + else if (NameIs(child, "testsuite")) + { + ReadJUnitSuite(child, tests); + } + } + } + + void + ReadXUnitTest(pugi::xml_node node, std::vector<TestResult> &tests) + { + TestResult test = ImportedTest(node.attribute("name").value(), + node.attribute("type").value()); + test.file = node.attribute("source-file").value(); + test.line = node.attribute("source-line").as_ullong(); + for (const auto trait : node.child("traits").children("trait")) + { + test.tags.push_back( + std::string{ trait.attribute("name").value() } + "=" + + trait.attribute("value").value()); + } + const std::string_view result + = node.attribute("result").value(); + const std::int64_t duration = ImportedMicroseconds( + NumberOf(node.attribute("time").value())); + Attempt attempt; + if (result == "Pass") + { + attempt = ImportedAttempt(Outcome::kPassed, duration); + } + else if (result == "Fail") + { + const auto failure = node.child("failure"); + attempt + = ImportedAttempt(Outcome::kFailed, + duration, + TextOf(failure.child("message")), + TextOf(failure.child("stack-trace"))); + attempt.assertions.front().macro + = failure.attribute("exception-type").value(); + } + else if (result == "Skip") + { + attempt = ImportedAttempt(Outcome::kSkipped, + 0, + TextOf(node.child("reason"))); + } + else + { + attempt = ImportedAttempt(Outcome::kNotRun, 0); + } + if (attempt.standardOutput.empty()) + { + attempt.standardOutput = TextOf(node.child("output")); + } + test.attempts.push_back(std::move(attempt)); + tests.push_back(std::move(test)); + } + + /// A failure outside any test, such as in a fixture. + void + ReadXUnitErrors(pugi::xml_node assembly, + std::vector<TestResult> &tests) + { + for (const auto error : + assembly.child("errors").children("error")) + { + const auto failure = error.child("failure"); + TestResult test = ImportedTest( + std::string{ error.attribute("type").value() } + " " + + error.attribute("name").value(), + assembly.attribute("name").value()); + test.attempts.push_back( + ImportedAttempt(Outcome::kFailed, + 0, + TextOf(failure.child("message")), + TextOf(failure.child("stack-trace")), + FailureCategory::kInfrastructure)); + tests.push_back(std::move(test)); + } + } + + void + ReadXUnitAssembly(pugi::xml_node assembly, + std::vector<TestResult> &tests) + { + ReadXUnitErrors(assembly, tests); + for (const auto child : assembly.children()) + { + if (NameIs(child, "test")) + { + ReadXUnitTest(child, tests); + } + else if (NameIs(child, "collection")) + { + for (const auto test : child.children("test")) + { + ReadXUnitTest(test, tests); + } + } + } + } + + /// `[days.]hours:minutes:seconds[.fraction]`. + std::int64_t + TrxDuration(std::string_view text) + { + double seconds = 0.0; + std::string field; + const auto addField = [&] { + seconds = seconds * 60.0 + NumberOf(field.c_str()); + field.clear(); + }; + double days = 0.0; + for (std::size_t index = 0; index < text.size(); ++index) + { + if (text[index] == ':') + { + addField(); + } + else if (text[index] == '.' + && text.find(':', index) != std::string_view::npos) + { + days = NumberOf(field.c_str()); + field.clear(); + } + else + { + field.push_back(text[index]); + } + } + addField(); + return ImportedMicroseconds(seconds + days * 86400.0); + } + + struct TrxOutcome final + { + Outcome outcome = Outcome::kNotRun; + FailureCategory category = FailureCategory::kNone; + }; + + TrxOutcome + TrxOutcomeOf(std::string_view outcome) + { + if (outcome == "Passed" || outcome == "PassedButRunAborted" + || outcome == "Warning" || outcome == "Completed") + { + return { Outcome::kPassed, FailureCategory::kNone }; + } + if (outcome == "Failed") + { + return { Outcome::kFailed, FailureCategory::kAssertion }; + } + if (outcome == "Error") + { + return { Outcome::kFailed, + FailureCategory::kUnexpectedException }; + } + if (outcome == "Timeout") + { + return { Outcome::kTimedOut, FailureCategory::kTimeout }; + } + if (outcome == "Aborted") + { + return { Outcome::kCrashed, FailureCategory::kCrash }; + } + if (outcome == "Disconnected") + { + return { Outcome::kCrashed, + FailureCategory::kInfrastructure }; + } + if (outcome == "NotExecuted" || outcome == "Inconclusive" + || outcome == "NotRunnable") + { + return { Outcome::kSkipped, FailureCategory::kNone }; + } + return {}; + } + + /// The class of a test: the type without its assembly. + std::string + TrxClass(std::string_view className) + { + return std::string{ className.substr(0, className.find(',')) }; + } + + void + ReadTrxResult(pugi::xml_node result, + const std::map<std::string, std::string> &classes, + std::vector<TestResult> &tests) + { + // A data-driven test: the rows are what ran. + if (const auto inner = result.child("InnerResults")) + { + for (const auto row : inner.children()) + { + ReadTrxResult(row, classes, tests); + } + return; + } + const auto found + = classes.find(result.attribute("testId").value()); + TestResult test = ImportedTest( + result.attribute("testName").value(), + found == classes.end() ? std::string{} : found->second); + const auto [outcome, category] + = TrxOutcomeOf(result.attribute("outcome").value()); + const auto output = result.child("Output"); + const auto error = output.child("ErrorInfo"); + std::string message = TextOf(error.child("Message")); + Attempt attempt = ImportedAttempt( + outcome, + TrxDuration(result.attribute("duration").value()), + outcome == Outcome::kSkipped + ? TextOf(output.child("StdOut")) + : std::move(message), + TextOf(error.child("StackTrace")), + category); + if (outcome != Outcome::kSkipped) + { + attempt.standardOutput = TextOf(output.child("StdOut")); + } + attempt.standardError = TextOf(output.child("StdErr")); + test.attempts.push_back(std::move(attempt)); + tests.push_back(std::move(test)); + } + } // namespace ImportXml + } // namespace + + Expected<std::vector<TestResult>> + ParseJUnit(std::string_view text) + { + pugi::xml_document document; + if (auto loaded = ImportXml::Load(document, text); !loaded) + { + return loaded.GetError(); + } + const pugi::xml_node root = document.document_element(); + std::vector<TestResult> tests; + if (ImportXml::NameIs(root, "testsuites") + || ImportXml::NameIs(root, "testsuite")) + { + ImportXml::ReadJUnitSuite(root, tests); + return tests; + } + return Error{ "a JUnit report has testsuites or testsuite as its root, " + "not " + + std::string{ root.name() } }; + } + + Expected<std::vector<TestResult>> + ParseXUnit(std::string_view text) + { + pugi::xml_document document; + if (auto loaded = ImportXml::Load(document, text); !loaded) + { + return loaded.GetError(); + } + const pugi::xml_node root = document.document_element(); + std::vector<TestResult> tests; + if (ImportXml::NameIs(root, "assembly")) + { + ImportXml::ReadXUnitAssembly(root, tests); + return tests; + } + if (ImportXml::NameIs(root, "assemblies")) + { + for (const auto assembly : root.children("assembly")) + { + ImportXml::ReadXUnitAssembly(assembly, tests); + } + return tests; + } + return Error{ "an xUnit report has assemblies or assembly as its root, " + "not " + + std::string{ root.name() } }; + } + + Expected<std::vector<TestResult>> + ParseTrx(std::string_view text) + { + pugi::xml_document document; + if (auto loaded = ImportXml::Load(document, text); !loaded) + { + return loaded.GetError(); + } + const pugi::xml_node root = document.document_element(); + if (!ImportXml::NameIs(root, "TestRun")) + { + return Error{ "a TRX report has TestRun as its root, not " + + std::string{ root.name() } }; + } + std::map<std::string, std::string> classes; + for (const auto definition : root.child("TestDefinitions").children()) + { + classes[definition.attribute("id").value()] = ImportXml::TrxClass( + definition.child("TestMethod").attribute("className").value()); + } + std::vector<TestResult> tests; + for (const auto result : root.child("Results").children()) + { + ImportXml::ReadTrxResult(result, classes, tests); + } + return tests; + } +} // namespace Progmasoft::Catch3::Runner +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + JsonValue + Strings(const std::vector<std::string> &values) + { + JsonValue array = JsonValue::Array(); + for (const auto &value : values) + { + array.Append(value); + } + return array; + } + + std::vector<std::string> + StringsFrom(const JsonValue &value) + { + std::vector<std::string> strings; + for (const auto &item : value.Items()) + { + strings.push_back(item.AsString()); + } + return strings; + } + + std::size_t + Size(const JsonValue &value) + { + return static_cast<std::size_t>(value.AsUnsigned()); + } + + /// Adds a member only when it says something, so reports and + /// protocol messages stay small. + void + SetText(JsonValue &object, std::string_view key, const std::string &text) + { + if (!text.empty()) + { + object.Set(key, text); + } + } + } // namespace + + JsonValue + ToJson(const AssertionRecord &assertion) + { + JsonValue value = JsonValue::Object(); + value.Set("file", assertion.file); + value.Set("line", assertion.line); + SetText(value, "macro", assertion.macro); + SetText(value, "expression", assertion.expression); + SetText(value, "expansion", assertion.expansion); + SetText(value, "message", assertion.message); + if (!assertion.context.empty()) + { + value.Set("context", Strings(assertion.context)); + } + if (!assertion.sections.empty()) + { + value.Set("sections", Strings(assertion.sections)); + } + value.Set("category", CategoryName(assertion.category)); + value.Set("passed", assertion.passed); + if (assertion.allowed) + { + value.Set("allowed", true); + } + return value; + } + + AssertionRecord + AssertionFromJson(const JsonValue &value) + { + AssertionRecord assertion; + assertion.file = value["file"].AsString(); + assertion.line = Size(value["line"]); + assertion.macro = value["macro"].AsString(); + assertion.expression = value["expression"].AsString(); + assertion.expansion = value["expansion"].AsString(); + assertion.message = value["message"].AsString(); + assertion.context = StringsFrom(value["context"]); + assertion.sections = StringsFrom(value["sections"]); + assertion.category = CategoryFromName(value["category"].AsString()) + .value_or(FailureCategory::kAssertion); + assertion.passed = value["passed"].AsBoolean(); + assertion.allowed = value["allowed"].AsBoolean(); + return assertion; + } + + JsonValue + ToJson(const BenchmarkRecord &benchmark) + { + JsonValue value = JsonValue::Object(); + value.Set("name", benchmark.name); + value.Set("samples", benchmark.samples); + value.Set("iterations", benchmark.iterations); + value.Set("meanNanoseconds", benchmark.meanNanoseconds); + value.Set("lowMeanNanoseconds", benchmark.lowMeanNanoseconds); + value.Set("highMeanNanoseconds", benchmark.highMeanNanoseconds); + value.Set("standardDeviationNanoseconds", + benchmark.standardDeviationNanoseconds); + value.Set("outlierVariance", benchmark.outlierVariance); + return value; + } + + BenchmarkRecord + BenchmarkFromJson(const JsonValue &value) + { + BenchmarkRecord benchmark; + benchmark.name = value["name"].AsString(); + benchmark.samples = Size(value["samples"]); + benchmark.iterations = Size(value["iterations"]); + benchmark.meanNanoseconds = value["meanNanoseconds"].AsNumber(); + benchmark.lowMeanNanoseconds = value["lowMeanNanoseconds"].AsNumber(); + benchmark.highMeanNanoseconds = value["highMeanNanoseconds"].AsNumber(); + benchmark.standardDeviationNanoseconds = + value["standardDeviationNanoseconds"].AsNumber(); + benchmark.outlierVariance = value["outlierVariance"].AsNumber(); + return benchmark; + } + + JsonValue + ToJson(const SectionRecord §ion) + { + JsonValue value = JsonValue::Object(); + value.Set("name", section.name); + value.Set("file", section.file); + value.Set("line", section.line); + value.Set("depth", section.depth); + value.Set("startMicroseconds", section.startMicroseconds); + value.Set("durationMicroseconds", section.durationMicroseconds); + return value; + } + SectionRecord SectionFromJson(const JsonValue &value) { @@ -16782,6 +18391,7 @@ namespace Progmasoft::Catch3::Runner #include <cstdio> #include <exception> #include <filesystem> +#include <iterator> #include <map> #include <random> #include <system_error> @@ -17264,6 +18874,215 @@ namespace Progmasoft::Catch3 .count(); } + /// Writes the reports that were asked for. One that is missing would + /// let a build server miss the run, so it fails the run. + bool + WriteReports(const Runner::Options &options, const RunResult &run) + { + bool written = true; + const auto write = [&](const std::string &path, + const std::string &contents) { + if (auto status = Runner::WriteFileAtomically(path, contents); + !status) + { + PrintError(status.GetError().Message()); + written = false; + } + }; + if (!options.junitPath.empty()) + { + write(options.junitPath, JUnitXmlWriter::ToString(run)); + } + if (!options.jsonPath.empty()) + { + write(options.jsonPath, Runner::ToJson(run).ToPrettyString()); + } + if (!options.tracePath.empty()) + { + write(options.tracePath, Runner::TraceJson(run)); + } + return written; + } + + void + SaveHistory(const std::filesystem::path &file, + const RunResult &run, + const std::string &commit, + const std::string &binaryFingerprint, + const std::map<std::string, std::string> &fingerprints) + { + if (run.tests.empty()) + { + return; + } + Runner::RunRecord record; + record.unixMilliseconds = run.startUnixMilliseconds; + record.commit = commit; + record.binaryFingerprint = binaryFingerprint; + record.seed = run.seed; + record.tests = run.tests.size(); + record.wallMicroseconds = run.wallMicroseconds; + for (const auto &test : run.tests) + { + record.failed += test.FailsRun() ? 1 : 0; + } + auto committed = Runner::History::Commit(file, + run, + record, + fingerprints, + {}); + if (!committed) + { + std::fprintf(stderr, + "warning: the history was not saved: %s\n", + committed.GetError().Message().c_str()); + } + } + + int + ExitCodeOf(const Runner::Options &options, + const RunResult &run, + bool reportMissing) + { + const bool quarantineCounts + = options.quarantine == Runner::QuarantineMode::kReport; + bool failed = false; + std::size_t ran = 0; + std::size_t skipped = 0; + for (const auto &test : run.tests) + { + if (test.outcome != Outcome::kUpToDate) + { + ++ran; + } + skipped += test.outcome == Outcome::kSkipped ? 1 : 0; + failed = failed || test.FailsRun() + || (quarantineCounts && test.quarantined + && IsFailure(test.outcome)); + } + if (failed) + { + return Catch::TestFailureExitCode; + } + if (reportMissing) + { + return Catch::UnspecifiedErrorExitCode; + } + if (run.tests.empty() && !options.allowNoTests) + { + return Catch::NoTestsRunExitCode; + } + if (ran > 0 && skipped == ran && !options.allowNoTests) + { + return Catch::AllTestsSkippedExitCode; + } + return 0; + } + + /// The file results are remembered in. + std::filesystem::path + HistoryFileOf(const Runner::Options &options, + const std::filesystem::path &program, + const std::string &programName) + { + std::string name = options.importName; + if (name.empty()) + { + name = options.mode == Runner::Mode::kImport ? "imported" + : programName; + } + return Runner::HistoryFile(HistoryDirectory(options, program), + name); + } + + /// Reports of other frameworks: `--import`. What they say goes + /// through the analysis, the history and the reports of a run. + int + Import(const Runner::Options &options, + const std::filesystem::path &program, + const std::string &programName) + { + std::vector<TestResult> imported; + for (const auto &file : options.importFiles) + { + auto text = Runner::ReadFile(file); + if (!text) + { + PrintError(text.GetError().Message()); + return Catch::UnspecifiedErrorExitCode; + } + auto tests = Runner::ParseImport(*text, options.importFormat); + if (!tests) + { + PrintError(tests.GetError().Context(file).Message()); + return Catch::UnspecifiedErrorExitCode; + } + imported.insert(imported.end(), + std::make_move_iterator(tests->begin()), + std::make_move_iterator(tests->end())); + } + + const std::filesystem::path historyFile + = HistoryFileOf(options, program, programName); + Runner::History history; + RunResult run; + run.binary = options.importName.empty() ? "imported" + : options.importName; + if (options.historyEnabled) + { + auto loaded = Runner::History::Load(historyFile); + if (loaded) + { + history = std::move(*loaded); + } + else + { + run.notes.push_back("the history could not be read: " + + loaded.GetError().Message()); + } + } + + run.tests = Runner::MergeImported(std::move(imported)); + run.candidates = run.tests.size(); + run.startUnixMilliseconds = UnixMilliseconds(); + std::error_code here; + const std::string revision + = options.importRevision.empty() + ? Runner::CurrentCommit( + std::filesystem::current_path(here)) + : options.importRevision; + std::map<std::string, std::string> fingerprints; + for (auto &test : run.tests) + { + run.wallMicroseconds += test.DurationMicroseconds(); + fingerprints[test.id] = revision; + const Runner::TestHistory *past = history.Find(test.id); + test.quarantined + = options.quarantine != Runner::QuarantineMode::kOff + && past != nullptr && past->quarantine.has_value(); + } + + Runner::AnalysisInputs analysis; + analysis.history = options.historyEnabled ? &history : nullptr; + Runner::Analyze(run, analysis); + + Runner::ConsoleReport console{ options, stdout }; + console.ImportStarting(run.tests.size(), + options.importFiles.size()); + for (const auto &test : run.tests) + { + console.TestFinished(test); + } + console.RunEnded(run); + + const bool reportMissing = !WriteReports(options, run); + if (options.historyEnabled) + { + SaveHistory(historyFile, run, revision, {}, fingerprints); + } + return ExitCodeOf(options, run, reportMissing); + } + int Run(const std::vector<std::string> &arguments) { @@ -17311,6 +19130,10 @@ namespace Progmasoft::Catch3 { return Runner::RunWorker(options, programName); } + if (options.mode == Runner::Mode::kImport) + { + return Import(options, program, programName); + } std::vector<std::string> notes; if (!options.configPath.empty() @@ -17376,8 +19199,7 @@ namespace Progmasoft::Catch3 } const std::filesystem::path historyFile - = Runner::HistoryFile(HistoryDirectory(options, program), - programName); + = HistoryFileOf(options, program, programName); Runner::History history; if (options.historyEnabled || options.mode == Runner::Mode::kAnalytics) @@ -17528,66 +19350,14 @@ namespace Progmasoft::Catch3 Runner::Analyze(run, analysis); console.RunEnded(run); - // A report that was asked for and is missing would let a build - // server miss the run, so it fails the run. - bool reportMissing = false; - if (!options.junitPath.empty()) - { - if (auto status = Runner::WriteFileAtomically( - options.junitPath, - JUnitXmlWriter::ToString(run)); - !status) - { - PrintError(status.GetError().Message()); - reportMissing = true; - } - } - if (!options.jsonPath.empty()) - { - if (auto status = Runner::WriteFileAtomically( - options.jsonPath, - Runner::ToJson(run).ToPrettyString()); - !status) - { - PrintError(status.GetError().Message()); - reportMissing = true; - } - } - if (!options.tracePath.empty()) - { - if (auto status - = Runner::WriteFileAtomically(options.tracePath, - Runner::TraceJson(run)); - !status) - { - PrintError(status.GetError().Message()); - reportMissing = true; - } - } - if (options.historyEnabled && !run.tests.empty()) + const bool reportMissing = !WriteReports(options, run); + if (options.historyEnabled) { - Runner::RunRecord record; - record.unixMilliseconds = run.startUnixMilliseconds; - record.commit = changes ? changes->commit : std::string{}; - record.binaryFingerprint = binaryFingerprint; - record.seed = run.seed; - record.tests = run.tests.size(); - record.wallMicroseconds = run.wallMicroseconds; - for (const auto &test : run.tests) - { - record.failed += test.FailsRun() ? 1 : 0; - } - auto committed = Runner::History::Commit(historyFile, - run, - record, - plan->fingerprints, - {}); - if (!committed) - { - std::fprintf(stderr, - "warning: the history was not saved: %s\n", - committed.GetError().Message().c_str()); - } + SaveHistory(historyFile, + run, + changes ? changes->commit : std::string{}, + binaryFingerprint, + plan->fingerprints); } if (!guard.empty()) { @@ -17603,39 +19373,7 @@ namespace Progmasoft::Catch3 { return Catch::UnmatchedTestSpecExitCode; } - const bool quarantineCounts - = options.quarantine == Runner::QuarantineMode::kReport; - bool failed = false; - std::size_t ran = 0; - std::size_t skipped = 0; - for (const auto &test : run.tests) - { - if (test.outcome != Outcome::kUpToDate) - { - ++ran; - } - skipped += test.outcome == Outcome::kSkipped ? 1 : 0; - failed = failed || test.FailsRun() - || (quarantineCounts && test.quarantined - && IsFailure(test.outcome)); - } - if (failed) - { - return Catch::TestFailureExitCode; - } - if (reportMissing) - { - return Catch::UnspecifiedErrorExitCode; - } - if (run.tests.empty() && !options.allowNoTests) - { - return Catch::NoTestsRunExitCode; - } - if (ran > 0 && skipped == ran && !options.allowNoTests) - { - return Catch::AllTestsSkippedExitCode; - } - return 0; + return ExitCodeOf(options, run, reportMissing); } } // namespace diff --git a/extras/catch_amalgamated.hpp b/extras/catch_amalgamated.hpp index 513c91ae..680dae0e 100644 --- a/extras/catch_amalgamated.hpp +++ b/extras/catch_amalgamated.hpp @@ -7,7 +7,7 @@ // SPDX-License-Identifier: BSL-1.0 // Catch3 v0.3.0, on the Catch2 v3.16.1 engine -// Generated: 2026-11-01 00:25:29.668428 +// Generated: 2026-11-01 14:22:22.117317 // ---------------------------------------------------------- // This file is an amalgamation of multiple different files. // You probably shouldn't edit it directly. diff --git a/extras/catch_amalgamated_third_party.hpp b/extras/catch_amalgamated_third_party.hpp index 353e0009..ff52646e 100644 --- a/extras/catch_amalgamated_third_party.hpp +++ b/extras/catch_amalgamated_third_party.hpp @@ -1,5 +1,5 @@ // Catch3 v0.3.0: yyjson and pugixml for catch_amalgamated.cpp -// Generated: 2026-11-01 00:25:29.854771 +// Generated: 2026-11-01 14:22:22.379981 // ---------------------------------------------------------- // This file is an amalgamation of src/Progmasoft/Runner/Yyjson.inl, // src/Progmasoft/PugiXml.inl and the libraries they include from @@ -19974,8 +19974,8 @@ yyjson_mut_val *yyjson_mut_merge_patch(yyjson_mut_doc *doc, # pragma warning(pop) #endif -// pugixml, compiled header-only into XmlWriter.cpp and nowhere else. Its -// namespace is renamed, so Catch3 cannot collide with a copy of pugixml that +// pugixml, compiled header-only into XmlWriter.cpp and ResultImportXml.cpp +// and nowhere else. Its namespace is renamed, so Catch3 cannot collide with a copy of pugixml that // the tests themselves link. It is vendored code: it is read as a system // header, outside the warnings Catch3 builds with. #if defined(__GNUC__) || defined(__clang__) diff --git a/premake5.lua b/premake5.lua index a1427cf3..7b917cfd 100644 --- a/premake5.lua +++ b/premake5.lua @@ -90,6 +90,7 @@ project "ProgmasoftPropertyTests" "tests/Progmasoft/RunnerOptions.tests.cpp", "tests/Progmasoft/RunnerConfig.tests.cpp", "tests/Progmasoft/RunnerAnalysis.tests.cpp", + "tests/Progmasoft/RunnerImport.tests.cpp", "tests/Progmasoft/RunnerOutput.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp", "tests/Progmasoft/Stateful.tests.cpp", diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index aee8c7a3..87b19651 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -310,6 +310,7 @@ set(PROGMASOFT_RUNNER_FILES "${CATCH_DIR}/src/Progmasoft/Runner/Process.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/ProjectConfig.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Protocol.hpp" + "${CATCH_DIR}/src/Progmasoft/Runner/ResultImport.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/ResultsJson.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Scheduler.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/SchedulerModel.hpp" @@ -329,6 +330,10 @@ set(PROGMASOFT_RUNNER_FILES "${CATCH_DIR}/src/Progmasoft/Runner/Process.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/ProjectConfig.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Protocol.cpp" + "${CATCH_DIR}/src/Progmasoft/Runner/ResultImport.cpp" + "${CATCH_DIR}/src/Progmasoft/Runner/ResultImportJson.cpp" + "${CATCH_DIR}/src/Progmasoft/Runner/ResultImportTap.cpp" + "${CATCH_DIR}/src/Progmasoft/Runner/ResultImportXml.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/ResultsJson.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Runner.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Scheduler.cpp" diff --git a/src/Progmasoft/Catch3/meson.build b/src/Progmasoft/Catch3/meson.build index 7b75dc6d..ccf93f29 100644 --- a/src/Progmasoft/Catch3/meson.build +++ b/src/Progmasoft/Catch3/meson.build @@ -297,6 +297,10 @@ runner_sources = files( '../Runner/Process.cpp', '../Runner/ProjectConfig.cpp', '../Runner/Protocol.cpp', + '../Runner/ResultImport.cpp', + '../Runner/ResultImportJson.cpp', + '../Runner/ResultImportTap.cpp', + '../Runner/ResultImportXml.cpp', '../Runner/ResultsJson.cpp', '../Runner/Runner.cpp', '../Runner/Scheduler.cpp', diff --git a/src/Progmasoft/PugiXml.inl b/src/Progmasoft/PugiXml.inl index 207bd17f..e93dd679 100644 --- a/src/Progmasoft/PugiXml.inl +++ b/src/Progmasoft/PugiXml.inl @@ -1,8 +1,8 @@ // Copyright (c) 2026 Progmasoft. // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 -// pugixml, compiled header-only into XmlWriter.cpp and nowhere else. Its -// namespace is renamed, so Catch3 cannot collide with a copy of pugixml that +// pugixml, compiled header-only into XmlWriter.cpp and ResultImportXml.cpp +// and nowhere else. Its namespace is renamed, so Catch3 cannot collide with a copy of pugixml that // the tests themselves link. It is vendored code: it is read as a system // header, outside the warnings Catch3 builds with. #if defined(__GNUC__) || defined(__clang__) diff --git a/src/Progmasoft/Runner/Analysis.cpp b/src/Progmasoft/Runner/Analysis.cpp index 802f1ed9..739ca2dc 100644 --- a/src/Progmasoft/Runner/Analysis.cpp +++ b/src/Progmasoft/Runner/Analysis.cpp @@ -350,6 +350,11 @@ namespace Progmasoft::Catch3::Runner test.analysis.insert( test.analysis.end(), findings.begin(), findings.end()); + // An imported failure ran somewhere else. + if (inputs.program.empty()) + { + continue; + } std::string replay = QuoteArgument(inputs.program) + " " + QuoteArgument(QuoteTestSpec(test.name)) + " --seed=" + std::to_string(failure->seed); diff --git a/src/Progmasoft/Runner/Console.cpp b/src/Progmasoft/Runner/Console.cpp index 44765311..91691967 100644 --- a/src/Progmasoft/Runner/Console.cpp +++ b/src/Progmasoft/Runner/Console.cpp @@ -202,6 +202,21 @@ namespace Progmasoft::Catch3::Runner std::fflush(output_); } + void + ConsoleReport::ImportStarting(std::size_t tests, std::size_t files) + { + total_ = tests; + if (options_.verbosity == Verbosity::kQuiet) + { + return; + } + Write(output_, + Paint("Catch3", kBold) + ": " + std::to_string(tests) + + " test(s) imported from " + std::to_string(files) + + " report(s)\n"); + std::fflush(output_); + } + void ConsoleReport::Retrying(const TestResult &test, const Attempt &attempt) { diff --git a/src/Progmasoft/Runner/Console.hpp b/src/Progmasoft/Runner/Console.hpp index e79fcf68..32c83976 100644 --- a/src/Progmasoft/Runner/Console.hpp +++ b/src/Progmasoft/Runner/Console.hpp @@ -25,6 +25,10 @@ namespace Progmasoft::Catch3::Runner std::uint32_t seed, const std::vector<std::string> ¬es); + /// The start of `--import`: results read from `files` reports. + void + ImportStarting(std::size_t tests, std::size_t files); + void TestFinished(const TestResult &test); diff --git a/src/Progmasoft/Runner/Options.cpp b/src/Progmasoft/Runner/Options.cpp index c59d7e40..4ae99eb3 100644 --- a/src/Progmasoft/Runner/Options.cpp +++ b/src/Progmasoft/Runner/Options.cpp @@ -683,6 +683,58 @@ namespace Progmasoft::Catch3::Runner options.replayFailures = true; return Status::Ok(); }}, + {"import", + '\0', + Takes::kValue, + "FILE", + "Importing results", + "read a report of another framework instead of running tests " + "(repeatable)", + [](Options &options, const std::optional<std::string> &value) { + options.mode = Mode::kImport; + options.importFiles.push_back(*value); + return Status::Ok(); + }}, + {"import-format", + '\0', + Takes::kValue, + "FORMAT", + "Importing results", + "auto (default), catch3, junit, xunit, trx, tap or ctrf", + [](Options &options, const std::optional<std::string> &value) { + return ReadChoice<ImportFormat>( + value, + "import-format", + {{"auto", ImportFormat::kAuto}, + {"catch3", ImportFormat::kCatch3}, + {"junit", ImportFormat::kJUnit}, + {"xunit", ImportFormat::kXUnit}, + {"trx", ImportFormat::kTrx}, + {"tap", ImportFormat::kTap}, + {"ctrf", ImportFormat::kCtrf}}, + options.importFormat); + }}, + {"import-name", + '\0', + Takes::kValue, + "NAME", + "Importing results", + "the history imported results go to (default: imported); " + "--analytics reads it too", + [](Options &options, const std::optional<std::string> &value) { + options.importName = *value; + return Status::Ok(); + }}, + {"import-revision", + '\0', + Takes::kValue, + "REVISION", + "Importing results", + "what the reports tested (default: the Git commit here)", + [](Options &options, const std::optional<std::string> &value) { + options.importRevision = *value; + return Status::Ok(); + }}, {"analytics", '\0', Takes::kOptionalValue, diff --git a/src/Progmasoft/Runner/Options.hpp b/src/Progmasoft/Runner/Options.hpp index 15bba45e..9f7f826e 100644 --- a/src/Progmasoft/Runner/Options.hpp +++ b/src/Progmasoft/Runner/Options.hpp @@ -21,6 +21,8 @@ namespace Progmasoft::Catch3::Runner kListTests, kListTags, kAnalytics, + /// Read the reports of other test frameworks. + kImport, kHelp, kVersion, /// Internal: run tests on behalf of a runner, over the protocol. @@ -78,6 +80,20 @@ namespace Progmasoft::Catch3::Runner kOff, }; + /// The reports `--import` reads. + enum class ImportFormat : std::uint8_t + { + /// Whichever the contents show. + kAuto, + /// Catch3's own: `--json` reports and catch3-results JSON Lines. + kCatch3, + kJUnit, + kXUnit, + kTrx, + kTap, + kCtrf, + }; + enum class ColorMode : std::uint8_t { kAuto, @@ -157,6 +173,15 @@ namespace Progmasoft::Catch3::Runner bool replayFailures = false; bool analyticsJson = false; + // Importing the reports of other frameworks. + std::vector<std::string> importFiles; + ImportFormat importFormat = ImportFormat::kAuto; + /// The history they go to; "imported" when empty. + std::string importName; + /// What was tested, so that a verdict that changes on one revision + /// counts as flaky; the Git commit here when empty. + std::string importRevision; + // Output. Verbosity verbosity = Verbosity::kNormal; ColorMode color = ColorMode::kAuto; diff --git a/src/Progmasoft/Runner/ProjectConfig.cpp b/src/Progmasoft/Runner/ProjectConfig.cpp index 54f89d9c..e382b5c5 100644 --- a/src/Progmasoft/Runner/ProjectConfig.cpp +++ b/src/Progmasoft/Runner/ProjectConfig.cpp @@ -113,6 +113,10 @@ namespace Progmasoft::Catch3::Runner "break", "replay", "analytics", + "import", + "import-format", + "import-name", + "import-revision", "verbose", "quiet", "premature-exit-guard-file", diff --git a/src/Progmasoft/Runner/ResultImport.cpp b/src/Progmasoft/Runner/ResultImport.cpp new file mode 100644 index 00000000..10a9c5a8 --- /dev/null +++ b/src/Progmasoft/Runner/ResultImport.cpp @@ -0,0 +1,305 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 +#include <Progmasoft/Runner/History.hpp> +#include <Progmasoft/Runner/Json.hpp> +#include <Progmasoft/Runner/ResultImport.hpp> +#include <algorithm> +#include <cctype> +#include <cmath> +#include <iterator> +#include <map> +#include <utility> + +namespace Progmasoft::Catch3::Runner +{ + std::string_view + ImportTrim(std::string_view text) + { + const auto blank = [](char character) { + return std::isspace(static_cast<unsigned char>(character)) != 0; + }; + while (!text.empty() && blank(text.front())) + { + text.remove_prefix(1); + } + while (!text.empty() && blank(text.back())) + { + text.remove_suffix(1); + } + return text; + } + + bool + ImportStartsWith(std::string_view text, std::string_view prefix) + { + return text.substr(0, prefix.size()) == prefix; + } + + std::vector<std::string_view> + ImportLines(std::string_view text) + { + std::vector<std::string_view> lines; + while (!text.empty()) + { + const auto end = text.find('\n'); + auto line = text.substr(0, end); + if (!line.empty() && line.back() == '\r') + { + line.remove_suffix(1); + } + lines.push_back(line); + if (end == std::string_view::npos) + { + break; + } + text.remove_prefix(end + 1); + } + return lines; + } + + namespace + { + namespace ImportText + { + /// Without a byte order mark and the white space before the + /// first character that counts. + std::string_view + Contents(std::string_view text) + { + if (ImportStartsWith(text, "\xEF\xBB\xBF")) + { + text.remove_prefix(3); + } + return ImportTrim(text); + } + + /// The name of the root element of an XML document. + std::string_view + RootElement(std::string_view text) + { + while (!text.empty()) + { + const auto open = text.find('<'); + if (open == std::string_view::npos + || open + 1 >= text.size()) + { + return {}; + } + text.remove_prefix(open + 1); + if (text.front() == '?' || text.front() == '!') + { + const auto close = text.find('>'); + if (close == std::string_view::npos) + { + return {}; + } + text.remove_prefix(close + 1); + continue; + } + return text.substr(0, text.find_first_of(" \t\r\n/>")); + } + return {}; + } + } // namespace ImportText + } // namespace + + std::optional<ImportFormat> + DetectImportFormat(std::string_view text) + { + const std::string_view contents = ImportText::Contents(text); + if (contents.empty()) + { + return std::nullopt; + } + if (contents.front() == '<') + { + const std::string_view root = ImportText::RootElement(contents); + if (root == "testsuites" || root == "testsuite") + { + return ImportFormat::kJUnit; + } + if (root == "assemblies" || root == "assembly") + { + return ImportFormat::kXUnit; + } + if (root == "TestRun") + { + return ImportFormat::kTrx; + } + return std::nullopt; + } + if (contents.front() == '{') + { + auto whole = ParseJson(contents); + if (!whole) + { + // JSON Lines: the first line says what the others are. + whole = ParseJson(contents.substr(0, contents.find('\n'))); + } + return whole ? ImportJsonFormat(*whole) : std::nullopt; + } + for (const auto raw : ImportLines(contents)) + { + const std::string_view line = ImportTrim(raw); + if (line.empty() || line.front() == '#') + { + continue; + } + if (ImportStartsWith(line, "TAP version") + || ImportStartsWith(line, "1..") || IsTapTestLine(line)) + { + return ImportFormat::kTap; + } + break; + } + return std::nullopt; + } + + Expected<std::vector<TestResult>> + ParseImport(std::string_view text, ImportFormat format) + { + const std::string_view contents = ImportText::Contents(text); + switch (format) + { + case ImportFormat::kAuto: + break; + case ImportFormat::kCatch3: + return ParseCatch3Report(contents); + case ImportFormat::kJUnit: + return ParseJUnit(contents); + case ImportFormat::kXUnit: + return ParseXUnit(contents); + case ImportFormat::kTrx: + return ParseTrx(contents); + case ImportFormat::kTap: + return ParseTap(contents); + case ImportFormat::kCtrf: + return ParseCtrf(contents); + } + const auto detected = DetectImportFormat(contents); + if (!detected) + { + return Error{ "its format is not one Catch3 knows; " + "--import-format names it" }; + } + return ParseImport(contents, *detected); + } + + std::vector<TestResult> + MergeImported(std::vector<TestResult> results) + { + std::vector<TestResult> tests; + std::map<std::string, std::size_t> positions; + for (auto &result : results) + { + if (result.id.empty()) + { + result.id = History::KeyOf(result); + } + const auto [position, added] + = positions.emplace(result.id, tests.size()); + if (added) + { + tests.push_back(std::move(result)); + continue; + } + TestResult &test = tests[position->second]; + test.flaky = test.flaky || result.flaky; + std::move(result.attempts.begin(), + result.attempts.end(), + std::back_inserter(test.attempts)); + } + for (auto &test : tests) + { + std::size_t passes = 0; + std::size_t failures = 0; + for (std::size_t index = 0; index < test.attempts.size(); ++index) + { + Attempt &attempt = test.attempts[index]; + attempt.number = index + 1; + passes += attempt.outcome == Outcome::kPassed ? 1 : 0; + failures += IsFailure(attempt.outcome) ? 1 : 0; + } + if (test.attempts.empty()) + { + continue; + } + test.outcome = test.attempts.back().outcome; + test.category = test.attempts.back().category; + if (passes > 0 && failures > 0) + { + test.outcome = Outcome::kPassed; + test.category = FailureCategory::kNone; + test.flaky = true; + test.analysis.push_back( + "flaky: failed " + std::to_string(failures) + " of " + + std::to_string(test.attempts.size()) + " attempts"); + } + } + return tests; + } + + TestResult + ImportedTest(std::string name, std::string className) + { + TestResult test; + test.name = std::move(name); + test.className = std::move(className); + test.id = History::KeyOf(test); + return test; + } + + Attempt + ImportedAttempt(Outcome outcome, + std::int64_t durationMicroseconds, + std::string message, + std::string details, + FailureCategory category) + { + Attempt attempt; + attempt.outcome = outcome; + attempt.durationMicroseconds + = std::max<std::int64_t>(durationMicroseconds, 0); + if (outcome == Outcome::kSkipped) + { + attempt.standardOutput = std::move(message); + return attempt; + } + if (outcome != Outcome::kFailedAsExpected && !IsFailure(outcome)) + { + return attempt; + } + if (category == FailureCategory::kNone) + { + category = outcome == Outcome::kCrashed ? FailureCategory::kCrash + : outcome == Outcome::kTimedOut + ? FailureCategory::kTimeout + : FailureCategory::kAssertion; + } + if (message.empty()) + { + message = details.substr(0, details.find('\n')); + } + attempt.category = category; + attempt.assertionsFailed = 1; + AssertionRecord assertion; + assertion.message = std::move(message); + assertion.expansion = std::move(details); + assertion.category = category; + assertion.allowed = outcome == Outcome::kFailedAsExpected; + attempt.assertions.push_back(std::move(assertion)); + return attempt; + } + + std::int64_t + ImportedMicroseconds(double seconds) + { + // About thirty years; a longer duration is not a duration. + constexpr double kLongest = 1e9; + if (!std::isfinite(seconds) || seconds <= 0.0 || seconds > kLongest) + { + return 0; + } + return static_cast<std::int64_t>(std::llround(seconds * 1e6)); + } +} // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/ResultImport.hpp b/src/Progmasoft/Runner/ResultImport.hpp new file mode 100644 index 00000000..b9ddff27 --- /dev/null +++ b/src/Progmasoft/Runner/ResultImport.hpp @@ -0,0 +1,89 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 +#pragma once + +#include <Progmasoft/Catch3/Results.hpp> +#include <cstdint> +#include <optional> +#include <string> +#include <string_view> +#include <vector> + +#include "Expected.hpp" +#include "Json.hpp" +#include "Options.hpp" + +namespace Progmasoft::Catch3::Runner +{ + /// The format of a report, from its contents; nothing when none fits. + [[nodiscard]] std::optional<ImportFormat> + DetectImportFormat(std::string_view text); + + /// The results a report holds, in its order. A test the report names + /// twice comes back twice; `MergeImported` makes one test of them. + [[nodiscard]] Expected<std::vector<TestResult>> + ParseImport(std::string_view text, ImportFormat format); + + /// One test per identifier: results that share one are attempts at that + /// test, in the order given, and the verdict is worked out over them. A + /// test that both failed and passed is flaky, as it is in a run of the + /// runner. + [[nodiscard]] std::vector<TestResult> + MergeImported(std::vector<TestResult> results); + + /// A result without attempts yet. + [[nodiscard]] TestResult + ImportedTest(std::string name, std::string className); + + /// One attempt as a report describes it. A failure gets an assertion + /// record with `message` and `details`, so the analysis treats it as it + /// treats a failure of Catch3's own; `category` defaults by outcome. + [[nodiscard]] Attempt + ImportedAttempt(Outcome outcome, + std::int64_t durationMicroseconds, + std::string message = {}, + std::string details = {}, + FailureCategory category = FailureCategory::kNone); + + /// Seconds as a report gives them; what makes no sense as a duration + /// counts as zero. + [[nodiscard]] std::int64_t + ImportedMicroseconds(double seconds); + + [[nodiscard]] std::string_view + ImportTrim(std::string_view text); + + [[nodiscard]] bool + ImportStartsWith(std::string_view text, std::string_view prefix); + + /// The lines of a text, without their line endings. + [[nodiscard]] std::vector<std::string_view> + ImportLines(std::string_view text); + + /// A format of Catch3's own or CTRF, from a parsed document. + [[nodiscard]] std::optional<ImportFormat> + ImportJsonFormat(const JsonValue &value); + + /// Whether a line reports a test in TAP. + [[nodiscard]] bool + IsTapTestLine(std::string_view line); + + /// A `--json` report or catch3-results lines. + [[nodiscard]] Expected<std::vector<TestResult>> + ParseCatch3Report(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseCtrf(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseTap(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseJUnit(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseXUnit(std::string_view text); + + [[nodiscard]] Expected<std::vector<TestResult>> + ParseTrx(std::string_view text); +} // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/ResultImportJson.cpp b/src/Progmasoft/Runner/ResultImportJson.cpp new file mode 100644 index 00000000..2cdc2019 --- /dev/null +++ b/src/Progmasoft/Runner/ResultImportJson.cpp @@ -0,0 +1,279 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + +// The JSON reports: Catch3's own and CTRF. +#include <Progmasoft/Runner/Json.hpp> +#include <Progmasoft/Runner/ResultImport.hpp> +#include <Progmasoft/Runner/ResultsJson.hpp> +#include <utility> + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + namespace ImportJson + { + /// A format of Catch3's own or CTRF, from the parsed document. + std::optional<ImportFormat> + FormatOf(const JsonValue &value) + { + if (value["format"].AsString() == "catch3-run" + || value["format"].AsString() == "catch3-results") + { + return ImportFormat::kCatch3; + } + if (value["results"]["tests"].IsArray()) + { + return ImportFormat::kCtrf; + } + return std::nullopt; + } + + /// What a test owes the verdict of an earlier run is worked out + /// again here, from this run's point of view. + void + Forget(TestResult &test) + { + test.quarantined = false; + test.history = FailureHistory::kUnknown; + test.selectionReason.clear(); + test.signature.clear(); + test.analysis.clear(); + test.replayCommand.clear(); + test.predictedDurationMicroseconds = 0; + test.predictedFailureProbability = 0.0; + } + + Expected<TestResult> + ReadLine(const JsonValue &value) + { + TestResult test = TestResultFromJson(value); + Forget(test); + if (test.name.empty()) + { + test.name = test.id; + } + if (test.name.empty()) + { + return Error{ "a test needs an 'id' or a 'name'" }; + } + const std::string &outcome = value["outcome"].AsString(); + if (!OutcomeFromName(outcome)) + { + return Error{ "'" + test.name + "' has the outcome '" + + outcome + "'" }; + } + if (test.attempts.empty()) + { + Attempt attempt = ImportedAttempt( + test.outcome, + value["durationMicroseconds"].AsInteger(), + value["message"].AsString(), + value["details"].AsString(), + test.category); + attempt.standardOutput = value["standardOutput"].AsString(); + attempt.standardError = value["standardError"].AsString(); + test.attempts.push_back(std::move(attempt)); + } + return test; + } + + Expected<std::vector<TestResult>> + ParseLines(std::string_view text) + { + std::vector<TestResult> tests; + std::size_t number = 0; + bool started = false; + for (const auto line : ImportLines(text)) + { + ++number; + if (ImportTrim(line).empty()) + { + continue; + } + const std::string where = "line " + std::to_string(number); + auto value = ParseJson(line); + if (!value) + { + return value.GetError().Context(where); + } + if (!started) + { + started = true; + if ((*value)["format"].AsString() != "catch3-results") + { + return Error{ "the first line must name the format " + "catch3-results" }; + } + if ((*value)["version"].AsInteger() + > kResultsFormatVersion) + { + return Error{ "the format is newer than this " + "version of Catch3 reads" }; + } + continue; + } + auto test = ReadLine(*value); + if (!test) + { + return test.GetError().Context(where); + } + tests.push_back(std::move(*test)); + } + return tests; + } + + Expected<std::vector<TestResult>> + ParseCatch3(std::string_view text) + { + auto whole = ParseJson(text); + if (!whole) + { + return ParseLines(text); + } + if ((*whole)["format"].AsString() != "catch3-run") + { + return ParseLines(text); + } + auto run = RunResultFromJson(*whole); + if (!run) + { + return run.GetError(); + } + for (auto &test : run->tests) + { + Forget(test); + } + return std::move(run->tests); + } + + std::string + TextOf(const JsonValue &value) + { + if (!value.IsArray()) + { + return value.AsString(); + } + std::string text; + for (const auto &item : value.Items()) + { + text += item.AsString() + "\n"; + } + return text; + } + + Outcome + CtrfOutcome(const std::string &status) + { + if (status == "passed") + { + return Outcome::kPassed; + } + if (status == "failed") + { + return Outcome::kFailed; + } + if (status == "skipped" || status == "pending") + { + return Outcome::kSkipped; + } + return Outcome::kNotRun; + } + + /// A test, or one retry of it: the two share these fields. + Attempt + CtrfAttempt(const JsonValue &value) + { + Attempt attempt = ImportedAttempt( + CtrfOutcome(value["status"].AsString()), + ImportedMicroseconds(value["duration"].AsNumber() / 1e3), + value["message"].AsString(), + value["trace"].AsString()); + attempt.standardOutput = TextOf(value["stdout"]); + attempt.standardError = TextOf(value["stderr"]); + return attempt; + } + + std::string + CtrfSuite(const JsonValue &suite) + { + if (!suite.IsArray()) + { + return suite.AsString(); + } + std::string text; + for (const auto &item : suite.Items()) + { + text += (text.empty() ? "" : " > ") + item.AsString(); + } + return text; + } + + Expected<std::vector<TestResult>> + ParseCtrf(std::string_view text) + { + auto document = ParseJson(text); + if (!document) + { + return document.GetError(); + } + const JsonValue &tests = (*document)["results"]["tests"]; + if (!tests.IsArray()) + { + return Error{ "a CTRF report has results.tests" }; + } + std::vector<TestResult> results; + for (const auto &item : tests.Items()) + { + const std::string &name = item["name"].AsString(); + if (name.empty()) + { + return Error{ "a CTRF test has no name" }; + } + TestResult test + = ImportedTest(name, CtrfSuite(item["suite"])); + test.file = item["filePath"].AsString(); + test.line + = static_cast<std::size_t>(item["line"].AsUnsigned()); + test.flaky = item["flaky"].AsBoolean(); + for (const auto &tag : item["tags"].Items()) + { + test.tags.push_back(tag.AsString()); + } + for (const auto &retry : item["retryAttempts"].Items()) + { + test.attempts.push_back(CtrfAttempt(retry)); + } + // The retries may or may not list the last attempt, which + // is the test itself. + Attempt last = CtrfAttempt(item); + if (test.attempts.empty() + || test.attempts.back().outcome != last.outcome) + { + test.attempts.push_back(std::move(last)); + } + results.push_back(std::move(test)); + } + return results; + } + } // namespace ImportJson + } // namespace + + std::optional<ImportFormat> + ImportJsonFormat(const JsonValue &value) + { + return ImportJson::FormatOf(value); + } + + Expected<std::vector<TestResult>> + ParseCatch3Report(std::string_view text) + { + return ImportJson::ParseCatch3(text); + } + + Expected<std::vector<TestResult>> + ParseCtrf(std::string_view text) + { + return ImportJson::ParseCtrf(text); + } +} // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/ResultImportTap.cpp b/src/Progmasoft/Runner/ResultImportTap.cpp new file mode 100644 index 00000000..a5f44046 --- /dev/null +++ b/src/Progmasoft/Runner/ResultImportTap.cpp @@ -0,0 +1,338 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + +// The Test Anything Protocol, versions 12 to 14. +#include <Progmasoft/Runner/ResultImport.hpp> +#include <algorithm> +#include <cctype> +#include <cstdlib> +#include <map> +#include <utility> + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + namespace ImportTap + { + std::string + Lowered(std::string_view text) + { + std::string lowered{ text }; + std::transform(lowered.begin(), + lowered.end(), + lowered.begin(), + [](unsigned char character) { + return static_cast<char>( + std::tolower(character)); + }); + return lowered; + } + + struct Line final + { + bool ok = false; + std::string description; + std::string directive; + std::string reason; + }; + + /// Where the `#` that starts a directive is: `\#` is part of the + /// description. + std::size_t + DirectiveStart(std::string_view text) + { + for (std::size_t index = 0; index < text.size(); ++index) + { + if (text[index] == '\\') + { + ++index; + } + else if (text[index] == '#') + { + return index; + } + } + return std::string_view::npos; + } + + std::string + Unescaped(std::string_view text) + { + std::string result; + for (std::size_t index = 0; index < text.size(); ++index) + { + if (text[index] == '\\' && index + 1 < text.size()) + { + ++index; + } + result.push_back(text[index]); + } + return result; + } + + /// Reads a test line; nothing when `text` is not one. + std::optional<Line> + ReadLine(std::string_view text) + { + Line line; + if (ImportStartsWith(text, "not ok")) + { + text.remove_prefix(6); + } + else if (ImportStartsWith(text, "ok")) + { + line.ok = true; + text.remove_prefix(2); + } + else + { + return std::nullopt; + } + if (!text.empty() && text.front() != ' ' + && text.front() != '\t') + { + return std::nullopt; + } + text = ImportTrim(text); + while (!text.empty() + && std::isdigit(static_cast<unsigned char>(text.front())) + != 0) + { + text.remove_prefix(1); + } + text = ImportTrim(text); + if (ImportStartsWith(text, "- ")) + { + text.remove_prefix(2); + } + const auto directive = DirectiveStart(text); + line.description + = Unescaped(ImportTrim(text.substr(0, directive))); + if (directive != std::string_view::npos) + { + const std::string_view rest + = ImportTrim(text.substr(directive + 1)); + const std::string word = Lowered(rest.substr(0, 4)); + if (word == "skip" || word == "todo") + { + line.directive = word; + const auto space = rest.find_first_of(" \t"); + line.reason = space == std::string_view::npos + ? std::string{} + : std::string{ ImportTrim( + rest.substr(space)) }; + } + } + return line; + } + + Outcome + OutcomeOf(const Line &line) + { + if (line.directive == "skip") + { + return Outcome::kSkipped; + } + if (line.ok) + { + return Outcome::kPassed; + } + return line.directive == "todo" ? Outcome::kFailedAsExpected + : Outcome::kFailed; + } + + std::string + Unquoted(std::string_view text) + { + text = ImportTrim(text); + if (text.size() >= 2 && text.front() == text.back() + && (text.front() == '"' || text.front() == '\'')) + { + text.remove_prefix(1); + text.remove_suffix(1); + } + return std::string{ text }; + } + + /// A YAML block after a test: its text goes into the failure, and + /// the two keys that tools agree on are read. + void + ApplyDiagnostics(TestResult &test, const std::string &block) + { + Attempt &attempt = test.attempts.back(); + for (const auto raw : ImportLines(block)) + { + const std::string_view line = ImportTrim(raw); + if (ImportStartsWith(line, "duration_ms:")) + { + const std::string value{ line.substr(12) }; + attempt.durationMicroseconds = ImportedMicroseconds( + std::strtod(value.c_str(), nullptr) / 1e3); + } + else if (ImportStartsWith(line, "message:") + && !attempt.assertions.empty()) + { + attempt.assertions.front().message + = Unquoted(line.substr(8)); + } + } + if (!attempt.assertions.empty()) + { + attempt.assertions.front().expansion += block; + } + } + + Expected<std::vector<TestResult>> + Parse(std::string_view text) + { + std::vector<TestResult> tests; + std::vector<std::string> subtests; + std::map<std::string, std::size_t> seen; + std::optional<std::size_t> planned; + std::size_t topLevel = 0; + std::string bailOut; + bool anything = false; + std::optional<std::string> block; + for (const auto raw : ImportLines(text)) + { + const std::string_view line = ImportTrim(raw); + if (block) + { + if (line == "...") + { + if (!tests.empty()) + { + ApplyDiagnostics(tests.back(), *block); + } + block.reset(); + } + else + { + block->append(raw).push_back('\n'); + } + continue; + } + if (line == "---") + { + block.emplace(); + continue; + } + std::size_t indent = 0; + for (const char character : raw) + { + if (character != ' ' && character != '\t') + { + break; + } + indent += character == '\t' ? 4 : 1; + } + const std::size_t depth = indent / 4; + if (ImportStartsWith(line, "# Subtest:")) + { + subtests.resize(depth); + subtests.emplace_back(ImportTrim(line.substr(10))); + continue; + } + if (ImportStartsWith(line, "Bail out!")) + { + anything = true; + bailOut = std::string{ line }; + break; + } + if (ImportStartsWith(line, "1..") && depth == 0) + { + anything = true; + const std::string count{ line.substr(3) }; + planned = static_cast<std::size_t>( + std::strtoull(count.c_str(), nullptr, 10)); + continue; + } + if (line.size() > 1 && line.front() == '#' && !tests.empty() + && !tests.back().attempts.back().assertions.empty()) + { + // Comments after a failure usually explain it. + auto &failure + = tests.back().attempts.back().assertions.front(); + failure.expansion + += std::string{ ImportTrim(line.substr(1)) } + "\n"; + continue; + } + const auto parsed = ReadLine(line); + if (!parsed) + { + continue; + } + anything = true; + topLevel += depth == 0 ? 1 : 0; + std::string parent; + for (std::size_t level = 0; + level < std::min(depth, subtests.size()); + ++level) + { + parent + += (parent.empty() ? "" : " > ") + subtests[level]; + } + std::string name = parsed->description; + if (name.empty()) + { + name = "test " + + std::to_string(++seen["unnamed\n" + parent]); + } + // TAP has no retries: a name that comes back in one + // stream is another test. + const std::size_t repeat = ++seen[parent + "\n" + name]; + if (repeat > 1) + { + name += " #" + std::to_string(repeat); + } + TestResult test + = ImportedTest(std::move(name), std::move(parent)); + test.attempts.push_back( + ImportedAttempt(OutcomeOf(*parsed), 0, parsed->reason)); + tests.push_back(std::move(test)); + } + if (!anything) + { + return Error{ "there are no TAP results in it" }; + } + // A stream that ends early must not read as a run that passed. + std::string incomplete; + if (!bailOut.empty()) + { + incomplete = bailOut; + } + else if (planned && *planned != topLevel) + { + incomplete = "planned " + std::to_string(*planned) + + " test(s), reported " + + std::to_string(topLevel); + } + if (!incomplete.empty()) + { + TestResult test = ImportedTest("the TAP stream", {}); + test.attempts.push_back( + ImportedAttempt(Outcome::kFailed, + 0, + std::move(incomplete), + {}, + FailureCategory::kInfrastructure)); + tests.push_back(std::move(test)); + } + return tests; + } + } // namespace ImportTap + } // namespace + + bool + IsTapTestLine(std::string_view line) + { + return ImportTap::ReadLine(line).has_value(); + } + + Expected<std::vector<TestResult>> + ParseTap(std::string_view text) + { + return ImportTap::Parse(text); + } +} // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/ResultImportXml.cpp b/src/Progmasoft/Runner/ResultImportXml.cpp new file mode 100644 index 00000000..ddbed989 --- /dev/null +++ b/src/Progmasoft/Runner/ResultImportXml.cpp @@ -0,0 +1,507 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + +// The XML reports: JUnit, xUnit.net v2 and Visual Studio's TRX. +#include <Progmasoft/Runner/ResultImport.hpp> +#include <cstdlib> +#include <map> +#include <string_view> +#include <utility> + +#include "../PugiXml.inl" + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + namespace ImportXml + { + Expected<bool> + Load(pugi::xml_document &document, std::string_view text) + { + const pugi::xml_parse_result parsed + = document.load_buffer(text.data(), text.size()); + if (!parsed) + { + return Error{ std::string{ "it is not valid XML: " } + + parsed.description() + " at byte " + + std::to_string(parsed.offset) }; + } + return true; + } + + double + NumberOf(const char *text) + { + std::string digits; + for (; *text != '\0'; ++text) + { + if (*text != ',') + { + digits.push_back(*text); + } + } + return std::strtod(digits.c_str(), nullptr); + } + + std::string + TextOf(pugi::xml_node node) + { + return node.text().get(); + } + + /// `error` and `failure` of JUnit and what Surefire writes for + /// retries, which have the same shape. + Attempt + JUnitFailure(pugi::xml_node node, FailureCategory category) + { + std::string details = TextOf(node); + if (const auto trace = node.child("stackTrace")) + { + details = TextOf(trace); + } + Attempt attempt + = ImportedAttempt(Outcome::kFailed, + ImportedMicroseconds(NumberOf( + node.attribute("time").value())), + node.attribute("message").value(), + std::move(details), + category); + attempt.assertions.front().macro + = node.attribute("type").value(); + attempt.standardOutput = TextOf(node.child("system-out")); + attempt.standardError = TextOf(node.child("system-err")); + return attempt; + } + + bool + NameIs(pugi::xml_node node, std::string_view name) + { + return name == node.name(); + } + + bool + NameEndsWithError(pugi::xml_node node) + { + return NameIs(node, "flakyError") || NameIs(node, "rerunError"); + } + + bool + NotRunStatus(pugi::xml_node testCase) + { + const std::string_view status + = testCase.attribute("status").value(); + const std::string_view result + = testCase.attribute("result").value(); + return status == "notrun" || status == "skipped" + || status == "disabled" || result == "skipped" + || result == "suppressed"; + } + + void + ReadJUnitCase(pugi::xml_node testCase, + pugi::xml_node suite, + std::vector<TestResult> &tests) + { + TestResult test + = ImportedTest(testCase.attribute("name").value(), + testCase.attribute("classname").value()); + test.file = testCase.attribute("file").value(); + for (auto outer = suite; test.file.empty() && outer; + outer = outer.parent()) + { + test.file = outer.attribute("file").value(); + } + test.line = testCase.attribute("line").as_ullong(); + + std::vector<Attempt> failures; + std::optional<Attempt> skipped; + for (const auto child : testCase.children()) + { + if (NameIs(child, "flakyFailure") + || NameIs(child, "rerunFailure") + || NameEndsWithError(child)) + { + test.attempts.push_back(JUnitFailure( + child, + NameEndsWithError(child) + ? FailureCategory::kUnexpectedException + : FailureCategory::kAssertion)); + } + else if (NameIs(child, "failure")) + { + failures.push_back( + JUnitFailure(child, FailureCategory::kAssertion)); + } + else if (NameIs(child, "error")) + { + failures.push_back(JUnitFailure( + child, + FailureCategory::kUnexpectedException)); + } + else if (NameIs(child, "skipped")) + { + std::string reason = child.attribute("message").value(); + skipped = ImportedAttempt( + Outcome::kSkipped, + 0, + reason.empty() ? TextOf(child) : std::move(reason)); + } + } + + const std::int64_t duration = ImportedMicroseconds( + NumberOf(testCase.attribute("time").value())); + Attempt verdict; + if (!failures.empty()) + { + // Several failures are one attempt with several findings. + verdict = std::move(failures.front()); + for (std::size_t index = 1; index < failures.size(); + ++index) + { + auto &record = failures[index].assertions.front(); + verdict.assertions.push_back(std::move(record)); + ++verdict.assertionsFailed; + } + verdict.durationMicroseconds = duration; + } + else if (skipped) + { + verdict = std::move(*skipped); + } + else if (NotRunStatus(testCase)) + { + verdict = ImportedAttempt(Outcome::kSkipped, 0); + } + else + { + verdict = ImportedAttempt(Outcome::kPassed, duration); + } + if (verdict.standardOutput.empty()) + { + verdict.standardOutput + = TextOf(testCase.child("system-out")); + } + if (verdict.standardError.empty()) + { + verdict.standardError + = TextOf(testCase.child("system-err")); + } + test.attempts.push_back(std::move(verdict)); + tests.push_back(std::move(test)); + } + + void + ReadJUnitSuite(pugi::xml_node suite, std::vector<TestResult> &tests) + { + for (const auto child : suite.children()) + { + if (NameIs(child, "testcase")) + { + ReadJUnitCase(child, suite, tests); + } + else if (NameIs(child, "testsuite")) + { + ReadJUnitSuite(child, tests); + } + } + } + + void + ReadXUnitTest(pugi::xml_node node, std::vector<TestResult> &tests) + { + TestResult test = ImportedTest(node.attribute("name").value(), + node.attribute("type").value()); + test.file = node.attribute("source-file").value(); + test.line = node.attribute("source-line").as_ullong(); + for (const auto trait : node.child("traits").children("trait")) + { + test.tags.push_back( + std::string{ trait.attribute("name").value() } + "=" + + trait.attribute("value").value()); + } + const std::string_view result + = node.attribute("result").value(); + const std::int64_t duration = ImportedMicroseconds( + NumberOf(node.attribute("time").value())); + Attempt attempt; + if (result == "Pass") + { + attempt = ImportedAttempt(Outcome::kPassed, duration); + } + else if (result == "Fail") + { + const auto failure = node.child("failure"); + attempt + = ImportedAttempt(Outcome::kFailed, + duration, + TextOf(failure.child("message")), + TextOf(failure.child("stack-trace"))); + attempt.assertions.front().macro + = failure.attribute("exception-type").value(); + } + else if (result == "Skip") + { + attempt = ImportedAttempt(Outcome::kSkipped, + 0, + TextOf(node.child("reason"))); + } + else + { + attempt = ImportedAttempt(Outcome::kNotRun, 0); + } + if (attempt.standardOutput.empty()) + { + attempt.standardOutput = TextOf(node.child("output")); + } + test.attempts.push_back(std::move(attempt)); + tests.push_back(std::move(test)); + } + + /// A failure outside any test, such as in a fixture. + void + ReadXUnitErrors(pugi::xml_node assembly, + std::vector<TestResult> &tests) + { + for (const auto error : + assembly.child("errors").children("error")) + { + const auto failure = error.child("failure"); + TestResult test = ImportedTest( + std::string{ error.attribute("type").value() } + " " + + error.attribute("name").value(), + assembly.attribute("name").value()); + test.attempts.push_back( + ImportedAttempt(Outcome::kFailed, + 0, + TextOf(failure.child("message")), + TextOf(failure.child("stack-trace")), + FailureCategory::kInfrastructure)); + tests.push_back(std::move(test)); + } + } + + void + ReadXUnitAssembly(pugi::xml_node assembly, + std::vector<TestResult> &tests) + { + ReadXUnitErrors(assembly, tests); + for (const auto child : assembly.children()) + { + if (NameIs(child, "test")) + { + ReadXUnitTest(child, tests); + } + else if (NameIs(child, "collection")) + { + for (const auto test : child.children("test")) + { + ReadXUnitTest(test, tests); + } + } + } + } + + /// `[days.]hours:minutes:seconds[.fraction]`. + std::int64_t + TrxDuration(std::string_view text) + { + double seconds = 0.0; + std::string field; + const auto addField = [&] { + seconds = seconds * 60.0 + NumberOf(field.c_str()); + field.clear(); + }; + double days = 0.0; + for (std::size_t index = 0; index < text.size(); ++index) + { + if (text[index] == ':') + { + addField(); + } + else if (text[index] == '.' + && text.find(':', index) != std::string_view::npos) + { + days = NumberOf(field.c_str()); + field.clear(); + } + else + { + field.push_back(text[index]); + } + } + addField(); + return ImportedMicroseconds(seconds + days * 86400.0); + } + + struct TrxOutcome final + { + Outcome outcome = Outcome::kNotRun; + FailureCategory category = FailureCategory::kNone; + }; + + TrxOutcome + TrxOutcomeOf(std::string_view outcome) + { + if (outcome == "Passed" || outcome == "PassedButRunAborted" + || outcome == "Warning" || outcome == "Completed") + { + return { Outcome::kPassed, FailureCategory::kNone }; + } + if (outcome == "Failed") + { + return { Outcome::kFailed, FailureCategory::kAssertion }; + } + if (outcome == "Error") + { + return { Outcome::kFailed, + FailureCategory::kUnexpectedException }; + } + if (outcome == "Timeout") + { + return { Outcome::kTimedOut, FailureCategory::kTimeout }; + } + if (outcome == "Aborted") + { + return { Outcome::kCrashed, FailureCategory::kCrash }; + } + if (outcome == "Disconnected") + { + return { Outcome::kCrashed, + FailureCategory::kInfrastructure }; + } + if (outcome == "NotExecuted" || outcome == "Inconclusive" + || outcome == "NotRunnable") + { + return { Outcome::kSkipped, FailureCategory::kNone }; + } + return {}; + } + + /// The class of a test: the type without its assembly. + std::string + TrxClass(std::string_view className) + { + return std::string{ className.substr(0, className.find(',')) }; + } + + void + ReadTrxResult(pugi::xml_node result, + const std::map<std::string, std::string> &classes, + std::vector<TestResult> &tests) + { + // A data-driven test: the rows are what ran. + if (const auto inner = result.child("InnerResults")) + { + for (const auto row : inner.children()) + { + ReadTrxResult(row, classes, tests); + } + return; + } + const auto found + = classes.find(result.attribute("testId").value()); + TestResult test = ImportedTest( + result.attribute("testName").value(), + found == classes.end() ? std::string{} : found->second); + const auto [outcome, category] + = TrxOutcomeOf(result.attribute("outcome").value()); + const auto output = result.child("Output"); + const auto error = output.child("ErrorInfo"); + std::string message = TextOf(error.child("Message")); + Attempt attempt = ImportedAttempt( + outcome, + TrxDuration(result.attribute("duration").value()), + outcome == Outcome::kSkipped + ? TextOf(output.child("StdOut")) + : std::move(message), + TextOf(error.child("StackTrace")), + category); + if (outcome != Outcome::kSkipped) + { + attempt.standardOutput = TextOf(output.child("StdOut")); + } + attempt.standardError = TextOf(output.child("StdErr")); + test.attempts.push_back(std::move(attempt)); + tests.push_back(std::move(test)); + } + } // namespace ImportXml + } // namespace + + Expected<std::vector<TestResult>> + ParseJUnit(std::string_view text) + { + pugi::xml_document document; + if (auto loaded = ImportXml::Load(document, text); !loaded) + { + return loaded.GetError(); + } + const pugi::xml_node root = document.document_element(); + std::vector<TestResult> tests; + if (ImportXml::NameIs(root, "testsuites") + || ImportXml::NameIs(root, "testsuite")) + { + ImportXml::ReadJUnitSuite(root, tests); + return tests; + } + return Error{ "a JUnit report has testsuites or testsuite as its root, " + "not " + + std::string{ root.name() } }; + } + + Expected<std::vector<TestResult>> + ParseXUnit(std::string_view text) + { + pugi::xml_document document; + if (auto loaded = ImportXml::Load(document, text); !loaded) + { + return loaded.GetError(); + } + const pugi::xml_node root = document.document_element(); + std::vector<TestResult> tests; + if (ImportXml::NameIs(root, "assembly")) + { + ImportXml::ReadXUnitAssembly(root, tests); + return tests; + } + if (ImportXml::NameIs(root, "assemblies")) + { + for (const auto assembly : root.children("assembly")) + { + ImportXml::ReadXUnitAssembly(assembly, tests); + } + return tests; + } + return Error{ "an xUnit report has assemblies or assembly as its root, " + "not " + + std::string{ root.name() } }; + } + + Expected<std::vector<TestResult>> + ParseTrx(std::string_view text) + { + pugi::xml_document document; + if (auto loaded = ImportXml::Load(document, text); !loaded) + { + return loaded.GetError(); + } + const pugi::xml_node root = document.document_element(); + if (!ImportXml::NameIs(root, "TestRun")) + { + return Error{ "a TRX report has TestRun as its root, not " + + std::string{ root.name() } }; + } + std::map<std::string, std::string> classes; + for (const auto definition : root.child("TestDefinitions").children()) + { + classes[definition.attribute("id").value()] = ImportXml::TrxClass( + definition.child("TestMethod").attribute("className").value()); + } + std::vector<TestResult> tests; + for (const auto result : root.child("Results").children()) + { + ImportXml::ReadTrxResult(result, classes, tests); + } + return tests; + } +} // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/Runner.cpp b/src/Progmasoft/Runner/Runner.cpp index 6dde30c7..ae711293 100644 --- a/src/Progmasoft/Runner/Runner.cpp +++ b/src/Progmasoft/Runner/Runner.cpp @@ -17,6 +17,7 @@ #include <cstdio> #include <exception> #include <filesystem> +#include <iterator> #include <map> #include <random> #include <system_error> @@ -34,6 +35,7 @@ #include "Planner.hpp" #include "Process.hpp" #include "ProjectConfig.hpp" +#include "ResultImport.hpp" #include "ResultsJson.hpp" #include "Scheduler.hpp" @@ -514,6 +516,215 @@ namespace Progmasoft::Catch3 .count(); } + /// Writes the reports that were asked for. One that is missing would + /// let a build server miss the run, so it fails the run. + bool + WriteReports(const Runner::Options &options, const RunResult &run) + { + bool written = true; + const auto write = [&](const std::string &path, + const std::string &contents) { + if (auto status = Runner::WriteFileAtomically(path, contents); + !status) + { + PrintError(status.GetError().Message()); + written = false; + } + }; + if (!options.junitPath.empty()) + { + write(options.junitPath, JUnitXmlWriter::ToString(run)); + } + if (!options.jsonPath.empty()) + { + write(options.jsonPath, Runner::ToJson(run).ToPrettyString()); + } + if (!options.tracePath.empty()) + { + write(options.tracePath, Runner::TraceJson(run)); + } + return written; + } + + void + SaveHistory(const std::filesystem::path &file, + const RunResult &run, + const std::string &commit, + const std::string &binaryFingerprint, + const std::map<std::string, std::string> &fingerprints) + { + if (run.tests.empty()) + { + return; + } + Runner::RunRecord record; + record.unixMilliseconds = run.startUnixMilliseconds; + record.commit = commit; + record.binaryFingerprint = binaryFingerprint; + record.seed = run.seed; + record.tests = run.tests.size(); + record.wallMicroseconds = run.wallMicroseconds; + for (const auto &test : run.tests) + { + record.failed += test.FailsRun() ? 1 : 0; + } + auto committed = Runner::History::Commit(file, + run, + record, + fingerprints, + {}); + if (!committed) + { + std::fprintf(stderr, + "warning: the history was not saved: %s\n", + committed.GetError().Message().c_str()); + } + } + + int + ExitCodeOf(const Runner::Options &options, + const RunResult &run, + bool reportMissing) + { + const bool quarantineCounts + = options.quarantine == Runner::QuarantineMode::kReport; + bool failed = false; + std::size_t ran = 0; + std::size_t skipped = 0; + for (const auto &test : run.tests) + { + if (test.outcome != Outcome::kUpToDate) + { + ++ran; + } + skipped += test.outcome == Outcome::kSkipped ? 1 : 0; + failed = failed || test.FailsRun() + || (quarantineCounts && test.quarantined + && IsFailure(test.outcome)); + } + if (failed) + { + return Catch::TestFailureExitCode; + } + if (reportMissing) + { + return Catch::UnspecifiedErrorExitCode; + } + if (run.tests.empty() && !options.allowNoTests) + { + return Catch::NoTestsRunExitCode; + } + if (ran > 0 && skipped == ran && !options.allowNoTests) + { + return Catch::AllTestsSkippedExitCode; + } + return 0; + } + + /// The file results are remembered in. + std::filesystem::path + HistoryFileOf(const Runner::Options &options, + const std::filesystem::path &program, + const std::string &programName) + { + std::string name = options.importName; + if (name.empty()) + { + name = options.mode == Runner::Mode::kImport ? "imported" + : programName; + } + return Runner::HistoryFile(HistoryDirectory(options, program), + name); + } + + /// Reports of other frameworks: `--import`. What they say goes + /// through the analysis, the history and the reports of a run. + int + Import(const Runner::Options &options, + const std::filesystem::path &program, + const std::string &programName) + { + std::vector<TestResult> imported; + for (const auto &file : options.importFiles) + { + auto text = Runner::ReadFile(file); + if (!text) + { + PrintError(text.GetError().Message()); + return Catch::UnspecifiedErrorExitCode; + } + auto tests = Runner::ParseImport(*text, options.importFormat); + if (!tests) + { + PrintError(tests.GetError().Context(file).Message()); + return Catch::UnspecifiedErrorExitCode; + } + imported.insert(imported.end(), + std::make_move_iterator(tests->begin()), + std::make_move_iterator(tests->end())); + } + + const std::filesystem::path historyFile + = HistoryFileOf(options, program, programName); + Runner::History history; + RunResult run; + run.binary = options.importName.empty() ? "imported" + : options.importName; + if (options.historyEnabled) + { + auto loaded = Runner::History::Load(historyFile); + if (loaded) + { + history = std::move(*loaded); + } + else + { + run.notes.push_back("the history could not be read: " + + loaded.GetError().Message()); + } + } + + run.tests = Runner::MergeImported(std::move(imported)); + run.candidates = run.tests.size(); + run.startUnixMilliseconds = UnixMilliseconds(); + std::error_code here; + const std::string revision + = options.importRevision.empty() + ? Runner::CurrentCommit( + std::filesystem::current_path(here)) + : options.importRevision; + std::map<std::string, std::string> fingerprints; + for (auto &test : run.tests) + { + run.wallMicroseconds += test.DurationMicroseconds(); + fingerprints[test.id] = revision; + const Runner::TestHistory *past = history.Find(test.id); + test.quarantined + = options.quarantine != Runner::QuarantineMode::kOff + && past != nullptr && past->quarantine.has_value(); + } + + Runner::AnalysisInputs analysis; + analysis.history = options.historyEnabled ? &history : nullptr; + Runner::Analyze(run, analysis); + + Runner::ConsoleReport console{ options, stdout }; + console.ImportStarting(run.tests.size(), + options.importFiles.size()); + for (const auto &test : run.tests) + { + console.TestFinished(test); + } + console.RunEnded(run); + + const bool reportMissing = !WriteReports(options, run); + if (options.historyEnabled) + { + SaveHistory(historyFile, run, revision, {}, fingerprints); + } + return ExitCodeOf(options, run, reportMissing); + } + int Run(const std::vector<std::string> &arguments) { @@ -561,6 +772,10 @@ namespace Progmasoft::Catch3 { return Runner::RunWorker(options, programName); } + if (options.mode == Runner::Mode::kImport) + { + return Import(options, program, programName); + } std::vector<std::string> notes; if (!options.configPath.empty() @@ -626,8 +841,7 @@ namespace Progmasoft::Catch3 } const std::filesystem::path historyFile - = Runner::HistoryFile(HistoryDirectory(options, program), - programName); + = HistoryFileOf(options, program, programName); Runner::History history; if (options.historyEnabled || options.mode == Runner::Mode::kAnalytics) @@ -778,66 +992,14 @@ namespace Progmasoft::Catch3 Runner::Analyze(run, analysis); console.RunEnded(run); - // A report that was asked for and is missing would let a build - // server miss the run, so it fails the run. - bool reportMissing = false; - if (!options.junitPath.empty()) - { - if (auto status = Runner::WriteFileAtomically( - options.junitPath, - JUnitXmlWriter::ToString(run)); - !status) - { - PrintError(status.GetError().Message()); - reportMissing = true; - } - } - if (!options.jsonPath.empty()) - { - if (auto status = Runner::WriteFileAtomically( - options.jsonPath, - Runner::ToJson(run).ToPrettyString()); - !status) - { - PrintError(status.GetError().Message()); - reportMissing = true; - } - } - if (!options.tracePath.empty()) + const bool reportMissing = !WriteReports(options, run); + if (options.historyEnabled) { - if (auto status - = Runner::WriteFileAtomically(options.tracePath, - Runner::TraceJson(run)); - !status) - { - PrintError(status.GetError().Message()); - reportMissing = true; - } - } - if (options.historyEnabled && !run.tests.empty()) - { - Runner::RunRecord record; - record.unixMilliseconds = run.startUnixMilliseconds; - record.commit = changes ? changes->commit : std::string{}; - record.binaryFingerprint = binaryFingerprint; - record.seed = run.seed; - record.tests = run.tests.size(); - record.wallMicroseconds = run.wallMicroseconds; - for (const auto &test : run.tests) - { - record.failed += test.FailsRun() ? 1 : 0; - } - auto committed = Runner::History::Commit(historyFile, - run, - record, - plan->fingerprints, - {}); - if (!committed) - { - std::fprintf(stderr, - "warning: the history was not saved: %s\n", - committed.GetError().Message().c_str()); - } + SaveHistory(historyFile, + run, + changes ? changes->commit : std::string{}, + binaryFingerprint, + plan->fingerprints); } if (!guard.empty()) { @@ -853,39 +1015,7 @@ namespace Progmasoft::Catch3 { return Catch::UnmatchedTestSpecExitCode; } - const bool quarantineCounts - = options.quarantine == Runner::QuarantineMode::kReport; - bool failed = false; - std::size_t ran = 0; - std::size_t skipped = 0; - for (const auto &test : run.tests) - { - if (test.outcome != Outcome::kUpToDate) - { - ++ran; - } - skipped += test.outcome == Outcome::kSkipped ? 1 : 0; - failed = failed || test.FailsRun() - || (quarantineCounts && test.quarantined - && IsFailure(test.outcome)); - } - if (failed) - { - return Catch::TestFailureExitCode; - } - if (reportMissing) - { - return Catch::UnspecifiedErrorExitCode; - } - if (run.tests.empty() && !options.allowNoTests) - { - return Catch::NoTestsRunExitCode; - } - if (ran > 0 && skipped == ran && !options.allowNoTests) - { - return Catch::AllTestsSkippedExitCode; - } - return 0; + return ExitCodeOf(options, run, reportMissing); } } // namespace diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 33bbc4b5..23185534 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -164,6 +164,7 @@ add_executable(ProgmasoftSelfTest Progmasoft/RunnerOptions.tests.cpp Progmasoft/RunnerConfig.tests.cpp Progmasoft/RunnerAnalysis.tests.cpp + Progmasoft/RunnerImport.tests.cpp Progmasoft/RunnerOutput.tests.cpp Progmasoft/Snapshot.tests.cpp Progmasoft/Stateful.tests.cpp diff --git a/tests/Progmasoft/BUILD.bazel b/tests/Progmasoft/BUILD.bazel index a070acf2..0571d862 100644 --- a/tests/Progmasoft/BUILD.bazel +++ b/tests/Progmasoft/BUILD.bazel @@ -18,6 +18,7 @@ cc_test( "RunnerOptions.tests.cpp", "RunnerConfig.tests.cpp", "RunnerAnalysis.tests.cpp", + "RunnerImport.tests.cpp", "RunnerOutput.tests.cpp", "Snapshot.tests.cpp", "Stateful.tests.cpp", diff --git a/tests/Progmasoft/RunnerImport.tests.cpp b/tests/Progmasoft/RunnerImport.tests.cpp new file mode 100644 index 00000000..4eab27f4 --- /dev/null +++ b/tests/Progmasoft/RunnerImport.tests.cpp @@ -0,0 +1,596 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0 + +// Reading the reports of other test frameworks, and what `--import` does with +// them. + +#include <Progmasoft/Catch3/Matchers/String.hpp> +#include <Progmasoft/Runner/History.hpp> +#include <Progmasoft/Runner/Options.hpp> +#include <Progmasoft/Runner/ResultImport.hpp> +#include <Progmasoft/Runner/ResultsJson.hpp> +#include <limits> +#include <string> +#include <string_view> +#include <vector> + +#include "RunnerFixtureClient.hpp" + +namespace +{ + namespace Runner = Progmasoft::Catch3::Runner; + using namespace Progmasoft::Catch3; + using Catch::Matchers::ContainsSubstring; + using FixtureClient::ExitCode; + using FixtureClient::RunFixture; + using FixtureClient::Scratch; + + std::vector<TestResult> + Import(std::string_view text, + Runner::ImportFormat format = Runner::ImportFormat::kAuto) + { + auto tests = Runner::ParseImport(text, format); + INFO((tests ? std::string{} : tests.GetError().Message())); + REQUIRE(tests); + return Runner::MergeImported(std::move(*tests)); + } + + const TestResult & + Named(const std::vector<TestResult> &tests, const std::string &name) + { + for (const auto &test : tests) + { + if (test.name == name) + { + return test; + } + } + FAIL("no result for '" << name << "'"); + return tests.front(); + } + + constexpr std::string_view kJUnit = R"(<?xml version="1.0"?> +<!-- written by some tool --> +<testsuites> + <testsuite name="outer" file="outer.py"> + <testsuite name="inner"> + <testcase classname="dates" name="parses" time="0.25"/> + <testcase classname="dates" name="rounds" time="1,000.5" line="7"> + <failure message="1.5 != 2" type="AssertionError">at rounds:7</failure> + <system-out>said something</system-out> + </testcase> + <testcase classname="dates" name="throws"> + <error type="ValueError"><![CDATA[Traceback +ValueError: no]]></error> + </testcase> + <testcase classname="dates" name="ignored"> + <skipped message="not today"/> + </testcase> + <testcase classname="dates" name="disabled" status="notrun"/> + <testcase classname="dates" name="retried" time="2"> + <flakyFailure message="first try" type="E" time="1"> + <stackTrace>at retried:1</stackTrace> + </flakyFailure> + </testcase> + <testcase classname="dates" name="always"> + <failure message="last"/> + <rerunFailure message="again"/> + </testcase> + </testsuite> + </testsuite> +</testsuites>)"; + + constexpr std::string_view kXUnit = R"(<assemblies> + <assembly name="Tests.dll"> + <errors> + <error type="fixture" name="Dates.Setup"> + <failure><message>no database</message></failure> + </error> + </errors> + <collection name="Dates"> + <test name="Dates.Parses" type="Dates" method="Parses" time="0.5" result="Pass"> + <traits><trait name="kind" value="unit"/></traits> + </test> + <test name="Dates.Rounds" type="Dates" time="0.1" result="Fail" + source-file="Dates.cs" source-line="12"> + <failure exception-type="Xunit.EqualException"> + <message>not equal</message> + <stack-trace>at Dates.Rounds</stack-trace> + </failure> + </test> + <test name="Dates.Later" type="Dates" result="Skip"><reason>later</reason></test> + </collection> + </assembly> +</assemblies>)"; + + constexpr std::string_view kTrx + = R"xml(<?xml version="1.0" encoding="utf-8"?> +<TestRun xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"> + <TestDefinitions> + <UnitTest id="1"><TestMethod className="Ns.Dates, Tests, Version=1.0" name="Parses"/></UnitTest> + <UnitTest id="2"><TestMethod className="Ns.Dates" name="Rounds"/></UnitTest> + <UnitTest id="3"><TestMethod className="Ns.Dates" name="Rows"/></UnitTest> + </TestDefinitions> + <Results> + <UnitTestResult testId="1" testName="Parses" outcome="Passed" duration="00:01:02.5000000"/> + <UnitTestResult testId="2" testName="Rounds" outcome="Failed" duration="1.00:00:00"> + <Output> + <StdOut>said something</StdOut> + <ErrorInfo><Message>Assert.Equal</Message><StackTrace>at Rounds</StackTrace></ErrorInfo> + </Output> + </UnitTestResult> + <UnitTestResult testId="2" testName="Rounds" outcome="Passed" duration="00:00:01"/> + <UnitTestResult testId="3" testName="Rows" outcome="Failed"> + <InnerResults> + <UnitTestResult testId="3" testName="Rows (1)" outcome="Passed"/> + <UnitTestResult testId="3" testName="Rows (2)" outcome="Timeout"/> + <UnitTestResult testId="3" testName="Rows (3)" outcome="Aborted"/> + <UnitTestResult testId="3" testName="Rows (4)" outcome="NotExecuted"/> + </InnerResults> + </UnitTestResult> + </Results> +</TestRun>)xml"; + + constexpr std::string_view kTap = R"(TAP version 14 +1..6 +ok 1 - parses +not ok 2 - rounds + --- + message: "1.5 != 2" + duration_ms: 1500 + ... +# it was close +ok 3 - later # SKIP not today +not ok 4 known bug # TODO fix it +ok 5 +# Subtest: group + ok 1 - parses + 1..1 +ok 6 - parses +)"; + + constexpr std::string_view kCtrf = R"({ + "results": { + "tool": { "name": "vitest" }, + "tests": [ + { "name": "parses", "status": "passed", "duration": 250, "suite": ["dates", "parse"] }, + { "name": "rounds", "status": "failed", "duration": 10, "message": "1.5 != 2", + "trace": "at rounds", "filePath": "dates.test.ts", "line": 7, "tags": ["@unit"], + "stdout": ["one", "two"] }, + { "name": "later", "status": "pending" }, + { "name": "retried", "status": "passed", "flaky": true, "retries": 1, + "retryAttempts": [ { "attempt": 1, "status": "failed", "message": "first" } ] }, + { "name": "listed", "status": "passed", + "retryAttempts": [ { "status": "failed" }, { "status": "passed" } ] } + ] + } +})"; + + constexpr std::string_view kLines + = R"({"format": "catch3-results", "version": 1} +{"name": "parses", "className": "Dates", "outcome": "passed", "durationMicroseconds": 1200} + +{"name": "rounds", "outcome": "failed", "message": "1.5 != 2", "details": "at rounds"} +{"id": "custom-id", "name": "crashes", "outcome": "crashed"} +)"; +} // namespace + +TEST_CASE("The format of a report shows in its contents", + "[progmasoft][runner][import]") +{ + using Runner::DetectImportFormat; + using Runner::ImportFormat; + CHECK(DetectImportFormat(kJUnit) == ImportFormat::kJUnit); + CHECK(DetectImportFormat("\xEF\xBB\xBF <testsuite/>") + == ImportFormat::kJUnit); + CHECK(DetectImportFormat(kXUnit) == ImportFormat::kXUnit); + CHECK(DetectImportFormat(kTrx) == ImportFormat::kTrx); + CHECK(DetectImportFormat(kTap) == ImportFormat::kTap); + CHECK(DetectImportFormat("# comment\nnot ok 1 - x\n") + == ImportFormat::kTap); + CHECK(DetectImportFormat("1..0 # SKIP nothing\n") == ImportFormat::kTap); + CHECK(DetectImportFormat(kCtrf) == ImportFormat::kCtrf); + CHECK(DetectImportFormat(kLines) == ImportFormat::kCatch3); + CHECK(DetectImportFormat(R"({"format": "catch3-run", "tests": []})") + == ImportFormat::kCatch3); + + CHECK_FALSE(DetectImportFormat("")); + CHECK_FALSE(DetectImportFormat("<html></html>")); + CHECK_FALSE(DetectImportFormat("{\"other\": 1}")); + CHECK_FALSE(DetectImportFormat("okay then")); + CHECK_FALSE(DetectImportFormat("{ not json")); +} + +TEST_CASE("Reports that cannot be read say why", "[progmasoft][runner][import]") +{ + using Runner::ImportFormat; + const auto error = [](std::string_view text, ImportFormat format) { + auto tests = Runner::ParseImport(text, format); + REQUIRE_FALSE(tests); + return tests.GetError().Message(); + }; + CHECK_THAT(error("hello", ImportFormat::kAuto), + ContainsSubstring("--import-format")); + CHECK_THAT(error("<testsuite>", ImportFormat::kJUnit), + ContainsSubstring("not valid XML")); + CHECK_THAT(error("<html/>", ImportFormat::kJUnit), + ContainsSubstring("html")); + CHECK_THAT(error("<html/>", ImportFormat::kXUnit), + ContainsSubstring("html")); + CHECK_THAT(error("<html/>", ImportFormat::kTrx), ContainsSubstring("html")); + CHECK_THAT(error("hello\n", ImportFormat::kTap), + ContainsSubstring("no TAP")); + CHECK_THAT(error("{}", ImportFormat::kCtrf), + ContainsSubstring("results.tests")); + CHECK_THAT(error(R"({"results": {"tests": [{"status": "passed"}]}})", + ImportFormat::kCtrf), + ContainsSubstring("no name")); + CHECK_THAT(error("{\"name\": \"x\"}\n", ImportFormat::kCatch3), + ContainsSubstring("catch3-results")); + CHECK_THAT(error("{\"format\": \"catch3-results\", \"version\": 99}\n", + ImportFormat::kCatch3), + ContainsSubstring("newer")); + CHECK_THAT( + error("{\"format\": \"catch3-results\"}\n{\"outcome\": \"passed\"}\n", + ImportFormat::kCatch3), + ContainsSubstring("line 2")); + CHECK_THAT(error("{\"format\": \"catch3-results\"}\n" + "{\"name\": \"x\", \"outcome\": \"green\"}\n", + ImportFormat::kCatch3), + ContainsSubstring("green")); + CHECK_THAT(error("{\"format\": \"catch3-results\"}\n{ broken\n", + ImportFormat::kCatch3), + ContainsSubstring("line 2")); +} + +TEST_CASE("JUnit reports are imported", "[progmasoft][runner][import]") +{ + const auto tests = Import(kJUnit); + REQUIRE(tests.size() == 7); + + const auto &parses = Named(tests, "parses"); + CHECK(parses.id == "dates::parses"); + CHECK(parses.className == "dates"); + CHECK(parses.outcome == Outcome::kPassed); + CHECK(parses.file == "outer.py"); + CHECK(parses.attempts.front().durationMicroseconds == 250000); + + const auto &rounds = Named(tests, "rounds"); + CHECK(rounds.outcome == Outcome::kFailed); + CHECK(rounds.category == FailureCategory::kAssertion); + CHECK(rounds.line == 7); + const auto &attempt = rounds.attempts.front(); + CHECK(attempt.durationMicroseconds == 1000500000); + CHECK(attempt.standardOutput == "said something"); + REQUIRE(attempt.assertions.size() == 1); + CHECK(attempt.assertions.front().message == "1.5 != 2"); + CHECK(attempt.assertions.front().macro == "AssertionError"); + CHECK(attempt.assertions.front().expansion == "at rounds:7"); + + const auto &throws = Named(tests, "throws"); + CHECK(throws.category == FailureCategory::kUnexpectedException); + CHECK(throws.attempts.front().assertions.front().message == "Traceback"); + + CHECK(Named(tests, "ignored").outcome == Outcome::kSkipped); + CHECK(Named(tests, "ignored").attempts.front().standardOutput + == "not today"); + CHECK(Named(tests, "disabled").outcome == Outcome::kSkipped); + + const auto &retried = Named(tests, "retried"); + CHECK(retried.outcome == Outcome::kPassed); + CHECK(retried.flaky); + REQUIRE(retried.attempts.size() == 2); + CHECK(retried.attempts.front().outcome == Outcome::kFailed); + CHECK(retried.attempts.front().number == 1); + CHECK(retried.attempts.back().number == 2); + CHECK(retried.attempts.back().durationMicroseconds == 2000000); + + const auto &always = Named(tests, "always"); + CHECK(always.outcome == Outcome::kFailed); + CHECK_FALSE(always.flaky); + CHECK(always.attempts.size() == 2); +} + +TEST_CASE("Several failures of one JUnit test are one attempt", + "[progmasoft][runner][import]") +{ + const auto tests = Import(R"(<testsuite><testcase name="both"> + <failure message="one"/><error message="two"/> + </testcase></testsuite>)"); + REQUIRE(tests.size() == 1); + CHECK(tests.front().id == "both"); + REQUIRE(tests.front().attempts.size() == 1); + CHECK(tests.front().attempts.front().assertions.size() == 2); + CHECK(tests.front().attempts.front().assertionsFailed == 2); +} + +TEST_CASE("xUnit reports are imported", "[progmasoft][runner][import]") +{ + const auto tests = Import(kXUnit); + REQUIRE(tests.size() == 4); + + const auto &parses = Named(tests, "Dates.Parses"); + CHECK(parses.outcome == Outcome::kPassed); + CHECK(parses.className == "Dates"); + CHECK(parses.tags == std::vector<std::string>{ "kind=unit" }); + + const auto &rounds = Named(tests, "Dates.Rounds"); + CHECK(rounds.outcome == Outcome::kFailed); + CHECK(rounds.file == "Dates.cs"); + CHECK(rounds.line == 12); + const auto &assertion = rounds.attempts.front().assertions.front(); + CHECK(assertion.message == "not equal"); + CHECK(assertion.macro == "Xunit.EqualException"); + + CHECK(Named(tests, "Dates.Later").outcome == Outcome::kSkipped); + CHECK(Named(tests, "fixture Dates.Setup").category + == FailureCategory::kInfrastructure); +} + +TEST_CASE("One assembly is an xUnit report too", "[progmasoft][runner][import]") +{ + const auto tests = Import( + R"(<assembly><test name="a" result="Pass"/><test name="b" result="NotRun"/></assembly>)"); + REQUIRE(tests.size() == 2); + CHECK(Named(tests, "b").outcome == Outcome::kNotRun); +} + +TEST_CASE("TRX reports are imported", "[progmasoft][runner][import]") +{ + const auto tests = Import(kTrx); + REQUIRE(tests.size() == 6); + + const auto &parses = Named(tests, "Parses"); + CHECK(parses.className == "Ns.Dates"); + CHECK(parses.attempts.front().durationMicroseconds == 62500000); + + // The same test twice is a retry. + const auto &rounds = Named(tests, "Rounds"); + CHECK(rounds.outcome == Outcome::kPassed); + CHECK(rounds.flaky); + REQUIRE(rounds.attempts.size() == 2); + CHECK(rounds.attempts.front().durationMicroseconds == 86400000000); + CHECK(rounds.attempts.front().standardOutput == "said something"); + CHECK(rounds.attempts.front().assertions.front().message == "Assert.Equal"); + + CHECK(Named(tests, "Rows (1)").outcome == Outcome::kPassed); + CHECK(Named(tests, "Rows (2)").outcome == Outcome::kTimedOut); + CHECK(Named(tests, "Rows (2)").category == FailureCategory::kTimeout); + CHECK(Named(tests, "Rows (3)").outcome == Outcome::kCrashed); + CHECK(Named(tests, "Rows (4)").outcome == Outcome::kSkipped); +} + +TEST_CASE("TAP streams are imported", "[progmasoft][runner][import]") +{ + const auto tests = Import(kTap); + REQUIRE(tests.size() == 7); + + CHECK(Named(tests, "parses").outcome == Outcome::kPassed); + + const auto &rounds = Named(tests, "rounds"); + CHECK(rounds.outcome == Outcome::kFailed); + CHECK(rounds.attempts.front().durationMicroseconds == 1500000); + const auto &assertion = rounds.attempts.front().assertions.front(); + CHECK(assertion.message == "1.5 != 2"); + CHECK_THAT(assertion.expansion, ContainsSubstring("duration_ms")); + CHECK_THAT(assertion.expansion, ContainsSubstring("it was close")); + + CHECK(Named(tests, "later").outcome == Outcome::kSkipped); + CHECK(Named(tests, "known bug").outcome == Outcome::kFailedAsExpected); + CHECK_FALSE(Named(tests, "known bug").FailsRun()); + CHECK(Named(tests, "test 1").outcome == Outcome::kPassed); + + // A name in a subtest is its own, and one that comes back is another test. + bool nested = false; + for (const auto &test : tests) + { + nested = nested || test.id == "group::parses"; + } + CHECK(nested); + CHECK(Named(tests, "parses #2").outcome == Outcome::kPassed); +} + +TEST_CASE("TAP keeps escaped hashes in descriptions", + "[progmasoft][runner][import]") +{ + const auto tests = Import("ok 1 - issue \\#12 # skip\n"); + REQUIRE(tests.size() == 1); + CHECK(tests.front().name == "issue #12"); + CHECK(tests.front().outcome == Outcome::kSkipped); +} + +TEST_CASE("A TAP stream that ends early fails", "[progmasoft][runner][import]") +{ + const auto bailed = Import("1..2\nok 1 - a\nBail out! no database\n"); + CHECK(Named(bailed, "the TAP stream").outcome == Outcome::kFailed); + CHECK(Named(bailed, "the TAP stream").category + == FailureCategory::kInfrastructure); + CHECK(Named(bailed, "the TAP stream") + .attempts.front() + .assertions.front() + .message + == "Bail out! no database"); + + const auto shorter = Import("1..3\nok 1 - a\n"); + CHECK_THAT(Named(shorter, "the TAP stream") + .attempts.front() + .assertions.front() + .message, + ContainsSubstring("planned 3")); + + CHECK(Import("1..0 # SKIP nothing here\n").empty()); + CHECK(Import("ok 1 - a\nok 2 - b\n").size() == 2); +} + +TEST_CASE("CTRF reports are imported", "[progmasoft][runner][import]") +{ + const auto tests = Import(kCtrf); + REQUIRE(tests.size() == 5); + + const auto &parses = Named(tests, "parses"); + CHECK(parses.className == "dates > parse"); + CHECK(parses.attempts.front().durationMicroseconds == 250000); + + const auto &rounds = Named(tests, "rounds"); + CHECK(rounds.outcome == Outcome::kFailed); + CHECK(rounds.file == "dates.test.ts"); + CHECK(rounds.line == 7); + CHECK(rounds.tags == std::vector<std::string>{ "@unit" }); + CHECK(rounds.attempts.front().standardOutput == "one\ntwo\n"); + CHECK(rounds.attempts.front().assertions.front().expansion == "at rounds"); + + CHECK(Named(tests, "later").outcome == Outcome::kSkipped); + + const auto &retried = Named(tests, "retried"); + CHECK(retried.flaky); + CHECK(retried.outcome == Outcome::kPassed); + CHECK(retried.attempts.size() == 2); + + // A list of attempts that holds the last one does not count it twice. + CHECK(Named(tests, "listed").attempts.size() == 2); +} + +TEST_CASE("catch3-results lines are imported", "[progmasoft][runner][import]") +{ + const auto tests = Import(kLines); + REQUIRE(tests.size() == 3); + CHECK(Named(tests, "parses").id == "Dates::parses"); + CHECK(Named(tests, "parses").attempts.front().durationMicroseconds == 1200); + const auto &rounds = Named(tests, "rounds"); + CHECK(rounds.category == FailureCategory::kAssertion); + CHECK(rounds.attempts.front().assertions.front().expansion == "at rounds"); + CHECK(Named(tests, "crashes").id == "custom-id"); + CHECK(Named(tests, "crashes").category == FailureCategory::kCrash); +} + +TEST_CASE("Catch3's own report is imported without its old conclusions", + "[progmasoft][runner][import]") +{ + RunResult run; + TestResult test; + test.id = "a"; + test.name = "a"; + test.outcome = Outcome::kFailed; + test.quarantined = true; + test.signature = "sig-old"; + test.analysis = { "old" }; + test.replayCommand = "tests a"; + test.attempts.push_back(Attempt{}); + test.attempts.back().outcome = Outcome::kFailed; + run.tests.push_back(test); + + const auto tests = Import(Runner::ToJson(run).ToString()); + REQUIRE(tests.size() == 1); + CHECK(tests.front().outcome == Outcome::kFailed); + CHECK_FALSE(tests.front().quarantined); + CHECK(tests.front().signature.empty()); + CHECK(tests.front().analysis.empty()); + CHECK(tests.front().replayCommand.empty()); +} + +TEST_CASE("Results that share an identifier are attempts", + "[progmasoft][runner][import]") +{ + const auto one = Runner::ParseImport( + "<testsuite><testcase name=\"a\"><failure/></testcase>" + "<testcase name=\"b\"/></testsuite>", + Runner::ImportFormat::kJUnit); + const auto two + = Runner::ParseImport("<testsuite><testcase name=\"a\"/></testsuite>", + Runner::ImportFormat::kJUnit); + REQUIRE(one); + REQUIRE(two); + std::vector<TestResult> all = *one; + all.insert(all.end(), two->begin(), two->end()); + + const auto tests = Runner::MergeImported(std::move(all)); + REQUIRE(tests.size() == 2); + CHECK(tests.front().name == "a"); + CHECK(tests.front().flaky); + CHECK(tests.front().outcome == Outcome::kPassed); + CHECK(tests.front().category == FailureCategory::kNone); + CHECK_THAT(tests.front().analysis.front(), + ContainsSubstring("failed 1 of 2")); + CHECK_FALSE(tests.back().flaky); +} + +TEST_CASE("Durations that make no sense count as zero", + "[progmasoft][runner][import]") +{ + CHECK(Runner::ImportedMicroseconds(1.5) == 1500000); + CHECK(Runner::ImportedMicroseconds(-1.0) == 0); + CHECK(Runner::ImportedMicroseconds(1e30) == 0); + CHECK(Runner::ImportedMicroseconds(std::numeric_limits<double>::quiet_NaN()) + == 0); + CHECK(Runner::ImportedAttempt(Outcome::kPassed, -5).durationMicroseconds + == 0); +} + +TEST_CASE("Importing goes through the history and the reports of a run", + "[progmasoft][runner][import]") +{ + const Scratch scratch; + const auto write = [&](const std::string &name, std::string_view text) { + const auto path = scratch.Path() / name; + REQUIRE(Runner::WriteFileAtomically(path, text)); + return path.string(); + }; + const auto report + = write("failing.xml", + "<testsuite><testcase classname=\"c\" name=\"t\">" + "<failure message=\"no\"/></testcase></testsuite>"); + const auto retry = write("passing.tap", "1..1\nok 1 - c::x\n"); + const std::string history = "--history=" + (scratch.Path() / "h").string(); + + const auto first = RunFixture(scratch, + { "--import=" + report, + "--import=" + retry, + "--import-name=web", + "--import-revision=one", + history }); + INFO(first.output); + CHECK(ExitCode(first) == 42); + CHECK_THAT(first.output, ContainsSubstring("imported from 2 report(s)")); + CHECK(first.report["binary"].AsString() == "web"); + CHECK(first.report["tests"].Items().size() == 2); + CHECK(first.Test("t")["replayCommand"].AsString().empty()); + + // The same test passes on the same revision: that is flakiness. + const auto second + = RunFixture(scratch, + { "--import=" + + write("fixed.xml", + "<testsuite><testcase classname=\"c\" " + "name=\"t\"/></testsuite>"), + "--import-name=web", + "--import-revision=one", + history }); + INFO(second.output); + CHECK(ExitCode(second) == 0); + + const auto third = RunFixture(scratch, + { "--import=" + report, + "--import-name=web", + "--import-revision=one", + history }); + INFO(third.output); + CHECK_THAT(third.Analysis("t"), ContainsSubstring("intermittent")); +} + +TEST_CASE("Importing a file that is not a report fails", + "[progmasoft][runner][import]") +{ + const Scratch scratch; + const auto path = scratch.Path() / "notes.txt"; + REQUIRE(Runner::WriteFileAtomically(path, "hello")); + const auto run + = RunFixture(scratch, { "--import=" + path.string(), "--no-history" }); + CHECK_FALSE(ExitCode(run) == 0); + CHECK_THAT(run.output, ContainsSubstring("notes.txt")); + + const auto missing + = RunFixture(scratch, + { "--import=" + (scratch.Path() / "none.xml").string(), + "--no-history" }); + CHECK_FALSE(ExitCode(missing) == 0); +} diff --git a/tests/Progmasoft/RunnerOptions.tests.cpp b/tests/Progmasoft/RunnerOptions.tests.cpp index 95c081ce..37d35d33 100644 --- a/tests/Progmasoft/RunnerOptions.tests.cpp +++ b/tests/Progmasoft/RunnerOptions.tests.cpp @@ -309,7 +309,9 @@ TEST_CASE("The help lists every option", "[progmasoft][runner][options]") "--benchmark-warmup-time", "--history", "--no-history", "--quarantine", "--replay", - "--analytics", "--verbose", + "--analytics", "--import", + "--import-format", "--import-name", + "--import-revision", "--verbose", "--quiet", "--color", "--junit", "--json", "--trace", "--premature-exit-guard-file", diff --git a/tests/meson.build b/tests/meson.build index 6ec807d7..26ae225d 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -82,6 +82,7 @@ progmasoft_property_tests = executable( 'Progmasoft/RunnerOptions.tests.cpp', 'Progmasoft/RunnerConfig.tests.cpp', 'Progmasoft/RunnerAnalysis.tests.cpp', + 'Progmasoft/RunnerImport.tests.cpp', 'Progmasoft/RunnerOutput.tests.cpp', 'Progmasoft/Snapshot.tests.cpp', 'Progmasoft/Stateful.tests.cpp', diff --git a/tools/scripts/generateAmalgamatedFiles.py b/tools/scripts/generateAmalgamatedFiles.py index b3151506..c927c75b 100755 --- a/tools/scripts/generateAmalgamatedFiles.py +++ b/tools/scripts/generateAmalgamatedFiles.py @@ -166,7 +166,7 @@ concatenated_runner_headers = set() # files; in the amalgamation they are one header, included where the first # of them was. third_party_wrappers = ['Runner/Yyjson.inl', 'PugiXml.inl'] -wrapper_include_parser = re.compile(r'\s*#include "([A-Za-z/]+\.inl)".*') +wrapper_include_parser = re.compile(r'\s*#include "([A-Za-z./]+\.inl)".*') third_party_files = { 'yyjson.c': 'yyjson/yyjson.c', 'yyjson.h': 'yyjson/yyjson.h', diff --git a/xmake.lua b/xmake.lua index fd94efb8..5a74dd58 100644 --- a/xmake.lua +++ b/xmake.lua @@ -61,6 +61,7 @@ target("ProgmasoftPropertyTests") "tests/Progmasoft/RunnerOptions.tests.cpp", "tests/Progmasoft/RunnerConfig.tests.cpp", "tests/Progmasoft/RunnerAnalysis.tests.cpp", + "tests/Progmasoft/RunnerImport.tests.cpp", "tests/Progmasoft/RunnerOutput.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp", "tests/Progmasoft/Stateful.tests.cpp",