Merge pull request #28 from Progmasoft/feature/result-import

Import the results of other test frameworks

29 files changed, 4556 insertions, 197 deletions

diff --git a/docs/progmasoft-roadmap.md b/docs/progmasoft-roadmap.md
index f376e9a0..5ab04264 100644
--- a/docs/progmasoft-roadmap.md
+++ b/docs/progmasoft-roadmap.md
@@ -48,6 +48,8 @@ Catch3 runner replaced Catch2's in every test binary:
 - every failure gets a category, a signature and an analysis;
 - reports are console, JUnit XML with flaky and rerun elements, JSON and a
   trace; the discovery listing has a fixed, versioned format;
+- reports of other frameworks are imported, so their tests get the same
+  history, flaky detection, quarantine and analytics;
 - every test gets a temporary directory of its own;
 - its command line is parsed by a hand-written, typed parser, like the other
   Progmasoft tools.
diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md
index 300c541e..8134ae18 100644
--- a/docs/progmasoft-runner.md
+++ b/docs/progmasoft-runner.md
@@ -5,6 +5,7 @@
 [How a run works](#how-a-run-works)<br>
 [Retries, flaky tests and order dependence](#retries-flaky-tests-and-order-dependence)<br>
 [History, quarantine and analytics](#history-quarantine-and-analytics)<br>
+[Importing the results of other frameworks](#importing-the-results-of-other-frameworks)<br>
 [Choosing and ordering tests](#choosing-and-ordering-tests)<br>
 [Tags the runner reads](#tags-the-runner-reads)<br>
 [Time limits](#time-limits)<br>
@@ -133,6 +134,73 @@ Under `bazel test` there is no history unless `CATCH3_HISTORY` names a
 directory that outlives the sandbox.
 
 
+## Importing the results of other frameworks
+
+`--import=FILE` reads the report of another test framework instead of running
+tests, so that tests Catch3 does not run get the same history, flaky detection,
+quarantine and analytics. Any Catch3 test binary can do it, and the option can
+be repeated:
+
+```sh
+./tests --import=results/pytest.xml --import=results/vitest.ctrf.json \
+        --import-name=web --junit=merged.xml
+```
+
+| `--import-format=` | Reads |
+| --- | --- |
+| `auto` | whichever the contents show (the default) |
+| `catch3` | a `--json` report of Catch3, or [`catch3-results`](#the-catch3-results-format) |
+| `junit` | JUnit XML, with Surefire's reruns; Gradle, pytest, Jest, GoogleTest and most others write it |
+| `xunit` | xUnit.net v2 XML |
+| `trx` | Visual Studio's TRX, with data-driven rows |
+| `tap` | TAP 12, 13 and 14, with subtests and YAML diagnostics |
+| `ctrf` | [CTRF](https://ctrf.io) JSON, with retries |
+
+The results go into a history of their own, named by `--import-name` and
+`imported` by default; `--analytics --import-name=NAME` reads it. The console,
+`--junit`, `--json`, `--trace`, `--quarantine`, `--history` and the exit code
+work as they do for a run. Reports that were read count as one run.
+
+A test is identified by its class and its name. Results with the same
+identifier, in one report or in several, are attempts at one test: a test that
+both failed and passed is flaky, passes and is remembered as flaky, which is how
+a report of a rerun is imported. A failure stays a failure when a later report
+skips the test. Reports of the same tests on different platforms are not reruns
+of each other: import each with an `--import-name` of its own, or a failure on one
+reads as flakiness.
+
+TAP names tests only within one stream, so the path given to `--import` is the
+class of its tests: keep it the same from run to run for the history to follow
+them. A name that comes back in one stream is another test and gets a number.
+
+`--import-revision=REVISION` names what the reports tested, and a verdict that
+changes on one revision then counts as flaky too. Without it only retries in the
+reports do, because a working directory can have edits or be another checkout.
+
+Nothing is run, so an imported failure has no replay command, and only what the
+report says is known: it has no order, seed or sections. A TAP stream that bails
+out or reports fewer tests than it planned adds a failed test called
+`the TAP stream`, so that it does not read as a run that passed.
+
+### The catch3-results format
+
+For a framework without a report of these kinds, `catch3-results` is the
+smallest thing to write: JSON Lines with one test per line, after a line that
+names the format.
+
+```json
+{"format": "catch3-results", "version": 1}
+{"name": "parses a date", "className": "Dates", "outcome": "passed", "durationMicroseconds": 1200}
+{"name": "rounds down", "outcome": "failed", "message": "1.5 != 2", "details": "at Dates.rounds:12"}
+```
+
+`name` or `id`, and `outcome`, are required; `outcome` is a name from the JSON
+report, such as `passed`, `failed`, `skipped`, `crashed` or `timed-out`. The other
+members are `id`, `className`, `file`, `line`, `tags`, `category`, `message`,
+`details`, `standardOutput` and `standardError`. A line can give `attempts`
+instead, as the JSON report does.
+
+
 ## Choosing and ordering tests
 
 Test specs work as in Catch2: names with `*` as a wildcard, `[tags]`, `~`
@@ -265,6 +333,8 @@ version.
   `quarantined`, `resources`, `dependsOn`, `timeoutSeconds` and `retries`.
   CMake's `catch3_discover_tests` reads it.
 * **Run report**, `--json`: `"format": "catch3-run"`, version 1.
+* **Imported results**, `--import`: `"format": "catch3-results"`, version 1;
+  see [its format](#the-catch3-results-format).
 * **Analytics**, `--analytics=json`: `"format": "catch3-analytics"`,
   version 1.
 * **History**: `"format": "catch3-history"`, version 1. It is the runner's
diff --git a/docs/release-notes.md b/docs/release-notes.md
index f44d1369..e1567d97 100644
--- a/docs/release-notes.md
+++ b/docs/release-notes.md
@@ -7,11 +7,22 @@ based on Catch2 3.16.1; Catch2's own history is in
 [its release notes](https://github.com/catchorg/Catch2/blob/devel/docs/release-notes.md).
 
 **Contents**<br>
+[Unreleased](#unreleased)<br>
 [0.3.0](#030)<br>
 [0.2.0](#020)<br>
 [0.1.0](#010)<br>
 
 
+## Unreleased
+
+### The Catch3 runner
+
+* `--import=FILE` reads the reports of other test frameworks instead of running
+  tests: JUnit, xUnit, TRX, TAP, CTRF, Catch3's own, and a small JSON Lines
+  format for anything else. Their tests get the history, flaky detection,
+  quarantine, analytics and reports of a run.
+
+
 ## 0.3.0
 
 Released 2026-11-01.
diff --git a/extras/catch_amalgamated.cpp b/extras/catch_amalgamated.cpp
index 7556181e..01b07618 100644
--- a/extras/catch_amalgamated.cpp
+++ b/extras/catch_amalgamated.cpp
@@ -7,7 +7,7 @@
 // SPDX-License-Identifier: BSL-1.0
 
 //  Catch3 v0.3.0, on the Catch2 v3.16.1 engine
-//  Generated: 2026-11-01 00:25:29.768099
+//  Generated: 2026-11-01 14:47:44.734580
 //  ----------------------------------------------------------
 //  This file is an amalgamation of multiple different files.
 //  You probably shouldn't edit it directly.
@@ -965,6 +965,8 @@ namespace Progmasoft::Catch3::Runner
         kListTests,
         kListTags,
         kAnalytics,
+        /// Read the reports of other test frameworks.
+        kImport,
         kHelp,
         kVersion,
         /// Internal: run tests on behalf of a runner, over the protocol.
@@ -1022,6 +1024,20 @@ namespace Progmasoft::Catch3::Runner
         kOff,
     };
 
+    /// The reports `--import` reads.
+    enum class ImportFormat : std::uint8_t
+    {
+        /// Whichever the contents show.
+        kAuto,
+        /// Catch3's own: `--json` reports and catch3-results JSON Lines.
+        kCatch3,
+        kJUnit,
+        kXUnit,
+        kTrx,
+        kTap,
+        kCtrf,
+    };
+
     enum class ColorMode : std::uint8_t
     {
         kAuto,
@@ -1101,6 +1117,15 @@ namespace Progmasoft::Catch3::Runner
         bool replayFailures = false;
         bool analyticsJson = false;
 
+        // Importing the reports of other frameworks.
+        std::vector<std::string> importFiles;
+        ImportFormat importFormat = ImportFormat::kAuto;
+        /// The history they go to; "imported" when empty.
+        std::string importName;
+        /// What was tested: a verdict that changes on one revision counts
+        /// as flaky. Without it, only retries in the reports do.
+        std::string importRevision;
+
         // Output.
         Verbosity verbosity = Verbosity::kNormal;
         ColorMode color = ColorMode::kAuto;
@@ -1268,6 +1293,10 @@ namespace Progmasoft::Catch3::Runner
                     std::uint32_t seed,
                     const std::vector<std::string> &notes);
 
+        /// The start of `--import`: results read from `files` reports.
+        void
+        ImportStarting(std::size_t tests, std::size_t files);
+
         void
         TestFinished(const TestResult &test);
 
@@ -1856,6 +1885,96 @@ namespace Progmasoft::Catch3::Runner
 // Copyright (c) 2026 Progmasoft.
 // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0
 
+#include <cstdint>
+#include <optional>
+#include <string>
+#include <string_view>
+#include <vector>
+
+
+namespace Progmasoft::Catch3::Runner
+{
+    /// The format of a report, from its contents; nothing when none fits.
+    [[nodiscard]] std::optional<ImportFormat>
+    DetectImportFormat(std::string_view text);
+
+    /// The results a report holds, in its order. A test the report names
+    /// twice comes back twice; `MergeImported` makes one test of them.
+    /// `source` names the report, for a format whose tests have no identity
+    /// of their own.
+    [[nodiscard]] Expected<std::vector<TestResult>>
+    ParseImport(std::string_view text,
+                ImportFormat format,
+                const std::string &source = {});
+
+    /// One test per identifier: results that share one are attempts at that
+    /// test, in the order given, and the verdict is worked out over them. A
+    /// test that both failed and passed is flaky, as it is in a run of the
+    /// runner.
+    [[nodiscard]] std::vector<TestResult>
+    MergeImported(std::vector<TestResult> results);
+
+    /// A result without attempts yet.
+    [[nodiscard]] TestResult
+    ImportedTest(std::string name, std::string className);
+
+    /// One attempt as a report describes it. A failure gets an assertion
+    /// record with `message` and `details`, so the analysis treats it as it
+    /// treats a failure of Catch3's own; `category` defaults by outcome.
+    [[nodiscard]] Attempt
+    ImportedAttempt(Outcome outcome,
+                    std::int64_t durationMicroseconds,
+                    std::string message = {},
+                    std::string details = {},
+                    FailureCategory category = FailureCategory::kNone);
+
+    /// Seconds as a report gives them; what makes no sense as a duration
+    /// counts as zero.
+    [[nodiscard]] std::int64_t
+    ImportedMicroseconds(double seconds);
+
+    [[nodiscard]] std::string_view
+    ImportTrim(std::string_view text);
+
+    [[nodiscard]] bool
+    ImportStartsWith(std::string_view text, std::string_view prefix);
+
+    /// The lines of a text, without their line endings.
+    [[nodiscard]] std::vector<std::string_view>
+    ImportLines(std::string_view text);
+
+    /// A format of Catch3's own or CTRF, from a parsed document.
+    [[nodiscard]] std::optional<ImportFormat>
+    ImportJsonFormat(const JsonValue &value);
+
+    /// Whether a line reports a test in TAP.
+    [[nodiscard]] bool
+    IsTapTestLine(std::string_view line);
+
+    /// A `--json` report or catch3-results lines.
+    [[nodiscard]] Expected<std::vector<TestResult>>
+    ParseCatch3Report(std::string_view text);
+
+    [[nodiscard]] Expected<std::vector<TestResult>>
+    ParseCtrf(std::string_view text);
+
+    /// `source` becomes the class of its tests: TAP names them only within
+    /// one stream.
+    [[nodiscard]] Expected<std::vector<TestResult>>
+    ParseTap(std::string_view text, const std::string &source);
+
+    [[nodiscard]] Expected<std::vector<TestResult>>
+    ParseJUnit(std::string_view text);
+
+    [[nodiscard]] Expected<std::vector<TestResult>>
+    ParseXUnit(std::string_view text);
+
+    [[nodiscard]] Expected<std::vector<TestResult>>
+    ParseTrx(std::string_view text);
+} // namespace Progmasoft::Catch3::Runner
+// Copyright (c) 2026 Progmasoft.
+// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0
+
 
 
 namespace Progmasoft::Catch3::Runner
@@ -9702,6 +9821,11 @@ namespace Progmasoft::Catch3::Runner
             test.analysis.insert(
                 test.analysis.end(), findings.begin(), findings.end());
 
+            // An imported failure ran somewhere else.
+            if (inputs.program.empty())
+            {
+                continue;
+            }
             std::string replay = QuoteArgument(inputs.program) + " " +
                                  QuoteArgument(QuoteTestSpec(test.name)) +
                                  " --seed=" + std::to_string(failure->seed);
@@ -10202,6 +10326,21 @@ namespace Progmasoft::Catch3::Runner
     }
 
     void
+    ConsoleReport::ImportStarting(std::size_t tests, std::size_t files)
+    {
+        total_ = tests;
+        if (options_.verbosity == Verbosity::kQuiet)
+        {
+            return;
+        }
+        Write(output_,
+              Paint("Catch3", kBold) + ": " + std::to_string(tests)
+                  + " test(s) imported from " + std::to_string(files)
+                  + " report(s)\n");
+        std::fflush(output_);
+    }
+
+    void
     ConsoleReport::Retrying(const TestResult &test, const Attempt &attempt)
     {
         if (options_.verbosity != Verbosity::kVerbose)
@@ -13375,6 +13514,59 @@ namespace Progmasoft::Catch3::Runner
                      options.replayFailures = true;
                      return Status::Ok();
                  }},
+                {"import",
+                 '\0',
+                 Takes::kValue,
+                 "FILE",
+                 "Importing results",
+                 "read a report of another framework instead of running tests "
+                 "(repeatable)",
+                 [](Options &options, const std::optional<std::string> &value) {
+                     options.mode = Mode::kImport;
+                     options.importFiles.push_back(*value);
+                     return Status::Ok();
+                 }},
+                {"import-format",
+                 '\0',
+                 Takes::kValue,
+                 "FORMAT",
+                 "Importing results",
+                 "auto (default), catch3, junit, xunit, trx, tap or ctrf",
+                 [](Options &options, const std::optional<std::string> &value) {
+                     return ReadChoice<ImportFormat>(
+                         value,
+                         "import-format",
+                         {{"auto", ImportFormat::kAuto},
+                          {"catch3", ImportFormat::kCatch3},
+                          {"junit", ImportFormat::kJUnit},
+                          {"xunit", ImportFormat::kXUnit},
+                          {"trx", ImportFormat::kTrx},
+                          {"tap", ImportFormat::kTap},
+                          {"ctrf", ImportFormat::kCtrf}},
+                         options.importFormat);
+                 }},
+                {"import-name",
+                 '\0',
+                 Takes::kValue,
+                 "NAME",
+                 "Importing results",
+                 "the history imported results go to (default: imported); "
+                 "--analytics reads it too",
+                 [](Options &options, const std::optional<std::string> &value) {
+                     options.importName = *value;
+                     return Status::Ok();
+                 }},
+                {"import-revision",
+                 '\0',
+                 Takes::kValue,
+                 "REVISION",
+                 "Importing results",
+                 "what the reports tested; a verdict that changes on one "
+                 "revision counts as flaky",
+                 [](Options &options, const std::optional<std::string> &value) {
+                     options.importRevision = *value;
+                     return Status::Ok();
+                 }},
                 {"analytics",
                  '\0',
                  Takes::kOptionalValue,
@@ -15881,6 +16073,10 @@ namespace Progmasoft::Catch3::Runner
             "break",
             "replay",
             "analytics",
+            "import",
+            "import-format",
+            "import-name",
+            "import-revision",
             "verbose",
             "quiet",
             "premature-exit-guard-file",
@@ -16379,6 +16575,1438 @@ namespace Progmasoft::Catch3::Runner
 } // namespace Progmasoft::Catch3::Runner
 // Copyright (c) 2026 Progmasoft.
 // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-2.0
+#include <algorithm>
+#include <cctype>
+#include <cmath>
+#include <iterator>
+#include <map>
+#include <utility>
+
+namespace Progmasoft::Catch3::Runner
+{
+    std::string_view
+    ImportTrim(std::string_view text)
+    {
+        const auto blank = [](char character) {
+            return std::isspace(static_cast<unsigned char>(character)) != 0;
+        };
+        while (!text.empty() && blank(text.front()))
+        {
+            text.remove_prefix(1);
+        }
+        while (!text.empty() && blank(text.back()))
+        {
+            text.remove_suffix(1);
+        }
+        return text;
+    }
+
+    bool
+    ImportStartsWith(std::string_view text, std::string_view prefix)
+    {
+        return text.substr(0, prefix.size()) == prefix;
+    }
+
+    std::vector<std::string_view>
+    ImportLines(std::string_view text)
+    {
+        std::vector<std::string_view> lines;
+        while (!text.empty())
+        {
+            const auto end = text.find('\n');
+            auto line = text.substr(0, end);
+            if (!line.empty() && line.back() == '\r')
+            {
+                line.remove_suffix(1);
+            }
+            lines.push_back(line);
+            if (end == std::string_view::npos)
+            {
+                break;
+            }
+            text.remove_prefix(end + 1);
+        }
+        return lines;
+    }
+
+    namespace
+    {
+        namespace ImportText
+        {
+            /// Without a byte order mark and the white space before the
+            /// first character that counts.
+            std::string_view
+            Contents(std::string_view text)
+            {
+                if (ImportStartsWith(text, "\xEF\xBB\xBF"))
+                {
+                    text.remove_prefix(3);
+                }
+                return ImportTrim(text);
+            }
+
+            /// The name of the root element of an XML document.
+            std::string_view
+            RootElement(std::string_view text)
+            {
+                while (!text.empty())
+                {
+                    const auto open = text.find('<');
+                    if (open == std::string_view::npos
+                        || open + 1 >= text.size())
+                    {
+                        return {};
+                    }
+                    text.remove_prefix(open + 1);
+                    if (text.front() == '?' || text.front() == '!')
+                    {
+                        const auto close = text.find('>');
+                        if (close == std::string_view::npos)
+                        {
+                            return {};
+                        }
+                        text.remove_prefix(close + 1);
+                        continue;
+                    }
+                    return text.substr(0, text.find_first_of(" \t\r\n/>"));
+                }
+                return {};
+            }
+        } // namespace ImportText
+    } // namespace
+
+    std::optional<ImportFormat>
+    DetectImportFormat(std::string_view text)
+    {
+        const std::string_view contents = ImportText::Contents(text);
+        if (contents.empty())
+        {
+            return std::nullopt;
+        }
+        if (contents.front() == '<')
+        {
+            const std::string_view root = ImportText::RootElement(contents);
+            if (root == "testsuites" || root == "testsuite")
+            {
+                return ImportFormat::kJUnit;
+            }
+            if (root == "assemblies" || root == "assembly")
+            {
+                return ImportFormat::kXUnit;
+            }
+            if (root == "TestRun")
+            {
+                return ImportFormat::kTrx;
+            }
+            return std::nullopt;
+        }
+        if (contents.front() == '{')
+        {
+            auto whole = ParseJson(contents);
+            if (!whole)
+            {
+                // JSON Lines: the first line says what the others are.
+                whole = ParseJson(contents.substr(0, contents.find('\n')));
+            }
+            return whole ? ImportJsonFormat(*whole) : std::nullopt;
+        }
+        for (const auto raw : ImportLines(contents))
+        {
+            const std::string_view line = ImportTrim(raw);
+            if (line.empty() || line.front() == '#')
+            {
+                continue;
+            }
+            if (ImportStartsWith(line, "TAP version")
+                || ImportStartsWith(line, "1..") || IsTapTestLine(line))
+            {
+                return ImportFormat::kTap;
+            }
+            break;
+        }
+        return std::nullopt;
+    }
+
+    Expected<std::vector<TestResult>>
+    ParseImport(std::string_view text,
+                ImportFormat format,
+                const std::string &source)
+    {
+        const std::string_view contents = ImportText::Contents(text);
+        switch (format)
+        {
+            case ImportFormat::kAuto:
+                break;
+            case ImportFormat::kCatch3:
+                return ParseCatch3Report(contents);
+            case ImportFormat::kJUnit:
+                return ParseJUnit(contents);
+            case ImportFormat::kXUnit:
+                return ParseXUnit(contents);
+            case ImportFormat::kTrx:
+                return ParseTrx(contents);
+            case ImportFormat::kTap:
+                return ParseTap(contents, source);
+            case ImportFormat::kCtrf:
+                return ParseCtrf(contents);
+        }
+        const auto detected = DetectImportFormat(contents);
+        if (!detected)
+        {
+            return Error{ "its format is not one Catch3 knows; "
+                          "--import-format names it" };
+        }
+        return ParseImport(contents, *detected, source);
+    }
+
+    std::vector<TestResult>
+    MergeImported(std::vector<TestResult> results)
+    {
+        std::vector<TestResult> tests;
+        std::map<std::string, std::size_t> positions;
+        for (auto &result : results)
+        {
+            if (result.id.empty())
+            {
+                result.id = History::KeyOf(result);
+            }
+            const auto [position, added]
+                = positions.emplace(result.id, tests.size());
+            if (added)
+            {
+                tests.push_back(std::move(result));
+                continue;
+            }
+            TestResult &test = tests[position->second];
+            test.flaky = test.flaky || result.flaky;
+            std::move(result.attempts.begin(),
+                      result.attempts.end(),
+                      std::back_inserter(test.attempts));
+        }
+        for (auto &test : tests)
+        {
+            std::size_t passes = 0;
+            std::size_t failures = 0;
+            for (std::size_t index = 0; index < test.attempts.size(); ++index)
+            {
+                Attempt &attempt = test.attempts[index];
+                attempt.number = index + 1;
+                passes += attempt.outcome == Outcome::kPassed ? 1 : 0;
+                failures += IsFailure(attempt.outcome) ? 1 : 0;
+            }
+            if (test.attempts.empty())
+            {
+                continue;
+            }
+            // A report of a skip says nothing about an earlier failure.
+            const auto verdict
+                = std::find_if(test.attempts.rbegin(),
+                               test.attempts.rend(),
+                               [](const Attempt &attempt) {
+                                   return IsFailure(attempt.outcome);
+                               });
+            const Attempt &last = verdict != test.attempts.rend()
+                                      ? *verdict
+                                      : test.attempts.back();
+            test.outcome = last.outcome;
+            test.category = last.category;
+            if (passes > 0 && failures > 0)
+            {
+                test.outcome = Outcome::kPassed;
+                test.category = FailureCategory::kNone;
+                test.flaky = true;
+                test.analysis.push_back(
+                    "flaky: failed " + std::to_string(failures) + " of "
+                    + std::to_string(test.attempts.size()) + " attempts");
+            }
+        }
+        return tests;
+    }
+
+    TestResult
+    ImportedTest(std::string name, std::string className)
+    {
+        TestResult test;
+        test.name = std::move(name);
+        test.className = std::move(className);
+        test.id = History::KeyOf(test);
+        return test;
+    }
+
+    Attempt
+    ImportedAttempt(Outcome outcome,
+                    std::int64_t durationMicroseconds,
+                    std::string message,
+                    std::string details,
+                    FailureCategory category)
+    {
+        Attempt attempt;
+        attempt.outcome = outcome;
+        attempt.durationMicroseconds
+            = std::max<std::int64_t>(durationMicroseconds, 0);
+        if (outcome == Outcome::kSkipped)
+        {
+            attempt.standardOutput = std::move(message);
+            return attempt;
+        }
+        if (outcome != Outcome::kFailedAsExpected && !IsFailure(outcome))
+        {
+            return attempt;
+        }
+        if (category == FailureCategory::kNone)
+        {
+            category = outcome == Outcome::kCrashed ? FailureCategory::kCrash
+                       : outcome == Outcome::kTimedOut
+                           ? FailureCategory::kTimeout
+                           : FailureCategory::kAssertion;
+        }
+        if (message.empty())
+        {
+            message = details.substr(0, details.find(