catch3
Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.
Clonegit clone https://git.progmasoft.com/catch3
Read the options of a project from catch3.json
- Commit
f41a9c3c01b4df1100e24a63987716cb4dbf28f2- Author
- Leitwolf11 ·
- Parents
1d48419b7ef6
Changes
docs/command-line.md | 12 + docs/progmasoft-runner.md | 49 +++ docs/release-notes.md | 4 + extras/catch_amalgamated.cpp | 696 ++++++++++++++++++++++++++++-- extras/catch_amalgamated.hpp | 2 +- extras/catch_amalgamated_third_party.hpp | 2 +- fuzzing/CMakeLists.txt | 2 +- fuzzing/Progmasoft/fuzz_ProjectConfig.cpp | 30 ++ premake5.lua | 1 + src/CMakeLists.txt | 2 + src/Progmasoft/Catch3/meson.build | 1 + src/Progmasoft/Runner/Options.cpp | 78 ++-- src/Progmasoft/Runner/Options.hpp | 19 +- src/Progmasoft/Runner/ProjectConfig.cpp | 548 +++++++++++++++++++++++ src/Progmasoft/Runner/ProjectConfig.hpp | 44 ++ src/Progmasoft/Runner/Runner.cpp | 12 +- tests/CMakeLists.txt | 1 + tests/Progmasoft/BUILD.bazel | 1 + tests/Progmasoft/RunnerConfig.tests.cpp | 423 ++++++++++++++++++ tests/meson.build | 1 + xmake.lua | 1 + 21 files changed, 1863 insertions(+), 66 deletions(-) diff --git a/docs/command-line.md b/docs/command-line.md index 29abc11a..4bd46f3b 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -8,6 +8,7 @@ [How tests behave](#how-tests-behave)<br> [History and analysis](#history-and-analysis)<br> [Output](#output)<br> +[Configuration](#configuration)<br> [Information](#information)<br> [Environment](#environment)<br> @@ -122,6 +123,17 @@ Without a spec, every test runs except hidden ones, whose tags start with | `--premature-exit-guard-file=FILE` | create FILE, private to the user, when the run starts and remove it when the run ends normally | +## Configuration + +A project can keep its options in a [`catch3.json`](progmasoft-runner.md#project-configuration); +options on the command line win over it. + +| Option | | +| --- | --- | +| `--config=FILE` | read this `catch3.json` instead of searching for one | +| `--no-config` | read no `catch3.json` | + + ## Information | Option | | diff --git a/docs/progmasoft-runner.md b/docs/progmasoft-runner.md index baf5521a..300c541e 100644 --- a/docs/progmasoft-runner.md +++ b/docs/progmasoft-runner.md @@ -12,6 +12,7 @@ [What a failure report says](#what-a-failure-report-says)<br> [Reports and traces](#reports-and-traces)<br> [Formats other tools read](#formats-other-tools-read)<br> +[Project configuration](#project-configuration)<br> [Bazel and CTest](#bazel-and-ctest)<br> [Exit codes](#exit-codes)<br> [Coming from Catch2's runner](#coming-from-catch2s-runner)<br> @@ -273,6 +274,54 @@ The messages between the runner and its workers are internal and change with the binary. +## Project configuration + +Options a project always wants go in a `catch3.json`: + +```json +{ + "version": 1, + "tests": ["~[slow]"], + "jobs": 8, + "timeout": 60, + "warn": ["no-assertions"], + "junit": "build/test-results/junit.xml" +} +``` + +The runner looks for the file in the working directory and in each +directory above it, up to the root of the Git repository, and reads the +nearest one. Outside a repository it looks only in the working directory. +`--config=FILE` names a file instead, and `--no-config` reads none. + +The keys are the long options, and their values go through the same checks +as on the command line, with the key named in the message: + +| Value | Keys | +| --- | --- | +| `true` turns the switch on | `incremental`, `filenames-as-tags`, `in-process`, `isolate`, `success`, `no-throw`, `allow-no-tests`, `skip-benchmarks`, `benchmark-no-analysis` | +| `false` turns it off | `adaptive-timeout`, `bisect` | +| a string | `select`, `changes-since`, `order`, `quarantine`, `color` | +| a whole number | `jobs`, `repeat`, `fail-fast`, `slowest`, `benchmark-samples`, `benchmark-resamples`, `benchmark-warmup-time` | +| a whole number or a string | `retries` (`"auto"`, `"off"`), `seed` (`"random"`) | +| a number | `timeout` (seconds), `benchmark-confidence-interval` | +| a path, relative to the file | `junit`, `json`, `trace`, `shard-durations`; `history`, or `false` for none | +| an array of strings | `tests` (test specs), `warn` | +| `"quiet"`, `"normal"` or `"verbose"` | `verbosity` | + +`"version": 1` is required; `"$schema"` may name a schema for editors. +The file is strict JSON: no comments and no trailing commas. A key it does +not know is an error, with a suggestion when it looks like a known one. +Options that describe one run, such as `--shard`, `--section`, `--list` or +`--replay`, belong on the command line only. + +Options on the command line win over the file, and test specs on the +command line replace the file's `tests` rather than adding to them. A +switch the file turns on stays on; `--no-config` leaves the file out. +`--verbose` names the file that was read. Workers, `--help` and +`--version` read no file. + + ## Bazel and CTest Under `bazel test` the runner reads Bazel's environment: `XML_OUTPUT_FILE` diff --git a/docs/release-notes.md b/docs/release-notes.md index aef49cf0..b18ee5be 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -33,6 +33,10 @@ line and reporters with [a runner of its own](progmasoft-runner.md#top): * Reports are console, JUnit XML, JSON and a Chrome trace; `--list=json` is a versioned discovery format. * Every test gets a temporary directory of its own. +* A project keeps its options in a + [`catch3.json`](progmasoft-runner.md#project-configuration), found up to + the root of the repository; the command line wins over it, and + `--config=FILE` and `--no-config` choose another file or none. * JSON is read and written with [yyjson](https://github.com/ibireme/yyjson) and JUnit XML is written with [pugixml](https://github.com/zeux/pugixml), both MIT and vendored in `third_party/`. They are compiled into Catch3's diff --git a/extras/catch_amalgamated.cpp b/extras/catch_amalgamated.cpp index a9cdb04b..b87c46a4 100644 --- a/extras/catch_amalgamated.cpp +++ b/extras/catch_amalgamated.cpp @@ -7,7 +7,7 @@ // SPDX-License-Identifier: BSL-1.0 // Catch3 v0.3.0, on the Catch2 v3.16.1 engine -// Generated: 2026-10-25 13:52:54.556111 +// Generated: 2026-10-29 19:10:35.449209 // ---------------------------------------------------------- // This file is an amalgamation of multiple different files. // You probably shouldn't edit it directly. @@ -950,8 +950,10 @@ namespace Progmasoft::Catch3::Runner #include <chrono> #include <cstdint> +#include <cstddef> #include <optional> #include <string> +#include <string_view> #include <vector> namespace Progmasoft::Catch3::Runner @@ -1109,11 +1111,24 @@ namespace Progmasoft::Catch3::Runner /// Created when the run starts and removed when it ends normally, so /// a file left behind shows that the run ended early. std::string exitGuardPath; + + // Configuration. + /// The `catch3.json` named by --config; after ParseCommandLine, the + /// one that was read, if any. + std::string configPath; + bool noConfig = false; }; - /// Reads a command line. `arguments` does not include the program. + /// Reads a command line on top of `options`. `arguments` does not include + /// the program. [[nodiscard]] Expected<Options> - ParseOptions(const std::vector<std::string> &arguments); + ParseOptions(const std::vector<std::string> &arguments, + Options options = {}); + + /// How many single-character edits turn `left` into `right`, for + /// suggesting the name that was meant. + [[nodiscard]] std::size_t + EditDistance(std::string_view left, std::string_view right); /// The help text, for `--help`. [[nodiscard]] std::string @@ -1727,6 +1742,47 @@ namespace Progmasoft::Catch3::Runner // Copyright (c) 2026 Progmasoft. // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <filesystem> +#include <optional> +#include <string> +#include <string_view> +#include <vector> + + +namespace Progmasoft::Catch3::Runner +{ + /// The name of the project configuration file. + inline constexpr std::string_view kProjectConfigName = "catch3.json"; + + /// The `catch3.json` that applies in `start`: the nearest one in `start` + /// or a directory above it, up to the root of the Git repository that + /// holds `start`. Outside a repository only `start` itself is searched, + /// so a stray file in a home directory never applies. + [[nodiscard]] std::optional<std::filesystem::path> + FindProjectConfig(const std::filesystem::path &start); + + /// Applies the text of a `catch3.json` to `options`. Its keys are the long + /// options of the command line, and its values go through the same checks; + /// relative paths are read from `directory`, the file's own. `file` names + /// the file in messages. + [[nodiscard]] Expected<Options> + ApplyProjectConfig(std::string_view text, + const std::filesystem::path &directory, + std::string_view file, + Options options = {}); + + /// Reads the command line together with the `catch3.json` it selects: + /// --config names one, --no-config reads none, and otherwise + /// FindProjectConfig searches from `workingDirectory`. Options on the + /// command line win over the file, and test specs on the command line + /// replace those of the file. Workers, --help and --version read no file. + [[nodiscard]] Expected<Options> + ParseCommandLine(const std::vector<std::string> &arguments, + const std::filesystem::path &workingDirectory); +} // namespace Progmasoft::Catch3::Runner +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 + #include <mutex> #include <optional> @@ -12776,32 +12832,6 @@ namespace Progmasoft::Catch3::Runner names + ", not '" + value.value_or("") + "'"}; } - std::size_t - EditDistance(std::string_view left, std::string_view right) - { - std::vector<std::size_t> previous(right.size() + 1); - std::vector<std::size_t> current(right.size() + 1); - for (std::size_t index = 0; index <= right.size(); ++index) - { - previous[index] = index; - } - for (std::size_t row = 1; row <= left.size(); ++row) - { - current[0] = row; - for (std::size_t column = 1; column <= right.size(); ++column) - { - const std::size_t substitution = - previous[column - 1] + - (left[row - 1] == right[column - 1] ? 0 : 1); - current[column] = std::min({previous[column] + 1, - current[column - 1] + 1, - substitution}); - } - std::swap(previous, current); - } - return previous[right.size()]; - } - const std::vector<OptionSpec> & Specs() { @@ -13453,6 +13483,29 @@ namespace Progmasoft::Catch3::Runner } return ReadCount(value, "slowest", options.slowest, 1); }}, + // Configuration. + {"config", + '\0', + Takes::kValue, + "FILE", + "Configuration", + "read this catch3.json instead of searching for one", + [](Options &options, const std::optional<std::string> &value) { + options.configPath = *value; + options.noConfig = false; + return Status::Ok(); + }}, + {"no-config", + '\0', + Takes::kNothing, + "", + "Configuration", + "read no catch3.json", + [](Options &options, const std::optional<std::string> &) { + options.noConfig = true; + options.configPath.clear(); + return Status::Ok(); + }}, // Information. {"list", 'l', @@ -13585,10 +13638,35 @@ namespace Progmasoft::Catch3::Runner } } // namespace + std::size_t + EditDistance(std::string_view left, std::string_view right) + { + std::vector<std::size_t> previous(right.size() + 1); + std::vector<std::size_t> current(right.size() + 1); + for (std::size_t index = 0; index <= right.size(); ++index) + { + previous[index] = index; + } + for (std::size_t row = 1; row <= left.size(); ++row) + { + current[0] = row; + for (std::size_t column = 1; column <= right.size(); ++column) + { + const std::size_t substitution = + previous[column - 1] + + (left[row - 1] == right[column - 1] ? 0 : 1); + current[column] = std::min({previous[column] + 1, + current[column - 1] + 1, + substitution}); + } + std::swap(previous, current); + } + return previous[right.size()]; + } + Expected<Options> - ParseOptions(const std::vector<std::string> &arguments) + ParseOptions(const std::vector<std::string> &arguments, Options options) { - Options options; bool onlySpecs = false; for (std::size_t index = 0; index < arguments.size(); ++index) { @@ -15683,6 +15761,551 @@ namespace Progmasoft::Catch3::Runner } // namespace Progmasoft::Catch3::Runner // Copyright (c) 2026 Progmasoft. // SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <cmath> +#include <cstdint> +#include <cstdio> +#include <set> +#include <system_error> +#include <utility> + + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + /// The version of the file format this runner reads. + constexpr double kConfigVersion = 1; + + /// What the value of a key is, and how it becomes arguments. + enum class ValueKind : std::uint8_t + { + /// true gives the option; false, the default, gives nothing. + kFlag, + /// false gives the negating option; true is the default. + kNegated, + /// A string, given as the value of the option. + kWord, + /// A file or directory, relative to the file's directory. + kPath, + /// A whole number. + kCount, + /// A whole number or a word, such as "auto". + kCountOrWord, + /// A number with a fraction, such as seconds. + kNumber, + /// An array of strings, each given to the option. + kWords, + /// A directory, or false for none. + kPathOrFalse, + /// "quiet", "normal" or "verbose". + kVerbosity, + /// An array of test specs. + kTests, + }; + + struct ConfigKey final + { + std::string_view name; + ValueKind kind; + /// The option it stands for, when it is not spelled like the key. + std::string_view option = {}; + }; + + const std::vector<ConfigKey> & + Keys() + { + static const auto *keys = new std::vector<ConfigKey>{ + { "tests", ValueKind::kTests }, + // Selecting tests. + { "select", ValueKind::kWord }, + { "changes-since", ValueKind::kWord }, + { "incremental", ValueKind::kFlag }, + { "shard-durations", ValueKind::kPath }, + { "filenames-as-tags", ValueKind::kFlag }, + // Running. + { "jobs", ValueKind::kCount }, + { "in-process", ValueKind::kFlag }, + { "isolate", ValueKind::kFlag }, + { "timeout", ValueKind::kNumber }, + { "adaptive-timeout", + ValueKind::kNegated, + "no-adaptive-timeout" }, + { "retries", ValueKind::kCountOrWord }, + { "repeat", ValueKind::kCount }, + { "fail-fast", ValueKind::kCount }, + { "order", ValueKind::kWord }, + { "seed", ValueKind::kCountOrWord }, + { "bisect", ValueKind::kNegated, "no-bisect" }, + // How tests behave. + { "success", ValueKind::kFlag }, + { "no-throw", ValueKind::kFlag }, + { "warn", ValueKind::kWords }, + { "allow-no-tests", ValueKind::kFlag }, + { "skip-benchmarks", ValueKind::kFlag }, + { "benchmark-samples", ValueKind::kCount }, + { "benchmark-resamples", ValueKind::kCount }, + { "benchmark-confidence-interval", ValueKind::kNumber }, + { "benchmark-no-analysis", ValueKind::kFlag }, + { "benchmark-warmup-time", ValueKind::kCount }, + // History and analysis. + { "history", ValueKind::kPathOrFalse }, + { "quarantine", ValueKind::kWord }, + // Output. + { "verbosity", ValueKind::kVerbosity }, + { "color", ValueKind::kWord }, + { "junit", ValueKind::kPath }, + { "json", ValueKind::kPath }, + { "trace", ValueKind::kPath }, + { "slowest", ValueKind::kCount }, + }; + return *keys; + } + + /// Options that describe one run rather than a project. + constexpr std::string_view kCommandLineOnly[] = { + "section", + "generator-index", + "path-filter", + "input-file", + "shard", + "break", + "replay", + "analytics", + "verbose", + "quiet", + "premature-exit-guard-file", + "list", + "list-tags", + "help", + "config", + "no-config", + "catch3-worker", + }; + + std::string + Quoted(std::string_view key) + { + return "\"" + std::string{ key } + "\""; + } + + std::string + UnknownKey(std::string_view key) + { + for (const std::string_view option : kCommandLineOnly) + { + if (key == option) + { + return Quoted(key) + + " belongs on the command line, not in a " + "catch3.json"; + } + } + std::string message = Quoted(key) + " is not a key of catch3.json"; + std::string_view closest; + std::size_t best = 3; + const auto consider = [&](std::string_view name) { + const std::size_t distance = EditDistance(key, name); + if (distance < best) + { + best = distance; + closest = name; + } + }; + consider("version"); + for (const auto &entry : Keys()) + { + consider(entry.name); + } + if (!closest.empty()) + { + message += "; did you mean " + Quoted(closest) + "?"; + } + return message; + } + + /// A whole number JSON can say exactly, or nothing. + std::optional<std::uint64_t> + WholeNumber(const JsonValue &value) + { + if (!value.IsNumber()) + { + return std::nullopt; + } + const double number = value.AsNumber(); + if (!std::isfinite(number) || number < 0 + || number != std::floor(number) || number > 9007199254740992.0) + { + return std::nullopt; + } + return static_cast<std::uint64_t>(number); + } + + /// A number in the form the command line reads: digits and a point. + std::string + DecimalText(double number) + { + char text[400]; + std::snprintf(text, sizeof(text), "%.17f", number); + std::string result{ text }; + for (char &character : result) + { + // The decimal separator of the locale a test may have set. + if (character == ',') + { + character = '.'; + } + } + return result; + } + + std::string + PathText(const std::string &value, + const std::filesystem::path &directory) + { + const std::filesystem::path path{ value }; + if (path.is_absolute() || directory.empty()) + { + return path.lexically_normal().string(); + } + return (directory / path).lexically_normal().string(); + } + + /// The arguments that one key and its value stand for. + Expected<std::vector<std::string>> + KeyArguments(const ConfigKey &key, + const JsonValue &value, + const std::filesystem::path &directory) + { + const std::string option + = "--" + + std::string{ key.option.empty() ? key.name : key.option }; + const auto wrong = [&](std::string_view expected) { + return Error{ Quoted(key.name) + " takes " + + std::string{ expected } + ", not " + + value.ToString() }; + }; + switch (key.kind) + { + case ValueKind::kFlag: + if (value.GetKind() != JsonValue::Kind::kBoolean) + { + return wrong("true or false"); + } + if (value.AsBoolean()) + { + return std::vector<std::string>{ option }; + } + return std::vector<std::string>{}; + case ValueKind::kNegated: + if (value.GetKind() != JsonValue::Kind::kBoolean) + { + return wrong("true or false"); + } + if (!value.AsBoolean()) + { + return std::vector<std::string>{ option }; + } + return std::vector<std::string>{}; + case ValueKind::kWord: + if (!value.IsString()) + { + return wrong("a string"); + } + return std::vector<std::string>{ option + "=" + + value.AsString() }; + case ValueKind::kPath: + if (!value.IsString() || value.AsString().empty()) + { + return wrong("a path"); + } + return std::vector<std::string>{ + option + "=" + PathText(value.AsString(), directory) + }; + case ValueKind::kCount: + if (const auto number = WholeNumber(value)) + { + return std::vector<std::string>{ + option + "=" + std::to_string(*number) + }; + } + return wrong("a whole number"); + case ValueKind::kCountOrWord: + if (value.IsString()) + { + return std::vector<std::string>{ option + "=" + + value.AsString() }; + } + if (const auto number = WholeNumber(value)) + { + return std::vector<std::string>{ + option + "=" + std::to_string(*number) + }; + } + return wrong("a whole number or a string"); + case ValueKind::kNumber: + if (!value.IsNumber() || !std::isfinite(value.AsNumber()) + || value.AsNumber() < 0) + { + return wrong("a number of at least 0"); + } + return std::vector<std::string>{ + option + "=" + DecimalText(value.AsNumber()) + }; + case ValueKind::kWords: + { + if (!value.IsArray()) + { + return wrong("an array of strings"); + } + std::vector<std::string> arguments; + for (const auto &item : value.Items()) + { + if (!item.IsString()) + { + return wrong("an array of strings"); + } + arguments.push_back(option + "=" + item.AsString()); + } + return arguments; + } + case ValueKind::kPathOrFalse: + if (value.GetKind() == JsonValue::Kind::kBoolean + && !value.AsBoolean()) + { + return std::vector<std::string>{ + "--no-" + std::string{ key.name } + }; + } + if (!value.IsString() || value.AsString().empty()) + { + return wrong("a path or false"); + } + return std::vector<std::string>{ + option + "=" + PathText(value.AsString(), directory) + }; + case ValueKind::kVerbosity: + if (value.IsString()) + { + if (value.AsString() == "quiet") + { + return std::vector<std::string>{ "--quiet" }; + } + if (value.AsString() == "verbose") + { + return std::vector<std::string>{ "--verbose" }; + } + if (value.AsString() == "normal") + { + return std::vector<std::string>{}; + } + } + return wrong("\"quiet\", \"normal\" or \"verbose\""); + case ValueKind::kTests: + { + if (!value.IsArray()) + { + return wrong("an array of test specs"); + } + // After --, every argument is a test spec, even one that + // starts with a dash. + std::vector<std::string> arguments{ "--" }; + for (const auto &item : value.Items()) + { + if (!item.IsString() || item.AsString().empty()) + { + return wrong("an array of test specs"); + } + arguments.push_back(item.AsString()); + } + return arguments; + } + } + return wrong("a known kind of value"); + } + } // namespace + + std::optional<std::filesystem::path> + FindProjectConfig(const std::filesystem::path &start) + { + std::error_code error; + std::filesystem::path directory + = std::filesystem::absolute(start, error).lexically_normal(); + if (error) + { + return std::nullopt; + } + if (directory.filename().empty() && directory.has_relative_path()) + { + directory = directory.parent_path(); + } + std::vector<std::filesystem::path> candidates; + for (std::filesystem::path current = directory;; + current = current.parent_path()) + { + candidates.push_back(current); + if (std::filesystem::exists(current / ".git", error)) + { + break; + } + if (!current.has_relative_path()) + { + // No repository holds the directory. + candidates = { directory }; + break; + } + } + for (const auto &candidate : candidates) + { + const auto file = candidate / kProjectConfigName; + if (std::filesystem::is_regular_file(file, error)) + { + return file; + } + } + return std::nullopt; + } + + Expected<Options> + ApplyProjectConfig(std::string_view text, + const std::filesystem::path &directory, + std::string_view file, + Options options) + { + const std::string prefix = std::string{ file } + ": "; + auto document = ParseJson(text); + if (!document) + { + return Error{ prefix + document.GetError().Message() }; + } + if (!document->IsObject()) + { + return Error{ prefix + "the file holds an object, not " + + document->ToString() }; + } + std::set<std::string_view> seen; + bool versioned = false; + for (const auto &[name, value] : document->Members()) + { + if (!seen.insert(name).second) + { + return Error{ prefix + Quoted(name) + " appears twice" }; + } + if (name == "version") + { + if (!value.IsNumber() || value.AsNumber() != kConfigVersion) + { + return Error{ + prefix + "version " + value.ToString() + + " is not known; this Catch3 reads version 1" + }; + } + versioned = true; + continue; + } + if (name == "$schema") + { + // For editors; the runner knows its own format. + continue; + } + const ConfigKey *key = nullptr; + for (const auto &entry : Keys()) + { + if (entry.name == name) + { + key = &entry; + break; + } + } + if (key == nullptr) + { + return Error{ prefix + UnknownKey(name) }; + } + auto arguments = KeyArguments(*key, value, directory); + if (!arguments) + { + return Error{ prefix + arguments.GetError().Message() }; + } + auto applied = ParseOptions(*arguments, std::move(options)); + if (!applied) + { + return Error{ prefix + Quoted(name) + ": " + + applied.GetError().Message() }; + } + options = std::move(*applied); + } + if (!versioned) + { + return Error{ prefix + "\"version\": 1 is missing" }; + } + return options; + } + + Expected<Options> + ParseCommandLine(const std::vector<std::string> &arguments, + const std::filesystem::path &workingDirectory) + { + auto commandLine = ParseOptions(arguments); + if (!commandLine) + { + return commandLine; + } + if (commandLine->noConfig || commandLine->mode == Mode::kWorker + || commandLine->mode == Mode::kHelp + || commandLine->mode == Mode::kVersion) + { + // A worker gets everything it needs from its runner. + return commandLine; + } + std::filesystem::path file; + if (!commandLine->configPath.empty()) + { + file = commandLine->configPath; + if (file.is_relative()) + { + file = workingDirectory / file; + } + } + else if (auto found = FindProjectConfig(workingDirectory)) + { + file = std::move(*found); + } + else + { + return commandLine; + } + file = file.lexically_normal(); + auto text = ReadFile(file); + if (!text) + { + return Error{ "cannot read the configuration " + file.string() + + ": " + text.GetError().Message() }; + } + auto configured + = ApplyProjectConfig(*text, file.parent_path(), file.string()); + if (!configured) + { + return configured; + } + Options base = std::move(*configured); + // Specs on the command line replace those of the file rather than + // widening them with "or". + std::vector<std::string> fileSpecs = std::move(base.testSpecs); + base.testSpecs.clear(); + auto parsed = ParseOptions(arguments, std::move(base)); + if (!parsed) + { + return parsed; + } + if (parsed->testSpecs.empty()) + { + parsed->testSpecs = std::move(fileSpecs); + } + parsed->configPath = file.string(); + return parsed; + } +} // namespace Progmasoft::Catch3::Runner +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 namespace Progmasoft::Catch3::Runner { @@ -16148,8 +16771,10 @@ namespace Progmasoft::Catch3::Runner #include <chrono> #include <cstdio> #include <exception> +#include <filesystem> #include <map> #include <random> +#include <system_error> namespace Progmasoft::Catch3 @@ -16641,7 +17266,9 @@ namespace Progmasoft::Catch3 { rest.assign(arguments.begin() + 1, arguments.end()); } - auto parsed = Runner::ParseOptions(rest); + std::error_code here; + auto parsed = Runner::ParseCommandLine( + rest, std::filesystem::current_path(here)); if (!parsed) { PrintError(parsed.GetError().Message()); @@ -16676,6 +17303,11 @@ namespace Progmasoft::Catch3 } std::vector<std::string> notes; + if (!options.configPath.empty() + && options.verbosity == Runner::Verbosity::kVerbose) + { + notes.push_back("configuration from " + options.configPath); + } if (BazelTest()) { ApplyBazel(options, notes); diff --git a/extras/catch_amalgamated.hpp b/extras/catch_amalgamated.hpp index 200d0b5e..07a42b88 100644 --- a/extras/catch_amalgamated.hpp +++ b/extras/catch_amalgamated.hpp @@ -7,7 +7,7 @@ // SPDX-License-Identifier: BSL-1.0 // Catch3 v0.3.0, on the Catch2 v3.16.1 engine -// Generated: 2026-10-25 13:52:54.193517 +// Generated: 2026-10-29 19:10:35.285105 // ---------------------------------------------------------- // This file is an amalgamation of multiple different files. // You probably shouldn't edit it directly. diff --git a/extras/catch_amalgamated_third_party.hpp b/extras/catch_amalgamated_third_party.hpp index d2c1e771..f5961365 100644 --- a/extras/catch_amalgamated_third_party.hpp +++ b/extras/catch_amalgamated_third_party.hpp @@ -1,5 +1,5 @@ // Catch3 v0.3.0: yyjson and pugixml for catch_amalgamated.cpp -// Generated: 2026-10-25 13:52:54.868632 +// Generated: 2026-10-29 19:10:35.543593 // ---------------------------------------------------------- // This file is an amalgamation of src/Progmasoft/Runner/Yyjson.inl, // src/Progmasoft/PugiXml.inl and the libraries they include from diff --git a/fuzzing/CMakeLists.txt b/fuzzing/CMakeLists.txt index 05591520..c5b71e9c 100644 --- a/fuzzing/CMakeLists.txt +++ b/fuzzing/CMakeLists.txt @@ -20,7 +20,7 @@ foreach(fuzzer TestSpecParser) endforeach() # Progmasoft's fuzzers live apart, under their own license header. -foreach(fuzzer RunnerJson RunnerOptions) +foreach(fuzzer ProjectConfig RunnerJson RunnerOptions) add_executable(fuzz_${fuzzer} Progmasoft/fuzz_${fuzzer}.cpp) target_link_libraries(fuzz_${fuzzer} PRIVATE fuzzhelper) endforeach() diff --git a/fuzzing/Progmasoft/fuzz_ProjectConfig.cpp b/fuzzing/Progmasoft/fuzz_ProjectConfig.cpp new file mode 100644 index 00000000..0c43f550 --- /dev/null +++ b/fuzzing/Progmasoft/fuzz_ProjectConfig.cpp @@ -0,0 +1,30 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 + +// The text of a catch3.json. Reading it must give options or an error, never +// crash, and the worker arguments derived from the options must parse again. + +#include <Progmasoft/Runner/Options.hpp> +#include <Progmasoft/Runner/ProjectConfig.hpp> + +#include <cstdint> +#include <cstdlib> +#include <string_view> + +extern "C" int +LLVMFuzzerTestOneInput(const std::uint8_t *data, std::size_t size) +{ + namespace Runner = Progmasoft::Catch3::Runner; + const std::string_view text{reinterpret_cast<const char *>(data), size}; + auto options = Runner::ApplyProjectConfig(text, "project", "catch3.json"); + if (options) + { + const auto worker = + Runner::ParseOptions(Runner::WorkerArguments(*options)); + if (!worker) + { + std::abort(); + } + } + return 0; +} diff --git a/premake5.lua b/premake5.lua index 311ba685..dc6fbb13 100644 --- a/premake5.lua +++ b/premake5.lua @@ -89,6 +89,7 @@ project "ProgmasoftPropertyTests" "tests/Progmasoft/RunnerUnits.tests.cpp", "tests/Progmasoft/RunnerPlanner.tests.cpp", "tests/Progmasoft/RunnerOptions.tests.cpp", + "tests/Progmasoft/RunnerConfig.tests.cpp", "tests/Progmasoft/RunnerAnalysis.tests.cpp", "tests/Progmasoft/RunnerOutput.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp", diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index a33fc0a3..f3b7812d 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -308,6 +308,7 @@ set(PROGMASOFT_RUNNER_FILES "${CATCH_DIR}/src/Progmasoft/Runner/Options.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Planner.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Process.hpp" + "${CATCH_DIR}/src/Progmasoft/Runner/ProjectConfig.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Protocol.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/ResultsJson.hpp" "${CATCH_DIR}/src/Progmasoft/Runner/Scheduler.hpp" @@ -326,6 +327,7 @@ set(PROGMASOFT_RUNNER_FILES "${CATCH_DIR}/src/Progmasoft/Runner/Options.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Planner.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Process.cpp" + "${CATCH_DIR}/src/Progmasoft/Runner/ProjectConfig.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Protocol.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/ResultsJson.cpp" "${CATCH_DIR}/src/Progmasoft/Runner/Runner.cpp" diff --git a/src/Progmasoft/Catch3/meson.build b/src/Progmasoft/Catch3/meson.build index 06faf162..70bf336d 100644 --- a/src/Progmasoft/Catch3/meson.build +++ b/src/Progmasoft/Catch3/meson.build @@ -295,6 +295,7 @@ runner_sources = files( '../Runner/Options.cpp', '../Runner/Planner.cpp', '../Runner/Process.cpp', + '../Runner/ProjectConfig.cpp', '../Runner/Protocol.cpp', '../Runner/ResultsJson.cpp', '../Runner/Runner.cpp', diff --git a/src/Progmasoft/Runner/Options.cpp b/src/Progmasoft/Runner/Options.cpp index bb36ab36..5ca48dd4 100644 --- a/src/Progmasoft/Runner/Options.cpp +++ b/src/Progmasoft/Runner/Options.cpp @@ -140,32 +140,6 @@ namespace Progmasoft::Catch3::Runner names + ", not '" + value.value_or("") + "'"}; } - std::size_t - EditDistance(std::string_view left, std::string_view right) - { - std::vector<std::size_t> previous(right.size() + 1); - std::vector<std::size_t> current(right.size() + 1); - for (std::size_t index = 0; index <= right.size(); ++index) - { - previous[index] = index; - } - for (std::size_t row = 1; row <= left.size(); ++row) - { - current[0] = row; - for (std::size_t column = 1; column <= right.size(); ++column) - { - const std::size_t substitution = - previous[column - 1] + - (left[row - 1] == right[column - 1] ? 0 : 1); - current[column] = std::min({previous[column] + 1, - current[column - 1] + 1, - substitution}); - } - std::swap(previous, current); - } - return previous[right.size()]; - } - const std::vector<OptionSpec> & Specs() { @@ -817,6 +791,29 @@ namespace Progmasoft::Catch3::Runner } return ReadCount(value, "slowest", options.slowest, 1); }}, + // Configuration. + {"config", + '\0', + Takes::kValue, + "FILE", + "Configuration", + "read this catch3.json instead of searching for one", + [](Options &options, const std::optional<std::string> &value) { + options.configPath = *value; + options.noConfig = false; + return Status::Ok(); + }}, + {"no-config", + '\0', + Takes::kNothing, + "", + "Configuration", + "read no catch3.json", + [](Options &options, const std::optional<std::string> &) { + options.noConfig = true; + options.configPath.clear(); + return Status::Ok(); + }}, // Information. {"list", 'l', @@ -949,10 +946,35 @@ namespace Progmasoft::Catch3::Runner } } // namespace + std::size_t + EditDistance(std::string_view left, std::string_view right) + { + std::vector<std::size_t> previous(right.size() + 1); + std::vector<std::size_t> current(right.size() + 1); + for (std::size_t index = 0; index <= right.size(); ++index) + { + previous[index] = index; + } + for (std::size_t row = 1; row <= left.size(); ++row) + { + current[0] = row; + for (std::size_t column = 1; column <= right.size(); ++column) + { + const std::size_t substitution = + previous[column - 1] + + (left[row - 1] == right[column - 1] ? 0 : 1); + current[column] = std::min({previous[column] + 1, + current[column - 1] + 1, + substitution}); + } + std::swap(previous, current); + } + return previous[right.size()]; + } + Expected<Options> - ParseOptions(const std::vector<std::string> &arguments) + ParseOptions(const std::vector<std::string> &arguments, Options options) { - Options options; bool onlySpecs = false; for (std::size_t index = 0; index < arguments.size(); ++index) { diff --git a/src/Progmasoft/Runner/Options.hpp b/src/Progmasoft/Runner/Options.hpp index 2820c1c0..f7bf0445 100644 --- a/src/Progmasoft/Runner/Options.hpp +++ b/src/Progmasoft/Runner/Options.hpp @@ -6,8 +6,10 @@ #include <chrono> #include <cstdint> +#include <cstddef> #include <optional> #include <string> +#include <string_view> #include <vector> namespace Progmasoft::Catch3::Runner @@ -165,11 +167,24 @@ namespace Progmasoft::Catch3::Runner /// Created when the run starts and removed when it ends normally, so /// a file left behind shows that the run ended early. std::string exitGuardPath; + + // Configuration. + /// The `catch3.json` named by --config; after ParseCommandLine, the + /// one that was read, if any. + std::string configPath; + bool noConfig = false; }; - /// Reads a command line. `arguments` does not include the program. + /// Reads a command line on top of `options`. `arguments` does not include + /// the program. [[nodiscard]] Expected<Options> - ParseOptions(const std::vector<std::string> &arguments); + ParseOptions(const std::vector<std::string> &arguments, + Options options = {}); + + /// How many single-character edits turn `left` into `right`, for + /// suggesting the name that was meant. + [[nodiscard]] std::size_t + EditDistance(std::string_view left, std::string_view right); /// The help text, for `--help`. [[nodiscard]] std::string diff --git a/src/Progmasoft/Runner/ProjectConfig.cpp b/src/Progmasoft/Runner/ProjectConfig.cpp new file mode 100644 index 00000000..ea5d921f --- /dev/null +++ b/src/Progmasoft/Runner/ProjectConfig.cpp @@ -0,0 +1,548 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <cmath> +#include <cstdint> +#include <cstdio> +#include <set> +#include <system_error> +#include <utility> + +#include "Files.hpp" +#include "Json.hpp" +#include "ProjectConfig.hpp" + +namespace Progmasoft::Catch3::Runner +{ + namespace + { + /// The version of the file format this runner reads. + constexpr double kConfigVersion = 1; + + /// What the value of a key is, and how it becomes arguments. + enum class ValueKind : std::uint8_t + { + /// true gives the option; false, the default, gives nothing. + kFlag, + /// false gives the negating option; true is the default. + kNegated, + /// A string, given as the value of the option. + kWord, + /// A file or directory, relative to the file's directory. + kPath, + /// A whole number. + kCount, + /// A whole number or a word, such as "auto". + kCountOrWord, + /// A number with a fraction, such as seconds. + kNumber, + /// An array of strings, each given to the option. + kWords, + /// A directory, or false for none. + kPathOrFalse, + /// "quiet", "normal" or "verbose". + kVerbosity, + /// An array of test specs. + kTests, + }; + + struct ConfigKey final + { + std::string_view name; + ValueKind kind; + /// The option it stands for, when it is not spelled like the key. + std::string_view option = {}; + }; + + const std::vector<ConfigKey> & + Keys() + { + static const auto *keys = new std::vector<ConfigKey>{ + { "tests", ValueKind::kTests }, + // Selecting tests. + { "select", ValueKind::kWord }, + { "changes-since", ValueKind::kWord }, + { "incremental", ValueKind::kFlag }, + { "shard-durations", ValueKind::kPath }, + { "filenames-as-tags", ValueKind::kFlag }, + // Running. + { "jobs", ValueKind::kCount }, + { "in-process", ValueKind::kFlag }, + { "isolate", ValueKind::kFlag }, + { "timeout", ValueKind::kNumber }, + { "adaptive-timeout", + ValueKind::kNegated, + "no-adaptive-timeout" }, + { "retries", ValueKind::kCountOrWord }, + { "repeat", ValueKind::kCount }, + { "fail-fast", ValueKind::kCount }, + { "order", ValueKind::kWord }, + { "seed", ValueKind::kCountOrWord }, + { "bisect", ValueKind::kNegated, "no-bisect" }, + // How tests behave. + { "success", ValueKind::kFlag }, + { "no-throw", ValueKind::kFlag }, + { "warn", ValueKind::kWords }, + { "allow-no-tests", ValueKind::kFlag }, + { "skip-benchmarks", ValueKind::kFlag }, + { "benchmark-samples", ValueKind::kCount }, + { "benchmark-resamples", ValueKind::kCount }, + { "benchmark-confidence-interval", ValueKind::kNumber }, + { "benchmark-no-analysis", ValueKind::kFlag }, + { "benchmark-warmup-time", ValueKind::kCount }, + // History and analysis. + { "history", ValueKind::kPathOrFalse }, + { "quarantine", ValueKind::kWord }, + // Output. + { "verbosity", ValueKind::kVerbosity }, + { "color", ValueKind::kWord }, + { "junit", ValueKind::kPath }, + { "json", ValueKind::kPath }, + { "trace", ValueKind::kPath }, + { "slowest", ValueKind::kCount }, + }; + return *keys; + } + + /// Options that describe one run rather than a project. + constexpr std::string_view kCommandLineOnly[] = { + "section", + "generator-index", + "path-filter", + "input-file", + "shard", + "break", + "replay", + "analytics", + "verbose", + "quiet", + "premature-exit-guard-file", + "list", + "list-tags", + "help", + "config", + "no-config", + "catch3-worker", + }; + + std::string + Quoted(std::string_view key) + { + return "\"" + std::string{ key } + "\""; + } + + std::string + UnknownKey(std::string_view key) + { + for (const std::string_view option : kCommandLineOnly) + { + if (key == option) + { + return Quoted(key) + + " belongs on the command line, not in a " + "catch3.json"; + } + } + std::string message = Quoted(key) + " is not a key of catch3.json"; + std::string_view closest; + std::size_t best = 3; + const auto consider = [&](std::string_view name) { + const std::size_t distance = EditDistance(key, name); + if (distance < best) + { + best = distance; + closest = name; + } + }; + consider("version"); + for (const auto &entry : Keys()) + { + consider(entry.name); + } + if (!closest.empty()) + { + message += "; did you mean " + Quoted(closest) + "?"; + } + return message; + } + + /// A whole number JSON can say exactly, or nothing. + std::optional<std::uint64_t> + WholeNumber(const JsonValue &value) + { + if (!value.IsNumber()) + { + return std::nullopt; + } + const double number = value.AsNumber(); + if (!std::isfinite(number) || number < 0 + || number != std::floor(number) || number > 9007199254740992.0) + { + return std::nullopt; + } + return static_cast<std::uint64_t>(number); + } + + /// A number in the form the command line reads: digits and a point. + std::string + DecimalText(double number) + { + char text[400]; + std::snprintf(text, sizeof(text), "%.17f", number); + std::string result{ text }; + for (char &character : result) + { + // The decimal separator of the locale a test may have set. + if (character == ',') + { + character = '.'; + } + } + return result; + } + + std::string + PathText(const std::string &value, + const std::filesystem::path &directory) + { + const std::filesystem::path path{ value }; + if (path.is_absolute() || directory.empty()) + { + return path.lexically_normal().string(); + } + return (directory / path).lexically_normal().string(); + } + + /// The arguments that one key and its value stand for. + Expected<std::vector<std::string>> + KeyArguments(const ConfigKey &key, + const JsonValue &value, + const std::filesystem::path &directory) + { + const std::string option + = "--" + + std::string{ key.option.empty() ? key.name : key.option }; + const auto wrong = [&](std::string_view expected) { + return Error{ Quoted(key.name) + " takes " + + std::string{ expected } + ", not " + + value.ToString() }; + }; + switch (key.kind) + { + case ValueKind::kFlag: + if (value.GetKind() != JsonValue::Kind::kBoolean) + { + return wrong("true or false"); + } + if (value.AsBoolean()) + { + return std::vector<std::string>{ option }; + } + return std::vector<std::string>{}; + case ValueKind::kNegated: + if (value.GetKind() != JsonValue::Kind::kBoolean) + { + return wrong("true or false"); + } + if (!value.AsBoolean()) + { + return std::vector<std::string>{ option }; + } + return std::vector<std::string>{}; + case ValueKind::kWord: + if (!value.IsString()) + { + return wrong("a string"); + } + return std::vector<std::string>{ option + "=" + + value.AsString() }; + case ValueKind::kPath: + if (!value.IsString() || value.AsString().empty()) + { + return wrong("a path"); + } + return std::vector<std::string>{ + option + "=" + PathText(value.AsString(), directory) + }; + case ValueKind::kCount: + if (const auto number = WholeNumber(value)) + { + return std::vector<std::string>{ + option + "=" + std::to_string(*number) + }; + } + return wrong("a whole number"); + case ValueKind::kCountOrWord: + if (value.IsString()) + { + return std::vector<std::string>{ option + "=" + + value.AsString() }; + } + if (const auto number = WholeNumber(value)) + { + return std::vector<std::string>{ + option + "=" + std::to_string(*number) + }; + } + return wrong("a whole number or a string"); + case ValueKind::kNumber: + if (!value.IsNumber() || !std::isfinite(value.AsNumber()) + || value.AsNumber() < 0) + { + return wrong("a number of at least 0"); + } + return std::vector<std::string>{ + option + "=" + DecimalText(value.AsNumber()) + }; + case ValueKind::kWords: + { + if (!value.IsArray()) + { + return wrong("an array of strings"); + } + std::vector<std::string> arguments; + for (const auto &item : value.Items()) + { + if (!item.IsString()) + { + return wrong("an array of strings"); + } + arguments.push_back(option + "=" + item.AsString()); + } + return arguments; + } + case ValueKind::kPathOrFalse: + if (value.GetKind() == JsonValue::Kind::kBoolean + && !value.AsBoolean()) + { + return std::vector<std::string>{ + "--no-" + std::string{ key.name } + }; + } + if (!value.IsString() || value.AsString().empty()) + { + return wrong("a path or false"); + } + return std::vector<std::string>{ + option + "=" + PathText(value.AsString(), directory) + }; + case ValueKind::kVerbosity: + if (value.IsString()) + { + if (value.AsString() == "quiet") + { + return std::vector<std::string>{ "--quiet" }; + } + if (value.AsString() == "verbose") + { + return std::vector<std::string>{ "--verbose" }; + } + if (value.AsString() == "normal") + { + return std::vector<std::string>{}; + } + } + return wrong("\"quiet\", \"normal\" or \"verbose\""); + case ValueKind::kTests: + { + if (!value.IsArray()) + { + return wrong("an array of test specs"); + } + // After --, every argument is a test spec, even one that + // starts with a dash. + std::vector<std::string> arguments{ "--" }; + for (const auto &item : value.Items()) + { + if (!item.IsString() || item.AsString().empty()) + { + return wrong("an array of test specs"); + } + arguments.push_back(item.AsString()); + } + return arguments; + } + } + return wrong("a known kind of value"); + } + } // namespace + + std::optional<std::filesystem::path> + FindProjectConfig(const std::filesystem::path &start) + { + std::error_code error; + std::filesystem::path directory + = std::filesystem::absolute(start, error).lexically_normal(); + if (error) + { + return std::nullopt; + } + if (directory.filename().empty() && directory.has_relative_path()) + { + directory = directory.parent_path(); + } + std::vector<std::filesystem::path> candidates; + for (std::filesystem::path current = directory;; + current = current.parent_path()) + { + candidates.push_back(current); + if (std::filesystem::exists(current / ".git", error)) + { + break; + } + if (!current.has_relative_path()) + { + // No repository holds the directory. + candidates = { directory }; + break; + } + } + for (const auto &candidate : candidates) + { + const auto file = candidate / kProjectConfigName; + if (std::filesystem::is_regular_file(file, error)) + { + return file; + } + } + return std::nullopt; + } + + Expected<Options> + ApplyProjectConfig(std::string_view text, + const std::filesystem::path &directory, + std::string_view file, + Options options) + { + const std::string prefix = std::string{ file } + ": "; + auto document = ParseJson(text); + if (!document) + { + return Error{ prefix + document.GetError().Message() }; + } + if (!document->IsObject()) + { + return Error{ prefix + "the file holds an object, not " + + document->ToString() }; + } + std::set<std::string_view> seen; + bool versioned = false; + for (const auto &[name, value] : document->Members()) + { + if (!seen.insert(name).second) + { + return Error{ prefix + Quoted(name) + " appears twice" }; + } + if (name == "version") + { + if (!value.IsNumber() || value.AsNumber() != kConfigVersion) + { + return Error{ + prefix + "version " + value.ToString() + + " is not known; this Catch3 reads version 1" + }; + } + versioned = true; + continue; + } + if (name == "$schema") + { + // For editors; the runner knows its own format. + continue; + } + const ConfigKey *key = nullptr; + for (const auto &entry : Keys()) + { + if (entry.name == name) + { + key = &entry; + break; + } + } + if (key == nullptr) + { + return Error{ prefix + UnknownKey(name) }; + } + auto arguments = KeyArguments(*key, value, directory); + if (!arguments) + { + return Error{ prefix + arguments.GetError().Message() }; + } + auto applied = ParseOptions(*arguments, std::move(options)); + if (!applied) + { + return Error{ prefix + Quoted(name) + ": " + + applied.GetError().Message() }; + } + options = std::move(*applied); + } + if (!versioned) + { + return Error{ prefix + "\"version\": 1 is missing" }; + } + return options; + } + + Expected<Options> + ParseCommandLine(const std::vector<std::string> &arguments, + const std::filesystem::path &workingDirectory) + { + auto commandLine = ParseOptions(arguments); + if (!commandLine) + { + return commandLine; + } + if (commandLine->noConfig || commandLine->mode == Mode::kWorker + || commandLine->mode == Mode::kHelp + || commandLine->mode == Mode::kVersion) + { + // A worker gets everything it needs from its runner. + return commandLine; + } + std::filesystem::path file; + if (!commandLine->configPath.empty()) + { + file = commandLine->configPath; + if (file.is_relative()) + { + file = workingDirectory / file; + } + } + else if (auto found = FindProjectConfig(workingDirectory)) + { + file = std::move(*found); + } + else + { + return commandLine; + } + file = file.lexically_normal(); + auto text = ReadFile(file); + if (!text) + { + return Error{ "cannot read the configuration " + file.string() + + ": " + text.GetError().Message() }; + } + auto configured + = ApplyProjectConfig(*text, file.parent_path(), file.string()); + if (!configured) + { + return configured; + } + Options base = std::move(*configured); + // Specs on the command line replace those of the file rather than + // widening them with "or". + std::vector<std::string> fileSpecs = std::move(base.testSpecs); + base.testSpecs.clear(); + auto parsed = ParseOptions(arguments, std::move(base)); + if (!parsed) + { + return parsed; + } + if (parsed->testSpecs.empty()) + { + parsed->testSpecs = std::move(fileSpecs); + } + parsed->configPath = file.string(); + return parsed; + } +} // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/ProjectConfig.hpp b/src/Progmasoft/Runner/ProjectConfig.hpp new file mode 100644 index 00000000..c715b1ed --- /dev/null +++ b/src/Progmasoft/Runner/ProjectConfig.hpp @@ -0,0 +1,44 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <filesystem> +#include <optional> +#include <string> +#include <string_view> +#include <vector> + +#include "Expected.hpp" +#include "Options.hpp" + +namespace Progmasoft::Catch3::Runner +{ + /// The name of the project configuration file. + inline constexpr std::string_view kProjectConfigName = "catch3.json"; + + /// The `catch3.json` that applies in `start`: the nearest one in `start` + /// or a directory above it, up to the root of the Git repository that + /// holds `start`. Outside a repository only `start` itself is searched, + /// so a stray file in a home directory never applies. + [[nodiscard]] std::optional<std::filesystem::path> + FindProjectConfig(const std::filesystem::path &start); + + /// Applies the text of a `catch3.json` to `options`. Its keys are the long + /// options of the command line, and its values go through the same checks; + /// relative paths are read from `directory`, the file's own. `file` names + /// the file in messages. + [[nodiscard]] Expected<Options> + ApplyProjectConfig(std::string_view text, + const std::filesystem::path &directory, + std::string_view file, + Options options = {}); + + /// Reads the command line together with the `catch3.json` it selects: + /// --config names one, --no-config reads none, and otherwise + /// FindProjectConfig searches from `workingDirectory`. Options on the + /// command line win over the file, and test specs on the command line + /// replace those of the file. Workers, --help and --version read no file. + [[nodiscard]] Expected<Options> + ParseCommandLine(const std::vector<std::string> &arguments, + const std::filesystem::path &workingDirectory); +} // namespace Progmasoft::Catch3::Runner diff --git a/src/Progmasoft/Runner/Runner.cpp b/src/Progmasoft/Runner/Runner.cpp index 7e69206a..62acac2c 100644 --- a/src/Progmasoft/Runner/Runner.cpp +++ b/src/Progmasoft/Runner/Runner.cpp @@ -16,8 +16,10 @@ #include <chrono> #include <cstdio> #include <exception> +#include <filesystem> #include <map> #include <random> +#include <system_error> #include "Analysis.hpp" #include "Catalog.hpp" @@ -31,6 +33,7 @@ #include "Options.hpp" #include "Planner.hpp" #include "Process.hpp" +#include "ProjectConfig.hpp" #include "ResultsJson.hpp" #include "Scheduler.hpp" @@ -523,7 +526,9 @@ namespace Progmasoft::Catch3 { rest.assign(arguments.begin() + 1, arguments.end()); } - auto parsed = Runner::ParseOptions(rest); + std::error_code here; + auto parsed = Runner::ParseCommandLine( + rest, std::filesystem::current_path(here)); if (!parsed) { PrintError(parsed.GetError().Message()); @@ -558,6 +563,11 @@ namespace Progmasoft::Catch3 } std::vector<std::string> notes; + if (!options.configPath.empty() + && options.verbosity == Runner::Verbosity::kVerbose) + { + notes.push_back("configuration from " + options.configPath); + } if (BazelTest()) { ApplyBazel(options, notes); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 5d152378..5af76cab 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -164,6 +164,7 @@ add_executable(ProgmasoftSelfTest Progmasoft/RunnerUnits.tests.cpp Progmasoft/RunnerPlanner.tests.cpp Progmasoft/RunnerOptions.tests.cpp + Progmasoft/RunnerConfig.tests.cpp Progmasoft/RunnerAnalysis.tests.cpp Progmasoft/RunnerOutput.tests.cpp Progmasoft/Snapshot.tests.cpp diff --git a/tests/Progmasoft/BUILD.bazel b/tests/Progmasoft/BUILD.bazel index bc3eda98..9627f65d 100644 --- a/tests/Progmasoft/BUILD.bazel +++ b/tests/Progmasoft/BUILD.bazel @@ -16,6 +16,7 @@ cc_test( "RunnerUnits.tests.cpp", "RunnerPlanner.tests.cpp", "RunnerOptions.tests.cpp", + "RunnerConfig.tests.cpp", "RunnerAnalysis.tests.cpp", "RunnerOutput.tests.cpp", "Snapshot.tests.cpp", diff --git a/tests/Progmasoft/RunnerConfig.tests.cpp b/tests/Progmasoft/RunnerConfig.tests.cpp new file mode 100644 index 00000000..4dce1f79 --- /dev/null +++ b/tests/Progmasoft/RunnerConfig.tests.cpp @@ -0,0 +1,423 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 + +// catch3.json: where it is found, what its keys set, how the command line +// wins over it and how a wrong file is explained. + +#include <Progmasoft/Catch3/Matchers/String.hpp> +#include <Progmasoft/Catch3/TestMacros.hpp> +#include <Progmasoft/Runner/Files.hpp> +#include <Progmasoft/Runner/Options.hpp> +#include <Progmasoft/Runner/ProjectConfig.hpp> +#include <chrono> +#include <filesystem> +#include <fstream> +#include <string> +#include <system_error> +#include <vector> + +namespace +{ + namespace Runner = Progmasoft::Catch3::Runner; + namespace fs = std::filesystem; + using Catch::Matchers::ContainsSubstring; + using std::chrono::milliseconds; + + /// A directory of its own under the system's temporary directory, + /// removed with everything in it at the end of the test. + class Scratch final + { + public: + Scratch() + { + std::error_code error; + root_ = fs::temp_directory_path(error) + / ("catch3-config-" + Runner::UniqueName()); + fs::create_directories(root_, error); + REQUIRE_FALSE(error); + } + + Scratch(const Scratch &) = delete; + Scratch & + operator=(const Scratch &) = delete; + + ~Scratch() + { + std::error_code error; + fs::remove_all(root_, error); + } + + [[nodiscard]] const fs::path & + Root() const noexcept + { + return root_; + } + + fs::path + Directory(const fs::path &relative) const + { + std::error_code error; + fs::create_directories(root_ / relative, error); + REQUIRE_FALSE(error); + return root_ / relative; + } + + fs::path + Write(const fs::path &relative, const std::string &text) const + { + const auto file = root_ / relative; + std::error_code error; + fs::create_directories(file.parent_path(), error); + std::ofstream stream(file, std::ios::binary); + stream << text; + REQUIRE(stream); + return file; + } + + private: + fs::path root_; + }; + + Runner::Options + Apply(const std::string &text, const fs::path &directory = {}) + { + auto options + = Runner::ApplyProjectConfig(text, directory, "catch3.json"); + INFO((options ? std::string{} : options.GetError().Message())); + REQUIRE(options); + return *options; + } + + std::string + ApplyError(const std::string &text) + { + auto options = Runner::ApplyProjectConfig(text, {}, "catch3.json"); + REQUIRE_FALSE(options); + return options.GetError().Message(); + } + + bool + InsideRepository(fs::path directory) + { + std::error_code error; + for (;; directory = directory.parent_path()) + { + if (fs::exists(directory / ".git", error)) + { + return true; + } + if (!directory.has_relative_path()) + { + return false; + } + } + } +} // namespace + +TEST_CASE("catch3.json is found up to the root of the repository", + "[progmasoft][runner][config]") +{ + const Scratch scratch; + scratch.Directory(".git"); + const auto deep = scratch.Directory("a/b/c"); + + CHECK_FALSE(Runner::FindProjectConfig(deep)); + + const auto top = scratch.Write("catch3.json", "{\"version\": 1}"); + CHECK(Runner::FindProjectConfig(deep) == top); + CHECK(Runner::FindProjectConfig(scratch.Root()) == top); + + // The nearest file wins. + const auto near = scratch.Write("a/b/catch3.json", "{\"version\": 1}"); + CHECK(Runner::FindProjectConfig(deep) == near); + CHECK(Runner::FindProjectConfig(scratch.Root() / "a") == top); +} + +TEST_CASE("The search stops at the repository it starts in", + "[progmasoft][runner][config]") +{ + const Scratch scratch; + scratch.Write("catch3.json", "{\"version\": 1}"); + scratch.Directory("project/.git"); + const auto inner = scratch.Directory("project/tests"); + CHECK_FALSE(Runner::FindProjectConfig(inner)); +} + +TEST_CASE("Outside a repository only the working directory is searched", + "[progmasoft][runner][config]") +{ + const Scratch scratch; + if (InsideRepository(scratch.Root())) + { + SKIP("the temporary directory is inside a Git repository"); + } + scratch.Write("catch3.json", "{\"version\": 1}"); + const auto below = scratch.Directory("below"); + CHECK_FALSE(Runner::FindProjectConfig(below)); + CHECK(Runner::FindProjectConfig(scratch.Root()) + == scratch.Root() / "catch3.json"); +} + +TEST_CASE("Keys set the option of the same name", + "[progmasoft][runner][config]") +{ + const auto options = Apply(R"({ + "version": 1, + "$schema": "https://catch3.progmasoft.com/schema/catch3-v1.json", + "tests": ["[unit]", "~[slow]"], + "select": "smart", + "changes-since": "origin/devel", + "incremental": true, + "filenames-as-tags": true, + "jobs": 6, + "isolate": true, + "timeout": 2.5, + "adaptive-timeout": false, + "retries": 3, + "repeat": 2, + "fail-fast": 4, + "order": "random", + "seed": 77, + "bisect": false, + "success": true, + "no-throw": true, + "warn": ["no-assertions", "infinite-generators"], + "allow-no-tests": true, + "skip-benchmarks": true, + "benchmark-samples": 10, + "benchmark-resamples": 1000, + "benchmark-confidence-interval": 0.9, + "benchmark-no-analysis": true, + "benchmark-warmup-time": 5, + "quarantine": "report", + "verbosity": "quiet", + "color": "never", + "slowest": 3 + })"); + CHECK(options.testSpecs == std::vector<std::string>{ "[unit]", "~[slow]" }); + CHECK(options.selection == Runner::Selection::kSmart); + CHECK(options.changesSince == "origin/devel"); + CHECK(options.incremental); + CHECK(options.filenamesAsTags); + CHECK(options.jobs == 6); + CHECK(options.isolate); + CHECK(options.timeout == milliseconds{ 2500 }); + CHECK(options.timeoutGiven); + CHECK_FALSE(options.adaptiveTimeout); + CHECK(options.retryMode == Runner::RetryMode::kAlways); + CHECK(options.retries == 3); + CHECK(options.repeat == 2); + CHECK(options.failFast == 4); + CHECK(options.order == Runner::Order::kRandom); + CHECK(options.seed == 77); + CHECK(options.seedGiven); + CHECK_FALSE(options.bisectOrder); + CHECK(options.includeSuccessful); + CHECK(options.noThrow); + CHECK(options.warnNoAssertions); + CHECK(options.warnInfiniteGenerators); + CHECK_FALSE(options.warnUnmatchedSpecs); + CHECK(options.allowNoTests); + CHECK(options.skipBenchmarks); + CHECK(options.benchmarkSamples == 10); + CHECK(options.benchmarkResamples == 1000); + CHECK(options.benchmarkConfidenceInterval == 0.9); + CHECK(options.benchmarkNoAnalysis); + CHECK(options.benchmarkWarmupTime == milliseconds{ 5 }); + CHECK(options.quarantine == Runner::QuarantineMode::kReport); + CHECK(options.verbosity == Runner::Verbosity::kQuiet); + CHECK(options.color == Runner::ColorMode::kNever); + CHECK(options.slowest == 3); +} + +TEST_CASE("Words and defaults read like the command line", + "[progmasoft][runner][config]") +{ + CHECK(Apply(R"({"version": 1, "retries": "off"})").retryMode + == Runner::RetryMode::kOff); + CHECK_FALSE(Apply(R"({"version": 1, "seed": "random"})").seedGiven); + CHECK_FALSE(Apply(R"({"version": 1, "history": false})").historyEnabled); + CHECK(Apply(R"({"version": 1, "verbosity": "verbose"})").verbosity + == Runner::Verbosity::kVerbose); + CHECK(Apply(R"({"version": 1, "verbosity": "normal"})").verbosity + == Runner::Verbosity::kNormal); + + // false is the default of a switch, and true of a negated one. + const auto defaults = Apply(R"({"version": 1, "isolate": false, + "adaptive-timeout": true, + "bisect": true})"); + CHECK_FALSE(defaults.isolate); + CHECK(defaults.adaptiveTimeout); + CHECK(defaults.bisectOrder); +} + +TEST_CASE("Relative paths are read from the directory of the file", + "[progmasoft][runner][config]") +{ + const fs::path directory = fs::path{ "project" } / "tests"; + const auto options = Apply(R"({ + "version": 1, + "junit": "out/junit.xml", + "json": "../report.json", + "trace": "trace.json", + "history": ".catch3", + "shard-durations": "durations.json" + })", + directory); + CHECK(fs::path{ options.junitPath } + == (directory / "out/junit.xml").lexically_normal()); + CHECK(fs::path{ options.jsonPath } + == fs::path{ "project/report.json" }.lexically_normal()); + CHECK(fs::path{ options.tracePath } + == (directory / "trace.json").lexically_normal()); + CHECK(fs::path{ options.historyDirectory } + == (directory / ".catch3").lexically_normal()); + CHECK(fs::path{ options.shardDurations } + == (directory / "durations.json").lexically_normal()); + + const auto absolute = (fs::temp_directory_path() / "j.xml").string(); + CHECK(fs::path{ Apply( + "{\"version\": 1, \"junit\": " + + [&] { + std::string quoted; + for (const char character : absolute) + { + if (character == '\\' || character == '"') + { + quoted += '\\'; + } + quoted += character; + } + return "\"" + quoted + "\""; + }() + + "}", + directory) + .junitPath } + == fs::path{ absolute }.lexically_normal()); +} + +TEST_CASE("A wrong catch3.json is explained", "[progmasoft][runner][config]") +{ + CHECK_THAT(ApplyError("{}"), + ContainsSubstring("\"version\": 1 is missing")); + CHECK_THAT(ApplyError(R"({"version": 2})"), + ContainsSubstring("this Catch3 reads version 1")); + CHECK_THAT(ApplyError("[1]"), ContainsSubstring("holds an object")); + CHECK_THAT(ApplyError(R"({"version": 1,})"), + ContainsSubstring("catch3.json: ")); + CHECK_THAT(ApplyError(R"({"version": 1, "version": 1})"), + ContainsSubstring("\"version\" appears twice")); + + CHECK_THAT(ApplyError(R"({"version": 1, "jbos": 4})"), + ContainsSubstring("did you mean \"jobs\"?")); + CHECK_THAT(ApplyError(R"({"version": 1, "zzzzzzzz": 4})"), + ContainsSubstring("\"zzzzzzzz\" is not a key of catch3.json") + && !ContainsSubstring("did you mean")); + CHECK_THAT(ApplyError(R"({"version": 1, "shard": "0/2"})"), + ContainsSubstring("belongs on the command line")); + + CHECK_THAT(ApplyError(R"({"version": 1, "jobs": "4"})"), + ContainsSubstring("\"jobs\" takes a whole number")); + CHECK_THAT(ApplyError(R"({"version": 1, "jobs": -1})"), + ContainsSubstring("\"jobs\" takes a whole number")); + CHECK_THAT(ApplyError(R"({"version": 1, "jobs": 1.5})"), + ContainsSubstring("\"jobs\" takes a whole number")); + CHECK_THAT(ApplyError(R"({"version": 1, "isolate": 1})"), + ContainsSubstring("\"isolate\" takes true or false")); + CHECK_THAT(ApplyError(R"({"version": 1, "timeout": -2})"), + ContainsSubstring("\"timeout\" takes a number of at least 0")); + CHECK_THAT(ApplyError(R"({"version": 1, "tests": "[unit]"})"), + ContainsSubstring("an array of test specs")); + CHECK_THAT(ApplyError(R"({"version": 1, "warn": ["no-assertions", 3]})"), + ContainsSubstring("an array of strings")); + CHECK_THAT(ApplyError(R"({"version": 1, "verbosity": "loud"})"), + ContainsSubstring("\"quiet\", \"normal\" or \"verbose\"")); + CHECK_THAT(ApplyError(R"({"version": 1, "history": true})"), + ContainsSubstring("a path or false")); + + // The checks of the command line apply, named by the key. + CHECK_THAT(ApplyError(R"({"version": 1, "order": "sideways"})"), + ContainsSubstring("\"order\": --order takes one of")); + CHECK_THAT(ApplyError(R"({"version": 1, "repeat": 0})"), + ContainsSubstring("\"repeat\": --repeat takes a whole number")); +} + +TEST_CASE("The command line wins over catch3.json", + "[progmasoft][runner][config]") +{ + const Scratch scratch; + scratch.Directory(".git"); + const auto file = scratch.Write("catch3.json", R"({ + "version": 1, + "jobs": 3, + "order": "name", + "tests": ["[unit]"], + "junit": "junit.xml" + })"); + const auto here = scratch.Directory("build"); + + const auto fromFile = Runner::ParseCommandLine({}, here); + REQUIRE(fromFile); + CHECK(fromFile->jobs == 3); + CHECK(fromFile->order == Runner::Order::kName); + CHECK(fromFile->testSpecs == std::vector<std::string>{ "[unit]" }); + CHECK(fs::path{ fromFile->junitPath } + == (scratch.Root() / "junit.xml").lexically_normal()); + CHECK(fs::path{ fromFile->configPath } == file.lexically_normal()); + + const auto overridden = Runner::ParseCommandLine( + { "--jobs=8", "Some test", "--order", "declared" }, + here); + REQUIRE(overridden); + CHECK(overridden->jobs == 8); + CHECK(overridden->order == Runner::Order::kDeclared); + // Specs on the command line replace those of the file. + CHECK(overridden->testSpecs == std::vector<std::string>{ "Some test" }); + CHECK_FALSE(overridden->junitPath.empty()); + + const auto ignored = Runner::ParseCommandLine({ "--no-config" }, here); + REQUIRE(ignored); + CHECK(ignored->jobs == 0); + CHECK(ignored->configPath.empty()); + CHECK(ignored->testSpecs.empty()); +} + +TEST_CASE("--config names the file, and some modes read none", + "[progmasoft][runner][config]") +{ + const Scratch scratch; + scratch.Write("ci/catch3.json", R"({"version": 1, "jobs": 2})"); + scratch.Write("broken.json", R"({"version": 1, "jobs": "many"})"); + + const auto named = Runner::ParseCommandLine({ "--config=ci/catch3.json" }, + scratch.Root()); + REQUIRE(named); + CHECK(named->jobs == 2); + + const auto broken = Runner::ParseCommandLine({ "--config", "broken.json" }, + scratch.Root()); + REQUIRE_FALSE(broken); + CHECK_THAT(broken.GetError().Message(), + ContainsSubstring("broken.json: \"jobs\" takes a whole number")); + + const auto missing + = Runner::ParseCommandLine({ "--config=absent.json" }, scratch.Root()); + REQUIRE_FALSE(missing); + CHECK_THAT(missing.GetError().Message(), + ContainsSubstring("cannot read the configuration")); + + // A broken file does not stand in the way of help, the version or a + // worker, which takes everything from its runner. + for (const char *mode : { "--help", "--version", "--catch3-worker" }) + { + CAPTURE(mode); + CHECK(Runner::ParseCommandLine({ "--config=broken.json", mode }, + scratch.Root())); + } + // The last of --config and --no-config decides. + CHECK(Runner::ParseCommandLine({ "--config=broken.json", "--no-config" }, + scratch.Root())); + CHECK_FALSE( + Runner::ParseCommandLine({ "--no-config", "--config=broken.json" }, + scratch.Root())); +} diff --git a/tests/meson.build b/tests/meson.build index 4a8c00e4..340b61cf 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -80,6 +80,7 @@ progmasoft_property_tests = executable( 'Progmasoft/RunnerUnits.tests.cpp', 'Progmasoft/RunnerPlanner.tests.cpp', 'Progmasoft/RunnerOptions.tests.cpp', + 'Progmasoft/RunnerConfig.tests.cpp', 'Progmasoft/RunnerAnalysis.tests.cpp', 'Progmasoft/RunnerOutput.tests.cpp', 'Progmasoft/Snapshot.tests.cpp', diff --git a/xmake.lua b/xmake.lua index 4e51fc71..91c66992 100644 --- a/xmake.lua +++ b/xmake.lua @@ -63,6 +63,7 @@ target("ProgmasoftPropertyTests") "tests/Progmasoft/RunnerUnits.tests.cpp", "tests/Progmasoft/RunnerPlanner.tests.cpp", "tests/Progmasoft/RunnerOptions.tests.cpp", + "tests/Progmasoft/RunnerConfig.tests.cpp", "tests/Progmasoft/RunnerAnalysis.tests.cpp", "tests/Progmasoft/RunnerOutput.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp",