diff options
| author | Leitwolf11 | 2026-10-23 23:25:41 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-10-23 23:25:41 +0300 |
| commit | 93d4e7dbebab867e0a7f59fc25308460a7a76280 (patch) | |
| tree | 54c568a3063d49e21f039b6d38f40f51be7991f0 /fuzzing | |
| parent | 83c6fc3ac13abe83fa39f7b5f8559ccaf4412233 (diff) | |
| download | catch3-93d4e7dbebab867e0a7f59fc25308460a7a76280.tar.gz | |
Replace Catch2's runner with the Catch3 runner
Catch3 keeps Catch2's API for writing tests (test cases, sections,
assertions, matchers, generators, listeners, benchmarks) and replaces its
session, command line and reporters with a runner of its own.
The runner starts the test binary again as worker processes that take the
next test when they are free. A crash, an exit or a hang ends one test, not
the run. On top of that it adds:
- retries that tell deterministic, flaky and order-dependent failures
apart, and a bisection that names the test leaving state behind;
- a history of verdicts and durations, kept under a file lock, that drives
automatic quarantine, adaptive time limits, predictive ordering,
incremental runs, failure replay and analytics;
- selection by change, past failure or prediction, dependency closure,
and sharding by a stable hash or by recorded durations;
- tags for serial and isolated tests, shared resources, dependencies,
time limits and retries;
- a failure category, a signature, an analysis and a replay command for
every failure;
- console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome
trace reports, and a versioned discovery listing (--list=json);
- a temporary directory per test (TMPDIR, TMP, TEMP and
TestTemporaryDirectory());
- Bazel's test environment: XML output file, test filter, sharding, seed,
premature exit file and undeclared outputs.
The command line is parsed by a hand-written, typed parser. Catch::Session
remains as a thin facade over the runner for custom main functions.
The result model in Progmasoft/Catch3/Results.hpp was rewritten around
tests and attempts, and the JUnit writer now writes it. The property
engine's example database is updated under a file lock with an atomic
rename, so parallel workers can write it at the same time.
Removed: Clara, the command-line and reporter-spec parsers, every reporter
and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width,
the JSON and XML writers, sharding in the test registry, approval tests and
their baselines, Conan packaging, and the unused third_party/clara.hpp.
Listeners keep working; reporters' list and skip events are gone.
The Catch2 library is now compiled as C++20; code that includes only
Catch2's headers still needs C++14. Duplicate test cases are still refused
at start-up. Version 0.3.0.
ExtraTests now turn exceptions off for clang-cl with MSVC-style options,
which -fno-exceptions did not do.
Diffstat (limited to 'fuzzing')
| -rw-r--r-- | fuzzing/CMakeLists.txt | 6 | ||||
| -rw-r--r-- | fuzzing/fuzz_RunnerJson.cpp | 29 | ||||
| -rw-r--r-- | fuzzing/fuzz_RunnerOptions.cpp | 50 | ||||
| -rw-r--r-- | fuzzing/fuzz_XmlWriter.cpp | 22 | ||||
| -rw-r--r-- | fuzzing/fuzz_textflow.cpp | 54 |
5 files changed, 82 insertions, 79 deletions
diff --git a/fuzzing/CMakeLists.txt b/fuzzing/CMakeLists.txt index 8370fe39..6ba940ce 100644 --- a/fuzzing/CMakeLists.txt +++ b/fuzzing/CMakeLists.txt @@ -7,14 +7,14 @@ add_library(fuzzhelper NullOStream.h NullOStream.cpp) target_link_libraries(fuzzhelper PUBLIC Catch2::Catch2) -# use C++17 so we can get string_view -target_compile_features(fuzzhelper PUBLIC cxx_std_17) +# The runner fuzzers include its C++20 headers. +target_compile_features(fuzzhelper PUBLIC cxx_std_20) # This should be possible to set from the outside to be oss-fuzz compatible, # fix later. For now, target libFuzzer only. target_link_options(fuzzhelper PUBLIC "-fsanitize=fuzzer") -foreach(fuzzer TestSpecParser XmlWriter textflow) +foreach(fuzzer TestSpecParser RunnerJson RunnerOptions) add_executable(fuzz_${fuzzer} fuzz_${fuzzer}.cpp) target_link_libraries(fuzz_${fuzzer} PRIVATE fuzzhelper) endforeach() diff --git a/fuzzing/fuzz_RunnerJson.cpp b/fuzzing/fuzz_RunnerJson.cpp new file mode 100644 index 00000000..4cc3770c --- /dev/null +++ b/fuzzing/fuzz_RunnerJson.cpp @@ -0,0 +1,29 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 + +// The runner reads JSON from its workers, from history files and from +// reports; none of them may crash it. A document that parses must also +// print and parse back to the same text. + +#include <Progmasoft/Runner/Json.hpp> + +#include <cstdint> +#include <cstdlib> +#include <string_view> + +extern "C" int +LLVMFuzzerTestOneInput(const std::uint8_t *data, std::size_t size) +{ + const std::string_view text{reinterpret_cast<const char *>(data), size}; + auto parsed = Progmasoft::Catch3::Runner::ParseJson(text); + if (parsed) + { + const auto printed = parsed->ToString(); + auto again = Progmasoft::Catch3::Runner::ParseJson(printed); + if (!again || again->ToString() != printed) + { + std::abort(); + } + } + return 0; +} diff --git a/fuzzing/fuzz_RunnerOptions.cpp b/fuzzing/fuzz_RunnerOptions.cpp new file mode 100644 index 00000000..2a10ba91 --- /dev/null +++ b/fuzzing/fuzz_RunnerOptions.cpp @@ -0,0 +1,50 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 + +// The command line of a test binary: arguments separated by NUL bytes. The +// parser must answer with options or an error, never crash, and the worker +// arguments it derives must parse again. + +#include <Progmasoft/Runner/Options.hpp> + +#include <cstdint> +#include <cstdlib> +#include <string> +#include <vector> + +extern "C" int +LLVMFuzzerTestOneInput(const std::uint8_t *data, std::size_t size) +{ + std::vector<std::string> arguments(1); + for (std::size_t index = 0; index < size; ++index) + { + if (data[index] == 0) + { + arguments.emplace_back(); + } + else + { + arguments.back().push_back(static_cast<char>(data[index])); + } + } + // Reading a file named by --input-file is not what is fuzzed here. + for (const auto &argument : arguments) + { + if (argument.find("input-file") != std::string::npos || + (argument.size() >= 2 && argument[0] == '-' && argument[1] == 'f')) + { + return 0; + } + } + auto options = Progmasoft::Catch3::Runner::ParseOptions(arguments); + if (options) + { + const auto worker = Progmasoft::Catch3::Runner::ParseOptions( + Progmasoft::Catch3::Runner::WorkerArguments(*options)); + if (!worker) + { + std::abort(); + } + } + return 0; +} diff --git a/fuzzing/fuzz_XmlWriter.cpp b/fuzzing/fuzz_XmlWriter.cpp deleted file mode 100644 index 70c4ed80..00000000 --- a/fuzzing/fuzz_XmlWriter.cpp +++ /dev/null @@ -1,22 +0,0 @@ - -// Copyright Catch2 Authors -// Distributed under the Boost Software License, Version 1.0. -// (See accompanying file LICENSE.txt or copy at -// https://www.boost.org/LICENSE_1_0.txt) - -// SPDX-License-Identifier: BSL-1.0 -//By Paul Dreik 2020 - -#include <catch2/internal/catch_xmlwriter.hpp> - -#include "NullOStream.h" - -extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) { - - std::string buf(Data,Data+Size); - NullOStream nul; - Catch::XmlEncode encode(buf); - encode.encodeTo(nul); - return 0; -} - diff --git a/fuzzing/fuzz_textflow.cpp b/fuzzing/fuzz_textflow.cpp deleted file mode 100644 index f3119379..00000000 --- a/fuzzing/fuzz_textflow.cpp +++ /dev/null @@ -1,54 +0,0 @@ - -// Copyright Catch2 Authors -// Distributed under the Boost Software License, Version 1.0. -// (See accompanying file LICENSE.txt or copy at -// https://www.boost.org/LICENSE_1_0.txt) - -// SPDX-License-Identifier: BSL-1.0 -//By Paul Dreik 2020 - -#include <catch2/internal/catch_textflow.hpp> - -#include "NullOStream.h" - -#include <string> -#include <string_view> -#include <cstdint> - - -template<class Callback> -void split(const char *Data, size_t Size, Callback callback) { - - using namespace std::literals; - constexpr auto sep="\n~~~\n"sv; - - std::string_view remainder(Data,Size); - for (;;) { - auto pos=remainder.find(sep); - if(pos==std::string_view::npos) { - //not found. use the remainder and exit - callback(remainder); - return; - } else { - //found. invoke callback on the first part, then proceed with the rest. - callback(remainder.substr(0,pos)); - remainder=remainder.substr(pos+sep.size()); - } - } -} - -extern "C" int LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) { - - Catch::TextFlow::Columns columns; - - // break the input on separator - split((const char*)Data,Size,[&](std::string_view word) { - columns+=Catch::TextFlow::Column(std::string(word)); - }); - - NullOStream nul; - nul << columns; - - return 0; -} - |