catch3
Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.
Clonegit clone https://git.progmasoft.com/catch3
Add composite property generators, single-trial replay and stable counterexample printing
- Commit
5d55a3226360fae1eaa5ef91908b0fa7d7d07234- Author
- Leitwolf11 ·
- Parents
febb5b1e2441
Changes
docs/progmasoft-catch3.md | 66 +++++- docs/why-catch3.md | 2 +- premake5.lua | 1 + src/Progmasoft/BUILD.bazel | 5 + src/Progmasoft/Catch3.hpp | 4 + src/Progmasoft/Catch3/Detail/Sequence.hpp | 97 ++++++++ src/Progmasoft/Catch3/Generators/Element.hpp | 63 +++++ src/Progmasoft/Catch3/Generators/Text.hpp | 93 ++++++++ src/Progmasoft/Catch3/Generators/Tuple.hpp | 83 +++++++ src/Progmasoft/Catch3/Generators/Vector.hpp | 60 +++++ src/Progmasoft/Catch3/Property.hpp | 97 +++++++- src/Progmasoft/meson.build | 5 + tests/CMakeLists.txt | 1 + tests/Progmasoft/BUILD.bazel | 1 + tests/Progmasoft/Generators.tests.cpp | 336 +++++++++++++++++++++++++++ tests/meson.build | 1 + xmake.lua | 1 + 17 files changed, 906 insertions(+), 10 deletions(-) diff --git a/docs/progmasoft-catch3.md b/docs/progmasoft-catch3.md index 43a73f03..55114f32 100644 --- a/docs/progmasoft-catch3.md +++ b/docs/progmasoft-catch3.md @@ -33,7 +33,10 @@ The current property API is split into these headers: bounded sampling, inclusive integer ranges, and per-trial seed derivation. - `<Progmasoft/Catch3/Generator.hpp>`: user-defined generation/shrinking pair. - `<Progmasoft/Catch3/Generators/Integer.hpp>` and `Boolean.hpp`: built-in - strategies with deterministic shrink candidates. + scalar strategies with deterministic shrink candidates. +- `<Progmasoft/Catch3/Generators/Vector.hpp>`, `Tuple.hpp`, `Element.hpp` and + `Text.hpp`: composite strategies described under + [Composite generators](#composite-generators). - `<Progmasoft/Catch3/Property.hpp>`: trial execution and structured results. - `<Progmasoft/Catch3/Assertions.hpp>`: `CATCH3_CHECK_PROPERTY`, which reports the failing trial, replay seed, reason, and minimized counterexample through @@ -53,10 +56,63 @@ minimal value. A false predicate or an exception from the predicate fails the property. Exceptions from generation or shrinking are reported as execution errors, distinct from an ordinary failing property. -The API currently accepts one generated value per property. Tuple strategies, -state-machine commands, parallel trials, and adaptive shrinking are deliberately -not implied by this first module; they need their own deterministic contracts and -tests before being added. +A failing report ends with the line +`replay: set PropertyOptions::ReplayTrialSeed to <seed>`. Setting that option +runs exactly one trial with the given seed, including its shrinking, and +ignores `Seed` and `Trials`; the reported trial number is then 1. This +reproduces a failure from a CI log without rerunning the trials before it. + +```cpp +Progmasoft::Catch3::PropertyOptions{ .ReplayTrialSeed = 1234567890123456789ULL } +``` + +### Composite generators + +A property still receives one generated value. Several arguments are passed +as one tuple: + +```cpp +using namespace Progmasoft::Catch3; + +CATCH3_CHECK_PROPERTY( + Tuple(Vector(Integer<int>(0, 100), 0, 20), Text("abc", 1, 8)), + [](const std::tuple<std::vector<int>, std::string> &input) { + return Accepts(std::get<0>(input), std::get<1>(input)); + }, + PropertyOptions{ .Trials = 500 }); +``` + +- `Tuple(generators...)` draws its components strictly left to right from the + trial's random source and shrinks one component at a time: every candidate + of the first component, then every candidate of the second, and so on. +- `Vector(element, minimumSize, maximumSize)` draws a length in the inclusive + range and then the elements in index order. `Text(alphabet, minimumLength, + maximumLength)` does the same for byte strings over the given alphabet; + `PrintableAscii()` is the default alphabet. Both shrink in a fixed order: + the shortest allowed prefix, the first and second half, the value with one + element removed, and finally the value with one element simplified. A + shrunk value is never shorter than its minimum, each step offers at most + 256 candidates, and a size limit above 1,048,576 elements is rejected. +- `Element(values)` picks uniformly from a fixed non-empty list. The list + order is the simplicity order: a value shrinks to the values listed before + it. `Text` treats its alphabet the same way, so put the simplest value or + character first. + +Counterexamples of tuples, pairs and vectors are printed element by element, +for example `{ 7, { true, false }, "text" }`. This does not depend on Catch2's +optional tuple or range string makers, so the report is the same whatever +configuration macros a client defines. Scalars still use `Catch::StringMaker`, +including user specializations. + +Invalid factory arguments, such as an empty alphabet or a minimum above its +maximum, throw `std::invalid_argument`; a client built without exceptions +gets the message on standard error followed by `std::abort`. + +State-machine commands, parallel trials, mapped or filtered generators, and +adaptive shrinking are deliberately not implied by this module; they need +their own deterministic contracts and tests before being added. Shrinking a +composite value is greedy and local like scalar shrinking: it finds a small +counterexample, not a provably smallest one. ## Snapshots diff --git a/docs/why-catch3.md b/docs/why-catch3.md index 30dd5159..7fbeef44 100644 --- a/docs/why-catch3.md +++ b/docs/why-catch3.md @@ -25,7 +25,7 @@ Catch2 runner. The current property runner handles one generated value per property. Shrinking is deterministic and bounded, but greedy rather than globally minimal. There is -not yet built-in support for state-machine properties, generated tuples, +not yet built-in support for state-machine properties, mapped or filtered generators, discard/classification policies, parallel execution, or a standalone Catch3 runner. Those features are not implied by the initial API. diff --git a/premake5.lua b/premake5.lua index 7a7d5e2a..ca773a60 100644 --- a/premake5.lua +++ b/premake5.lua @@ -81,6 +81,7 @@ project "ProgmasoftPropertyTests" targetdir "build/premake/bin/%{cfg.buildcfg}" objdir "build/premake/obj/ProgmasoftPropertyTests/%{cfg.buildcfg}" files { + "tests/Progmasoft/Generators.tests.cpp", "tests/Progmasoft/Property.tests.cpp", "tests/Progmasoft/Results.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp", diff --git a/src/Progmasoft/BUILD.bazel b/src/Progmasoft/BUILD.bazel index ed49083c..14a67bf6 100644 --- a/src/Progmasoft/BUILD.bazel +++ b/src/Progmasoft/BUILD.bazel @@ -15,9 +15,14 @@ cc_library( "Catch3.hpp", "Catch3/Assertions.hpp", "Catch3/Detail/InvalidArgument.hpp", + "Catch3/Detail/Sequence.hpp", "Catch3/Generator.hpp", "Catch3/Generators/Boolean.hpp", + "Catch3/Generators/Element.hpp", "Catch3/Generators/Integer.hpp", + "Catch3/Generators/Text.hpp", + "Catch3/Generators/Tuple.hpp", + "Catch3/Generators/Vector.hpp", "Catch3/Property.hpp", "Catch3/Random.hpp", "Catch3/Results.hpp", diff --git a/src/Progmasoft/Catch3.hpp b/src/Progmasoft/Catch3.hpp index d2d34e74..5fca369a 100644 --- a/src/Progmasoft/Catch3.hpp +++ b/src/Progmasoft/Catch3.hpp @@ -8,7 +8,11 @@ #include <Progmasoft/Catch3/Assertions.hpp> #include <Progmasoft/Catch3/Generator.hpp> #include <Progmasoft/Catch3/Generators/Boolean.hpp> +#include <Progmasoft/Catch3/Generators/Element.hpp> #include <Progmasoft/Catch3/Generators/Integer.hpp> +#include <Progmasoft/Catch3/Generators/Text.hpp> +#include <Progmasoft/Catch3/Generators/Tuple.hpp> +#include <Progmasoft/Catch3/Generators/Vector.hpp> #include <Progmasoft/Catch3/Property.hpp> #include <Progmasoft/Catch3/Random.hpp> #include <Progmasoft/Catch3/Results.hpp> diff --git a/src/Progmasoft/Catch3/Detail/Sequence.hpp b/src/Progmasoft/Catch3/Detail/Sequence.hpp new file mode 100644 index 00000000..b1446a79 --- /dev/null +++ b/src/Progmasoft/Catch3/Detail/Sequence.hpp @@ -0,0 +1,97 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <cstddef> +#include <iterator> +#include <utility> +#include <vector> + +namespace Progmasoft::Catch3::Detail +{ + + /// Upper bound on the candidates one sequence value offers per shrink + /// step. The property runner re-shrinks after every accepted candidate, + /// so a bounded list still reaches small counterexamples while keeping + /// each step linear in this constant instead of in the sequence length. + inline constexpr std::size_t kMaximumSequenceCandidates = 256; + + /// Largest sequence a built-in generator will create. A size range above + /// this is rejected instead of silently allocating gigabytes in a test. + inline constexpr std::size_t kMaximumSequenceLength = std::size_t{ 1 } + << 20U; + + /** + * @brief Ordered shrink candidates shared by Vector and Text. + * + * Candidates are produced in a fixed order, simplest first: + * + * 1. the shortest allowed prefix, + * 2. the first half and then the second half, + * 3. the value with one element removed, position by position, + * 4. the value with one element replaced by each of that element's own + * shrink candidates, position by position. + * + * No candidate is shorter than @p minimumSize, every candidate differs + * from @p value in length or in exactly one element, and the list is cut + * at kMaximumSequenceCandidates. The same candidate may appear twice, + * for example when removing the last element equals the shortest prefix; + * the runner simply tests it again. + */ + template<typename Sequence, typename ShrinkElement> + [[nodiscard]] std::vector<Sequence> + ShrinkSequence(const Sequence &value, + std::size_t minimumSize, + ShrinkElement &&shrinkElement) + { + using Element = typename Sequence::value_type; + using Difference = typename std::iterator_traits< + typename Sequence::const_iterator>::difference_type; + + std::vector<Sequence> candidates; + const std::size_t size = value.size(); + const auto full = [&candidates] { + return candidates.size() >= kMaximumSequenceCandidates; + }; + const auto at = [&value](std::size_t index) { + return value.begin() + static_cast<Difference>(index); + }; + + if (size > minimumSize) + { + candidates.push_back(Sequence(at(0), at(minimumSize))); + + const std::size_t kept = size - size / 2; + if (kept > minimumSize && kept < size) + { + candidates.push_back(Sequence(at(0), at(kept))); + candidates.push_back(Sequence(at(size - kept), at(size))); + } + + for (std::size_t index = 0; index < size && !full(); ++index) + { + Sequence shorter(at(0), at(index)); + shorter.insert(shorter.end(), at(index + 1), at(size)); + candidates.push_back(std::move(shorter)); + } + } + + for (std::size_t index = 0; index < size && !full(); ++index) + { + const Element current = static_cast<Element>(*at(index)); + for (const auto &simpler : shrinkElement(current)) + { + if (full()) + { + break; + } + Sequence replaced = value; + *(replaced.begin() + static_cast<Difference>(index)) + = static_cast<Element>(simpler); + candidates.push_back(std::move(replaced)); + } + } + return candidates; + } + +} // namespace Progmasoft::Catch3::Detail diff --git a/src/Progmasoft/Catch3/Generators/Element.hpp b/src/Progmasoft/Catch3/Generators/Element.hpp new file mode 100644 index 00000000..087b6355 --- /dev/null +++ b/src/Progmasoft/Catch3/Generators/Element.hpp @@ -0,0 +1,63 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Generator.hpp> +#include <algorithm> +#include <concepts> +#include <cstddef> +#include <initializer_list> +#include <iterator> +#include <memory> +#include <utility> +#include <vector> + +namespace Progmasoft::Catch3 +{ + + /// Picks uniformly from a fixed, non-empty list of values. + /// + /// The list order is the simplicity order: a value shrinks to the values + /// listed before it, earliest first, so place the simplest case first. + /// A value that is not in the list has no candidates. + template<typename ValueType> + requires std::equality_comparable<ValueType> + [[nodiscard]] Generator<ValueType> + Element(std::vector<ValueType> values) + { + if (values.empty()) + { + Detail::RejectInvalidArgument( + "Element requires at least one value"); + } + + // Both callbacks share one immutable copy of the list. + const auto shared + = std::make_shared<const std::vector<ValueType>>(std::move(values)); + return MakeGenerator<ValueType>( + [shared](Random &random) { + const std::size_t index + = random.Between(std::size_t{ 0 }, shared->size() - 1); + return static_cast<ValueType>((*shared)[index]); + }, + [shared](const ValueType &value) { + const auto found + = std::find(shared->begin(), shared->end(), value); + if (found == shared->end()) + { + return std::vector<ValueType>{}; + } + return std::vector<ValueType>(shared->begin(), found); + }); + } + + /// Convenience overload for a literal list of values. + template<typename ValueType> + requires std::equality_comparable<ValueType> + [[nodiscard]] Generator<ValueType> + Element(std::initializer_list<ValueType> values) + { + return Element(std::vector<ValueType>(values)); + } + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Generators/Text.hpp b/src/Progmasoft/Catch3/Generators/Text.hpp new file mode 100644 index 00000000..05df8c00 --- /dev/null +++ b/src/Progmasoft/Catch3/Generators/Text.hpp @@ -0,0 +1,93 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Detail/Sequence.hpp> +#include <Progmasoft/Catch3/Generator.hpp> +#include <cstddef> +#include <memory> +#include <string> +#include <string_view> +#include <utility> +#include <vector> + +namespace Progmasoft::Catch3 +{ + + /// The 95 printable ASCII characters, space through tilde, with the + /// lower-case letter `a` first so that it is the simplest character. + [[nodiscard]] inline std::string + PrintableAscii() + { + std::string alphabet = "a"; + for (char character = ' '; character <= '~'; ++character) + { + if (character != 'a') + { + alphabet.push_back(character); + } + } + return alphabet; + } + + /// Generates byte strings whose characters are drawn from @p alphabet + /// and whose length lies in the inclusive range. + /// + /// The alphabet order is the simplicity order: shrinking first shortens + /// the string and then replaces single characters with earlier alphabet + /// characters, earliest first. Characters are bytes; the generator does + /// not interpret an encoding, so supply an alphabet that is valid for + /// the code under test. + [[nodiscard]] inline Generator<std::string> + Text(std::string alphabet = PrintableAscii(), + std::size_t minimumLength = 0, + std::size_t maximumLength = 32) + { + if (alphabet.empty()) + { + Detail::RejectInvalidArgument( + "Text requires a non-empty alphabet"); + } + if (minimumLength > maximumLength) + { + Detail::RejectInvalidArgument( + "Text requires minimumLength <= maximumLength"); + } + if (maximumLength > Detail::kMaximumSequenceLength) + { + Detail::RejectInvalidArgument( + "Text maximumLength exceeds the supported sequence length"); + } + + const auto shared + = std::make_shared<const std::string>(std::move(alphabet)); + return MakeGenerator<std::string>( + [shared, minimumLength, maximumLength](Random &random) { + const std::size_t length + = random.Between(minimumLength, maximumLength); + std::string text; + text.reserve(length); + for (std::size_t index = 0; index < length; ++index) + { + text.push_back((*shared)[random.Between( + std::size_t{ 0 }, + shared->size() - 1)]); + } + return text; + }, + [shared, minimumLength](const std::string &value) { + return Detail::ShrinkSequence( + value, + minimumLength, + [&shared](char character) { + const std::size_t position = shared->find(character); + if (position == std::string::npos) + { + return std::string{}; + } + return shared->substr(0, position); + }); + }); + } + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Generators/Tuple.hpp b/src/Progmasoft/Catch3/Generators/Tuple.hpp new file mode 100644 index 00000000..df58da05 --- /dev/null +++ b/src/Progmasoft/Catch3/Generators/Tuple.hpp @@ -0,0 +1,83 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Generator.hpp> +#include <cstddef> +#include <tuple> +#include <utility> +#include <vector> + +namespace Progmasoft::Catch3 +{ + namespace Detail + { + + template<std::size_t Index, typename... ValueTypes> + void + AppendComponentCandidates( + const std::tuple<Generator<ValueTypes>...> &generators, + const std::tuple<ValueTypes...> &value, + std::vector<std::tuple<ValueTypes...>> &candidates) + { + for (const auto &simpler : + std::get<Index>(generators).Shrink(std::get<Index>(value))) + { + std::tuple<ValueTypes...> replaced = value; + std::get<Index>(replaced) = simpler; + candidates.push_back(std::move(replaced)); + } + } + + template<typename... ValueTypes, std::size_t... Indices> + [[nodiscard]] std::vector<std::tuple<ValueTypes...>> + ShrinkTuple(const std::tuple<Generator<ValueTypes>...> &generators, + const std::tuple<ValueTypes...> &value, + std::index_sequence<Indices...>) + { + std::vector<std::tuple<ValueTypes...>> candidates; + // The comma fold visits components strictly left to right. + (AppendComponentCandidates<Indices>(generators, value, candidates), + ...); + return candidates; + } + + } // namespace Detail + + /// Combines independent generators into one generator of tuples, which + /// is how a property receives several generated arguments. + /// + /// Components are drawn strictly left to right from the trial's random + /// source, so a tuple is reproducible from its replay seed. Shrinking + /// changes one component at a time: all candidates of the first + /// component, then all of the second, and so on. Combined with the + /// runner's greedy restart this minimizes every component without + /// trying the cross product of candidates. + template<typename... ValueTypes> + requires(sizeof...(ValueTypes) > 0) + [[nodiscard]] Generator<std::tuple<ValueTypes...>> + Tuple(Generator<ValueTypes>... generators) + { + using Value = std::tuple<ValueTypes...>; + const std::tuple<Generator<ValueTypes>...> components{ std::move( + generators)... }; + + return MakeGenerator<Value>( + [components](Random &random) { + return std::apply( + [&random](const Generator<ValueTypes> &...component) { + // Braced initialization fixes the evaluation order; + // a function-call argument list would not. + return Value{ component.Generate(random)... }; + }, + components); + }, + [components](const Value &value) { + return Detail::ShrinkTuple( + components, + value, + std::index_sequence_for<ValueTypes...>{}); + }); + } + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Generators/Vector.hpp b/src/Progmasoft/Catch3/Generators/Vector.hpp new file mode 100644 index 00000000..65c81e3b --- /dev/null +++ b/src/Progmasoft/Catch3/Generators/Vector.hpp @@ -0,0 +1,60 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Detail/Sequence.hpp> +#include <Progmasoft/Catch3/Generator.hpp> +#include <cstddef> +#include <utility> +#include <vector> + +namespace Progmasoft::Catch3 +{ + + /// Generates vectors whose length lies in the inclusive size range and + /// whose elements come from @p element, drawn in index order. + /// + /// Shrinking first shortens the vector, then simplifies single elements + /// with the element generator's own candidates; see + /// Detail::ShrinkSequence for the exact order. A shrunk vector never + /// becomes shorter than @p minimumSize, so a property may rely on it. + template<typename ValueType> + [[nodiscard]] Generator<std::vector<ValueType>> + Vector(Generator<ValueType> element, + std::size_t minimumSize = 0, + std::size_t maximumSize = 32) + { + if (minimumSize > maximumSize) + { + Detail::RejectInvalidArgument( + "Vector requires minimumSize <= maximumSize"); + } + if (maximumSize > Detail::kMaximumSequenceLength) + { + Detail::RejectInvalidArgument( + "Vector maximumSize exceeds the supported sequence length"); + } + + return MakeGenerator<std::vector<ValueType>>( + [element, minimumSize, maximumSize](Random &random) { + const std::size_t size + = random.Between(minimumSize, maximumSize); + std::vector<ValueType> values; + values.reserve(size); + for (std::size_t index = 0; index < size; ++index) + { + values.push_back(element.Generate(random)); + } + return values; + }, + [element, minimumSize](const std::vector<ValueType> &value) { + return Detail::ShrinkSequence( + value, + minimumSize, + [&element](const ValueType &item) { + return element.Shrink(item); + }); + }); + } + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Property.hpp b/src/Progmasoft/Catch3/Property.hpp index 1a853bf1..57f7ac7b 100644 --- a/src/Progmasoft/Catch3/Property.hpp +++ b/src/Progmasoft/Catch3/Property.hpp @@ -12,8 +12,10 @@ #include <optional> #include <sstream> #include <string> +#include <tuple> #include <type_traits> #include <utility> +#include <vector> namespace Progmasoft::Catch3 { @@ -24,6 +26,11 @@ namespace Progmasoft::Catch3 std::size_t Trials = 100; std::uint64_t Seed = 0xC47C4A53ULL; std::size_t MaxShrinkSteps = 1000; + /// When set, runs exactly one trial with this seed instead of + /// deriving Trials seeds from Seed. Use the "replay seed" of a + /// reported failure to reproduce that single trial, including its + /// shrinking, without rerunning the trials before it. + std::optional<std::uint64_t> ReplayTrialSeed = std::nullopt; }; enum class PropertyStatus : std::uint8_t @@ -88,7 +95,9 @@ namespace Progmasoft::Catch3 << Failure->TrialSeed << ", shrink checks " << Failure->ShrinkSteps << "\ncounterexample: " << Failure->Counterexample - << "\nreason: " << Failure->Reason; + << "\nreason: " << Failure->Reason + << "\nreplay: set PropertyOptions::ReplayTrialSeed to " + << Failure->TrialSeed; } return output.str(); } @@ -97,11 +106,87 @@ namespace Progmasoft::Catch3 namespace Detail { + template<typename ValueType> + [[nodiscard]] std::string + StringifyPropertyValue(const ValueType &value); + + /// Renders one counterexample. Scalars use Catch2's StringMaker, so + /// user specializations apply. Tuples, pairs and vectors are + /// rendered here, element by element, because Catch2 only prints + /// tuples when a configuration macro is defined before its headers + /// and a counterexample must not degrade to "{?}" depending on + /// include order. + template<typename ValueType> + struct PropertyValuePrinter final + { + [[nodiscard]] static std::string + Print(const ValueType &value) + { + return Catch::StringMaker<ValueType>::convert(value); + } + }; + + template<typename... ValueTypes> + struct PropertyValuePrinter<std::tuple<ValueTypes...>> final + { + [[nodiscard]] static std::string + Print(const std::tuple<ValueTypes...> &value) + { + std::string text = "{"; + bool first = true; + std::apply( + [&text, &first](const ValueTypes &...component) { + const auto append = [&text, &first](std::string item) { + text += first ? " " : ", "; + text += item; + first = false; + }; + (append(StringifyPropertyValue(component)), ...); + }, + value); + text += first ? "}" : " }"; + return text; + } + }; + + template<typename FirstType, typename SecondType> + struct PropertyValuePrinter<std::pair<FirstType, SecondType>> final + { + [[nodiscard]] static std::string + Print(const std::pair<FirstType, SecondType> &value) + { + return "{ " + StringifyPropertyValue(value.first) + ", " + + StringifyPropertyValue(value.second) + " }"; + } + }; + + template<typename ElementType, typename AllocatorType> + struct PropertyValuePrinter<std::vector<ElementType, AllocatorType>> + final + { + [[nodiscard]] static std::string + Print(const std::vector<ElementType, AllocatorType> &value) + { + std::string text = "{"; + bool first = true; + for (const auto &element : value) + { + text += first ? " " : ", "; + // The cast turns std::vector<bool>'s proxy into a value. + text += StringifyPropertyValue( + static_cast<ElementType>(element)); + first = false; + } + text += first ? "}" : " }"; + return text; + } + }; + template<typename ValueType> [[nodiscard]] std::string StringifyPropertyValue(const ValueType &value) { - return Catch::StringMaker<std::remove_cvref_t<ValueType>>::convert( + return PropertyValuePrinter<std::remove_cvref_t<ValueType>>::Print( value); } @@ -178,11 +263,15 @@ namespace Progmasoft::Catch3 } PropertyResult result; - for (std::size_t trial = 0; trial < options.Trials; ++trial) + const std::size_t trials + = options.ReplayTrialSeed.has_value() ? 1 : options.Trials; + for (std::size_t trial = 0; trial < trials; ++trial) { result.TrialsRun = trial + 1; const std::uint64_t trialSeed - = Random::DeriveSeed(options.Seed, trial); + = options.ReplayTrialSeed.has_value() + ? *options.ReplayTrialSeed + : Random::DeriveSeed(options.Seed, trial); Random random(trialSeed); std::optional<ValueType> sample; diff --git a/src/Progmasoft/meson.build b/src/Progmasoft/meson.build index f9acedb2..1f9afbda 100644 --- a/src/Progmasoft/meson.build +++ b/src/Progmasoft/meson.build @@ -21,6 +21,7 @@ if get_option('install') install_headers('Catch3.hpp', subdir: 'Progmasoft') install_headers('Catch3/Assertions.hpp', subdir: 'Progmasoft/Catch3') install_headers('Catch3/Detail/InvalidArgument.hpp', subdir: 'Progmasoft/Catch3/Detail') + install_headers('Catch3/Detail/Sequence.hpp', subdir: 'Progmasoft/Catch3/Detail') install_headers('Catch3/Generator.hpp', subdir: 'Progmasoft/Catch3') install_headers('Catch3/Property.hpp', subdir: 'Progmasoft/Catch3') install_headers('Catch3/Random.hpp', subdir: 'Progmasoft/Catch3') @@ -29,4 +30,8 @@ if get_option('install') install_headers('Catch3/XmlWriter.hpp', subdir: 'Progmasoft/Catch3') install_headers('Catch3/Generators/Boolean.hpp', subdir: 'Progmasoft/Catch3/Generators') install_headers('Catch3/Generators/Integer.hpp', subdir: 'Progmasoft/Catch3/Generators') + install_headers('Catch3/Generators/Element.hpp', subdir: 'Progmasoft/Catch3/Generators') + install_headers('Catch3/Generators/Text.hpp', subdir: 'Progmasoft/Catch3/Generators') + install_headers('Catch3/Generators/Tuple.hpp', subdir: 'Progmasoft/Catch3/Generators') + install_headers('Catch3/Generators/Vector.hpp', subdir: 'Progmasoft/Catch3/Generators') endif diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 78e56f1d..79660687 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -158,6 +158,7 @@ target_link_libraries(SelfTest PRIVATE Catch2WithMain) # Exercise the C++20 Progmasoft layer independently so the upstream Catch2 # SelfTest continues to compile at its supported baseline language level. add_executable(ProgmasoftSelfTest + Progmasoft/Generators.tests.cpp Progmasoft/Property.tests.cpp Progmasoft/Results.tests.cpp Progmasoft/Snapshot.tests.cpp diff --git a/tests/Progmasoft/BUILD.bazel b/tests/Progmasoft/BUILD.bazel index 18912383..d2eeca92 100644 --- a/tests/Progmasoft/BUILD.bazel +++ b/tests/Progmasoft/BUILD.bazel @@ -7,6 +7,7 @@ package(default_visibility = ["//visibility:public"]) cc_test( name = "property_tests", srcs = [ + "Generators.tests.cpp", "Property.tests.cpp", "Results.tests.cpp", "Snapshot.tests.cpp", diff --git a/tests/Progmasoft/Generators.tests.cpp b/tests/Progmasoft/Generators.tests.cpp new file mode 100644 index 00000000..553a8962 --- /dev/null +++ b/tests/Progmasoft/Generators.tests.cpp @@ -0,0 +1,336 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <Progmasoft/Catch3.hpp> + +#include <algorithm> +#include <cstddef> +#include <cstdint> +#include <numeric> +#include <stdexcept> +#include <string> +#include <tuple> +#include <utility> +#include <vector> + +namespace { + + using Progmasoft::Catch3::CheckProperty; + using Progmasoft::Catch3::Element; + using Progmasoft::Catch3::Integer; + using Progmasoft::Catch3::PropertyOptions; + using Progmasoft::Catch3::PropertyStatus; + using Progmasoft::Catch3::Random; + using Progmasoft::Catch3::Text; + using Progmasoft::Catch3::Tuple; + using Progmasoft::Catch3::Vector; + +} // namespace + +TEST_CASE( "Vector generators respect size and element bounds", + "[progmasoft][generator][vector]" ) { + const auto generator = Vector( Integer<int>( -5, 5 ), 2, 9 ); + Random random( 0x51A7E ); + bool sawMinimum = false; + bool sawMaximum = false; + for ( int index = 0; index < 500; ++index ) { + const std::vector<int> values = generator.Generate( random ); + REQUIRE( values.size() >= 2 ); + REQUIRE( values.size() <= 9 ); + sawMinimum = sawMinimum || values.size() == 2; + sawMaximum = sawMaximum || values.size() == 9; + for ( const int value : values ) { + REQUIRE( value >= -5 ); + REQUIRE( value <= 5 ); + } + } + // Both ends of the inclusive size range are reachable. + CHECK( sawMinimum ); + CHECK( sawMaximum ); + + Random first( 77 ); + Random second( 77 ); + CHECK( generator.Generate( first ) == generator.Generate( second ) ); +} + +TEST_CASE( "Vector shrinking shortens first and never drops below its minimum", + "[progmasoft][generator][vector]" ) { + const auto generator = Vector( Integer<int>( 0, 100 ), 2, 16 ); + const std::vector<int> value{ 9, 8, 7, 6, 5 }; + const auto candidates = generator.Shrink( value ); + + REQUIRE( candidates.size() >= 4 ); + // Shortest allowed prefix, then the two halves, then single removals. + CHECK( candidates[0] == std::vector<int>{ 9, 8 } ); + CHECK( candidates[1] == std::vector<int>{ 9, 8, 7 } ); + CHECK( candidates[2] == std::vector<int>{ 7, 6, 5 } ); + CHECK( candidates[3] == std::vector<int>{ 8, 7, 6, 5 } ); + for ( const auto& candidate : candidates ) { + CHECK( candidate.size() >= 2 ); + CHECK( candidate.size() <= value.size() ); + CHECK( candidate != value ); + } + // Element simplification keeps the length and changes one position. + CHECK( std::find( candidates.begin(), + candidates.end(), + std::vector<int>{ 0, 8, 7, 6, 5 } ) != + candidates.end() ); + + // A value already at its minimum length only simplifies elements. + for ( const auto& candidate : + generator.Shrink( std::vector<int>{ 3, 0 } ) ) { + CHECK( candidate.size() == 2 ); + } + CHECK( generator.Shrink( std::vector<int>{ 0, 0 } ).empty() ); +} + +TEST_CASE( "Vector shrinking offers a bounded number of candidates", + "[progmasoft][generator][vector]" ) { + const auto generator = Vector( Integer<int>( 0, 1000 ), 0, 5000 ); + const std::vector<int> large( 4000, 999 ); + const auto candidates = generator.Shrink( large ); + CHECK( candidates.size() == + Progmasoft::Catch3::Detail::kMaximumSequenceCandidates ); + CHECK( candidates.front().empty() ); +} + +TEST_CASE( "Vector properties shrink to a small counterexample", + "[progmasoft][property][vector]" ) { + const auto result = CheckProperty( + Vector( Integer<int>( 0, 100 ), 0, 20 ), + []( const std::vector<int>& values ) { + return std::accumulate( values.begin(), values.end(), 0 ) < 10; + }, + PropertyOptions{ .Trials = 200, .Seed = 0xF00D } ); + + REQUIRE( result.Status == PropertyStatus::kFailed ); + REQUIRE( result.Failure.has_value() ); + // The smallest failing input is one element equal to the threshold. + CHECK( result.Failure->Counterexample == "{ 10 }" ); +} + +TEST_CASE( "Vector<bool> values generate, shrink and print", + "[progmasoft][generator][vector]" ) { + const auto generator = Vector( Progmasoft::Catch3::Boolean(), 0, 8 ); + const auto result = CheckProperty( + generator, + []( const std::vector<bool>& flags ) { + return std::count( flags.begin(), flags.end(), true ) < 2; + }, + PropertyOptions{ .Trials = 300, .Seed = 0xB001 } ); + REQUIRE( result.Status == PropertyStatus::kFailed ); + REQUIRE( result.Failure.has_value() ); + CHECK( result.Failure->Counterexample == "{ true, true }" ); +} + +TEST_CASE( "Tuple generators draw components left to right", + "[progmasoft][generator][tuple]" ) { + const auto left = Integer<int>( 0, 1000 ); + const auto right = Integer<std::int64_t>( -9, 9 ); + const auto generator = Tuple( left, right, Progmasoft::Catch3::Boolean() ); + + Random combined( 0xABCDEF ); + Random manual( 0xABCDEF ); + for ( int index = 0; index < 50; ++index ) { + const auto value = generator.Generate( combined ); + // The same seed consumed by hand in declaration order. + const int expectedLeft = left.Generate( manual ); + const std::int64_t expectedRight = right.Generate( manual ); + const bool expectedFlag = ( manual.Next() & 1U ) != 0; + CHECK( std::get<0>( value ) == expectedLeft ); + CHECK( std::get<1>( value ) == expectedRight ); + CHECK( std::get<2>( value ) == expectedFlag ); + } +} + +TEST_CASE( "Tuple shrinking changes one component at a time, first to last", + "[progmasoft][generator][tuple]" ) { + const auto generator = + Tuple( Integer<int>( 0, 100 ), Integer<int>( 0, 100 ) ); + const std::tuple<int, int> value{ 4, 6 }; + const auto candidates = generator.Shrink( value ); + + const auto firstCandidates = Integer<int>( 0, 100 ).Shrink( 4 ); + const auto secondCandidates = Integer<int>( 0, 100 ).Shrink( 6 ); + REQUIRE( candidates.size() == + firstCandidates.size() + secondCandidates.size() ); + for ( std::size_t index = 0; index < firstCandidates.size(); ++index ) { + CHECK( candidates[index] == + std::tuple<int, int>{ firstCandidates[index], 6 } ); + } + for ( std::size_t index = 0; index < secondCandidates.size(); ++index ) { + CHECK( candidates[firstCandidates.size() + index] == + std::tuple<int, int>{ 4, secondCandidates[index] } ); + } + CHECK( generator.Shrink( std::tuple<int, int>{ 0, 0 } ).empty() ); +} + +TEST_CASE( "Tuple properties minimize every component", + "[progmasoft][property][tuple]" ) { + const auto result = CheckProperty( + Tuple( Integer<int>( 0, 1000 ), Integer<int>( 0, 1000 ) ), + []( const std::tuple<int, int>& value ) { + return std::get<0>( value ) < 50 || std::get<1>( value ) < 50; + }, + PropertyOptions{ .Trials = 500, .Seed = 0x7007 } ); + + REQUIRE( result.Status == PropertyStatus::kFailed ); + REQUIRE( result.Failure.has_value() ); + CHECK( result.Failure->Counterexample == "{ 50, 50 }" ); +} + +TEST_CASE( "Element generators pick listed values and shrink toward the front", + "[progmasoft][generator][element]" ) { + const auto generator = Element<std::string>( { "none", "low", "high" } ); + Random random( 0xE1E ); + std::vector<int> seen( 3, 0 ); + for ( int index = 0; index < 300; ++index ) { + const std::string value = generator.Generate( random ); + if ( value == "none" ) { + ++seen[0]; + } else if ( value == "low" ) { + ++seen[1]; + } else { + REQUIRE( value == "high" ); + ++seen[2]; + } + } + CHECK( seen[0] > 0 ); + CHECK( seen[1] > 0 ); + CHECK( seen[2] > 0 ); + + CHECK( generator.Shrink( "none" ).empty() ); + CHECK( generator.Shrink( "low" ) == std::vector<std::string>{ "none" } ); + CHECK( generator.Shrink( "high" ) == + std::vector<std::string>{ "none", "low" } ); + // A value outside the list has no simpler listed form. + CHECK( generator.Shrink( "other" ).empty() ); + + const auto single = Element<int>( { 7 } ); + CHECK( single.Generate( random ) == 7 ); +} + +TEST_CASE( "Text generators stay inside their alphabet and length range", + "[progmasoft][generator][text]" ) { + const auto generator = Text( "abc", 1, 6 ); + Random random( 0x7E87 ); + for ( int index = 0; index < 300; ++index ) { + const std::string value = generator.Generate( random ); + REQUIRE( value.size() >= 1 ); + REQUIRE( value.size() <= 6 ); + CHECK( value.find_first_not_of( "abc" ) == std::string::npos ); + } + + const std::string printable = Progmasoft::Catch3::PrintableAscii(); + CHECK( printable.size() == 95 ); + CHECK( printable.front() == 'a' ); + std::string sorted = printable; + std::sort( sorted.begin(), sorted.end() ); + CHECK( std::adjacent_find( sorted.begin(), sorted.end() ) == + sorted.end() ); + CHECK( sorted.front() == ' ' ); + CHECK( sorted.back() == '~' ); +} + +TEST_CASE( "Text properties shrink to the shortest simplest string", + "[progmasoft][property][text]" ) { + const auto result = CheckProperty( + Text( "abcxyz", 0, 24 ), + []( const std::string& value ) { + return value.find( 'x' ) == std::string::npos; + }, + PropertyOptions{ .Trials = 300, .Seed = 0x7E47 } ); + + REQUIRE( result.Status == PropertyStatus::kFailed ); + REQUIRE( result.Failure.has_value() ); + CHECK( result.Failure->Counterexample == "\"x\"" ); + + // Characters simplify toward the front of the alphabet. + const auto candidates = Text( "abc", 2, 4 ).Shrink( "cb" ); + CHECK( candidates == + std::vector<std::string>{ "ab", "bb", "ca" } ); +} + +TEST_CASE( "A reported replay seed reproduces exactly that trial", + "[progmasoft][property][replay]" ) { + const auto generator = + Tuple( Integer<int>( 0, 1000 ), Vector( Integer<int>( 0, 9 ), 0, 6 ) ); + const auto predicate = + []( const std::tuple<int, std::vector<int>>& value ) { + return std::get<0>( value ) < 600 || + std::get<1>( value ).size() < 2; + }; + + const auto original = CheckProperty( + generator, predicate, PropertyOptions{ .Trials = 500, .Seed = 0x5EED } ); + REQUIRE( original.Status == PropertyStatus::kFailed ); + REQUIRE( original.Failure.has_value() ); + // The failure was not on the first trial, so replay skips real work. + REQUIRE( original.Failure->Trial > 1 ); + CHECK( original.Describe().find( + "replay: set PropertyOptions::ReplayTrialSeed to " + + std::to_string( original.Failure->TrialSeed ) ) != + std::string::npos ); + + // A different root seed and trial count prove that only the replay seed + // decides what runs. + const auto replayed = CheckProperty( + generator, + predicate, + PropertyOptions{ .Trials = 3, + .Seed = 1, + .ReplayTrialSeed = original.Failure->TrialSeed } ); + REQUIRE( replayed.Status == PropertyStatus::kFailed ); + REQUIRE( replayed.Failure.has_value() ); + CHECK( replayed.TrialsRun == 1 ); + CHECK( replayed.Failure->Trial == 1 ); + CHECK( replayed.Failure->TrialSeed == original.Failure->TrialSeed ); + CHECK( replayed.Failure->Counterexample == + original.Failure->Counterexample ); + CHECK( replayed.Failure->ShrinkSteps == original.Failure->ShrinkSteps ); + + // Replaying a seed whose trial passes reports one passing trial. + const auto passing = CheckProperty( + Integer<int>( 0, 10 ), + []( int value ) { return value <= 10; }, + PropertyOptions{ .Trials = 50, .ReplayTrialSeed = 12345 } ); + CHECK( passing.Succeeded() ); + CHECK( passing.TrialsRun == 1 ); +} + +TEST_CASE( "Counterexamples print nested structures without Catch2 macros", + "[progmasoft][property][printing]" ) { + using Progmasoft::Catch3::Detail::StringifyPropertyValue; + + CHECK( StringifyPropertyValue( std::vector<int>{} ) == "{}" ); + CHECK( StringifyPropertyValue( std::vector<int>{ 1, 2, 3 } ) == + "{ 1, 2, 3 }" ); + CHECK( StringifyPropertyValue( std::tuple<>{} ) == "{}" ); + CHECK( StringifyPropertyValue( std::pair<int, bool>{ 4, false } ) == + "{ 4, false }" ); + CHECK( StringifyPropertyValue( + std::tuple<int, std::vector<bool>, std::string>{ + 7, { true, false }, "text" } ) == + "{ 7, { true, false }, \"text\" }" ); + CHECK( StringifyPropertyValue( + std::vector<std::tuple<int, int>>{ { 1, 2 }, { 3, 4 } } ) == + "{ { 1, 2 }, { 3, 4 } }" ); +} + +TEST_CASE( "Generator factories reject impossible configurations", + "[progmasoft][generator]" ) { + CHECK_THROWS_AS( Vector( Integer<int>( 0, 1 ), 3, 2 ), + std::invalid_argument ); + CHECK_THROWS_AS( + Vector( Integer<int>( 0, 1 ), + 0, + Progmasoft::Catch3::Detail::kMaximumSequenceLength + 1 ), + std::invalid_argument ); + CHECK_THROWS_AS( Element( std::vector<int>{} ), std::invalid_argument ); + CHECK_THROWS_AS( Text( "", 0, 4 ), std::invalid_argument ); + CHECK_THROWS_AS( Text( "ab", 5, 4 ), std::invalid_argument ); + CHECK_THROWS_AS( + Text( "ab", + 0, + Progmasoft::Catch3::Detail::kMaximumSequenceLength + 1 ), + std::invalid_argument ); +} diff --git a/tests/meson.build b/tests/meson.build index d96b6f9f..c4abaf29 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -82,6 +82,7 @@ test('SelfTest', self_test) progmasoft_property_tests = executable( 'ProgmasoftPropertyTests', [ + 'Progmasoft/Generators.tests.cpp', 'Progmasoft/Property.tests.cpp', 'Progmasoft/Results.tests.cpp', 'Progmasoft/Snapshot.tests.cpp', diff --git a/xmake.lua b/xmake.lua index 82fc0726..a865f98d 100644 --- a/xmake.lua +++ b/xmake.lua @@ -50,6 +50,7 @@ target("ProgmasoftPropertyTests") set_kind("binary") set_languages("cxx20") add_files( + "tests/Progmasoft/Generators.tests.cpp", "tests/Progmasoft/Property.tests.cpp", "tests/Progmasoft/Results.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp",