catch3
Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.
Clonegit clone https://git.progmasoft.com/catch3
Add a native property-testing engine with choice-sequence shrinking
- Commit
afac14ac5a62d8d2e7efd2184ad34e16593eddfd- Author
- Leitwolf11 ·
- Parents
2c23a508c95c
Changes
.clang-format | 4 +- docs/Readme.md | 3 +- docs/progmasoft-catch3.md | 17 +- docs/progmasoft-properties.md | 230 ++++++++++ docs/progmasoft-roadmap.md | 90 ++++ docs/why-catch3.md | 14 +- premake5.lua | 6 + src/CMakeLists.txt | 3 + src/Progmasoft/BUILD.bazel | 12 + src/Progmasoft/Catch3.hpp | 8 + src/Progmasoft/Catch3/Assertions.hpp | 33 ++ src/Progmasoft/Catch3/ForAll.hpp | 191 ++++++++ src/Progmasoft/Catch3/Property.hpp | 100 ++++- src/Progmasoft/Catch3/Relations.hpp | 99 +++++ src/Progmasoft/Catch3/Stateful.hpp | 157 +++++++ src/Progmasoft/Catch3/Strategy/ChoiceSource.hpp | 191 ++++++++ src/Progmasoft/Catch3/Strategy/Containers.hpp | 391 +++++++++++++++++ src/Progmasoft/Catch3/Strategy/Context.hpp | 145 +++++++ src/Progmasoft/Catch3/Strategy/Engine.hpp | 85 ++++ src/Progmasoft/Catch3/Strategy/Numeric.hpp | 224 ++++++++++ src/Progmasoft/Catch3/Strategy/Strategy.hpp | 303 +++++++++++++ src/Progmasoft/ChoiceShrinker.cpp | 384 ++++++++++++++++ src/Progmasoft/Context.cpp | 122 ++++++ src/Progmasoft/PropertyEngine.cpp | 553 ++++++++++++++++++++++++ src/Progmasoft/meson.build | 18 +- tests/CMakeLists.txt | 3 + tests/Progmasoft/BUILD.bazel | 3 + tests/Progmasoft/ForAll.tests.cpp | 314 ++++++++++++++ tests/Progmasoft/Stateful.tests.cpp | 129 ++++++ tests/Progmasoft/Strategy.tests.cpp | 312 +++++++++++++ tests/meson.build | 3 + xmake.lua | 6 + 32 files changed, 4136 insertions(+), 17 deletions(-) diff --git a/.clang-format b/.clang-format index 907da4cb..05a21827 100644 --- a/.clang-format +++ b/.clang-format @@ -2,9 +2,9 @@ # SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 --- -# Visual X# C++20 format profile, pinned to LLVM/Clang 23.1.0. +# Progmasoft C++ Style format profile, pinned to LLVM/Clang 23.1.0. # Naming is intentionally outside clang-format and remains governed by the -# existing Visual X# PascalCase/camelCase conventions. +# PascalCase/camelCase conventions of Progmasoft C++ Style. Language: Cpp BasedOnStyle: LLVM Standard: c++20 diff --git a/docs/Readme.md b/docs/Readme.md index 89995f45..d8e9730e 100644 --- a/docs/Readme.md +++ b/docs/Readme.md @@ -3,7 +3,8 @@ Catch3 combines the upstream Catch2-compatible runner and API with Progmasoft's C++20 extensions. Start with the [tutorial](tutorial.md#top), -then read [what Catch3 adds](progmasoft-catch3.md) and +then read [what Catch3 adds](progmasoft-catch3.md), +[native property testing](progmasoft-properties.md), [the roadmap](progmasoft-roadmap.md) and [why the two API families remain distinct](why-catch3.md). The compatibility pages below retain real `catch2/` include paths, `Catch2::` CMake targets, and upstream version history; renaming those would make examples incorrect. diff --git a/docs/progmasoft-catch3.md b/docs/progmasoft-catch3.md index 55114f32..30ea7a15 100644 --- a/docs/progmasoft-catch3.md +++ b/docs/progmasoft-catch3.md @@ -8,6 +8,14 @@ namespace. ## Property testing +Catch3 has two property APIs. [Native property testing](progmasoft-properties.md#top) +with `ForAll` and strategies is the full engine: every combination of +strategies shrinks, failures replay from a token and from an example database, +and it adds state machines, statistics, targeted search and differential and +metamorphic oracles. New tests should use it. The value-shrinking +`CheckProperty` API described here remains supported for existing tests. +[The roadmap](progmasoft-roadmap.md#top) lists what comes next. + `<Progmasoft/Catch3.hpp>` is the umbrella header. It brings in Catch2's `catch_all.hpp` as well as the Progmasoft extensions, so it is convenient but increases compile time. Include individual headers when a translation unit only @@ -108,11 +116,10 @@ Invalid factory arguments, such as an empty alphabet or a minimum above its maximum, throw `std::invalid_argument`; a client built without exceptions gets the message on standard error followed by `std::abort`. -State-machine commands, parallel trials, mapped or filtered generators, and -adaptive shrinking are deliberately not implied by this module; they need -their own deterministic contracts and tests before being added. Shrinking a -composite value is greedy and local like scalar shrinking: it finds a small -counterexample, not a provably smallest one. +Mapped, filtered and dependent generators and state machines are not part of +this module; the strategies of [native property testing](progmasoft-properties.md#top) +provide them. Shrinking a composite value here is greedy and local like scalar +shrinking: it finds a small counterexample, not a provably smallest one. ## Snapshots diff --git a/docs/progmasoft-properties.md b/docs/progmasoft-properties.md new file mode 100644 index 00000000..8042b85b --- /dev/null +++ b/docs/progmasoft-properties.md @@ -0,0 +1,230 @@ +<a id="top"></a> +# Native property testing + +`<Progmasoft/Catch3/ForAll.hpp>` is Catch3's own property-testing engine. It +needs no third-party library. Catch2's support for RapidCheck through +RapidCheck's own Catch integration keeps working as before, but the Progmasoft +additions do not build on RapidCheck. The engine goes beyond it where property +tests usually stop being practical: + +- **Shrinking of the recorded choices.** Every combination of strategies + shrinks, including `Map`, `Filter`, `FlatMap` and state machines, and a + shrunk input always satisfies the filters and dependencies that built it. +- **An example database.** Failures found once are replayed first on the next + run, on any machine that shares the directory, until they are fixed. +- **Targeted search.** `Target` scores guide a search towards inputs that + random sampling rarely reaches. +- **Coverage requirements.** `Cover` fails a property whose inputs reach an + interesting case too rarely. +- **Ready-made oracles.** `Equivalent`, `Metamorphic`, `RoundTrip` and + `Idempotent` check properties that need no expected result. +- **No exceptions required.** `CATCH3_PROPERTY_ASSERT` returns from the body; + the engine never throws. + +The older `CheckProperty` and `Generator` API of +[Progmasoft Catch3 additions](progmasoft-catch3.md#top) remains supported +and unchanged. + +## A first property + +```cpp +#include <Progmasoft/Catch3.hpp> + +using namespace Progmasoft::Catch3; + +TEST_CASE( "reversing twice gives the original", "[property]" ) { + CATCH3_CHECK_FOR_ALL( + Gen::Vectors( Gen::Integers<int>() ), + []( const std::vector<int>& values ) { + auto copy = values; + std::reverse( copy.begin(), copy.end() ); + std::reverse( copy.begin(), copy.end() ); + return copy == values; + } ); +} +``` + +`ForAll([options,] strategy..., body)` draws one value from each strategy and +passes them to the body. The body returns `bool`, or returns `void` and uses +the property macros: + +```cpp +CATCH3_CHECK_FOR_ALL( + Gen::Integers<int>( -1000, 1000 ), Gen::Integers<int>( 1, 50 ), + []( int value, int divisor ) { + CATCH3_PROPERTY_ASSUME( value % divisor == 0 ); // otherwise discard + Note( "quotient", value / divisor ); // part of the report + CATCH3_PROPERTY_ASSERT( value / divisor * divisor == value ); + } ); +``` + +`CATCH3_CHECK_FOR_ALL` and `CATCH3_REQUIRE_FOR_ALL` report through Catch2's +normal assertion path. `ForAll` itself returns a `PropertyResult` for code +that wants to inspect it. Catch2's own `CHECK` and `REQUIRE` should not be +used inside a property body: the engine runs the body many times while it +shrinks, and every run would be reported. + +## Strategies + +A `Strategy<T>` builds values of `T`. All strategies live in +`Progmasoft::Catch3::Gen`: + +| Strategy | Values | Shrinks towards | +| --- | --- | --- | +| `Integers<T>(min, max)` | integers of the range | the value closest to zero | +| `Booleans()` | `true`, `false` | `false` | +| `Floats<T>(FloatOptions)` | finite values of the range; NaN and infinities on request | zero or the bound closest to it, then integral values | +| `Characters(alphabet)`, `PrintableCharacters()` | characters | the first character of the alphabet | +| `Strings(character, range)`, `Utf8Strings(range, min, max)` | byte strings; UTF-8 text without surrogates | shorter text, simpler characters | +| `Vectors(element, range)`, `UniqueVectors`, `Sets`, `Maps` | collections | fewer and simpler elements | +| `Optionals(value)` | `std::nullopt` or a value | `std::nullopt` | +| `Tuples(a, b, ...)`, `Construct<T>(a, b, ...)` | tuples, and objects built with braces | each component | +| `Permutations(values)` | orderings of a list | the original order | +| `Just(value)`, `SampledFrom(values)` | fixed values | the first value | +| `OneOf(a, b, ...)`, `Frequency({ {weight, strategy}, ... })` | one of several strategies | the first strategy | +| `Lazy(factory)`, `Recursive(base, extend, depth)` | self-referential and recursive structures | a leaf | + +Every strategy has `Map(function)`, `Filter(predicate)` and +`FlatMap(function)`. `FlatMap` draws a value and then a second strategy that +depends on it, for example a size and then a vector of exactly that size. + +A `SizeRange{ minimum, maximum }` bounds a collection. Without a maximum the +length grows with the size of the trial, which rises from 0 to 100 over the +first half of the trials, so the first trials try small inputs. + +## How shrinking works + +Each strategy builds its value from a sequence of choices, small integers that +a `ChoiceSource` hands out. During a trial the engine records them. When a +trial fails, the engine does not shrink the value; it shrinks the recorded +choices and replays them through the same strategies. Choice 0 is always the +simplest decision, and a shorter sequence is simpler than a longer one. Five +passes repeat until none helps or `MaxShrinkSteps` replays are used: + +1. delete blocks of 16, 8, 4, 2 and 1 choices, which drops collection + elements and command steps; +2. set blocks to zero; +3. lower each choice by binary search, then by single steps for conditions + that are not monotone; +4. swap neighbours that are out of order; +5. move a unit from a choice to a later one, for sums such as `x + y > 10`. + +Because the strategies are replayed, the shrunk input is always one the +strategies can produce: a filter is still satisfied, a dependent size still +matches its vector, a state machine still respects its preconditions. That is +the difference from shrinking values directly. + +The report shows the minimized input, the reason, the notes, and a replay +token: + +```text +property failed after 37 trials; failing trial 37, root seed 3359259219, shrink checks 112 +counterexample: { 0, 100 } +reason: property returned false +replay: set PropertyOptions::Replay to "c3.3c.0-0-2s" +``` + +Setting `PropertyOptions::Replay` to the token runs exactly that input once, +without trials or shrinking. + +## Options + +| Field | Default | Meaning | +| --- | --- | --- | +| `Trials` | 100 | valid trials to run | +| `Seed` | fixed | root seed; every trial derives its own seed from it | +| `MaxShrinkSteps` | 1000 | replays the shrinker may use | +| `Replay` | none | run only the input of a replay token | +| `MaxDiscardRatio` | 10 | discarded trials allowed per requested trial before the property gives up | +| `TargetSteps` | 500 | steps of targeted search when the body calls `Target` | +| `DatabasePath` | none | directory of the example database; otherwise `CATCH3_PROPERTY_DATABASE` | +| `Name` | test name and line | names the property in the database | +| `ShowStatistics` | false | adds the `Classify` and `Cover` counts to a passing report | + +## Statistics and coverage requirements + +`Classify(condition, label)` counts how often a condition held. `Cover(percent, +condition, label)` also requires a percentage: a property whose inputs rarely +reach an interesting case fails with `kInsufficientCoverage` instead of +passing on easy inputs. + +```cpp +Cover( 10.0, values.empty(), "empty" ); +Cover( 30.0, values.size() > 10, "long" ); +``` + +## Targeted search + +`Target(score, label)` reports a score. After the regular trials the engine +takes the highest-scoring input of each label and mutates its choices one at a +time, keeping a mutation when the score rises. Inputs that random sampling +almost never reaches, such as a long chain of operations or a value near an +overflow, become reachable when a score measures the distance to them. + +## The example database + +With `DatabasePath`, or the `CATCH3_PROPERTY_DATABASE` environment variable, +the engine stores the replay token of every minimized failure in a file named +after the property. The next run replays those tokens before any new trial, so +a failure found once, on any machine that shares the directory, is checked +first until it is fixed; a stored example that passes is removed. The engine +never fails a test because the directory is missing or unwritable. + +## Model-based testing + +`<Progmasoft/Catch3/Stateful.hpp>` runs random sequences of commands against a +model and the system under test side by side: + +```cpp +using Model = std::deque<int>; + +Command<Model, Queue> push{ + "push", + []( const Model& ) { + return Gen::Integers<int>( 0, 100 ).Map( []( int value ) { + return Action<Model, Queue>{ + "push(" + std::to_string( value ) + ")", + [value]( Model& model, Queue& queue ) { + model.push_back( value ); + queue.Push( value ); + CATCH3_PROPERTY_ASSERT( queue.Size() == model.size() ); + } }; + } ); + } }; + +const auto result = CheckStateful<Model, Queue>( + options, [] { return Model{}; }, + []( const Model& ) { return Queue{}; }, + { push, pop } ); +``` + +A command builds its action from the current model, so arguments can depend on +the state, and its precondition decides whether it may run. A failing sequence +shrinks to fewer steps with simpler arguments; the report lists the steps, for +example `{ push(0), push(0), push(0), push(0) }`. + +## Ready-made oracles + +`<Progmasoft/Catch3/Relations.hpp>` provides bodies for properties that need +no specification of the expected result: + +| Body | Checks | +| --- | --- | +| `Equivalent(reference, candidate)` | differential testing: both return the same result | +| `Metamorphic(function, transform, relation)` | `relation(f(x), f(transform(x)))` | +| `RoundTrip(encode, decode)` | `decode(encode(x)) == x` | +| `Idempotent(function)` | `f(f(x)) == f(x)` | + +```cpp +CATCH3_CHECK_FOR_ALL( Gen::Vectors( Gen::Integers<int>() ), + Equivalent( ReferenceSort, FastSort ) ); +``` + +A mismatch reports both results as notes. + +## Compatibility + +The engine is C++20 and part of the `Progmasoft::Catch3` library target. It +does not change Catch2's runner, reporters, command line, or the `Catch` +namespace, and it needs no exceptions: a client built without them gets the +same results, except that an exception thrown by a body cannot be caught. diff --git a/docs/progmasoft-roadmap.md b/docs/progmasoft-roadmap.md new file mode 100644 index 00000000..4b18a6c7 --- /dev/null +++ b/docs/progmasoft-roadmap.md @@ -0,0 +1,90 @@ +<a id="top"></a> +# Catch3 roadmap + +Catch3 grows from a Catch2-compatible test framework into a testing toolkit: +unit testing, property testing, mocking, coverage, mutation testing, test +generation and test selection, driven by a runner of its own. Two rules hold +for every step: + +- **Catch2 stays compatible.** The `Catch` namespace, the `catch2/` headers, + the macros, the command line and the reporters keep working unchanged. New + capabilities live under `Progmasoft::Catch3` and `Progmasoft/Catch3/`. +- **Nothing is claimed before it is tested.** A feature is listed as done when + its tests run in CI on Windows, Linux and macOS. + +The steps are ordered by what each one builds on. + +| Step | Capability | State | +| --- | --- | --- | +| 1 | Native property testing: choice-sequence shrinking, combinators, state machines, statistics, targeted search, example database, differential and metamorphic oracles | done; see [Native property testing](progmasoft-properties.md#top) | +| 2 | The `catch3` runner | planned | +| 3 | Mocking | planned | +| 4 | Per-test code coverage | planned | +| 5 | Change-impact test selection and test prioritization | planned | +| 6 | Mutation testing | planned | +| 7 | Coverage-, mutation- and history-guided test generation | planned | + +## 2. The `catch3` runner + +Catch2's runner executes the tests of one binary in one process. The features +that follow need more: they run many binaries, each test in isolation, under +coverage or mutation, and decide which tests to run and in which order. The +`catch3` runner is a separate program that drives Catch2 test binaries from +the outside, so existing binaries need no change: + +- it discovers test cases with `--list-tests --reporter json` and runs them + with `--reporter json`, both stable parts of Catch2's command line; +- it runs tests in parallel processes, shards them across machines, enforces + per-test timeouts, isolates crashes, and retries flaky tests with a record + of every attempt; +- it keeps a history of results and durations, the input for prioritization; +- it writes JUnit XML, JSON and a terminal report, and keeps Catch2's exit + codes; +- its command line is parsed by a hand-written, typed parser, like the other + Progmasoft tools. + +## 3. Mocking + +Mocks that report through Catch2's assertion path: expectations with call +counts, argument matchers built on Catch2's matchers, sequences, and actions, +for virtual interfaces and for free functions behind a seam. Unexpected calls +and unmet expectations fail the test case that owns the mock. + +## 4. Per-test code coverage + +The runner runs each test case of an instrumented binary on its own and keeps +one coverage profile per test: LLVM source-based coverage with Clang and GCC's +gcov format with GCC. The result is a map from each test to the lines and +branches it reaches, plus the usual combined report. + +## 5. Change-impact selection and prioritization + +With the per-test map, the runner selects the tests whose covered lines a +change touches, taking the change from Git, and runs those first. Ordering +uses failure history, duration and coverage of the change, so that a failing +run fails early. A full run remains one option away, and the selection is +reported, never silent. + +## 6. Mutation testing + +The runner makes small changes to the code under test, such as a flipped +comparison or a removed call, rebuilds, and runs the tests that cover the +changed line. A mutation that no test detects points at a missing assertion. +The per-test coverage map keeps this affordable: only covering tests run for +each mutant. + +## 7. Test generation + +The property engine already records choice sequences. Test generation feeds +those sequences from three sources: + +- **coverage-guided:** choice sequences that reach new coverage are kept and + mutated further, as a fuzzer keeps inputs, so property tests explore code + that random inputs miss; +- **mutation-guided:** inputs are searched for that tell a surviving mutant + from the original program, and become new regression examples; +- **incremental:** examples found in earlier runs, kept in the example + database, seed the next run, so the search continues where it stopped. + +Generated examples are written as ordinary Catch2 test cases or stored replay +tokens, so they run without the generator. diff --git a/docs/why-catch3.md b/docs/why-catch3.md index 8c900ac7..9be7a5f7 100644 --- a/docs/why-catch3.md +++ b/docs/why-catch3.md @@ -15,19 +15,19 @@ Catch2 runner. | Area | Catch2 compatibility engine | Progmasoft Catch3 additions | | --- | --- | --- | -| Test execution | Catch2's existing runner, reporters, command-line interface, and test registration | `CATCH3_CHECK_PROPERTY` reports property outcomes through Catch2's assertion path | +| Test execution | Catch2's existing runner, reporters, command-line interface, and test registration | `CATCH3_CHECK_FOR_ALL` and `CATCH3_CHECK_PROPERTY` report property outcomes through Catch2's assertion path | | Public include path | Existing `catch2/...` headers | `Progmasoft/Catch3.hpp` and focused `Progmasoft/Catch3/...` headers | | C++ baseline | Existing Catch2 target remains at C++14 | The added property API requires C++20 | | Randomness | Catch2's existing behavior is unchanged | A specified SplitMix64 sequence provides deterministic trial seeds and unbiased bounded integer sampling | -| Generated values | No change to existing Catch2 generators | Initial integer/boolean strategies, custom generator callbacks, and bounded greedy shrinking | +| Generated values | No change to existing Catch2 generators | Strategies for numbers, text, collections and structures with `Map`, `Filter` and `FlatMap`; shrinking of recorded choices, so every combination shrinks; replay tokens and an example database; state machines, `Classify`/`Cover` statistics, targeted search, and differential and metamorphic oracles | | Build identity | Existing `Catch2` target and APIs remain available | Separate `Progmasoft::Catch3` CMake target and `//src/Progmasoft:catch3` Bazel target | | Source terms | Catch2-originated source retains its upstream BSL-1.0 terms | Progmasoft-authored additions state their own SPDX license in each file | -The current property runner handles one generated value per property. Shrinking -is deterministic and bounded, but greedy rather than globally minimal. There is -not yet built-in support for state-machine properties, mapped or filtered generators, -discard/classification policies, parallel execution, or a standalone Catch3 -runner. Those features are not implied by the initial API. +Properties take any number of generated values. Shrinking is deterministic and +bounded, but local rather than globally minimal. Parallel trials and a +standalone Catch3 runner do not exist yet; [the roadmap](progmasoft-roadmap.md#top) +orders them with mocking, per-test coverage, test selection, mutation testing +and test generation. ## Compatibility policy diff --git a/premake5.lua b/premake5.lua index ca773a60..7b5b1835 100644 --- a/premake5.lua +++ b/premake5.lua @@ -62,6 +62,9 @@ project "ProgmasoftCatch3" targetdir "build/premake/bin/%{cfg.buildcfg}" objdir "build/premake/obj/ProgmasoftCatch3/%{cfg.buildcfg}" files { + "src/Progmasoft/ChoiceShrinker.cpp", + "src/Progmasoft/Context.cpp", + "src/Progmasoft/PropertyEngine.cpp", "src/Progmasoft/Results.cpp", "src/Progmasoft/Results.hpp", "src/Progmasoft/Snapshot.cpp", @@ -83,8 +86,11 @@ project "ProgmasoftPropertyTests" files { "tests/Progmasoft/Generators.tests.cpp", "tests/Progmasoft/Property.tests.cpp", + "tests/Progmasoft/ForAll.tests.cpp", "tests/Progmasoft/Results.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp", + "tests/Progmasoft/Stateful.tests.cpp", + "tests/Progmasoft/Strategy.tests.cpp", "src/Progmasoft/**.hpp", } includedirs { "src", generated_include } diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 5eec18e9..cc3af6e7 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -363,6 +363,9 @@ add_library(Catch2::Catch2 ALIAS Catch2) # Progmasoft's compiled C++20 utilities are separate from Catch2's # compatibility engine, which retains its existing C++14 baseline. add_library(ProgmasoftCatch3 STATIC + "${CATCH_DIR}/src/Progmasoft/ChoiceShrinker.cpp" + "${CATCH_DIR}/src/Progmasoft/Context.cpp" + "${CATCH_DIR}/src/Progmasoft/PropertyEngine.cpp" "${CATCH_DIR}/src/Progmasoft/Results.cpp" "${CATCH_DIR}/src/Progmasoft/Snapshot.cpp" "${CATCH_DIR}/src/Progmasoft/XmlWriter.cpp" diff --git a/src/Progmasoft/BUILD.bazel b/src/Progmasoft/BUILD.bazel index 14a67bf6..b053c277 100644 --- a/src/Progmasoft/BUILD.bazel +++ b/src/Progmasoft/BUILD.bazel @@ -7,6 +7,9 @@ package(default_visibility = ["//visibility:public"]) cc_library( name = "catch3", srcs = [ + "ChoiceShrinker.cpp", + "Context.cpp", + "PropertyEngine.cpp", "Results.cpp", "Snapshot.cpp", "XmlWriter.cpp", @@ -16,6 +19,7 @@ cc_library( "Catch3/Assertions.hpp", "Catch3/Detail/InvalidArgument.hpp", "Catch3/Detail/Sequence.hpp", + "Catch3/ForAll.hpp", "Catch3/Generator.hpp", "Catch3/Generators/Boolean.hpp", "Catch3/Generators/Element.hpp", @@ -25,8 +29,16 @@ cc_library( "Catch3/Generators/Vector.hpp", "Catch3/Property.hpp", "Catch3/Random.hpp", + "Catch3/Relations.hpp", "Catch3/Results.hpp", "Catch3/Snapshot.hpp", + "Catch3/Stateful.hpp", + "Catch3/Strategy/ChoiceSource.hpp", + "Catch3/Strategy/Containers.hpp", + "Catch3/Strategy/Context.hpp", + "Catch3/Strategy/Engine.hpp", + "Catch3/Strategy/Numeric.hpp", + "Catch3/Strategy/Strategy.hpp", "Catch3/XmlWriter.hpp", ], copts = select({ diff --git a/src/Progmasoft/Catch3.hpp b/src/Progmasoft/Catch3.hpp index 5fca369a..a1db1db7 100644 --- a/src/Progmasoft/Catch3.hpp +++ b/src/Progmasoft/Catch3.hpp @@ -6,6 +6,7 @@ #include <catch2/catch_all.hpp> #include <Progmasoft/Catch3/Assertions.hpp> +#include <Progmasoft/Catch3/ForAll.hpp> #include <Progmasoft/Catch3/Generator.hpp> #include <Progmasoft/Catch3/Generators/Boolean.hpp> #include <Progmasoft/Catch3/Generators/Element.hpp> @@ -15,6 +16,13 @@ #include <Progmasoft/Catch3/Generators/Vector.hpp> #include <Progmasoft/Catch3/Property.hpp> #include <Progmasoft/Catch3/Random.hpp> +#include <Progmasoft/Catch3/Relations.hpp> #include <Progmasoft/Catch3/Results.hpp> #include <Progmasoft/Catch3/Snapshot.hpp> +#include <Progmasoft/Catch3/Stateful.hpp> +#include <Progmasoft/Catch3/Strategy/ChoiceSource.hpp> +#include <Progmasoft/Catch3/Strategy/Containers.hpp> +#include <Progmasoft/Catch3/Strategy/Context.hpp> +#include <Progmasoft/Catch3/Strategy/Numeric.hpp> +#include <Progmasoft/Catch3/Strategy/Strategy.hpp> #include <Progmasoft/Catch3/XmlWriter.hpp> diff --git a/src/Progmasoft/Catch3/Assertions.hpp b/src/Progmasoft/Catch3/Assertions.hpp index 668c086d..61fef04a 100644 --- a/src/Progmasoft/Catch3/Assertions.hpp +++ b/src/Progmasoft/Catch3/Assertions.hpp @@ -5,8 +5,14 @@ #include <catch2/catch_message.hpp> #include <catch2/catch_test_macros.hpp> +#include <catch2/interfaces/catch_interfaces_capture.hpp> + +#include <Progmasoft/Catch3/ForAll.hpp> #include <Progmasoft/Catch3/Property.hpp> #include <Progmasoft/Catch3/Snapshot.hpp> +#include <Progmasoft/Catch3/Strategy/Context.hpp> + +#include <string> /// Runs a property and reports its replay seed and minimized counterexample. #define CATCH3_CHECK_PROPERTY( ... ) \ @@ -25,3 +31,30 @@ INFO( catch3SnapshotResult.Message ); \ CHECK( catch3SnapshotResult.Succeeded() ); \ } while ( false ) + +/// Runs ForAll( [options,] strategy..., body ) under the name of the +/// current test case, which keys the example database, and reports the +/// result through Catch2: the minimized counterexample, its notes and its +/// replay token on failure. +#define CATCH3_CHECK_FOR_ALL( ... ) \ + do { \ + const ::Progmasoft::Catch3::Detail::PropertyNameScope catch3Name( \ + ::Catch::getResultCapture().getCurrentTestName() + ":" + \ + std::to_string( __LINE__ ) ); \ + const auto catch3ForAllResult = \ + ::Progmasoft::Catch3::ForAll( __VA_ARGS__ ); \ + INFO( catch3ForAllResult.Describe() ); \ + CHECK( catch3ForAllResult.Succeeded() ); \ + } while ( false ) + +/// As CATCH3_CHECK_FOR_ALL, but ends the test case on failure. +#define CATCH3_REQUIRE_FOR_ALL( ... ) \ + do { \ + const ::Progmasoft::Catch3::Detail::PropertyNameScope catch3Name( \ + ::Catch::getResultCapture().getCurrentTestName() + ":" + \ + std::to_string( __LINE__ ) ); \ + const auto catch3ForAllResult = \ + ::Progmasoft::Catch3::ForAll( __VA_ARGS__ ); \ + INFO( catch3ForAllResult.Describe() ); \ + REQUIRE( catch3ForAllResult.Succeeded() ); \ + } while ( false ) diff --git a/src/Progmasoft/Catch3/ForAll.hpp b/src/Progmasoft/Catch3/ForAll.hpp new file mode 100644 index 00000000..e41bfdb6 --- /dev/null +++ b/src/Progmasoft/Catch3/ForAll.hpp @@ -0,0 +1,191 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Property.hpp> +#include <Progmasoft/Catch3/Strategy/Containers.hpp> +#include <Progmasoft/Catch3/Strategy/Context.hpp> +#include <Progmasoft/Catch3/Strategy/Engine.hpp> +#include <Progmasoft/Catch3/Strategy/Strategy.hpp> +#include <exception> +#include <functional> +#include <optional> +#include <string> +#include <tuple> +#include <type_traits> +#include <utility> + +namespace Progmasoft::Catch3 +{ + + namespace Detail + { + + /// Runs a body on one input and turns what happened into a verdict. + /// A body returns bool, or void and fails through + /// CATCH3_PROPERTY_ASSERT; an exception from it is a failure. + template<typename Body, typename Tuple> + void + RunBody(Body &&body, const Tuple &values, TrialOutcome &outcome) + { + TrialRecordScope scope(outcome.Record); +#if defined(__cpp_exceptions) + try + { +#endif + using Result = decltype(std::apply(body, values)); + if constexpr (std::is_void_v<Result>) + { + std::apply(body, values); + } + else + { + if (!static_cast<bool>(std::apply(body, values))) + { + RecordPropertyFailureMessage( + "property returned false"); + } + } +#if defined(__cpp_exceptions) + } + catch (const std::exception &exception) + { + RecordPropertyFailureMessage( + std::string("property threw std::exception: ") + + exception.what()); + } + catch (...) + { + RecordPropertyFailureMessage( + "property threw a non-standard exception"); + } +#endif + if (outcome.Record.Failure.has_value()) + { + outcome.Verdict = TrialVerdict::kFailed; + outcome.Reason = *outcome.Record.Failure; + } + else if (outcome.Record.Discarded) + { + outcome.Verdict = TrialVerdict::kDiscarded; + } + } + + /// Draws a value, catching what a strategy throws. + template<typename ValueType> + [[nodiscard]] std::optional<ValueType> + DrawGuarded(const Strategy<ValueType> &strategy, + ChoiceSource &source, + TrialOutcome &outcome) + { +#if defined(__cpp_exceptions) + try + { +#endif + auto value = strategy.Draw(source); + if (!value.has_value()) + { + outcome.Verdict = TrialVerdict::kInvalid; + } + return value; +#if defined(__cpp_exceptions) + } + catch (const std::exception &exception) + { + outcome.Verdict = TrialVerdict::kErrored; + outcome.Reason = std::string("strategy threw std::exception: ") + + exception.what(); + } + catch (...) + { + outcome.Verdict = TrialVerdict::kErrored; + outcome.Reason = "strategy threw a non-standard exception"; + } + return std::nullopt; +#endif + } + + template<typename... ValueTypes> + [[nodiscard]] std::string + DescribeArguments(const std::tuple<ValueTypes...> &values) + { + if constexpr (sizeof...(ValueTypes) == 1) + { + return DescribeValue(std::get<0>(values)); + } + else + { + return DescribeValue(values); + } + } + + template<typename Body, typename... ValueTypes> + [[nodiscard]] PropertyResult + ForAllWith(const PropertyOptions &options, + Body body, + Strategy<ValueTypes>... strategies) + { + static_assert(sizeof...(ValueTypes) > 0, + "ForAll needs at least one strategy"); + const auto input = Gen::Tuples(std::move(strategies)...); + const TrialRunner runner = + [&](ChoiceSource &source, bool describe) -> TrialOutcome { + TrialOutcome outcome; + auto values = DrawGuarded(input, source, outcome); + if (!values.has_value()) + { + return outcome; + } + if (describe) + { + outcome.Description = DescribeArguments(*values); + } + RunBody(body, std::as_const(*values), outcome); + return outcome; + }; + return RunPropertyEngine(options, runner); + } + + template<typename Type> + inline constexpr bool kIsPropertyOptions + = std::is_same_v<std::remove_cvref_t<Type>, PropertyOptions>; + + } // namespace Detail + + /// Checks that a property holds for values drawn from strategies. + /// + /// ForAll([options,] strategy..., body) + /// + /// body receives one value per strategy, as const references. It + /// returns bool, or returns void and uses CATCH3_PROPERTY_ASSERT and + /// CATCH3_PROPERTY_ASSUME; Note, Classify, Cover and Target may be + /// called from it. A failure is minimized by shrinking the choices the + /// strategies made, and the report carries a replay token for exactly + /// the minimized input. + template<typename... Arguments> + [[nodiscard]] PropertyResult + ForAll(Arguments &&...arguments) + { + constexpr std::size_t kCount = sizeof...(Arguments); + static_assert(kCount >= 2, "ForAll needs strategies and a body"); + auto all = std::forward_as_tuple(std::forward<Arguments>(arguments)...); + using First = std::tuple_element_t<0, std::tuple<Arguments...>>; + if constexpr (Detail::kIsPropertyOptions<First>) + { + return [&]<std::size_t... Index>(std::index_sequence<Index...>) { + return Detail::ForAllWith(std::get<0>(all), + std::get<kCount - 1U>(all), + std::get<Index + 1U>(all)...); + }(std::make_index_sequence<kCount - 2U>{}); + } + else + { + return [&]<std::size_t... Index>(std::index_sequence<Index...>) { + return Detail::ForAllWith(PropertyOptions{}, + std::get<kCount - 1U>(all), + std::get<Index>(all)...); + }(std::make_index_sequence<kCount - 1U>{}); + } + } + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Property.hpp b/src/Progmasoft/Catch3/Property.hpp index 57f7ac7b..bd055148 100644 --- a/src/Progmasoft/Catch3/Property.hpp +++ b/src/Progmasoft/Catch3/Property.hpp @@ -31,6 +31,30 @@ namespace Progmasoft::Catch3 /// reported failure to reproduce that single trial, including its /// shrinking, without rerunning the trials before it. std::optional<std::uint64_t> ReplayTrialSeed = std::nullopt; + + // The fields below are read by ForAll and the other strategy-based + // checks; CheckProperty with a Generator ignores them. + + /// A replay token from a failure report: runs exactly the reported, + /// minimized counterexample and nothing else. + std::optional<std::string> Replay = std::nullopt; + /// Trials that may be discarded, by a filter that found nothing or + /// by CATCH3_PROPERTY_ASSUME, per requested trial before the + /// property gives up. + std::size_t MaxDiscardRatio = 10; + /// Extra trials that search for a higher score after the regular + /// ones, when the property calls Target. 0 disables the search. + std::size_t TargetSteps = 500; + /// Directory of the example database. Failing examples are stored + /// there and replayed first on the next run. Empty: the directory + /// named by CATCH3_PROPERTY_DATABASE, or no database when that is + /// unset. + std::string DatabasePath; + /// Names the property in the database. The CATCH3_ macros fill it + /// with the test case name and the source line. + std::string Name; + /// Adds the Classify and Cover statistics to a passing report. + bool ShowStatistics = false; }; enum class PropertyStatus : std::uint8_t @@ -39,6 +63,20 @@ namespace Progmasoft::Catch3 kFailed, kErrored, kInvalidConfiguration, + /// A Cover requirement was not met. + kInsufficientCoverage, + /// Too many trials were discarded to trust the result. + kGaveUp, + }; + + /// How often one Classify or Cover label held. + struct LabelStatistic final + { + std::string Label; + std::size_t Count = 0; + double Percentage = 0.0; + /// The percentage a Cover call requires; 0 for Classify. + double Required = 0.0; }; struct PropertyFailure final @@ -49,6 +87,11 @@ namespace Progmasoft::Catch3 std::size_t ShrinkSteps = 0; std::string Counterexample; std::string Reason; + /// Replay token of the minimized counterexample; empty for + /// CheckProperty with a Generator, whose replay is TrialSeed. + std::string Replay; + /// What the property recorded with Note for the counterexample. + std::vector<std::string> Notes; }; /// Stable, machine-readable outcome with a deterministic human-readable @@ -59,6 +102,10 @@ namespace Progmasoft::Catch3 std::size_t TrialsRun = 0; std::optional<PropertyFailure> Failure; std::string Error; + std::size_t Discarded = 0; + std::vector<LabelStatistic> Statistics; + /// Set when Statistics should be part of a passing report. + bool ShowStatistics = false; [[nodiscard]] bool Succeeded() const noexcept @@ -74,6 +121,23 @@ namespace Progmasoft::Catch3 { case PropertyStatus::kPassed: output << "property passed: " << TrialsRun << " trials"; + if (Discarded != 0) + { + output << ", " << Discarded << " discarded"; + } + if (ShowStatistics) + { + DescribeStatistics(output); + } + return output.str(); + case PropertyStatus::kInsufficientCoverage: + output << "property coverage insufficient after " + << TrialsRun << " trials: " << Error; + DescribeStatistics(output); + return output.str(); + case PropertyStatus::kGaveUp: + output << "property gave up after " << TrialsRun + << " trials: " << Error; return output.str(); case PropertyStatus::kInvalidConfiguration: output << "invalid property configuration: " << Error; @@ -88,7 +152,7 @@ namespace Progmasoft::Catch3 break; } - if (Failure.has_value()) + if (Failure.has_value() && Failure->Replay.empty()) { output << "; failing trial " << Failure->Trial << ", root seed " << Failure->RootSeed << ", replay seed " @@ -99,8 +163,42 @@ namespace Progmasoft::Catch3 << "\nreplay: set PropertyOptions::ReplayTrialSeed to " << Failure->TrialSeed; } + else if (Failure.has_value()) + { + output << "; failing trial " << Failure->Trial << ", root seed " + << Failure->RootSeed << ", shrink checks " + << Failure->ShrinkSteps + << "\ncounterexample: " << Failure->Counterexample + << "\nreason: " << Failure->Reason; + for (const auto ¬e : Failure->Notes) + { + output << "\nnote: " << note; + } + output << "\nreplay: set PropertyOptions::Replay to \"" + << Failure->Replay << '"'; + } return output.str(); } + + private: + void + DescribeStatistics(std::ostringstream &output) const + { + for (const auto &statistic : Statistics) + { + output << "\n " << statistic.Label << ": " + << statistic.Count << " (" + << static_cast<long long>(statistic.Percentage * 10.0 + + 0.5) + / 10.0 + << "%"; + if (statistic.Required > 0.0) + { + output << ", required " << statistic.Required << "%"; + } + output << ')'; + } + } }; namespace Detail diff --git a/src/Progmasoft/Catch3/Relations.hpp b/src/Progmasoft/Catch3/Relations.hpp new file mode 100644 index 00000000..d3c2ec62 --- /dev/null +++ b/src/Progmasoft/Catch3/Relations.hpp @@ -0,0 +1,99 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/ForAll.hpp> +#include <Progmasoft/Catch3/Strategy/Context.hpp> +#include <functional> +#include <string> +#include <utility> + +// Ready-made property bodies for the oracles that need no specification of +// the expected result: differential, metamorphic and round-trip testing. +// Each one is passed to ForAll in place of a hand-written body: +// +// ForAll(Gen::Vectors(Gen::Integers<int>()), +// Equivalent(ReferenceSort, FastSort)); + +namespace Progmasoft::Catch3 +{ + + /// Differential testing: the candidate must return what the reference + /// returns for every input. A mismatch reports both results. + template<typename Reference, typename Candidate> + [[nodiscard]] auto + Equivalent(Reference reference, Candidate candidate) + { + return [reference, candidate](const auto &...input) mutable { + const auto expected = std::invoke(reference, input...); + const auto actual = std::invoke(candidate, input...); + if (!(expected == actual)) + { + Note("reference: " + Detail::DescribeValue(expected)); + Note("candidate: " + Detail::DescribeValue(actual)); + Detail::RecordPropertyFailureMessage( + "the candidate disagrees with the reference"); + } + }; + } + + /// Metamorphic testing: for every input x, relation(function(x), + /// function(transform(x))) must hold, for example that sorting a + /// shuffled list gives the same result, or that doubling a length + /// doubles a cost. + template<typename Function, typename Transform, typename Relation> + [[nodiscard]] auto + Metamorphic(Function function, Transform transform, Relation relation) + { + return [function, transform, relation](const auto &input) mutable { + const auto source = std::invoke(function, input); + const auto followUpInput = std::invoke(transform, input); + const auto followUp = std::invoke(function, followUpInput); + if (!static_cast<bool>(std::invoke(relation, source, followUp))) + { + Note("follow-up input: " + Detail::DescribeValue(followUpInput)); + Note("source output: " + Detail::DescribeValue(source)); + Note("follow-up output: " + Detail::DescribeValue(followUp)); + Detail::RecordPropertyFailureMessage( + "the metamorphic relation does not hold"); + } + }; + } + + /// decode(encode(x)) == x for every input. + template<typename Encode, typename Decode> + [[nodiscard]] auto + RoundTrip(Encode encode, Decode decode) + { + return [encode, decode](const auto &input) mutable { + const auto encoded = std::invoke(encode, input); + const auto decoded = std::invoke(decode, encoded); + if (!(decoded == input)) + { + Note("encoded: " + Detail::DescribeValue(encoded)); + Note("decoded: " + Detail::DescribeValue(decoded)); + Detail::RecordPropertyFailureMessage( + "decoding the encoded input did not give it back"); + } + }; + } + + /// function(function(x)) == function(x) for every input. + template<typename Function> + [[nodiscard]] auto + Idempotent(Function function) + { + return [function](const auto &input) mutable { + const auto once = std::invoke(function, input); + const auto twice = std::invoke(function, once); + if (!(twice == once)) + { + Note("once: " + Detail::DescribeValue(once)); + Note("twice: " + Detail::DescribeValue(twice)); + Detail::RecordPropertyFailureMessage( + "applying the function twice changed the result"); + } + }; + } + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Stateful.hpp b/src/Progmasoft/Catch3/Stateful.hpp new file mode 100644 index 00000000..6bd34caa --- /dev/null +++ b/src/Progmasoft/Catch3/Stateful.hpp @@ -0,0 +1,157 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/ForAll.hpp> +#include <Progmasoft/Catch3/Strategy/Containers.hpp> +#include <functional> +#include <optional> +#include <string> +#include <utility> +#include <vector> + +namespace Progmasoft::Catch3 +{ + + /// One step of a stateful test: a description for the report and the + /// code that applies the step to the model and to the system under + /// test. It checks the system against the model with + /// CATCH3_PROPERTY_ASSERT. + template<typename Model, typename System> + struct Action final + { + std::string Description; + std::function<void(Model &, System &)> Apply; + }; + + /// A kind of step. Make builds the step from the current model, so its + /// arguments can depend on the state, for example by choosing a key + /// that the model holds. Precondition says whether the step may run in + /// a state; an unset precondition always holds. + template<typename Model, typename System> + struct Command final + { + std::string Name; + std::function<Strategy<Action<Model, System>>(const Model &)> Make; + std::function<bool(const Model &)> Precondition = {}; + }; + + /// A command without arguments. + template<typename Model, typename System, typename Apply> + [[nodiscard]] Command<Model, System> + MakeCommand(std::string name, Apply apply) + { + return Command<Model, System>{ + name, + [name, apply](const Model &) { + return Gen::Just(Action<Model, System>{ name, apply }); + }, + }; + } + + namespace Detail + { + + template<typename Model, typename System> + [[nodiscard]] TrialOutcome + RunCommands(ChoiceSource &source, + bool describe, + const std::function<Model()> &initialModel, + const std::function<System(const Model &)> &makeSystem, + const std::vector<Command<Model, System>> &commands, + const SizeRange &steps) + { + TrialOutcome outcome; + Model model = initialModel(); + std::optional<System> system; + std::vector<std::string> log; + { + TrialRecordScope scope(outcome.Record); + system.emplace(makeSystem(model)); + } + const auto addStep = [&]() -> bool { + // The choice names a command; when its precondition does not + // hold, the next commands are tried in order, so that the + // same choice always leads to the same command. + const std::size_t first = static_cast<std::size_t>( + source.Draw(commands.size() - 1U)); + const Command<Model, System> *chosen = nullptr; + for (std::size_t offset = 0; offset < commands.size(); ++offset) + { + const auto &candidate + = commands[(first + offset) % commands.size()]; + if (!candidate.Precondition || candidate.Precondition(model)) + { + chosen = &candidate; + break; + } + } + if (chosen == nullptr) + { + return false; + } + auto action = DrawGuarded(chosen->Make(model), source, outcome); + if (!action.has_value()) + { + return false; + } + log.push_back(action->Description); + RunBody( + [&]() { action->Apply(model, *system); }, + std::tuple<>(), + outcome); + return outcome.Verdict == TrialVerdict::kPassed; + }; + (void)Detail::DrawElements(source, steps, addStep); + if (outcome.Verdict == TrialVerdict::kInvalid && !log.empty()) + { + // A step that could not be built ends the sequence; the + // steps before it still ran and passed. + outcome.Verdict = TrialVerdict::kPassed; + } + if (describe) + { + std::string text = "{"; + for (std::size_t index = 0; index < log.size(); ++index) + { + text += index == 0 ? " " : ", "; + text += log[index]; + } + outcome.Description = text + (log.empty() ? "}" : " }"); + } + return outcome; + } + + } // namespace Detail + + /// Model-based testing: runs random sequences of commands against a + /// model and the system under test, side by side. A failing sequence + /// shrinks to a short one with simple arguments, since dropping a step + /// is deleting its choices. + template<typename Model, typename System> + [[nodiscard]] PropertyResult + CheckStateful(const PropertyOptions &options, + std::function<Model()> initialModel, + std::function<System(const Model &)> makeSystem, + std::vector<Command<Model, System>> commands, + SizeRange steps = {}) + { + if (commands.empty()) + { + Detail::RejectInvalidArgument( + "CheckStateful requires at least one command"); + } + Detail::CheckRange(steps); + const Detail::TrialRunner runner + = [&](ChoiceSource &source, bool describe) { + return Detail::RunCommands(source, + describe, + initialModel, + makeSystem, + commands, + steps); + }; + return Detail::RunPropertyEngine(options, runner); + } + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Strategy/ChoiceSource.hpp b/src/Progmasoft/Catch3/Strategy/ChoiceSource.hpp new file mode 100644 index 00000000..aaa6c509 --- /dev/null +++ b/src/Progmasoft/Catch3/Strategy/ChoiceSource.hpp @@ -0,0 +1,191 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Random.hpp> +#include <cstddef> +#include <cstdint> +#include <optional> +#include <span> +#include <vector> + +namespace Progmasoft::Catch3 +{ + + /// The stream of decisions a strategy makes while it builds one value. + /// + /// Every random decision of a strategy is one choice: an integer in the + /// inclusive range [0, maximum]. During generation the choices come from + /// a Random and are recorded; during shrinking and replay they are read + /// back from a recorded sequence. A strategy therefore never shrinks + /// values itself. The engine shrinks the recorded choices, and every + /// strategy built from these choices, including mapped, filtered and + /// dependent ones, shrinks with it. + /// + /// The contract that makes this work: choice 0 is the simplest decision, + /// and a smaller choice never produces a more complex value. A shorter + /// sequence is simpler than a longer one. A read past the end of a + /// replayed sequence yields 0, the simplest decision, so deleting + /// choices is always a valid shrink. + class ChoiceSource final + { + public: + /// Generation: choices come from a seeded Random, scaled by size. + ChoiceSource(std::uint64_t seed, std::size_t size) noexcept + : random_(seed) + , size_(size) + {} + + /// Replay: choices are read from a recorded sequence. + ChoiceSource(std::span<const std::uint64_t> replay, + std::size_t size) noexcept + : random_(0) + , replay_(replay) + , replaying_(true) + , size_(size) + {} + + /// A uniform choice in [0, maximum]. + [[nodiscard]] std::uint64_t + Draw(std::uint64_t maximum) + { + if (replaying_) + { + return Record(ReadReplay(maximum)); + } + const std::uint64_t value + = maximum == UINT64_MAX ? random_.Next() + : random_.Below(maximum + 1U); + return Record(value); + } + + /// A choice in [0, maximum] whose generation favours small values + /// and the edges of the range, scaled by the current size. Replay + /// is identical to Draw: only the distribution differs, never the + /// meaning of a choice. + [[nodiscard]] std::uint64_t + DrawBiased(std::uint64_t maximum) + { + if (replaying_) + { + return Record(ReadReplay(maximum)); + } + const std::uint64_t roll = random_.Below(32); + std::uint64_t value = 0; + if (roll == 0) + { + value = 0; + } + else if (roll == 1) + { + value = maximum; + } + else if (roll < 4) + { + value = maximum == 0 ? 0 : random_.Below(2); + } + else if (roll < 8 || maximum <= 1) + { + value = maximum == UINT64_MAX ? random_.Next() + : random_.Below(maximum + 1U); + } + else + { + // Grows with the size so that early trials stay small. + const std::uint64_t limit = ScaledLimit(maximum); + value = random_.Below(limit + 1U); + } + return Record(value); + } + + /// A biased coin: true with roughly the given probability during + /// generation. false (choice 0) is the simpler answer. + [[nodiscard]] bool + DrawBoolean(double probabilityOfTrue) + { + if (replaying_) + { + return Record(ReadReplay(1)) == 1; + } + const double unit + = static_cast<double>(random_.Next() >> 11U) * 0x1.0p-53; + return Record(unit < probabilityOfTrue ? 1 : 0) == 1; + } + + /// The size of the current trial, from 0 to 100. Strategies use it + /// to keep early trials small. + [[nodiscard]] std::size_t + Size() const noexcept + { + return size_; + } + + /// The choices drawn so far. + [[nodiscard]] const std::vector<std::uint64_t> & + Choices() const noexcept + { + return choices_; + } + + /// True when a replay read past the end of its sequence. + [[nodiscard]] bool + Overran() const noexcept + { + return overran_; + } + + /// Stops generation when a strategy exceeds this many choices, so + /// that an unbounded recursive strategy cannot exhaust memory. + static constexpr std::size_t kMaximumChoices = 1U << 20U; + + [[nodiscard]] bool + Exhausted() const noexcept + { + return choices_.size() >= kMaximumChoices; + } + + private: + [[nodiscard]] std::uint64_t + ReadReplay(std::uint64_t maximum) noexcept + { + if (position_ >= replay_.size()) + { + overran_ = true; + return 0; + } + const std::uint64_t value = replay_[position_++]; + // A shrink may lower a choice that an earlier, changed choice + // now bounds more tightly; clamping keeps the replay valid. + return value > maximum ? maximum : value; + } + + [[nodiscard]] std::uint64_t + ScaledLimit(std::uint64_t maximum) const noexcept + { + // 2^(size / 4 + 2): 4 at size 0, about 2^27 at size 100. + const std::size_t bits = size_ / 4U + 2U; + if (bits >= 64U) + { + return maximum; + } + const std::uint64_t limit = (std::uint64_t{ 1 } << bits) - 1U; + return limit < maximum ? limit : maximum; + } + + std::uint64_t + Record(std::uint64_t value) + { + choices_.push_back(value); + return value; + } + + Random random_; + std::span<const std::uint64_t> replay_; + std::size_t position_ = 0; + bool replaying_ = false; + bool overran_ = false; + std::size_t size_; + std::vector<std::uint64_t> choices_; + }; + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Strategy/Containers.hpp b/src/Progmasoft/Catch3/Strategy/Containers.hpp new file mode 100644 index 00000000..477df896 --- /dev/null +++ b/src/Progmasoft/Catch3/Strategy/Containers.hpp @@ -0,0 +1,391 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Strategy/Strategy.hpp> +#include <algorithm> +#include <cstddef> +#include <cstdint> +#include <limits> +#include <map> +#include <optional> +#include <set> +#include <string> +#include <tuple> +#include <utility> +#include <vector> + +namespace Progmasoft::Catch3 +{ + + /// Bounds of a generated collection. + struct SizeRange final + { + std::size_t Minimum = 0; + /// No upper bound when unset; the length then grows with the size + /// of the trial, around a quarter of it on average. + std::optional<std::size_t> Maximum = std::nullopt; + }; + + namespace Detail + { + + /// Draws a collection element by element. After the minimum, each + /// further element is announced by a "more" choice, so that the + /// engine removes an element by deleting its choices and shortens a + /// collection by turning one "more" into 0. + template<typename AddElement> + [[nodiscard]] bool + DrawElements(ChoiceSource &source, + const SizeRange &range, + AddElement addElement) + { + const std::size_t maximum + = range.Maximum.value_or(std::numeric_limits<std::size_t>::max()); + const double average + = std::max(1.0, static_cast<double>(source.Size()) / 4.0); + const double probability = average / (average + 1.0); + std::size_t count = 0; + while (count < maximum) + { + if (count >= range.Minimum && !source.DrawBoolean(probability)) + { + break; + } + if (!addElement()) + { + return false; + } + ++count; + } + return true; + } + + inline void + CheckRange(const SizeRange &range) + { + if (range.Maximum.has_value() && *range.Maximum < range.Minimum) + { + Detail::RejectInvalidArgument( + "a size range requires Minimum <= Maximum"); + } + } + + } // namespace Detail + + namespace Gen + { + + /// Vectors of elements; they shrink by losing elements and by + /// shrinking the elements that remain. + template<typename ElementType> + [[nodiscard]] Strategy<std::vector<ElementType>> + Vectors(Strategy<ElementType> element, SizeRange range = {}) + { + Detail::CheckRange(range); + return Strategy<std::vector<ElementType>>( + [element, range](ChoiceSource &source) + -> std::optional<std::vector<ElementType>> { + std::vector<ElementType> result; + const bool complete + = Detail::DrawElements(source, range, [&]() { + auto value = element.Draw(source); + if (!value.has_value()) + { + return false; + } + result.push_back(std::move(*value)); + return true; + }); + if (!complete) + { + return std::nullopt; + } + return result; + }); + } + + /// Vectors whose elements are distinct under key. A duplicate is + /// drawn again up to maximumRetries times before the trial gives + /// up. + template<typename ElementType, typename Key = std::identity> + [[nodiscard]] Strategy<std::vector<ElementType>> + UniqueVectors(Strategy<ElementType> element, + SizeRange range = {}, + Key key = {}, + std::size_t maximumRetries = 50) + { + Detail::CheckRange(range); + return Strategy<std::vector<ElementType>>( + [element, range, key, maximumRetries](ChoiceSource &source) + -> std::optional<std::vector<ElementType>> { + std::vector<ElementType> result; + const auto isNew = [&](const ElementType &candidate) { + return std::none_of( + result.begin(), result.end(), [&](const auto &old) { + return std::invoke(key, old) + == std::invoke(key, candidate); + }); + }; + const bool complete + = Detail::DrawElements(source, range, [&]() { + for (std::size_t retry = 0; + retry <= maximumRetries; + ++retry) + { + auto value = element.Draw(source); + if (!value.has_value()) + { + return false; + } + if (isNew(*value)) + { + result.push_back(std::move(*value)); + return true; + } + } + return false; + }); + if (!complete || result.size() < range.Minimum) + { + return std::nullopt; + } + return result; + }); + } + + /// Ordered sets of distinct elements. + template<typename ElementType> + [[nodiscard]] Strategy<std::set<ElementType>> + Sets(Strategy<ElementType> element, SizeRange range = {}) + { + return UniqueVectors(std::move(element), range) + .Map([](std::vector<ElementType> values) { + return std::set<ElementType>(values.begin(), values.end()); + }); + } + + /// Ordered maps with distinct keys. + template<typename KeyType, typename ValueType> + [[nodiscard]] Strategy<std::map<KeyType, ValueType>> + Maps(Strategy<KeyType> key, + Strategy<ValueType> value, + SizeRange range = {}) + { + Detail::CheckRange(range); + return Strategy<std::map<KeyType, ValueType>>( + [key, value, range](ChoiceSource &source) + -> std::optional<std::map<KeyType, ValueType>> { + std::map<KeyType, ValueType> result; + const bool complete + = Detail::DrawElements(source, range, [&]() { + for (int retry = 0; retry <= 50; ++retry) + { + auto drawnKey = key.Draw(source); + if (!drawnKey.has_value()) + { + return false; + } + if (result.contains(*drawnKey)) + { + continue; + } + auto drawnValue = value.Draw(source); + if (!drawnValue.has_value()) + { + return false; + } + result.emplace(std::move(*drawnKey), + std::move(*drawnValue)); + return true; + } + return false; + }); + if (!complete || result.size() < range.Minimum) + { + return std::nullopt; + } + return result; + }); + } + + /// Strings whose characters come from character. + [[nodiscard]] inline Strategy<std::string> + Strings(Strategy<char> character, SizeRange range = {}) + { + return Vectors(std::move(character), range) + .Map([](std::vector<char> characters) { + return std::string(characters.begin(), characters.end()); + }); + } + + /// Characters of an alphabet; shrinks towards its first character. + [[nodiscard]] inline Strategy<char> + Characters(std::string alphabet) + { + return SampledFrom(std::vector<char>(alphabet.begin(), alphabet.end())); + } + + /// Printable ASCII, simplest first: letters and digits, then the + /// rest, so that counterexamples read well. + [[nodiscard]] inline Strategy<char> + PrintableCharacters() + { + return Characters("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" + "0123456789 !\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"); + } + + /// UTF-8 encoded text of Unicode scalar values in [minimum, maximum]; + /// surrogates are never produced. Code points shrink towards + /// minimum. + [[nodiscard]] inline Strategy<std::string> + Utf8Strings(SizeRange range = {}, + char32_t minimum = U' ', + char32_t maximum = 0x10FFFF) + { + if (minimum > maximum || maximum > 0x10FFFF) + { + Detail::RejectInvalidArgument( + "Utf8Strings requires minimum <= maximum <= U+10FFFF"); + } + const Strategy<char32_t> codePoint( + [minimum, maximum]( + ChoiceSource &source) -> std::optional<char32_t> { + std::uint64_t value + = minimum + source.DrawBiased(maximum - minimum); + if (value >= 0xD800 && value <= 0xDFFF) + { + value = minimum; + } + return static_cast<char32_t>(value); + }); + return Vectors(codePoint, range) + .Map([](const std::vector<char32_t> &codePoints) { + std::string text; + for (const char32_t code : codePoints) + { + if (code < 0x80) + { + text += static_cast<char>(code); + } + else if (code < 0x800) + { + text += static_cast<char>(0xC0 | (code >> 6U)); + text += static_cast<char>(0x80 | (code & 0x3FU)); + } + else if (code < 0x10000) + { + text += static_cast<char>(0xE0 | (code >> 12U)); + text += static_cast<char>( + 0x80 | ((code >> 6U) & 0x3FU)); + text += static_cast<char>(0x80 | (code & 0x3FU)); + } + else + { + text += static_cast<char>(0xF0 | (code >> 18U)); + text += static_cast<char>( + 0x80 | ((code >> 12U) & 0x3FU)); + text += static_cast<char>( + 0x80 | ((code >> 6U) & 0x3FU)); + text += static_cast<char>(0x80 | (code & 0x3FU)); + } + } + return text; + }); + } + + /// No value or a value; shrinks towards no value. + template<typename ValueType> + [[nodiscard]] Strategy<std::optional<ValueType>> + Optionals(Strategy<ValueType> value) + { + return Strategy<std::optional<ValueType>>( + [value](ChoiceSource &source) + -> std::optional<std::optional<ValueType>> { + if (source.Draw(1) == 0) + { + return std::optional<ValueType>(); + } + auto drawn = value.Draw(source); + if (!drawn.has_value()) + { + return std::nullopt; + } + return std::optional<ValueType>(std::move(*drawn)); + }); + } + + /// One value from each strategy, drawn left to right. + template<typename... ValueTypes> + [[nodiscard]] Strategy<std::tuple<ValueTypes...>> + Tuples(Strategy<ValueTypes>... strategies) + { + return Strategy<std::tuple<ValueTypes...>>( + [strategies...](ChoiceSource &source) + -> std::optional<std::tuple<ValueTypes...>> { + std::tuple<std::optional<ValueTypes>...> parts; + bool complete = true; + // A braced list fixes the evaluation order: left to + // right, which keeps the choice sequence stable. + std::apply( + [&](auto &...part) { + ((complete = complete + && (part = strategies.Draw(source)) + .has_value()), + ...); + }, + parts); + if (!complete) + { + return std::nullopt; + } + return std::apply( + [](auto &...part) { + return std::tuple<ValueTypes...>( + std::move(*part)...); + }, + parts); + }); + } + + /// Builds an object from one value of each strategy, as + /// ObjectType{ values... }. + template<typename ObjectType, typename... ValueTypes> + [[nodiscard]] Strategy<ObjectType> + Construct(Strategy<ValueTypes>... strategies) + { + return Tuples(std::move(strategies)...) + .Map([](std::tuple<ValueTypes...> values) { + return std::apply( + [](ValueTypes &...value) { + return ObjectType{ std::move(value)... }; + }, + values); + }); + } + + /// Permutations of a list; they shrink towards the original order. + template<typename ValueType> + [[nodiscard]] Strategy<std::vector<ValueType>> + Permutations(std::vector<ValueType> values) + { + return Strategy<std::vector<ValueType>>( + [values](ChoiceSource &source) + -> std::optional<std::vector<ValueType>> { + std::vector<ValueType> result = values; + // Each choice says how far ahead the next element comes + // from; choice 0 keeps it in place. + for (std::size_t index = 0; index + 1U < result.size(); + ++index) + { + const auto offset = static_cast<std::size_t>( + source.Draw(result.size() - index - 1U)); + std::swap(result[index], result[index + offset]); + } + return result; + }); + } + + } // namespace Gen + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/Catch3/Strategy/Context.hpp b/src/Progmasoft/Catch3/Strategy/Context.hpp new file mode 100644 index 00000000..57a87819 --- /dev/null +++ b/src/Progmasoft/Catch3/Strategy/Context.hpp @@ -0,0 +1,145 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <catch2/catch_tostring.hpp> +#include <cstddef> +#include <optional> +#include <string> +#include <string_view> +#include <vector> + +namespace Progmasoft::Catch3 +{ + + /// What one run of a property body recorded besides its verdict. + /// + /// The engine installs a fresh record before each run of the body, on + /// the running thread, and reads it afterwards. The functions below + /// write into the record of the current thread; called outside a + /// property they do nothing. + struct TrialRecord final + { + struct Coverage final + { + std::string Label; + double Required = 0.0; + bool Held = false; + }; + + struct Score final + { + std::string Label; + double Value = 0.0; + }; + + std::vector<std::string> Notes; + std::vector<std::string> Labels; + std::vector<Coverage> Covers; + std::vector<Score> Targets; + std::optional<std::string> Failure; + bool Discarded = false; + }; + + namespace Detail + { + + /// The record of the property body running on this thread, or null. + [[nodiscard]] TrialRecord * + CurrentTrialRecord() noexcept; + + /// Installs a record for the lifetime of the scope. + class TrialRecordScope final + { + public: + explicit TrialRecordScope(TrialRecord &record) noexcept; + ~TrialRecordScope(); + TrialRecordScope(const TrialRecordScope &) = delete; + TrialRecordScope &operator=(const TrialRecordScope &) = delete; + + private: + TrialRecord *previous_; + }; + + void + RecordPropertyFailure(std::string_view expression, + std::string_view file, + std::size_t line); + + /// Fails the current trial with a message, as the relation checks + /// do when they compare two results. + void + RecordPropertyFailureMessage(std::string message); + + /// The name a property gets when PropertyOptions::Name is empty; + /// the CATCH3_ macros set it to the test case name and the line. + [[nodiscard]] std::string + DefaultPropertyName(); + + class PropertyNameScope final + { + public: + explicit PropertyNameScope(std::string name); + ~PropertyNameScope(); + PropertyNameScope(const PropertyNameScope &) = delete; + PropertyNameScope &operator=(const PropertyNameScope &) = delete; + + private: + std::string previous_; + }; + + } // namespace Detail + + /// Adds a line to the failure report of the current trial. + void + Note(std::string text); + + /// Adds a value to the failure report, rendered as Catch2 renders it. + template<typename ValueType> + void + Note(std::string_view name, const ValueType &value) + { + Note(std::string(name) + " = " + + Catch::StringMaker<ValueType>::convert(value)); + } + + /// Counts the current trial under label; the counts are reported with + /// PropertyOptions::ShowStatistics and with a Cover failure. + void + Classify(bool condition, std::string label); + + /// Requires that condition holds in at least requiredPercentage of the + /// trials; otherwise the property fails with kInsufficientCoverage. + void + Cover(double requiredPercentage, bool condition, std::string label); + + /// Reports a score that the engine tries to maximize after the regular + /// trials: it mutates the highest-scoring input to search for a higher + /// score, which finds inputs that random sampling rarely reaches. + void + Target(double score, std::string label = {}); + +} // namespace Progmasoft::Catch3 + +/// Fails the current property trial, without unwinding, when the condition +/// is false. Use it in a property body that returns void. +#define CATCH3_PROPERTY_ASSERT( ... ) \ + do { \ + if ( !static_cast<bool>( __VA_ARGS__ ) ) { \ + ::Progmasoft::Catch3::Detail::RecordPropertyFailure( \ + #__VA_ARGS__, __FILE__, static_cast<std::size_t>( __LINE__ ) ); \ + return; \ + } \ + } while ( false ) + +/// Discards the current trial when the condition is false. A discarded +/// trial is neither a pass nor a failure; see PropertyOptions::MaxDiscardRatio. +#define CATCH3_PROPERTY_ASSUME( ... ) \ + do { \ + if ( !static_cast<bool>( __VA_ARGS__ ) ) { \ + if ( auto *catch3Record = \ + ::Progmasoft::Catch3::Detail::CurrentTrialRecord() ) \ + catch3Record->Discarded = true; \ + return; \ + } \ + } while ( false ) diff --git a/src/Progmasoft/Catch3/Strategy/Engine.hpp b/src/Progmasoft/Catch3/Strategy/Engine.hpp new file mode 100644 index 00000000..fbe8d340 --- /dev/null +++ b/src/Progmasoft/Catch3/Strategy/Engine.hpp @@ -0,0 +1,85 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Property.hpp> +#include <Progmasoft/Catch3/Strategy/ChoiceSource.hpp> +#include <Progmasoft/Catch3/Strategy/Context.hpp> +#include <cstddef> +#include <cstdint> +#include <functional> +#include <optional> +#include <span> +#include <string> +#include <string_view> +#include <vector> + +namespace Progmasoft::Catch3::Detail +{ + + enum class TrialVerdict : std::uint8_t + { + kPassed, + kFailed, + /// The body called CATCH3_PROPERTY_ASSUME with a false condition. + kDiscarded, + /// The strategies described no value, for example a filter that + /// found nothing. + kInvalid, + /// A strategy threw; the property cannot be judged. + kErrored, + }; + + struct TrialOutcome final + { + TrialVerdict Verdict = TrialVerdict::kPassed; + std::string Reason; + TrialRecord Record; + /// The input, rendered; only filled when the engine asks for it. + std::string Description; + }; + + /// Draws the input of one trial from the source and runs the body on + /// it. describe asks for TrialOutcome::Description. + using TrialRunner = std::function<TrialOutcome(ChoiceSource &, bool describe)>; + + /// Runs a strategy-based property: the regular trials, the targeted + /// search, the coverage requirements, the example database and the + /// shrinking of a failure. Independent of the value types, so that it + /// is compiled once. + [[nodiscard]] PropertyResult + RunPropertyEngine(const PropertyOptions &options, const TrialRunner &runner); + + /// A replay token: the size of the trial and its choices. + [[nodiscard]] std::string + EncodeChoices(std::size_t size, std::span<const std::uint64_t> choices); + + struct DecodedChoices final + { + std::size_t Size = 0; + std::vector<std::uint64_t> Choices; + }; + + [[nodiscard]] std::optional<DecodedChoices> + DecodeChoices(std::string_view token); + + /// Minimizes a failing choice sequence: deletes and zeroes blocks of + /// choices, lowers single choices by binary search, sorts neighbours + /// and moves value between neighbours. stillFails replays a candidate + /// and returns the choices the replay consumed when it fails too. + /// Returns the simplest failing sequence found within budget replays; + /// calls counts the replays made. + [[nodiscard]] std::vector<std::uint64_t> + ShrinkChoices( + std::vector<std::uint64_t> failing, + std::size_t budget, + const std::function<std::optional<std::vector<std::uint64_t>>( + const std::vector<std::uint64_t> &)> &stillFails, + std::size_t &calls); + + /// Shortlex order: shorter first, then lexicographically smaller. + [[nodiscard]] bool + IsSimpler(std::span<const std::uint64_t> candidate, + std::span<const std::uint64_t> current) noexcept; + +} // namespace Progmasoft::Catch3::Detail diff --git a/src/Progmasoft/Catch3/Strategy/Numeric.hpp b/src/Progmasoft/Catch3/Strategy/Numeric.hpp new file mode 100644 index 00000000..8f58b06d --- /dev/null +++ b/src/Progmasoft/Catch3/Strategy/Numeric.hpp @@ -0,0 +1,224 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Strategy/Strategy.hpp> +#include <array> +#include <bit> +#include <cmath> +#include <concepts> +#include <cstdint> +#include <limits> +#include <optional> +#include <type_traits> + +namespace Progmasoft::Catch3 +{ + + /// The range of Gen::Floats, and what it may produce besides finite + /// values inside the range. + struct FloatOptions final + { + std::optional<double> Minimum = std::nullopt; + std::optional<double> Maximum = std::nullopt; + bool AllowNaN = false; + bool AllowInfinity = false; + bool AllowNegativeZero = true; + }; + +} // namespace Progmasoft::Catch3 + +namespace Progmasoft::Catch3::Gen +{ + + /// Integers in [minimum, maximum]. They shrink towards the value of the + /// range closest to zero; generation favours small magnitudes, zero and + /// the bounds, scaled by the trial size. + template<std::integral IntegerType> + requires(!std::same_as<IntegerType, bool>) + [[nodiscard]] Strategy<IntegerType> + Integers(IntegerType minimum = std::numeric_limits<IntegerType>::lowest(), + IntegerType maximum = std::numeric_limits<IntegerType>::max()) + { + static_assert(sizeof(IntegerType) <= sizeof(std::uint64_t)); + if (minimum > maximum) + { + Detail::RejectInvalidArgument( + "Integers requires minimum <= maximum"); + } + return Strategy<IntegerType>( + [minimum, maximum]( + ChoiceSource &source) -> std::optional<IntegerType> { + // Magnitudes are computed in 64 bits; the casts are exact + // because every magnitude lies inside the requested range. + if (minimum >= 0) + { + const auto width = static_cast<std::uint64_t>(maximum) + - static_cast<std::uint64_t>(minimum); + return static_cast<IntegerType>( + static_cast<std::uint64_t>(minimum) + + source.DrawBiased(width)); + } + if (maximum <= 0) + { + // Two's complement arithmetic in 64 bits: maximum minus + // minimum may not fit a signed 64-bit integer. + const auto width = static_cast<std::uint64_t>( + static_cast<std::int64_t>(maximum)) + - static_cast<std::uint64_t>( + static_cast<std::int64_t>(minimum)); + const auto value = static_cast<std::uint64_t>( + static_cast<std::int64_t>(maximum)) + - source.DrawBiased(width); + return static_cast<IntegerType>( + static_cast<std::int64_t>(value)); + } + // The range crosses zero: a sign, then a magnitude. Choice 0 + // is non-negative, so values shrink to 0, never to minimum. + if (source.Draw(1) == 0) + { + return static_cast<IntegerType>(source.DrawBiased( + static_cast<std::uint64_t>(maximum))); + } + // -(minimum + 1) cannot overflow, and the magnitude of -1 is + // choice 0. + const auto negativeWidth = static_cast<std::uint64_t>( + -(static_cast<std::int64_t>(minimum) + 1)); + const auto magnitude = source.DrawBiased(negativeWidth); + return static_cast<IntegerType>( + -1 - static_cast<std::int64_t>(magnitude)); + }); + } + + /// true or false; shrinks towards false. + [[nodiscard]] inline Strategy<bool> + Booleans() + { + return Strategy<bool>( + [](ChoiceSource &source) -> std::optional<bool> { + return source.Draw(1) == 1; + }); + } + + /// Floating-point values. They shrink towards zero, or the bound + /// closest to it, then towards integral values and short fractions. + /// Generation mixes integral values, short binary fractions, uniform + /// values of the range, arbitrary bit patterns, and edge values such as + /// the smallest subnormal and the largest finite value. + template<std::floating_point FloatType> + [[nodiscard]] Strategy<FloatType> + Floats(FloatOptions options = {}) + { + constexpr double kLargest = std::numeric_limits<FloatType>::max(); + const double minimum = options.Minimum.value_or(-kLargest); + const double maximum = options.Maximum.value_or(kLargest); + if (!(minimum <= maximum) || std::isnan(minimum) || std::isnan(maximum)) + { + Detail::RejectInvalidArgument( + "Floats requires minimum <= maximum, neither NaN"); + } + return Strategy<FloatType>( + [options, minimum, maximum]( + ChoiceSource &source) -> std::optional<FloatType> { + const auto inRange = [&](double value) { + return value >= minimum && value <= maximum; + }; + const double anchor + = inRange(0.0) ? 0.0 + : (std::abs(minimum) < std::abs(maximum) + ? minimum + : maximum); + const auto finish = [&](double value) -> FloatType { + if (!inRange(value)) + { + value = anchor; + } + return static_cast<FloatType>(value); + }; + // The kind is the first choice: 0 is the anchor itself, and + // each later kind is less simple than the one before. + const std::uint64_t kinds = 5U + + (options.AllowInfinity ? 1U : 0U) + + (options.AllowNaN ? 1U : 0U); + const std::uint64_t kind = source.Draw(kinds); + const auto sign = [&]() { + return source.Draw(1) == 0 ? 1.0 : -1.0; + }; + switch (kind) + { + case 0: + return finish(anchor); + case 1: + // Integral values near zero. + return finish(sign() + * static_cast<double>( + source.DrawBiased(1ULL << 53U))); + case 2: + { + // Short binary fractions such as 2.75. + const double whole = static_cast<double>( + source.DrawBiased(1ULL << 20U)); + const double fraction + = static_cast<double>(source.Draw(255)) + / 256.0; + return finish(sign() * (whole + fraction)); + } + case 3: + { + // Uniform inside the range; finite ranges only. + const double unit + = static_cast<double>(source.Draw( + (1ULL << 53U) - 1U)) + * 0x1.0p-53; + const double value + = minimum * (1.0 - unit) + maximum * unit; + return finish(std::isfinite(value) ? value : anchor); + } + case 4: + { + static constexpr std::array<double, 9> kEdges = { + -0.0, + std::numeric_limits<double>::denorm_min(), + std::numeric_limits<double>::min(), + std::numeric_limits<double>::epsilon(), + 1.0, + 0.5, + static_cast<double>( + std::numeric_limits<FloatType>::max()), + static_cast<double>( + std::numeric_limits<FloatType>::lowest()), + static_cast<double>( + std::numeric_limits<FloatType>::min()), + }; + const double edge = kEdges[static_cast<std::size_t>( + source.Draw(kEdges.size() - 1U))]; + const double signedEdge = sign() * edge; + if (!options.AllowNegativeZero && signedEdge == 0.0 + && std::signbit(signedEdge)) + { + return finish(0.0); + } + return finish(signedEdge); + } + case 5: + { + // Any bit pattern; non-finite patterns fall back. + const auto bits = source.Draw(UINT64_MAX); + const double value = std::bit_cast<double>(bits); + return finish(std::isfinite(value) ? value : anchor); + } + default: + break; + } + const bool wantsInfinity + = options.AllowInfinity && kind == 6U; + if (wantsInfinity) + { + return static_cast<FloatType>( + sign() * std::numeric_limits<double>::infinity()); + } + return std::numeric_limits<FloatType>::quiet_NaN(); + }); + } + +} // namespace Progmasoft::Catch3::Gen diff --git a/src/Progmasoft/Catch3/Strategy/Strategy.hpp b/src/Progmasoft/Catch3/Strategy/Strategy.hpp new file mode 100644 index 00000000..5eb06d6f --- /dev/null +++ b/src/Progmasoft/Catch3/Strategy/Strategy.hpp @@ -0,0 +1,303 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#pragma once + +#include <Progmasoft/Catch3/Detail/InvalidArgument.hpp> +#include <Progmasoft/Catch3/Strategy/ChoiceSource.hpp> +#include <algorithm> +#include <concepts> +#include <cstddef> +#include <cstdint> +#include <functional> +#include <initializer_list> +#include <memory> +#include <optional> +#include <type_traits> +#include <utility> +#include <vector> + +namespace Progmasoft::Catch3 +{ + + /// Describes how to build values of one type from a ChoiceSource. + /// + /// A strategy draws choices and returns a value, or no value when the + /// choices do not describe one: a filter that found nothing, or a + /// source that ran out of room. Shrinking needs no code here; see + /// ChoiceSource. Strategies are cheap to copy and immutable. + template<typename ValueType> + class Strategy final + { + public: + using Value = ValueType; + using Function + = std::function<std::optional<ValueType>(ChoiceSource &)>; + + explicit Strategy(Function draw) + : draw_(std::make_shared<const Function>(std::move(draw))) + { + if (!*draw_) + { + Detail::RejectInvalidArgument("a strategy requires a function"); + } + } + + /// Builds one value, or nothing when the choices describe none. + [[nodiscard]] std::optional<ValueType> + Draw(ChoiceSource &source) const + { + if (source.Exhausted()) + { + return std::nullopt; + } + return (*draw_)(source); + } + + /// Transforms every value. The result shrinks with this strategy. + template<typename Transform> + [[nodiscard]] auto + Map(Transform transform) const + { + using Result = std::remove_cvref_t< + std::invoke_result_t<Transform &, ValueType>>; + Strategy self = *this; + return Strategy<Result>( + [self, transform]( + ChoiceSource &source) mutable -> std::optional<Result> { + auto value = self.Draw(source); + if (!value.has_value()) + { + return std::nullopt; + } + return std::invoke(transform, std::move(*value)); + }); + } + + /// Keeps only values that satisfy the predicate. Each value costs + /// one attempt; after maximumAttempts rejected values the strategy + /// gives up for this trial, which the runner counts as a discarded + /// trial and reports when it happens too often. + template<typename Predicate> + [[nodiscard]] Strategy + Filter(Predicate predicate, std::size_t maximumAttempts = 100) const + { + if (maximumAttempts == 0) + { + Detail::RejectInvalidArgument( + "Filter requires at least one attempt"); + } + Strategy self = *this; + return Strategy( + [self, predicate, maximumAttempts]( + ChoiceSource &source) mutable -> std::optional<ValueType> { + for (std::size_t attempt = 0; attempt < maximumAttempts; + ++attempt) + { + auto value = self.Draw(source); + if (!value.has_value()) + { + return std::nullopt; + } + if (std::invoke(predicate, std::as_const(*value))) + { + return value; + } + } + return std::nullopt; + }); + } + + /// Builds a value, then a second strategy from it, and draws from + /// that. Unlike value-based shrinking, both levels shrink: the + /// engine shrinks the shared choice sequence. + template<typename Bind> + [[nodiscard]] auto + FlatMap(Bind bind) const + { + using Next + = std::remove_cvref_t<std::invoke_result_t<Bind &, ValueType>>; + using Result = typename Next::Value; + Strategy self = *this; + return Strategy<Result>( + [self, bind]( + ChoiceSource &source) mutable -> std::optional<Result> { + auto value = self.Draw(source); + if (!value.has_value()) + { + return std::nullopt; + } + return std::invoke(bind, std::move(*value)).Draw(source); + }); + } + + private: + std::shared_ptr<const Function> draw_; + }; + + /// The value type of a strategy. + template<typename StrategyType> + using StrategyValue = typename std::remove_cvref_t<StrategyType>::Value; + + namespace Gen + { + + /// Always the same value; draws no choice. + template<typename ValueType> + [[nodiscard]] Strategy<std::remove_cvref_t<ValueType>> + Just(ValueType value) + { + using Result = std::remove_cvref_t<ValueType>; + return Strategy<Result>( + [value = Result(std::move(value))](ChoiceSource &) + -> std::optional<Result> { return value; }); + } + + /// One of the strategies, chosen uniformly. Shrinks towards the + /// first one, so list the simplest first. + template<typename ValueType> + [[nodiscard]] Strategy<ValueType> + OneOf(std::vector<Strategy<ValueType>> strategies) + { + if (strategies.empty()) + { + Detail::RejectInvalidArgument( + "OneOf requires at least one strategy"); + } + return Strategy<ValueType>( + [strategies = std::move(strategies)]( + ChoiceSource &source) -> std::optional<ValueType> { + const auto index = source.Draw(strategies.size() - 1U); + return strategies[static_cast<std::size_t>(index)].Draw( + source); + }); + } + + template<typename First, typename... Rest> + requires(std::same_as<StrategyValue<First>, StrategyValue<Rest>> + && ...) + [[nodiscard]] Strategy<StrategyValue<First>> + OneOf(First first, Rest... rest) + { + return OneOf(std::vector<Strategy<StrategyValue<First>>>{ + std::move(first), std::move(rest)... }); + } + + /// A strategy chosen with probability proportional to its weight. + /// Shrinks towards the first entry, whatever its weight. + template<typename ValueType> + [[nodiscard]] Strategy<ValueType> + Frequency( + std::vector<std::pair<std::uint32_t, Strategy<ValueType>>> entries) + { + std::uint64_t total = 0; + for (const auto &entry : entries) + { + total += entry.first; + } + if (entries.empty() || total == 0) + { + Detail::RejectInvalidArgument( + "Frequency requires a positive total weight"); + } + return Strategy<ValueType>( + [entries = std::move(entries), total]( + ChoiceSource &source) -> std::optional<ValueType> { + // The choice is an index into the cumulative weights, so + // choice 0 always selects the first entry. + std::uint64_t point = source.Draw(total - 1U); + for (const auto &entry : entries) + { + if (point < entry.first) + { + return entry.second.Draw(source); + } + point -= entry.first; + } + return entries.back().second.Draw(source); + }); + } + + /// A uniformly chosen element of a fixed list; shrinks towards the + /// first element. + template<typename ValueType> + [[nodiscard]] Strategy<ValueType> + SampledFrom(std::vector<ValueType> values) + { + if (values.empty()) + { + Detail::RejectInvalidArgument( + "SampledFrom requires at least one value"); + } + return Strategy<ValueType>( + [values = std::move(values)]( + ChoiceSource &source) -> std::optional<ValueType> { + return values[static_cast<std::size_t>( + source.Draw(values.size() - 1U))]; + }); + } + + template<typename ValueType> + [[nodiscard]] Strategy<ValueType> + SampledFrom(std::initializer_list<ValueType> values) + { + return SampledFrom(std::vector<ValueType>(values)); + } + + /// Defers building a strategy until it is drawn from, so that a + /// strategy can refer to itself. + template<typename Factory> + [[nodiscard]] auto + Lazy(Factory factory) + { + using Result = std::remove_cvref_t<std::invoke_result_t<Factory &>>; + using ValueType = typename Result::Value; + return Strategy<ValueType>( + [factory]( + ChoiceSource &source) mutable -> std::optional<ValueType> { + return std::invoke(factory).Draw(source); + }); + } + + /// A recursive structure: base builds the leaves, and extend builds a + /// node from a strategy for its children. The depth is bounded by + /// maximumDepth and grows with the size, so early trials are flat. + /// Recursion shrinks towards a leaf, since choosing a leaf is the + /// choice 0. + template<typename ValueType, typename Extend> + [[nodiscard]] Strategy<ValueType> + Recursive(Strategy<ValueType> base, + Extend extend, + std::size_t maximumDepth = 8) + { + struct Node final + { + Strategy<ValueType> base; + Extend extend; + std::size_t maximumDepth; + + std::optional<ValueType> + Draw(ChoiceSource &source, std::size_t depth) const + { + const std::size_t allowed = std::min<std::size_t>( + maximumDepth, 1U + source.Size() / 12U); + if (depth >= allowed || !source.DrawBoolean(0.5)) + { + return base.Draw(source); + } + const Node *self = this; + const Strategy<ValueType> children( + [self, depth](ChoiceSource &inner) { + return self->Draw(inner, depth + 1U); + }); + return std::invoke(extend, children).Draw(source); + } + }; + auto node = std::make_shared<const Node>( + Node{ std::move(base), std::move(extend), maximumDepth }); + return Strategy<ValueType>( + [node](ChoiceSource &source) { return node->Draw(source, 0); }); + } + + } // namespace Gen + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/ChoiceShrinker.cpp b/src/Progmasoft/ChoiceShrinker.cpp new file mode 100644 index 00000000..c1aac19c --- /dev/null +++ b/src/Progmasoft/ChoiceShrinker.cpp @@ -0,0 +1,384 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <Progmasoft/Catch3/Strategy/Engine.hpp> + +#include <algorithm> +#include <array> +#include <charconv> +#include <cstddef> +#include <cstdint> +#include <string> +#include <utility> + +namespace Progmasoft::Catch3::Detail +{ + + bool + IsSimpler(std::span<const std::uint64_t> candidate, + std::span<const std::uint64_t> current) noexcept + { + if (candidate.size() != current.size()) + { + return candidate.size() < current.size(); + } + return std::lexicographical_compare( + candidate.begin(), candidate.end(), current.begin(), current.end()); + } + + namespace + { + + constexpr std::string_view kTokenPrefix = "c3."; + + /// Replays candidates against the current best and keeps the + /// simplest failing one. Every replay counts against the budget. + class Shrinker final + { + public: + using StillFails + = std::function<std::optional<std::vector<std::uint64_t>>( + const std::vector<std::uint64_t> &)>; + + Shrinker(std::vector<std::uint64_t> failing, + std::size_t budget, + const StillFails &stillFails) + : current_(std::move(failing)) + , budget_(budget) + , stillFails_(stillFails) + {} + + [[nodiscard]] std::vector<std::uint64_t> + Run(std::size_t &calls) + { + bool improved = true; + while (improved && !Spent()) + { + improved = false; + improved |= DeleteBlocks(); + improved |= ZeroBlocks(); + improved |= MinimizeChoices(); + improved |= SortNeighbours(); + improved |= MoveBetweenNeighbours(); + } + calls = calls_; + return current_; + } + + private: + static constexpr std::array<std::size_t, 5> kBlockSizes + = { 16, 8, 4, 2, 1 }; + + [[nodiscard]] bool + Spent() const noexcept + { + return calls_ >= budget_; + } + + /// Replays a candidate; true when it fails and the sequence it + /// consumed is simpler than the current one, which it replaces. + bool + Try(const std::vector<std::uint64_t> &candidate) + { + if (Spent() || !IsSimpler(candidate, current_)) + { + return false; + } + ++calls_; + auto consumed = stillFails_(candidate); + if (!consumed.has_value() || !IsSimpler(*consumed, current_)) + { + return false; + } + current_ = std::move(*consumed); + return true; + } + + bool + DeleteBlocks() + { + bool improved = false; + for (const std::size_t block : kBlockSizes) + { + // From the end, so that a deletion does not move the + // blocks still to be tried. + for (std::size_t index = current_.size(); + index-- > 0 && !Spent();) + { + if (index + block > current_.size()) + { + continue; + } + auto candidate = current_; + candidate.erase( + candidate.begin() + + static_cast<std::ptrdiff_t>(index), + candidate.begin() + + static_cast<std::ptrdiff_t>(index + block)); + improved |= Try(candidate); + } + } + return improved; + } + + bool + ZeroBlocks() + { + bool improved = false; + for (const std::size_t block : kBlockSizes) + { + for (std::size_t index = 0; + index + block <= current_.size() && !Spent(); + ++index) + { + const auto first = current_.begin() + + static_cast<std::ptrdiff_t>(index); + if (std::all_of(first, + first + + static_cast<std::ptrdiff_t>(block), + [](std::uint64_t value) { + return value == 0; + })) + { + continue; + } + auto candidate = current_; + std::fill_n(candidate.begin() + + static_cast<std::ptrdiff_t>(index), + block, + 0U); + improved |= Try(candidate); + } + } + return improved; + } + + /// Lowers each choice as far as the failure allows, assuming + /// that a smaller choice is usually still failing when a larger + /// one is: a binary search between 0 and the current value. + bool + MinimizeChoices() + { + bool improved = false; + for (std::size_t index = 0; + index < current_.size() && !Spent(); + ++index) + { + std::uint64_t high = current_[index]; + if (high == 0) + { + continue; + } + if (TryValue(index, 0)) + { + improved = true; + continue; + } + std::uint64_t low = 0; + while (low + 1U < high && !Spent() + && index < current_.size()) + { + const std::uint64_t middle = low + (high - low) / 2U; + if (TryValue(index, middle)) + { + high = middle; + improved = true; + } + // Binary search assumes that failures are monotone + // in the choice; a filter or a parity condition + // breaks that, so the neighbour gets a chance too: + // with only even values passing a filter, an odd + // middle would otherwise stop the search. + else if (middle + 1U < high + && TryValue(index, middle + 1U)) + { + high = middle + 1U; + improved = true; + } + else + { + low = middle + 1U; + } + } + // A few single steps below the result, once: 104 -> 102 + // when only even values pass. The next round of passes + // repeats the search from there. + for (std::uint64_t step = 1; step <= 8 && !Spent() + && index < current_.size() + && current_[index] >= step; + ++step) + { + if (TryValue(index, current_[index] - step)) + { + improved = true; + break; + } + } + } + return improved; + } + + bool + TryValue(std::size_t index, std::uint64_t value) + { + if (index >= current_.size() || current_[index] <= value) + { + return false; + } + auto candidate = current_; + candidate[index] = value; + return Try(candidate); + } + + bool + SortNeighbours() + { + bool improved = false; + for (std::size_t index = 0; + index + 1U < current_.size() && !Spent(); + ++index) + { + if (current_[index] <= current_[index + 1U]) + { + continue; + } + auto candidate = current_; + std::swap(candidate[index], candidate[index + 1U]); + improved |= Try(candidate); + } + return improved; + } + + /// Moves one unit from a choice to one of the next few: the + /// sequence gets lexicographically smaller while a sum such as + /// x + y > 10 keeps failing. + bool + MoveBetweenNeighbours() + { + bool improved = false; + for (std::size_t index = 0; + index < current_.size() && !Spent(); + ++index) + { + for (std::size_t offset = 1; + offset <= 4 && index + offset < current_.size() + && !Spent(); + ++offset) + { + if (current_[index] == 0 + || current_[index + offset] == UINT64_MAX) + { + break; + } + auto candidate = current_; + --candidate[index]; + ++candidate[index + offset]; + improved |= Try(candidate); + } + } + return improved; + } + + std::vector<std::uint64_t> current_; + std::size_t budget_; + std::size_t calls_ = 0; + const StillFails &stillFails_; + }; + + } // namespace + + std::vector<std::uint64_t> + ShrinkChoices( + std::vector<std::uint64_t> failing, + std::size_t budget, + const std::function<std::optional<std::vector<std::uint64_t>>( + const std::vector<std::uint64_t> &)> &stillFails, + std::size_t &calls) + { + Shrinker shrinker(std::move(failing), budget, stillFails); + return shrinker.Run(calls); + } + + std::string + EncodeChoices(std::size_t size, std::span<const std::uint64_t> choices) + { + // c3.<size>.<choice>-<choice>-... with each number in base 36: short + // enough to paste from a CI log, and only characters that survive + // shells and C++ string literals unescaped. + std::string token(kTokenPrefix); + std::array<char, 24> buffer{}; + const auto append = [&](std::uint64_t value) { + const auto end = std::to_chars( + buffer.data(), buffer.data() + buffer.size(), + value, 36) + .ptr; + token.append(buffer.data(), end); + }; + append(size); + token += '.'; + for (std::size_t index = 0; index < choices.size(); ++index) + { + if (index != 0) + { + token += '-'; + } + append(choices[index]); + } + return token; + } + + std::optional<DecodedChoices> + DecodeChoices(std::string_view token) + { + if (!token.starts_with(kTokenPrefix)) + { + return std::nullopt; + } + token.remove_prefix(kTokenPrefix.size()); + const auto parse = [](std::string_view text) + -> std::optional<std::uint64_t> { + std::uint64_t value = 0; + const auto result = std::from_chars( + text.data(), text.data() + text.size(), value, 36); + if (text.empty() || result.ec != std::errc() + || result.ptr != text.data() + text.size()) + { + return std::nullopt; + } + return value; + }; + const auto dot = token.find('.'); + if (dot == std::string_view::npos) + { + return std::nullopt; + } + const auto size = parse(token.substr(0, dot)); + if (!size.has_value() || *size > 100U) + { + return std::nullopt; + } + DecodedChoices decoded; + decoded.Size = static_cast<std::size_t>(*size); + std::string_view rest = token.substr(dot + 1U); + while (!rest.empty()) + { + const auto dash = rest.find('-'); + const auto value = parse(rest.substr(0, dash)); + if (!value.has_value()) + { + return std::nullopt; + } + decoded.Choices.push_back(*value); + if (dash == std::string_view::npos) + { + break; + } + rest.remove_prefix(dash + 1U); + if (rest.empty()) + { + return std::nullopt; + } + } + return decoded; + } + +} // namespace Progmasoft::Catch3::Detail diff --git a/src/Progmasoft/Context.cpp b/src/Progmasoft/Context.cpp new file mode 100644 index 00000000..81b02b7d --- /dev/null +++ b/src/Progmasoft/Context.cpp @@ -0,0 +1,122 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <Progmasoft/Catch3/Strategy/Context.hpp> + +#include <string> +#include <utility> + +namespace Progmasoft::Catch3 +{ + + namespace + { + + thread_local TrialRecord *currentRecord = nullptr; + thread_local std::string defaultPropertyName; + + } // namespace + + namespace Detail + { + + TrialRecord * + CurrentTrialRecord() noexcept + { + return currentRecord; + } + + TrialRecordScope::TrialRecordScope(TrialRecord &record) noexcept + : previous_(currentRecord) + { + currentRecord = &record; + } + + TrialRecordScope::~TrialRecordScope() + { + currentRecord = previous_; + } + + void + RecordPropertyFailure(std::string_view expression, + std::string_view file, + std::size_t line) + { + if (currentRecord == nullptr) + { + return; + } + // Only the first failure of a trial counts: it is the one the + // body stopped at. + if (!currentRecord->Failure.has_value()) + { + currentRecord->Failure = "CATCH3_PROPERTY_ASSERT( " + + std::string(expression) + " ) at " + + std::string(file) + ":" + + std::to_string(line); + } + } + + void + RecordPropertyFailureMessage(std::string message) + { + if (currentRecord != nullptr && !currentRecord->Failure.has_value()) + { + currentRecord->Failure = std::move(message); + } + } + + std::string + DefaultPropertyName() + { + return defaultPropertyName; + } + + PropertyNameScope::PropertyNameScope(std::string name) + : previous_(std::exchange(defaultPropertyName, std::move(name))) + {} + + PropertyNameScope::~PropertyNameScope() + { + defaultPropertyName = std::move(previous_); + } + + } // namespace Detail + + void + Note(std::string text) + { + if (currentRecord != nullptr) + { + currentRecord->Notes.push_back(std::move(text)); + } + } + + void + Classify(bool condition, std::string label) + { + if (currentRecord != nullptr && condition) + { + currentRecord->Labels.push_back(std::move(label)); + } + } + + void + Cover(double requiredPercentage, bool condition, std::string label) + { + if (currentRecord != nullptr) + { + currentRecord->Covers.push_back( + { std::move(label), requiredPercentage, condition }); + } + } + + void + Target(double score, std::string label) + { + if (currentRecord != nullptr) + { + currentRecord->Targets.push_back({ std::move(label), score }); + } + } + +} // namespace Progmasoft::Catch3 diff --git a/src/Progmasoft/PropertyEngine.cpp b/src/Progmasoft/PropertyEngine.cpp new file mode 100644 index 00000000..c42b235f --- /dev/null +++ b/src/Progmasoft/PropertyEngine.cpp @@ -0,0 +1,553 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <Progmasoft/Catch3/Strategy/Engine.hpp> + +#include <algorithm> +#include <cstdlib> +#include <filesystem> +#include <fstream> +#include <map> +#include <sstream> +#include <string> +#include <system_error> +#include <utility> + +namespace Progmasoft::Catch3::Detail +{ + + namespace + { + + /// Stores failing examples so that the next run replays them first. + /// + /// One file per property, named by a hash of the property name, with + /// one replay token per line. The database never throws: an + /// unreadable or unwritable directory simply behaves as an empty + /// one, since a test must not fail because of its cache. + class ExampleDatabase final + { + public: + explicit ExampleDatabase(const PropertyOptions &options) + { + std::string directory = options.DatabasePath; + if (directory.empty()) + { + // The value only names a directory; it is never logged. + directory = ReadEnvironment("CATCH3_PROPERTY_DATABASE"); + } + if (directory.empty()) + { + return; + } + std::string name = options.Name; + if (name.empty()) + { + name = DefaultPropertyName(); + } + if (name.empty()) + { + name = "unnamed property"; + } + file_ = std::filesystem::path(directory) / (Hash(name) + ".txt"); + } + + [[nodiscard]] std::vector<std::string> + Load() const + { + std::vector<std::string> tokens; + if (file_.empty()) + { + return tokens; + } + std::ifstream input(file_); + std::string line; + while (std::getline(input, line)) + { + if (!line.empty() && line.back() == '\r') + { + line.pop_back(); + } + if (!line.empty()) + { + tokens.push_back(line); + } + } + return tokens; + } + + void + Save(const std::vector<std::string> &tokens) const + { + if (file_.empty()) + { + return; + } + std::error_code error; + if (tokens.empty()) + { + std::filesystem::remove(file_, error); + return; + } + std::filesystem::create_directories(file_.parent_path(), error); + std::ofstream output(file_, std::ios::trunc); + for (const auto &token : tokens) + { + output << token << '\n'; + } + } + + private: + [[nodiscard]] static std::string + ReadEnvironment(const char *name) + { +#if defined(_WIN32) + // The CRT deprecates getenv in favour of the copying form. + char *value = nullptr; + std::size_t length = 0; + if (_dupenv_s(&value, &length, name) != 0 || value == nullptr) + { + return {}; + } + std::string result(value); + std::free(value); + return result; +#else + const char *value = std::getenv(name); + return value == nullptr ? std::string() : std::string(value); +#endif + } + + [[nodiscard]] static std::string + Hash(std::string_view text) + { + // FNV-1a 64: stable across platforms and library versions. + std::uint64_t hash = 0xCBF29CE484222325ULL; + for (const char character : text) + { + hash ^= static_cast<unsigned char>(character); + hash *= 0x100000001B3ULL; + } + std::ostringstream name; + name << std::hex << hash; + return name.str(); + } + + std::filesystem::path file_; + }; + + struct LabelCount final + { + std::size_t Count = 0; + double Required = 0.0; + }; + + class Engine final + { + public: + Engine(const PropertyOptions &options, const TrialRunner &runner) + : options_(options) + , runner_(runner) + , database_(options) + {} + + PropertyResult + Run() + { + if (options_.Replay.has_value()) + { + return RunReplay(*options_.Replay); + } + if (options_.Trials == 0) + { + return Invalid("Trials must be greater than zero"); + } + if (auto stored = ReplayDatabase()) + { + return std::move(*stored); + } + if (auto failure = RunTrials()) + { + return std::move(*failure); + } + if (auto failure = SearchTargets()) + { + return std::move(*failure); + } + return Finish(); + } + + private: + struct Execution final + { + TrialOutcome Outcome; + std::vector<std::uint64_t> Choices; + }; + + Execution + Execute(ChoiceSource source, bool describe) + { + Execution execution{ runner_(source, describe), {} }; + execution.Choices = source.Choices(); + return execution; + } + + Execution + Replay(const std::vector<std::uint64_t> &choices, + std::size_t size, + bool describe) + { + return Execute(ChoiceSource(std::span(choices), size), describe); + } + + static PropertyResult + Invalid(std::string message) + { + PropertyResult result; + result.Status = PropertyStatus::kInvalidConfiguration; + result.Error = std::move(message); + return result; + } + + PropertyResult + RunReplay(const std::string &token) + { + const auto decoded = DecodeChoices(token); + if (!decoded.has_value()) + { + return Invalid("PropertyOptions::Replay is not a replay " + "token: " + + token); + } + auto execution = Replay(decoded->Choices, decoded->Size, true); + trialsRun_ = 1; + if (execution.Outcome.Verdict == TrialVerdict::kFailed) + { + return Report(std::move(execution), + decoded->Size, + 0, + 1, + 0); + } + PropertyResult result; + result.TrialsRun = 1; + if (execution.Outcome.Verdict == TrialVerdict::kErrored) + { + result.Status = PropertyStatus::kErrored; + result.Error = execution.Outcome.Reason; + } + return result; + } + + /// Replays the stored examples of this property. One that still + /// fails is shrunk further and reported; the others are dropped. + std::optional<PropertyResult> + ReplayDatabase() + { + auto tokens = database_.Load(); + std::vector<std::string> kept; + for (const auto &token : tokens) + { + const auto decoded = DecodeChoices(token); + if (!decoded.has_value()) + { + continue; + } + auto execution + = Replay(decoded->Choices, decoded->Size, false); + if (execution.Outcome.Verdict == TrialVerdict::kFailed) + { + ++trialsRun_; + return ShrinkAndReport(std::move(execution), + decoded->Size, + 0, + trialsRun_); + } + } + if (!tokens.empty()) + { + database_.Save(kept); + } + return std::nullopt; + } + + std::optional<PropertyResult> + RunTrials() + { + const std::size_t allowedDiscards + = options_.Trials * options_.MaxDiscardRatio; + std::size_t attempt = 0; + while (validTrials_ < options_.Trials) + { + const std::uint64_t seed + = Random::DeriveSeed(options_.Seed, attempt); + // The size grows over the first half of the trials and + // stays at its maximum for the second half. + const std::size_t size = std::min<std::size_t>( + 100, attempt * 200U / options_.Trials); + ++attempt; + auto execution = Execute(ChoiceSource(seed, size), false); + ++trialsRun_; + switch (execution.Outcome.Verdict) + { + case TrialVerdict::kPassed: + Count(execution); + ++validTrials_; + break; + case TrialVerdict::kFailed: + return ShrinkAndReport( + std::move(execution), size, seed, trialsRun_); + case TrialVerdict::kErrored: + { + PropertyResult result; + result.Status = PropertyStatus::kErrored; + result.TrialsRun = trialsRun_; + result.Error = execution.Outcome.Reason; + return result; + } + case TrialVerdict::kDiscarded: + case TrialVerdict::kInvalid: + if (++discarded_ > allowedDiscards) + { + PropertyResult result; + result.Status = PropertyStatus::kGaveUp; + result.TrialsRun = trialsRun_; + result.Discarded = discarded_; + result.Error + = std::to_string(discarded_) + + " trials were discarded, more than " + "MaxDiscardRatio allows, with only " + + std::to_string(validTrials_) + + " valid; loosen the filter or the " + "assumption, or build the input so " + "that it is valid by construction"; + return result; + } + break; + } + } + return std::nullopt; + } + + void + Count(const Execution &execution) + { + const auto &record = execution.Outcome.Record; + for (const auto &label : record.Labels) + { + ++labels_[label].Count; + } + for (const auto &cover : record.Covers) + { + auto &entry = covers_[cover.Label]; + entry.Required = std::max(entry.Required, cover.Required); + if (cover.Held) + { + ++entry.Count; + } + } + for (const auto &target : record.Targets) + { + auto found = best_.find(target.Label); + if (found == best_.end() || target.Value > found->second.first) + { + best_[target.Label] = { target.Value, execution.Choices }; + } + } + } + + /// Hill climbing on the choices of the best input of each + /// target: one mutated choice per step, kept when the score + /// rises. Inputs reached this way are regular trials, so a + /// failure among them is shrunk and reported as usual. + std::optional<PropertyResult> + SearchTargets() + { + if (best_.empty() || options_.TargetSteps == 0) + { + return std::nullopt; + } + Random random(options_.Seed ^ 0x7A26E7ULL); + std::vector<std::string> labels; + for (const auto &entry : best_) + { + labels.push_back(entry.first); + } + for (std::size_t step = 0; step < options_.TargetSteps; ++step) + { + const auto &label = labels[step % labels.size()]; + auto &[bestScore, bestChoices] = best_[label]; + if (bestChoices.empty()) + { + continue; + } + auto candidate = bestChoices; + auto &choice = candidate[static_cast<std::size_t>( + random.Below(candidate.size()))]; + switch (random.Below(6)) + { + case 0: + ++choice; + break; + case 1: + choice = choice == 0 ? 0 : choice - 1U; + break; + case 2: + choice = choice * 2U + 1U; + break; + case 3: + choice /= 2U; + break; + case 4: + choice = random.Below(choice + 2U); + break; + default: + choice = random.Next(); + break; + } + auto execution = Replay(candidate, 100, false); + ++trialsRun_; + if (execution.Outcome.Verdict == TrialVerdict::kFailed) + { + return ShrinkAndReport( + std::move(execution), 100, 0, trialsRun_); + } + if (execution.Outcome.Verdict != TrialVerdict::kPassed) + { + continue; + } + for (const auto &target : execution.Outcome.Record.Targets) + { + if (target.Label == label && target.Value > bestScore) + { + bestScore = target.Value; + bestChoices = execution.Choices; + } + } + } + return std::nullopt; + } + + PropertyResult + ShrinkAndReport(Execution failing, + std::size_t size, + std::uint64_t trialSeed, + std::size_t trial) + { + std::size_t calls = 0; + const auto minimized = ShrinkChoices( + failing.Choices, + options_.MaxShrinkSteps, + [&](const std::vector<std::uint64_t> &candidate) + -> std::optional<std::vector<std::uint64_t>> { + auto execution = Replay(candidate, size, false); + if (execution.Outcome.Verdict != TrialVerdict::kFailed) + { + return std::nullopt; + } + return std::move(execution.Choices); + }, + calls); + auto minimal = Replay(minimized, size, true); + if (minimal.Outcome.Verdict != TrialVerdict::kFailed) + { + // A flaky body: the minimized input passed on replay. + // Report the original input, which did fail. + minimal = Replay(failing.Choices, size, true); + } + return Report(std::move(minimal), size, trialSeed, trial, calls); + } + + PropertyResult + Report(Execution failing, + std::size_t size, + std::uint64_t trialSeed, + std::size_t trial, + std::size_t calls) + { + PropertyFailure failure; + failure.RootSeed = options_.Seed; + failure.TrialSeed = trialSeed; + failure.Trial = trial; + failure.ShrinkSteps = calls; + failure.Counterexample = failing.Outcome.Description; + failure.Reason = failing.Outcome.Reason; + failure.Replay = EncodeChoices(size, failing.Choices); + failure.Notes = std::move(failing.Outcome.Record.Notes); + + auto stored = database_.Load(); + if (std::find(stored.begin(), stored.end(), failure.Replay) + == stored.end()) + { + stored.insert(stored.begin(), failure.Replay); + database_.Save(stored); + } + + PropertyResult result; + result.Status = PropertyStatus::kFailed; + result.TrialsRun = trialsRun_; + result.Discarded = discarded_; + result.Failure = std::move(failure); + return result; + } + + PropertyResult + Finish() + { + PropertyResult result; + result.Status = PropertyStatus::kPassed; + result.TrialsRun = trialsRun_; + result.Discarded = discarded_; + result.ShowStatistics = options_.ShowStatistics; + const auto percentage = [this](std::size_t count) { + return validTrials_ == 0 + ? 0.0 + : 100.0 * static_cast<double>(count) + / static_cast<double>(validTrials_); + }; + for (const auto &[label, entry] : labels_) + { + result.Statistics.push_back( + { label, entry.Count, percentage(entry.Count), 0.0 }); + } + std::string missing; + for (const auto &[label, entry] : covers_) + { + const double reached = percentage(entry.Count); + result.Statistics.push_back( + { label, entry.Count, reached, entry.Required }); + if (reached < entry.Required) + { + missing += (missing.empty() ? "" : ", ") + label; + } + } + if (!missing.empty()) + { + result.Status = PropertyStatus::kInsufficientCoverage; + result.Error = "too few trials covered " + missing; + } + return result; + } + + const PropertyOptions &options_; + const TrialRunner &runner_; + ExampleDatabase database_; + std::size_t trialsRun_ = 0; + std::size_t validTrials_ = 0; + std::size_t discarded_ = 0; + std::map<std::string, LabelCount> labels_; + std::map<std::string, LabelCount> covers_; + std::map<std::string, + std::pair<double, std::vector<std::uint64_t>>> + best_; + }; + + } // namespace + + PropertyResult + RunPropertyEngine(const PropertyOptions &options, const TrialRunner &runner) + { + Engine engine(options, runner); + return engine.Run(); + } + +} // namespace Progmasoft::Catch3::Detail diff --git a/src/Progmasoft/meson.build b/src/Progmasoft/meson.build index 1f9afbda..0c838140 100644 --- a/src/Progmasoft/meson.build +++ b/src/Progmasoft/meson.build @@ -5,7 +5,14 @@ # own C++14 baseline for existing users. progmasoft_catch3 = static_library( 'ProgmasoftCatch3', - ['Results.cpp', 'Snapshot.cpp', 'XmlWriter.cpp'], + [ + 'ChoiceShrinker.cpp', + 'Context.cpp', + 'PropertyEngine.cpp', + 'Results.cpp', + 'Snapshot.cpp', + 'XmlWriter.cpp', + ], include_directories: '..', dependencies: catch2_dep, override_options: ['cpp_std=c++20'], @@ -28,6 +35,15 @@ if get_option('install') install_headers('Catch3/Results.hpp', subdir: 'Progmasoft/Catch3') install_headers('Catch3/Snapshot.hpp', subdir: 'Progmasoft/Catch3') install_headers('Catch3/XmlWriter.hpp', subdir: 'Progmasoft/Catch3') + install_headers('Catch3/ForAll.hpp', subdir: 'Progmasoft/Catch3') + install_headers('Catch3/Relations.hpp', subdir: 'Progmasoft/Catch3') + install_headers('Catch3/Stateful.hpp', subdir: 'Progmasoft/Catch3') + install_headers('Catch3/Strategy/ChoiceSource.hpp', subdir: 'Progmasoft/Catch3/Strategy') + install_headers('Catch3/Strategy/Containers.hpp', subdir: 'Progmasoft/Catch3/Strategy') + install_headers('Catch3/Strategy/Context.hpp', subdir: 'Progmasoft/Catch3/Strategy') + install_headers('Catch3/Strategy/Engine.hpp', subdir: 'Progmasoft/Catch3/Strategy') + install_headers('Catch3/Strategy/Numeric.hpp', subdir: 'Progmasoft/Catch3/Strategy') + install_headers('Catch3/Strategy/Strategy.hpp', subdir: 'Progmasoft/Catch3/Strategy') install_headers('Catch3/Generators/Boolean.hpp', subdir: 'Progmasoft/Catch3/Generators') install_headers('Catch3/Generators/Integer.hpp', subdir: 'Progmasoft/Catch3/Generators') install_headers('Catch3/Generators/Element.hpp', subdir: 'Progmasoft/Catch3/Generators') diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 79660687..3d4db3b3 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -160,8 +160,11 @@ target_link_libraries(SelfTest PRIVATE Catch2WithMain) add_executable(ProgmasoftSelfTest Progmasoft/Generators.tests.cpp Progmasoft/Property.tests.cpp + Progmasoft/ForAll.tests.cpp Progmasoft/Results.tests.cpp Progmasoft/Snapshot.tests.cpp + Progmasoft/Stateful.tests.cpp + Progmasoft/Strategy.tests.cpp ) target_link_libraries(ProgmasoftSelfTest PRIVATE Progmasoft::Catch3 Catch2WithMain) target_compile_features(ProgmasoftSelfTest PRIVATE cxx_std_20) diff --git a/tests/Progmasoft/BUILD.bazel b/tests/Progmasoft/BUILD.bazel index d2eeca92..853d99d4 100644 --- a/tests/Progmasoft/BUILD.bazel +++ b/tests/Progmasoft/BUILD.bazel @@ -7,10 +7,13 @@ package(default_visibility = ["//visibility:public"]) cc_test( name = "property_tests", srcs = [ + "ForAll.tests.cpp", "Generators.tests.cpp", "Property.tests.cpp", "Results.tests.cpp", "Snapshot.tests.cpp", + "Stateful.tests.cpp", + "Strategy.tests.cpp", ], copts = select({ "@platforms//os:windows": ["/std:c++20"], diff --git a/tests/Progmasoft/ForAll.tests.cpp b/tests/Progmasoft/ForAll.tests.cpp new file mode 100644 index 00000000..b88477a6 --- /dev/null +++ b/tests/Progmasoft/ForAll.tests.cpp @@ -0,0 +1,314 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <Progmasoft/Catch3.hpp> + +#include <algorithm> +#include <cstdint> +#include <filesystem> +#include <numeric> +#include <set> +#include <stdexcept> +#include <string> +#include <vector> + +namespace { + + using Progmasoft::Catch3::ForAll; + using Progmasoft::Catch3::PropertyOptions; + using Progmasoft::Catch3::PropertyStatus; + using Progmasoft::Catch3::SizeRange; + namespace Gen = Progmasoft::Catch3::Gen; + + PropertyOptions Quick( std::size_t trials = 200 ) { + PropertyOptions options; + options.Trials = trials; + options.Seed = 0xCA7C3; + options.MaxShrinkSteps = 5000; + return options; + } + +} // namespace + +TEST_CASE( "ForAll passes and counts its trials", + "[progmasoft][forall]" ) { + const auto result = ForAll( Quick( 150 ), + Gen::Integers<int>( -100, 100 ), + Gen::Integers<int>( -100, 100 ), + []( int left, int right ) { + return left + right == right + left; + } ); + CHECK( result.Status == PropertyStatus::kPassed ); + CHECK( result.TrialsRun == 150 ); + CHECK( result.Describe() == "property passed: 150 trials" ); +} + +TEST_CASE( "ForAll shrinks integers to the boundary of the failure", + "[progmasoft][forall]" ) { + const auto result = ForAll( + Quick(), Gen::Integers<int>(), []( int value ) { return value <= 1000; } ); + REQUIRE( result.Status == PropertyStatus::kFailed ); + REQUIRE( result.Failure.has_value() ); + CHECK( result.Failure->Counterexample == "1001 (0x3e9)" ); + CHECK( result.Failure->Reason == "property returned false" ); + CHECK( result.Failure->Replay.rfind( "c3.", 0 ) == 0 ); +} + +TEST_CASE( "ForAll shrinks collections to their essential element", + "[progmasoft][forall]" ) { + const auto result = ForAll( + Quick(), + Gen::Vectors( Gen::Integers<int>( 0, 100 ) ), + []( const std::vector<int>& values ) { + return std::none_of( values.begin(), values.end(), + []( int value ) { return value >= 10; } ); + } ); + REQUIRE( result.Status == PropertyStatus::kFailed ); + CHECK( result.Failure->Counterexample == "{ 10 }" ); +} + +TEST_CASE( "ForAll shrinks through dependent and filtered strategies", + "[progmasoft][forall]" ) { + const auto dependent = + Gen::Integers<int>( 0, 20 ).FlatMap( []( int size ) { + return Gen::Vectors( + Gen::Integers<int>( 0, 100 ), + SizeRange{ static_cast<std::size_t>( size ), + static_cast<std::size_t>( size ) } ); + } ); + const auto sized = ForAll( Quick(), dependent, + []( const std::vector<int>& values ) { + return values.size() < 3; + } ); + REQUIRE( sized.Status == PropertyStatus::kFailed ); + CHECK( sized.Failure->Counterexample == "{ 0, 0, 0 }" ); + + const auto even = Gen::Integers<int>( 0, 100000 ).Filter( + []( int value ) { return value % 2 == 0; } ); + const auto filtered = + ForAll( Quick(), even, []( int value ) { return value <= 100; } ); + INFO( filtered.Describe() ); + REQUIRE( filtered.Status == PropertyStatus::kFailed ); + CHECK( filtered.Failure->Counterexample == "102" ); + + const auto mapped = ForAll( + Quick(), + Gen::Integers<int>( 0, 1000 ).Map( []( int value ) { + return std::to_string( value ); + } ), + []( const std::string& text ) { return text.size() < 3; } ); + REQUIRE( mapped.Status == PropertyStatus::kFailed ); + CHECK( mapped.Failure->Counterexample == "\"100\"" ); +} + +TEST_CASE( "ForAll shrinks several arguments together", + "[progmasoft][forall]" ) { + const auto result = ForAll( Quick(), + Gen::Integers<int>( 0, 1000 ), + Gen::Integers<int>( 0, 1000 ), + []( int left, int right ) { + return left + right < 100; + } ); + REQUIRE( result.Status == PropertyStatus::kFailed ); + CHECK( result.Failure->Counterexample == "{ 0, 100 }" ); +} + +TEST_CASE( "A replay token reproduces exactly the reported input", + "[progmasoft][forall]" ) { + const auto body = []( const std::vector<int>& values ) { + return std::accumulate( values.begin(), values.end(), 0 ) < 50; + }; + const auto strategy = Gen::Vectors( Gen::Integers<int>( 0, 30 ) ); + const auto first = ForAll( Quick(), strategy, body ); + REQUIRE( first.Status == PropertyStatus::kFailed ); + + auto options = Quick(); + options.Replay = first.Failure->Replay; + const auto replayed = ForAll( options, strategy, body ); + REQUIRE( replayed.Status == PropertyStatus::kFailed ); + CHECK( replayed.TrialsRun == 1 ); + CHECK( replayed.Failure->Counterexample == first.Failure->Counterexample ); + CHECK( replayed.Failure->Replay == first.Failure->Replay ); + CHECK( first.Describe().find( "replay: set PropertyOptions::Replay to \"" + + first.Failure->Replay + "\"" ) != + std::string::npos ); + + // The same options give the same report. + CHECK( ForAll( Quick(), strategy, body ).Describe() == first.Describe() ); + + options.Replay = "not a token"; + CHECK( ForAll( options, strategy, body ).Status == + PropertyStatus::kInvalidConfiguration ); +} + +TEST_CASE( "Property bodies assert, assume and note without exceptions", + "[progmasoft][forall]" ) { + const auto failing = ForAll( + Quick(), Gen::Integers<int>( 0, 1000 ), []( int value ) { + Progmasoft::Catch3::Note( "half", value / 2 ); + CATCH3_PROPERTY_ASSERT( value < 500 ); + } ); + REQUIRE( failing.Status == PropertyStatus::kFailed ); + CHECK( failing.Failure->Counterexample == "500 (0x1f4)" ); + CHECK( failing.Failure->Reason.find( + "CATCH3_PROPERTY_ASSERT( value < 500 )" ) == 0 ); + CHECK( failing.Failure->Notes == std::vector<std::string>{ "half = 250" } ); + + const auto assumed = ForAll( + Quick(), Gen::Integers<int>( 0, 9 ), []( int value ) { + CATCH3_PROPERTY_ASSUME( value % 2 == 0 ); + CATCH3_PROPERTY_ASSERT( value % 2 == 0 ); + } ); + CHECK( assumed.Status == PropertyStatus::kPassed ); + CHECK( assumed.Discarded > 0 ); + + auto strict = Quick( 50 ); + strict.MaxDiscardRatio = 1; + const auto gaveUp = ForAll( strict, Gen::Integers<int>( 0, 9 ), + []( int ) { CATCH3_PROPERTY_ASSUME( false ); } ); + CHECK( gaveUp.Status == PropertyStatus::kGaveUp ); + CHECK( gaveUp.Describe().find( "property gave up" ) == 0 ); +} + +TEST_CASE( "Exceptions from bodies fail and from strategies error", + "[progmasoft][forall]" ) { + const auto throwing = ForAll( + Quick(), Gen::Integers<int>( 0, 100 ), []( int value ) -> bool { + if ( value > 10 ) { throw std::runtime_error( "too big" ); } + return true; + } ); + REQUIRE( throwing.Status == PropertyStatus::kFailed ); + CHECK( throwing.Failure->Counterexample == "11" ); + CHECK( throwing.Failure->Reason == + "property threw std::exception: too big" ); + + const auto broken = ForAll( + Quick(), + Gen::Integers<int>( 0, 10 ).Map( []( int ) -> int { + throw std::runtime_error( "cannot build" ); + } ), + []( int ) { return true; } ); + CHECK( broken.Status == PropertyStatus::kErrored ); + CHECK( broken.Error == "strategy threw std::exception: cannot build" ); +} + +TEST_CASE( "Classify and Cover report and enforce statistics", + "[progmasoft][forall]" ) { + auto options = Quick( 400 ); + options.ShowStatistics = true; + const auto covered = ForAll( + options, Gen::Integers<int>( -100, 100 ), []( int value ) { + Progmasoft::Catch3::Classify( value == 0, "zero" ); + Progmasoft::Catch3::Cover( 20.0, value < 0, "negative" ); + return true; + } ); + CHECK( covered.Status == PropertyStatus::kPassed ); + CHECK( covered.Describe().find( "negative:" ) != std::string::npos ); + + const auto uncovered = ForAll( + Quick( 400 ), Gen::Integers<int>( 0, 1000 ), []( int value ) { + Progmasoft::Catch3::Cover( 50.0, value == 7, "seven" ); + return true; + } ); + CHECK( uncovered.Status == PropertyStatus::kInsufficientCoverage ); + CHECK( uncovered.Describe().find( "required 50%" ) != std::string::npos ); +} + +TEST_CASE( "Targeted search runs after the regular trials", + "[progmasoft][forall]" ) { + auto options = Quick( 100 ); + options.TargetSteps = 300; + const auto result = ForAll( + options, Gen::Integers<std::int64_t>( 0, 1LL << 40 ), []( std::int64_t value ) { + Progmasoft::Catch3::Target( static_cast<double>( value ) ); + return value < ( 1LL << 39 ) + ( 1LL << 38 ); + } ); + // Climbing towards the largest values must reach the failing region. + REQUIRE( result.Status == PropertyStatus::kFailed ); + CHECK( result.Failure->Counterexample == "824633720832 (0xc000000000)" ); +} + +TEST_CASE( "The example database replays a stored failure first", + "[progmasoft][forall]" ) { + const auto directory = std::filesystem::temp_directory_path() / + "catch3-property-database-test"; + std::filesystem::remove_all( directory ); + auto options = Quick(); + options.DatabasePath = directory.string(); + options.Name = "database test"; + int threshold = 200; + const auto body = [&]( int value ) { return value < threshold; }; + + const auto first = ForAll( options, Gen::Integers<int>( 0, 1000 ), body ); + REQUIRE( first.Status == PropertyStatus::kFailed ); + CHECK( first.Failure->Counterexample == "200" ); + CHECK( !std::filesystem::is_empty( directory ) ); + + // A new seed would need its own trials; the stored example fails at + // once. + options.Seed = 0xD1FFE7; + const auto second = ForAll( options, Gen::Integers<int>( 0, 1000 ), body ); + REQUIRE( second.Status == PropertyStatus::kFailed ); + CHECK( second.TrialsRun == 1 ); + CHECK( second.Failure->Counterexample == "200" ); + + // Once fixed, the example passes and is dropped. + threshold = 2000; + const auto fixed = ForAll( options, Gen::Integers<int>( 0, 1000 ), body ); + CHECK( fixed.Status == PropertyStatus::kPassed ); + CHECK( std::filesystem::is_empty( directory ) ); + std::filesystem::remove_all( directory ); +} + +TEST_CASE( "Relations check differential, metamorphic and round-trip oracles", + "[progmasoft][forall][relations]" ) { + using namespace Progmasoft::Catch3; + const auto sorted = []( std::vector<int> values ) { + std::sort( values.begin(), values.end() ); + return values; + }; + // A "fast" sort that loses duplicates. + const auto lossy = []( const std::vector<int>& values ) { + const std::set<int> unique( values.begin(), values.end() ); + return std::vector<int>( unique.begin(), unique.end() ); + }; + const auto differential = ForAll( + Quick(), Gen::Vectors( Gen::Integers<int>( 0, 50 ) ), + Equivalent( sorted, lossy ) ); + REQUIRE( differential.Status == PropertyStatus::kFailed ); + CHECK( differential.Failure->Counterexample == "{ 0, 0 }" ); + CHECK( differential.Failure->Notes == + std::vector<std::string>{ "reference: { 0, 0 }", + "candidate: { 0 }" } ); + + const auto sum = []( const std::vector<int>& values ) { + return std::accumulate( values.begin(), values.end(), 0LL ); + }; + const auto reversed = []( std::vector<int> values ) { + std::reverse( values.begin(), values.end() ); + return values; + }; + CHECK( ForAll( Quick(), Gen::Vectors( Gen::Integers<int>( -9, 9 ) ), + Metamorphic( sum, reversed, std::equal_to<>() ) ) + .Succeeded() ); + + const auto encode = []( int value ) { return std::to_string( value ); }; + const auto decode = []( const std::string& text ) { + return std::stoi( text ); + }; + CHECK( ForAll( Quick(), Gen::Integers<int>(), RoundTrip( encode, decode ) ) + .Succeeded() ); + CHECK( ForAll( Quick(), Gen::Vectors( Gen::Integers<int>( 0, 9 ) ), + Idempotent( sorted ) ) + .Succeeded() ); +} + +TEST_CASE( "CATCH3_CHECK_FOR_ALL reports through Catch2", + "[progmasoft][forall]" ) { + CATCH3_CHECK_FOR_ALL( Gen::Integers<int>( 0, 100 ), + []( int value ) { return value >= 0; } ); + CATCH3_REQUIRE_FOR_ALL( Quick( 20 ), + Gen::Strings( Gen::PrintableCharacters() ), + []( const std::string& text ) { + return text.size() == text.length(); + } ); +} diff --git a/tests/Progmasoft/Stateful.tests.cpp b/tests/Progmasoft/Stateful.tests.cpp new file mode 100644 index 00000000..48e2c918 --- /dev/null +++ b/tests/Progmasoft/Stateful.tests.cpp @@ -0,0 +1,129 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <Progmasoft/Catch3.hpp> + +#include <cstddef> +#include <deque> +#include <string> +#include <vector> + +namespace { + + using Progmasoft::Catch3::Action; + using Progmasoft::Catch3::Command; + using Progmasoft::Catch3::PropertyOptions; + using Progmasoft::Catch3::PropertyStatus; + using Progmasoft::Catch3::SizeRange; + namespace Gen = Progmasoft::Catch3::Gen; + + // A queue with a bug: past three elements it silently drops the + // oldest one. + class SmallQueue { + public: + void Push( int value ) { + if ( values_.size() == 3 ) { values_.pop_front(); } + values_.push_back( value ); + } + int Pop() { + const int value = values_.front(); + values_.pop_front(); + return value; + } + [[nodiscard]] std::size_t Size() const { return values_.size(); } + + private: + std::deque<int> values_; + }; + + using Model = std::deque<int>; + + std::vector<Command<Model, SmallQueue>> QueueCommands() { + Command<Model, SmallQueue> push{ + "push", + []( const Model& ) { + return Gen::Integers<int>( 0, 100 ).Map( []( int value ) { + return Action<Model, SmallQueue>{ + "push(" + std::to_string( value ) + ")", + [value]( Model& model, SmallQueue& queue ) { + model.push_back( value ); + queue.Push( value ); + CATCH3_PROPERTY_ASSERT( queue.Size() == + model.size() ); + } }; + } ); + } }; + Command<Model, SmallQueue> pop{ + "pop", + []( const Model& ) { + return Gen::Just( Action<Model, SmallQueue>{ + "pop", []( Model& model, SmallQueue& queue ) { + const int expected = model.front(); + model.pop_front(); + CATCH3_PROPERTY_ASSERT( queue.Pop() == expected ); + } } ); + }, + []( const Model& model ) { return !model.empty(); } }; + return { push, pop }; + } + +} // namespace + +TEST_CASE( "Stateful tests find and shrink a failing command sequence", + "[progmasoft][stateful]" ) { + PropertyOptions options; + options.Trials = 300; + options.Seed = 0x57A7E; + options.MaxShrinkSteps = 5000; + const auto result = Progmasoft::Catch3::CheckStateful<Model, SmallQueue>( + options, + [] { return Model{}; }, + []( const Model& ) { return SmallQueue{}; }, + QueueCommands() ); + REQUIRE( result.Status == PropertyStatus::kFailed ); + // The shortest failure: the fourth push drops an element. + CHECK( result.Failure->Counterexample == + "{ push(0), push(0), push(0), push(0) }" ); + CHECK( result.Failure->Reason.find( + "CATCH3_PROPERTY_ASSERT( queue.Size() == model.size() )" ) == + 0 ); +} + +TEST_CASE( "Stateful tests pass for a correct system and respect preconditions", + "[progmasoft][stateful]" ) { + struct Correct { + std::deque<int> Values; + }; + Command<Model, Correct> push{ + "push", []( const Model& ) { + return Gen::Integers<int>( 0, 9 ).Map( []( int value ) { + return Action<Model, Correct>{ + "push", [value]( Model& model, Correct& system ) { + model.push_back( value ); + system.Values.push_back( value ); + } }; + } ); + } }; + Command<Model, Correct> pop{ + "pop", + []( const Model& ) { + return Gen::Just( Action<Model, Correct>{ + "pop", []( Model& model, Correct& system ) { + // The precondition guarantees an element. + CATCH3_PROPERTY_ASSERT( !model.empty() ); + CATCH3_PROPERTY_ASSERT( system.Values.front() == + model.front() ); + model.pop_front(); + system.Values.pop_front(); + } } ); + }, + []( const Model& model ) { return !model.empty(); } }; + PropertyOptions options; + options.Trials = 200; + const auto result = Progmasoft::Catch3::CheckStateful<Model, Correct>( + options, + [] { return Model{}; }, + []( const Model& ) { return Correct{}; }, + { push, pop }, + SizeRange{ 0, 50 } ); + CHECK( result.Status == PropertyStatus::kPassed ); +} diff --git a/tests/Progmasoft/Strategy.tests.cpp b/tests/Progmasoft/Strategy.tests.cpp new file mode 100644 index 00000000..5a1d5a2a --- /dev/null +++ b/tests/Progmasoft/Strategy.tests.cpp @@ -0,0 +1,312 @@ +// Copyright (c) 2026 Progmasoft. +// SPDX-License-Identifier: MPL-2.0 WITH AdditionRef-Progmasoft-Exception-1.1 +#include <Progmasoft/Catch3.hpp> + +#include <algorithm> +#include <cmath> +#include <cstdint> +#include <limits> +#include <map> +#include <optional> +#include <set> +#include <stdexcept> +#include <string> +#include <vector> + +namespace { + + using Progmasoft::Catch3::ChoiceSource; + using Progmasoft::Catch3::SizeRange; + using Progmasoft::Catch3::Strategy; + namespace Gen = Progmasoft::Catch3::Gen; + + template <typename ValueType> + std::vector<ValueType> Sample( const Strategy<ValueType>& strategy, + std::size_t count, + std::uint64_t seed ) { + std::vector<ValueType> values; + for ( std::size_t index = 0; index < count; ++index ) { + ChoiceSource source( seed + index, index % 101 ); + auto value = strategy.Draw( source ); + if ( value.has_value() ) { values.push_back( *value ); } + } + return values; + } + + // Draws once, then replays the recorded choices: the same value must + // come back, which is what shrinking and replay tokens rely on. + template <typename ValueType> + bool ReplaysIdentically( const Strategy<ValueType>& strategy, + std::uint64_t seed ) { + ChoiceSource original( seed, 60 ); + const auto value = strategy.Draw( original ); + const auto choices = original.Choices(); + ChoiceSource replay( choices, 60 ); + const auto again = strategy.Draw( replay ); + return value == again && replay.Choices() == choices && + !replay.Overran(); + } + +} // namespace + +TEST_CASE( "Choice sources replay, clamp and run out to zero", + "[progmasoft][strategy]" ) { + const std::vector<std::uint64_t> recorded{ 3, 99, 1 }; + ChoiceSource replay( recorded, 10 ); + CHECK( replay.Draw( 10 ) == 3 ); + CHECK( replay.Draw( 7 ) == 7 ); // clamped to the new maximum + CHECK( replay.DrawBoolean( 0.0 ) ); + CHECK_FALSE( replay.Overran() ); + CHECK( replay.Draw( 5 ) == 0 ); // past the end: the simplest choice + CHECK( replay.Overran() ); + CHECK( replay.Choices() == std::vector<std::uint64_t>{ 3, 7, 1, 0 } ); + + ChoiceSource generated( 42, 50 ); + for ( int index = 0; index < 1000; ++index ) { + CHECK( generated.Draw( 9 ) <= 9 ); + CHECK( generated.DrawBiased( 1000 ) <= 1000 ); + } + CHECK( generated.Choices().size() == 2000 ); +} + +TEST_CASE( "Integer strategies respect every kind of range", + "[progmasoft][strategy]" ) { + const auto crossing = Gen::Integers<int>( -50, 70 ); + for ( int value : Sample( crossing, 2000, 1 ) ) { + CHECK( value >= -50 ); + CHECK( value <= 70 ); + } + const auto negative = Gen::Integers<std::int64_t>( -900, -100 ); + for ( auto value : Sample( negative, 2000, 2 ) ) { + CHECK( value >= -900 ); + CHECK( value <= -100 ); + } + const auto positive = Gen::Integers<std::uint8_t>( 10, 20 ); + for ( auto value : Sample( positive, 2000, 3 ) ) { + CHECK( value >= 10 ); + CHECK( value <= 20 ); + } + const auto full = Gen::Integers<std::int64_t>(); + const auto values = Sample( full, 4000, 4 ); + CHECK( std::any_of( values.begin(), values.end(), []( auto value ) { + return value == std::numeric_limits<std::int64_t>::max(); + } ) ); + CHECK( std::any_of( values.begin(), values.end(), []( auto value ) { + return value < 0; + } ) ); + const auto upToZero = Gen::Integers<std::int64_t>( + std::numeric_limits<std::int64_t>::lowest(), 0 ); + for ( auto value : Sample( upToZero, 2000, 5 ) ) { CHECK( value <= 0 ); } + + // The simplest choices give the value closest to zero. + const std::vector<std::uint64_t> zeros; + ChoiceSource simplest( zeros, 0 ); + CHECK( crossing.Draw( simplest ) == 0 ); + ChoiceSource simplestNegative( zeros, 0 ); + CHECK( negative.Draw( simplestNegative ) == -100 ); + ChoiceSource simplestPositive( zeros, 0 ); + CHECK( positive.Draw( simplestPositive ) == 10 ); + + CHECK_THROWS_AS( Gen::Integers<int>( 2, 1 ), std::invalid_argument ); + for ( std::uint64_t seed = 0; seed < 50; ++seed ) { + CHECK( ReplaysIdentically( full, seed ) ); + } +} + +TEST_CASE( "Float strategies stay finite and in range unless allowed", + "[progmasoft][strategy]" ) { + using Progmasoft::Catch3::FloatOptions; + const auto bounded = Gen::Floats<double>( + FloatOptions{ .Minimum = -2.5, .Maximum = 1000.0 } ); + for ( double value : Sample( bounded, 3000, 6 ) ) { + CHECK( std::isfinite( value ) ); + CHECK( value >= -2.5 ); + CHECK( value <= 1000.0 ); + } + const auto everything = Gen::Floats<double>( + FloatOptions{ .AllowNaN = true, .AllowInfinity = true } ); + const auto values = Sample( everything, 3000, 7 ); + CHECK( std::any_of( values.begin(), values.end(), []( double value ) { + return std::isnan( value ); + } ) ); + CHECK( std::any_of( values.begin(), values.end(), []( double value ) { + return std::isinf( value ); + } ) ); + const auto positive = Gen::Floats<float>( + FloatOptions{ .Minimum = 1.0, .Maximum = 2.0 } ); + const std::vector<std::uint64_t> zeros; + ChoiceSource simplest( zeros, 0 ); + CHECK( positive.Draw( simplest ) == 1.0F ); + CHECK_THROWS_AS( Gen::Floats<double>( FloatOptions{ .Minimum = 2.0, + .Maximum = 1.0 } ), + std::invalid_argument ); +} + +TEST_CASE( "Collection strategies respect their size ranges", + "[progmasoft][strategy]" ) { + const auto vectors = Gen::Vectors( Gen::Integers<int>( 0, 9 ), + SizeRange{ 2, 5 } ); + for ( const auto& value : Sample( vectors, 1000, 8 ) ) { + CHECK( value.size() >= 2 ); + CHECK( value.size() <= 5 ); + } + const auto unique = Gen::UniqueVectors( Gen::Integers<int>( 0, 30 ), + SizeRange{ 0, 20 } ); + for ( const auto& value : Sample( unique, 1000, 9 ) ) { + CHECK( std::set<int>( value.begin(), value.end() ).size() == + value.size() ); + } + const auto sets = Gen::Sets( Gen::Integers<int>( 0, 3 ), + SizeRange{ 4, 4 } ); + for ( const auto& value : Sample( sets, 200, 10 ) ) { + CHECK( value == std::set<int>{ 0, 1, 2, 3 } ); + } + const auto maps = Gen::Maps( Gen::Integers<int>( 0, 100 ), + Gen::Booleans(), + SizeRange{ 1, 10 } ); + for ( const auto& value : Sample( maps, 500, 11 ) ) { + CHECK( !value.empty() ); + CHECK( value.size() <= 10 ); + } + const auto permutations = Gen::Permutations( std::vector<int>{ 1, 2, 3, 4, 5 } ); + for ( auto value : Sample( permutations, 300, 12 ) ) { + std::sort( value.begin(), value.end() ); + CHECK( value == std::vector<int>{ 1, 2, 3, 4, 5 } ); + } + const std::vector<std::uint64_t> zeros; + ChoiceSource simplest( zeros, 0 ); + CHECK( permutations.Draw( simplest ) == + std::vector<int>{ 1, 2, 3, 4, 5 } ); + CHECK_THROWS_AS( Gen::Vectors( Gen::Booleans(), SizeRange{ 3, 2 } ), + std::invalid_argument ); + for ( std::uint64_t seed = 0; seed < 50; ++seed ) { + CHECK( ReplaysIdentically( vectors, seed ) ); + CHECK( ReplaysIdentically( maps, seed ) ); + } +} + +TEST_CASE( "Text strategies produce valid UTF-8 without surrogates", + "[progmasoft][strategy]" ) { + const auto text = Gen::Utf8Strings( SizeRange{ 0, 30 }, 0, 0x10FFFF ); + for ( const auto& value : Sample( text, 1000, 13 ) ) { + for ( std::size_t index = 0; index + 1 < value.size(); ++index ) { + const auto lead = static_cast<unsigned char>( value[index] ); + const auto next = static_cast<unsigned char>( value[index + 1] ); + // ED A0..BF would encode a surrogate. + CHECK_FALSE( ( lead == 0xED && next >= 0xA0 ) ); + } + } + const auto printable = Gen::Strings( Gen::PrintableCharacters(), + SizeRange{ 1, 8 } ); + for ( const auto& value : Sample( printable, 500, 14 ) ) { + CHECK( !value.empty() ); + CHECK( value.size() <= 8 ); + CHECK( std::all_of( value.begin(), value.end(), []( char c ) { + return c >= 0x20 && c < 0x7F; + } ) ); + } +} + +TEST_CASE( "Combinators compose and keep replays stable", + "[progmasoft][strategy]" ) { + const auto even = Gen::Integers<int>( 0, 1000 ).Filter( + []( int value ) { return value % 2 == 0; } ); + for ( int value : Sample( even, 500, 15 ) ) { CHECK( value % 2 == 0 ); } + + const auto sized = Gen::Integers<int>( 0, 6 ).FlatMap( []( int size ) { + return Gen::Vectors( Gen::Just( size ), + SizeRange{ static_cast<std::size_t>( size ), + static_cast<std::size_t>( size ) } ); + } ); + for ( const auto& value : Sample( sized, 500, 16 ) ) { + CHECK( std::all_of( value.begin(), value.end(), [&]( int element ) { + return element == static_cast<int>( value.size() ); + } ) ); + } + + const auto mixed = Gen::OneOf( Gen::Just( std::string( "a" ) ), + Gen::Just( std::string( "b" ) ) ); + const auto mixedValues = Sample( mixed, 200, 17 ); + CHECK( std::count( mixedValues.begin(), mixedValues.end(), "a" ) > 0 ); + CHECK( std::count( mixedValues.begin(), mixedValues.end(), "b" ) > 0 ); + + const auto weighted = Gen::Frequency<int>( { { 1, Gen::Just( 0 ) }, + { 9, Gen::Just( 1 ) } } ); + const auto weightedValues = Sample( weighted, 2000, 18 ); + CHECK( std::count( weightedValues.begin(), weightedValues.end(), 1 ) > + std::count( weightedValues.begin(), weightedValues.end(), 0 ) ); + + const auto trees = Gen::Recursive( + Gen::Just( 1 ), + []( const Strategy<int>& children ) { + return Gen::Tuples( children, children ) + .Map( []( std::tuple<int, int> pair ) { + return std::get<0>( pair ) + std::get<1>( pair ); + } ); + }, + 4 ); + for ( int leaves : Sample( trees, 500, 19 ) ) { + CHECK( leaves >= 1 ); + CHECK( leaves <= 16 ); // depth 4 at most: 2^4 leaves + } + + const auto never = Gen::Integers<int>( 0, 10 ).Filter( + []( int ) { return false; }, 5 ); + ChoiceSource source( 1, 50 ); + CHECK_FALSE( never.Draw( source ).has_value() ); +} + +TEST_CASE( "Replay tokens round-trip and reject malformed text", + "[progmasoft][strategy]" ) { + using Progmasoft::Catch3::Detail::DecodeChoices; + using Progmasoft::Catch3::Detail::EncodeChoices; + const std::vector<std::uint64_t> choices{ + 0, 1, 35, 36, std::numeric_limits<std::uint64_t>::max() }; + const auto token = EncodeChoices( 73, choices ); + CHECK( token == "c3.21.0-1-z-10-3w5e11264sgsf" ); + const auto decoded = DecodeChoices( token ); + REQUIRE( decoded.has_value() ); + CHECK( decoded->Size == 73 ); + CHECK( decoded->Choices == choices ); + + const auto empty = DecodeChoices( EncodeChoices( 0, {} ) ); + REQUIRE( empty.has_value() ); + CHECK( empty->Choices.empty() ); + + for ( const char* malformed : + { "", "c3.", "c3.1", "x3.1.2", "c3.1.2-", "c3.1.-2", "c3.2t.1", + "c3.1.2--3", "c3.1.zzzzzzzzzzzzzzz" } ) { + CHECK_FALSE( DecodeChoices( malformed ).has_value() ); + } +} + +TEST_CASE( "The choice shrinker finds simple failing sequences", + "[progmasoft][strategy]" ) { + using Progmasoft::Catch3::Detail::IsSimpler; + using Progmasoft::Catch3::Detail::ShrinkChoices; + CHECK( IsSimpler( std::vector<std::uint64_t>{ 9 }, + std::vector<std::uint64_t>{ 0, 0 } ) ); + CHECK( IsSimpler( std::vector<std::uint64_t>{ 1, 9 }, + std::vector<std::uint64_t>{ 2, 0 } ) ); + CHECK_FALSE( IsSimpler( std::vector<std::uint64_t>{ 2, 0 }, + std::vector<std::uint64_t>{ 2, 0 } ) ); + + // Fails when the first two choices sum to at least 10. + const auto sumAtLeastTen = []( const std::vector<std::uint64_t>& choices ) + -> std::optional<std::vector<std::uint64_t>> { + std::vector<std::uint64_t> consumed = choices; + consumed.resize( 2, 0 ); + if ( consumed[0] + consumed[1] >= 10 ) { return consumed; } + return std::nullopt; + }; + std::size_t calls = 0; + const auto minimized = ShrinkChoices( + { 7, 9, 4, 4, 1 }, 10000, sumAtLeastTen, calls ); + // Shortlex: two choices, the first as small as the sum allows. + CHECK( minimized == std::vector<std::uint64_t>{ 0, 10 } ); + CHECK( calls > 0 ); + + std::size_t limitedCalls = 0; + (void)ShrinkChoices( { 7, 9, 4, 4, 1 }, 3, sumAtLeastTen, limitedCalls ); + CHECK( limitedCalls <= 3 ); +} diff --git a/tests/meson.build b/tests/meson.build index c4abaf29..4e258ee5 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -84,8 +84,11 @@ progmasoft_property_tests = executable( [ 'Progmasoft/Generators.tests.cpp', 'Progmasoft/Property.tests.cpp', + 'Progmasoft/ForAll.tests.cpp', 'Progmasoft/Results.tests.cpp', 'Progmasoft/Snapshot.tests.cpp', + 'Progmasoft/Stateful.tests.cpp', + 'Progmasoft/Strategy.tests.cpp', ], include_directories: '../src', dependencies: [progmasoft_catch3_dep, catch2_with_main_dep], diff --git a/xmake.lua b/xmake.lua index 385cf47d..9d956df7 100644 --- a/xmake.lua +++ b/xmake.lua @@ -39,6 +39,9 @@ target("ProgmasoftCatch3") set_languages("cxx20") add_headerfiles("src/Progmasoft/**.hpp", {prefixdir = ""}) add_files( + "src/Progmasoft/ChoiceShrinker.cpp", + "src/Progmasoft/Context.cpp", + "src/Progmasoft/PropertyEngine.cpp", "src/Progmasoft/Results.cpp", "src/Progmasoft/Snapshot.cpp", "src/Progmasoft/XmlWriter.cpp" @@ -52,8 +55,11 @@ target("ProgmasoftPropertyTests") add_files( "tests/Progmasoft/Generators.tests.cpp", "tests/Progmasoft/Property.tests.cpp", + "tests/Progmasoft/ForAll.tests.cpp", "tests/Progmasoft/Results.tests.cpp", "tests/Progmasoft/Snapshot.tests.cpp", + "tests/Progmasoft/Stateful.tests.cpp", + "tests/Progmasoft/Strategy.tests.cpp", "src/catch2/internal/catch_main.cpp" ) add_includedirs("src", generated_include)