catch3
Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.
Clonegit clone https://git.progmasoft.com/catch3
approved tests that were correct the first time (previous approvals were erroneous)
- Commit
714d01c07c540f2464a80a3ec7623fcb42c72710- Author
- Phil Nash ·
- Parents
c6990cdf9157
Changes
.../SelfTest/Baselines/console.std.approved.txt | 78 +++++++++++++++++++++- .../SelfTest/Baselines/console.sw.approved.txt | 72 +++++++++++++++++++- .../SelfTest/Baselines/console.swa4.approved.txt | 14 +++- projects/SelfTest/Baselines/junit.sw.approved.txt | 46 ++++++++++++- projects/SelfTest/Baselines/xml.sw.approved.txt | 52 ++++++++++++--- 5 files changed, 245 insertions(+), 17 deletions(-) diff --git a/projects/SelfTest/Baselines/console.std.approved.txt b/projects/SelfTest/Baselines/console.std.approved.txt index fe760fd7..e6bcf41e 100644 --- a/projects/SelfTest/Baselines/console.std.approved.txt +++ b/projects/SelfTest/Baselines/console.std.approved.txt @@ -3,6 +3,18 @@ <exe-name> is a <version> host application. Run with -? for options +------------------------------------------------------------------------------- +#748 - captures with unexpected exceptions + outside assertions +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with messages: + answer := 42 + expected exception + ------------------------------------------------------------------------------- #748 - captures with unexpected exceptions inside REQUIRE_NOTHROW @@ -115,6 +127,17 @@ TrickyTests.cpp:<line number>: FAILED: with expansion: {?} == {?} +------------------------------------------------------------------------------- +An unchecked exception reports the line of the last assertion +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: + {Unknown expression after the reported line} +due to unexpected exception with message: + unexpected exception + ------------------------------------------------------------------------------- Contains string matcher ------------------------------------------------------------------------------- @@ -148,6 +171,16 @@ ExceptionTests.cpp:<line number>: FAILED: due to unexpected exception with message: custom exception - not std +------------------------------------------------------------------------------- +Custom std-exceptions can be custom translated +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + custom std exception + ------------------------------------------------------------------------------- EndsWith string matcher ------------------------------------------------------------------------------- @@ -413,6 +446,16 @@ MiscTests.cpp:<line number>: warning: This one ran +------------------------------------------------------------------------------- +Non-std exceptions can be translated +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + custom exception + ------------------------------------------------------------------------------- Ordering comparison checks that should fail ------------------------------------------------------------------------------- @@ -599,6 +642,16 @@ with expansion: } " +------------------------------------------------------------------------------- +Unexpected exceptions can be translated +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + 3.14 + ------------------------------------------------------------------------------- Vector matchers that fail Contains (element) @@ -660,6 +713,16 @@ MatchersTests.cpp:<line number>: FAILED: with expansion: { 1, 2, 3 } Equals: { } +------------------------------------------------------------------------------- +When unchecked exceptions are thrown directly they are always failures +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + unexpected exception + ------------------------------------------------------------------------------- When unchecked exceptions are thrown during a CHECK the test should continue ------------------------------------------------------------------------------- @@ -694,6 +757,17 @@ ExceptionTests.cpp:<line number>: FAILED: due to unexpected exception with message: expected exception +------------------------------------------------------------------------------- +When unchecked exceptions are thrown from sections they are always failures + section name +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + unexpected exception + ------------------------------------------------------------------------------- Where the LHS is not a simple value ------------------------------------------------------------------------------- @@ -866,6 +940,6 @@ with expansion: "first" == "second" =============================================================================== -test cases: 167 | 125 passed | 38 failed | 4 failed as expected -assertions: 960 | 859 passed | 81 failed | 20 failed as expected +test cases: 167 | 119 passed | 44 failed | 4 failed as expected +assertions: 967 | 859 passed | 87 failed | 21 failed as expected diff --git a/projects/SelfTest/Baselines/console.sw.approved.txt b/projects/SelfTest/Baselines/console.sw.approved.txt index 954e80e6..ac178f80 100644 --- a/projects/SelfTest/Baselines/console.sw.approved.txt +++ b/projects/SelfTest/Baselines/console.sw.approved.txt @@ -14,6 +14,18 @@ PASSED: with message: yay +------------------------------------------------------------------------------- +#748 - captures with unexpected exceptions + outside assertions +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with messages: + answer := 42 + expected exception + ------------------------------------------------------------------------------- #748 - captures with unexpected exceptions inside REQUIRE_NOTHROW @@ -444,6 +456,11 @@ ExceptionTests.cpp:<line number>: PASSED: CHECK( 1 == 1 ) +ExceptionTests.cpp:<line number>: FAILED: + {Unknown expression after the reported line} +due to unexpected exception with message: + unexpected exception + ------------------------------------------------------------------------------- Anonymous test case 1 ------------------------------------------------------------------------------- @@ -998,6 +1015,16 @@ ExceptionTests.cpp:<line number>: FAILED: due to unexpected exception with message: custom exception - not std +------------------------------------------------------------------------------- +Custom std-exceptions can be custom translated +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + custom std exception + ------------------------------------------------------------------------------- Demonstrate that a non-const == is not used ------------------------------------------------------------------------------- @@ -4508,6 +4535,16 @@ MiscTests.cpp:<line number>: warning: This one ran +------------------------------------------------------------------------------- +Non-std exceptions can be translated +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + custom exception + ------------------------------------------------------------------------------- NotImplemented exception ------------------------------------------------------------------------------- @@ -7945,6 +7982,16 @@ PASSED: with expansion: true +------------------------------------------------------------------------------- +Unexpected exceptions can be translated +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + 3.14 + ------------------------------------------------------------------------------- Use a custom approx ------------------------------------------------------------------------------- @@ -8165,6 +8212,16 @@ ExceptionTests.cpp:<line number>: PASSED: REQUIRE_THROWS( thisThrows() ) +------------------------------------------------------------------------------- +When unchecked exceptions are thrown directly they are always failures +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + unexpected exception + ------------------------------------------------------------------------------- When unchecked exceptions are thrown during a CHECK the test should continue ------------------------------------------------------------------------------- @@ -8199,6 +8256,17 @@ ExceptionTests.cpp:<line number>: FAILED: due to unexpected exception with message: expected exception +------------------------------------------------------------------------------- +When unchecked exceptions are thrown from sections they are always failures + section name +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with message: + unexpected exception + ------------------------------------------------------------------------------- Where the LHS is not a simple value ------------------------------------------------------------------------------- @@ -9398,6 +9466,6 @@ MiscTests.cpp:<line number>: PASSED: =============================================================================== -test cases: 167 | 124 passed | 39 failed | 4 failed as expected -assertions: 962 | 859 passed | 83 failed | 20 failed as expected +test cases: 167 | 118 passed | 45 failed | 4 failed as expected +assertions: 969 | 859 passed | 89 failed | 21 failed as expected diff --git a/projects/SelfTest/Baselines/console.swa4.approved.txt b/projects/SelfTest/Baselines/console.swa4.approved.txt index e9a0cbb7..dcabf4e0 100644 --- a/projects/SelfTest/Baselines/console.swa4.approved.txt +++ b/projects/SelfTest/Baselines/console.swa4.approved.txt @@ -14,6 +14,18 @@ PASSED: with message: yay +------------------------------------------------------------------------------- +#748 - captures with unexpected exceptions + outside assertions +------------------------------------------------------------------------------- +ExceptionTests.cpp:<line number> +............................................................................... + +ExceptionTests.cpp:<line number>: FAILED: +due to unexpected exception with messages: + answer := 42 + expected exception + ------------------------------------------------------------------------------- #748 - captures with unexpected exceptions inside REQUIRE_NOTHROW @@ -133,5 +145,5 @@ ConditionTests.cpp:<line number>: FAILED: =============================================================================== test cases: 6 | 3 passed | 1 failed | 2 failed as expected -assertions: 17 | 11 passed | 4 failed | 2 failed as expected +assertions: 18 | 11 passed | 4 failed | 3 failed as expected diff --git a/projects/SelfTest/Baselines/junit.sw.approved.txt b/projects/SelfTest/Baselines/junit.sw.approved.txt index 45ba1ad8..b1b02add 100644 --- a/projects/SelfTest/Baselines/junit.sw.approved.txt +++ b/projects/SelfTest/Baselines/junit.sw.approved.txt @@ -1,7 +1,14 @@ <?xml version="1.0" encoding="UTF-8"?> <testsuitesspanner> - <testsuite name="<exe-name>" errors="8" failures="76" tests="963" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> + <testsuite name="<exe-name>" errors="15" failures="75" tests="970" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> <testcase classname="global" name="# A test name that starts with a #" time="{duration}"/> + <testcase classname="#748 - captures with unexpected exceptions" name="outside assertions" time="{duration}"> + <error type="TEST_CASE"> +expected exception +answer := 42 +ExceptionTests.cpp:<line number> + </error> + </testcase> <testcase classname="#748 - captures with unexpected exceptions" name="inside REQUIRE_NOTHROW" time="{duration}"> <error message="thisThrows()" type="REQUIRE_NOTHROW"> expected exception @@ -79,7 +86,12 @@ TrickyTests.cpp:<line number> <testcase classname="global" name="Absolute margin" time="{duration}"/> <testcase classname="global" name="AllOf matcher" time="{duration}"/> <testcase classname="global" name="An expression with side-effects should only be evaluated once" time="{duration}"/> - <testcase classname="global" name="An unchecked exception reports the line of the last assertion" time="{duration}"/> + <testcase classname="global" name="An unchecked exception reports the line of the last assertion" time="{duration}"> + <error message="{Unknown expression after the reported line}"> +unexpected exception +ExceptionTests.cpp:<line number> + </error> + </testcase> <testcase classname="global" name="Anonymous test case 1" time="{duration}"/> <testcase classname="global" name="AnyOf matcher" time="{duration}"/> <testcase classname="global" name="Approximate PI" time="{duration}"/> @@ -115,6 +127,12 @@ ExceptionTests.cpp:<line number> <testcase classname="global" name="Custom exceptions can be translated when testing for throwing as something else" time="{duration}"> <error message="throwCustom(), std::exception" type="REQUIRE_THROWS_AS"> custom exception - not std +ExceptionTests.cpp:<line number> + </error> + </testcase> + <testcase classname="global" name="Custom std-exceptions can be custom translated" time="{duration}"> + <error type="TEST_CASE"> +custom std exception ExceptionTests.cpp:<line number> </error> </testcase> @@ -282,6 +300,12 @@ ExceptionTests.cpp:<line number> </failure> </testcase> <testcase classname="global" name="Nice descriptive name" time="{duration}"/> + <testcase classname="global" name="Non-std exceptions can be translated" time="{duration}"> + <error type="TEST_CASE"> +custom exception +ExceptionTests.cpp:<line number> + </error> + </testcase> <testcase classname="global" name="NotImplemented exception" time="{duration}"/> <testcase classname="global" name="Objects that evaluated in boolean contexts can be checked" time="{duration}"/> <testcase classname="global" name="Operators at different namespace levels not hijacked by Koenig lookup" time="{duration}"/> @@ -497,6 +521,12 @@ MiscTests.cpp:<line number> <testcase classname="Tracker" name="start a generator/Start a new inner section/Re-enter for second generation" time="{duration}"/> <testcase classname="Tracker" name="start a generator/Fail an inner section" time="{duration}"/> <testcase classname="Tracker" name="start a generator/Fail an inner section/Re-enter for second generation" time="{duration}"/> + <testcase classname="global" name="Unexpected exceptions can be translated" time="{duration}"> + <error type="TEST_CASE"> +3.14 +ExceptionTests.cpp:<line number> + </error> + </testcase> <testcase classname="global" name="Use a custom approx" time="{duration}"/> <testcase classname="Variadic macros" name="Section with one argument" time="{duration}"/> <testcase classname="Vector matchers" name="Contains (element)" time="{duration}"/> @@ -533,6 +563,12 @@ MatchersTests.cpp:<line number> </failure> </testcase> <testcase classname="global" name="When checked exceptions are thrown they can be expected or unexpected" time="{duration}"/> + <testcase classname="global" name="When unchecked exceptions are thrown directly they are always failures" time="{duration}"> + <error type="TEST_CASE"> +unexpected exception +ExceptionTests.cpp:<line number> + </error> + </testcase> <testcase classname="global" name="When unchecked exceptions are thrown during a CHECK the test should continue" time="{duration}"> <error message="thisThrows() == 0" type="CHECK"> expected exception @@ -548,6 +584,12 @@ ExceptionTests.cpp:<line number> <testcase classname="global" name="When unchecked exceptions are thrown from functions they are always failures" time="{duration}"> <error message="thisThrows() == 0" type="CHECK"> expected exception +ExceptionTests.cpp:<line number> + </error> + </testcase> + <testcase classname="When unchecked exceptions are thrown from sections they are always failures" name="section name" time="{duration}"> + <error type="TEST_CASE"> +unexpected exception ExceptionTests.cpp:<line number> </error> </testcase> diff --git a/projects/SelfTest/Baselines/xml.sw.approved.txt b/projects/SelfTest/Baselines/xml.sw.approved.txt index 526f68aa..fed55cd0 100644 --- a/projects/SelfTest/Baselines/xml.sw.approved.txt +++ b/projects/SelfTest/Baselines/xml.sw.approved.txt @@ -6,7 +6,13 @@ </TestCase> <TestCase name="#748 - captures with unexpected exceptions" tags="[!shouldfail][!throws]" filename="projects/<exe-name>/ExceptionTests.cpp" > <Section name="outside assertions" filename="projects/<exe-name>/ExceptionTests.cpp" > - <OverallResults successes="0" failures="0" expectedFailures="0"/> + <Info> + answer := 42 + </Info> + <Exception filename="projects/<exe-name>/ExceptionTests.cpp" > + expected exception + </Exception> + <OverallResults successes="0" failures="1" expectedFailures="0"/> </Section> <Section name="inside REQUIRE_NOTHROW" filename="projects/<exe-name>/ExceptionTests.cpp" > <Info> @@ -506,7 +512,18 @@ 1 == 1 </Expanded> </Expression> - <OverallResult success="true"/> + <Expression success="false" filename="projects/<exe-name>/ExceptionTests.cpp" > + <Original> + {Unknown expression after the reported line} + </Original> + <Expanded> + {Unknown expression after the reported line} + </Expanded> + <Exception filename="projects/<exe-name>/ExceptionTests.cpp" > + unexpected exception + </Exception> + </Expression> + <OverallResult success="false"/> </TestCase> <TestCase name="Anonymous test case 1" filename="projects/<exe-name>/VariadicMacrosTests.cpp" > <OverallResult success="true"/> @@ -1110,7 +1127,10 @@ <OverallResult success="false"/> </TestCase> <TestCase name="Custom std-exceptions can be custom translated" tags="[!throws][.][failing][hide]" filename="projects/<exe-name>/ExceptionTests.cpp" > - <OverallResult success="true"/> + <Exception filename="projects/<exe-name>/ExceptionTests.cpp" > + custom std exception + </Exception> + <OverallResult success="false"/> </TestCase> <TestCase name="Demonstrate that a non-const == is not used" tags="[Tricky]" filename="projects/<exe-name>/TrickyTests.cpp" > <Expression success="true" type="REQUIRE" filename="projects/<exe-name>/TrickyTests.cpp" > @@ -4635,7 +4655,10 @@ re>" <OverallResult success="true"/> </TestCase> <TestCase name="Non-std exceptions can be translated" tags="[!throws][.][failing][hide]" filename="projects/<exe-name>/ExceptionTests.cpp" > - <OverallResult success="true"/> + <Exception filename="projects/<exe-name>/ExceptionTests.cpp" > + custom exception + </Exception> + <OverallResult success="false"/> </TestCase> <TestCase name="NotImplemented exception" tags="[!throws]" filename="projects/<exe-name>/ExceptionTests.cpp" > <Expression success="true" type="REQUIRE_THROWS" filename="projects/<exe-name>/ExceptionTests.cpp" > @@ -8586,7 +8609,10 @@ there" <OverallResult success="true"/> </TestCase> <TestCase name="Unexpected exceptions can be translated" tags="[!throws][.][failing][hide]" filename="projects/<exe-name>/ExceptionTests.cpp" > - <OverallResult success="true"/> + <Exception filename="projects/<exe-name>/ExceptionTests.cpp" > + 3.14 + </Exception> + <OverallResult success="false"/> </TestCase> <TestCase name="Use a custom approx" tags="[Approx][custom]" filename="projects/<exe-name>/ApproxTests.cpp" > <Expression success="true" type="REQUIRE" filename="projects/<exe-name>/ApproxTests.cpp" > @@ -8849,7 +8875,10 @@ there" <OverallResult success="true"/> </TestCase> <TestCase name="When unchecked exceptions are thrown directly they are always failures" tags="[!throws][.][failing][hide]" filename="projects/<exe-name>/ExceptionTests.cpp" > - <OverallResult success="true"/> + <Exception filename="projects/<exe-name>/ExceptionTests.cpp" > + unexpected exception + </Exception> + <OverallResult success="false"/> </TestCase> <TestCase name="When unchecked exceptions are thrown during a CHECK the test should continue" tags="[!throws][.][failing][hide]" filename="projects/<exe-name>/ExceptionTests.cpp" > <Expression success="false" type="CHECK" filename="projects/<exe-name>/ExceptionTests.cpp" > @@ -8895,9 +8924,12 @@ there" </TestCase> <TestCase name="When unchecked exceptions are thrown from sections they are always failures" tags="[!throws][.][failing][hide]" filename="projects/<exe-name>/ExceptionTests.cpp" > <Section name="section name" filename="projects/<exe-name>/ExceptionTests.cpp" > - <OverallResults successes="0" failures="0" expectedFailures="0"/> + <Exception filename="projects/<exe-name>/ExceptionTests.cpp" > + unexpected exception + </Exception> + <OverallResults successes="0" failures="1" expectedFailures="0"/> </Section> - <OverallResult success="true"/> + <OverallResult success="false"/> </TestCase> <TestCase name="When unchecked exceptions are thrown, but caught, they do not affect the test" tags="[!throws]" filename="projects/<exe-name>/ExceptionTests.cpp" > <OverallResult success="true"/> @@ -10082,7 +10114,7 @@ spanner <OverallResult success="true"/> </Section> <OverallResult success="true"/> </TestCase> - <OverallResults successes="859" failures="84" expectedFailures="20"/> + <OverallResults successes="859" failures="90" expectedFailures="21"/> </Group> - <OverallResults successes="859" failures="83" expectedFailures="20"/> + <OverallResults successes="859" failures="89" expectedFailures="21"/> </Catch>