catch3
Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.
Clonegit clone https://git.progmasoft.com/catch3
Add tests for #1514
- Commit
28db5ed4c98db6a47d7a04afbd6b7ad9ba73e401- Author
- Martin Hořeňovský ·
- Parents
7d2451f11945
Changes
projects/SelfTest/Baselines/compact.sw.approved.txt | 5 ++++- projects/SelfTest/Baselines/console.std.approved.txt | 16 ++++++++++++++-- projects/SelfTest/Baselines/console.sw.approved.txt | 16 ++++++++++++++-- projects/SelfTest/Baselines/console.swa4.approved.txt | 19 +++++++++++++------ projects/SelfTest/Baselines/junit.sw.approved.txt | 16 +++++++++++++++- projects/SelfTest/Baselines/xml.sw.approved.txt | 17 +++++++++++++++-- projects/SelfTest/UsageTests/Tricky.tests.cpp | 8 ++++++++ 7 files changed, 83 insertions(+), 14 deletions(-) diff --git a/projects/SelfTest/Baselines/compact.sw.approved.txt b/projects/SelfTest/Baselines/compact.sw.approved.txt index 31c7ee0d..494fba13 100644 --- a/projects/SelfTest/Baselines/compact.sw.approved.txt +++ b/projects/SelfTest/Baselines/compact.sw.approved.txt @@ -14,6 +14,9 @@ Compilation.tests.cpp:<line number>: passed: std::memcmp(uarr, "123", sizeof(uar Compilation.tests.cpp:<line number>: passed: std::memcmp(sarr, "456", sizeof(sarr)) == 0 for: 0 == 0 with 2 messages: 'uarr := "123"' and 'sarr := "456"' Compilation.tests.cpp:<line number>: passed: Compilation.tests.cpp:<line number>: passed: h1 == h2 for: [1403 helper] == [1403 helper] +Nor would this +This would not be caught previously +Tricky.tests.cpp:<line number>: failed: explicitly with 1 message: '1514' Compilation.tests.cpp:<line number>: passed: std::is_same<TypeList<int>, TypeList<int>>::value for: true Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42' with 1 message: 'expected exception' Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42'; expression was: thisThrows() with 1 message: 'expected exception' @@ -1459,5 +1462,5 @@ Misc.tests.cpp:<line number>: passed: v.size() == 5 for: 5 == 5 Misc.tests.cpp:<line number>: passed: v.capacity() >= 5 for: 5 >= 5 Misc.tests.cpp:<line number>: passed: Misc.tests.cpp:<line number>: passed: -Failed 69 test cases, failed 129 assertions. +Failed 70 test cases, failed 130 assertions. diff --git a/projects/SelfTest/Baselines/console.std.approved.txt b/projects/SelfTest/Baselines/console.std.approved.txt index 9be03dcd..2353c07e 100644 --- a/projects/SelfTest/Baselines/console.std.approved.txt +++ b/projects/SelfTest/Baselines/console.std.approved.txt @@ -1,3 +1,5 @@ +Nor would this +This would not be caught previously ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <exe-name> is a <version> host application. @@ -5,6 +7,16 @@ Run with -? for options Randomness seeded to: 1 +------------------------------------------------------------------------------- +#1514: stderr/stdout is not captured in tests aborted by an exception +------------------------------------------------------------------------------- +Tricky.tests.cpp:<line number> +............................................................................... + +Tricky.tests.cpp:<line number>: FAILED: +explicitly with message: + 1514 + ------------------------------------------------------------------------------- #748 - captures with unexpected exceptions outside assertions @@ -1174,6 +1186,6 @@ due to unexpected exception with message: Why would you throw a std::string? =============================================================================== -test cases: 246 | 186 passed | 56 failed | 4 failed as expected -assertions: 1380 | 1244 passed | 115 failed | 21 failed as expected +test cases: 247 | 186 passed | 57 failed | 4 failed as expected +assertions: 1381 | 1244 passed | 116 failed | 21 failed as expected diff --git a/projects/SelfTest/Baselines/console.sw.approved.txt b/projects/SelfTest/Baselines/console.sw.approved.txt index 23f91912..03b049f2 100644 --- a/projects/SelfTest/Baselines/console.sw.approved.txt +++ b/projects/SelfTest/Baselines/console.sw.approved.txt @@ -133,6 +133,18 @@ Compilation.tests.cpp:<line number>: PASSED: with expansion: [1403 helper] == [1403 helper] +Nor would this +This would not be caught previously +------------------------------------------------------------------------------- +#1514: stderr/stdout is not captured in tests aborted by an exception +------------------------------------------------------------------------------- +Tricky.tests.cpp:<line number> +............................................................................... + +Tricky.tests.cpp:<line number>: FAILED: +explicitly with message: + 1514 + ------------------------------------------------------------------------------- #1548 ------------------------------------------------------------------------------- @@ -10709,6 +10721,6 @@ Misc.tests.cpp:<line number> Misc.tests.cpp:<line number>: PASSED: =============================================================================== -test cases: 246 | 173 passed | 69 failed | 4 failed as expected -assertions: 1394 | 1244 passed | 129 failed | 21 failed as expected +test cases: 247 | 173 passed | 70 failed | 4 failed as expected +assertions: 1395 | 1244 passed | 130 failed | 21 failed as expected diff --git a/projects/SelfTest/Baselines/console.swa4.approved.txt b/projects/SelfTest/Baselines/console.swa4.approved.txt index 63d7e28f..770b3273 100644 --- a/projects/SelfTest/Baselines/console.swa4.approved.txt +++ b/projects/SelfTest/Baselines/console.swa4.approved.txt @@ -133,6 +133,18 @@ Compilation.tests.cpp:<line number>: PASSED: with expansion: [1403 helper] == [1403 helper] +Nor would this +This would not be caught previously +------------------------------------------------------------------------------- +#1514: stderr/stdout is not captured in tests aborted by an exception +------------------------------------------------------------------------------- +Tricky.tests.cpp:<line number> +............................................................................... + +Tricky.tests.cpp:<line number>: FAILED: +explicitly with message: + 1514 + ------------------------------------------------------------------------------- #1548 ------------------------------------------------------------------------------- @@ -328,12 +340,7 @@ Condition.tests.cpp:<line number>: FAILED: with expansion: false -Condition.tests.cpp:<line number>: FAILED: - CHECK_FALSE( true ) -with expansion: - !true - =============================================================================== -test cases: 16 | 13 passed | 1 failed | 2 failed as expected +test cases: 17 | 13 passed | 2 failed | 2 failed as expected assertions: 40 | 33 passed | 4 failed | 3 failed as expected diff --git a/projects/SelfTest/Baselines/junit.sw.approved.txt b/projects/SelfTest/Baselines/junit.sw.approved.txt index 2c8a6495..b99e0151 100644 --- a/projects/SelfTest/Baselines/junit.sw.approved.txt +++ b/projects/SelfTest/Baselines/junit.sw.approved.txt @@ -4,7 +4,7 @@ <property name="random-seed" value="1"/> </properties> loose text artifact - <testsuite name="<exe-name>" errors="17" failures="113" tests="1395" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> + <testsuite name="<exe-name>" errors="17" failures="114" tests="1396" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}"> <testcase classname="<exe-name>.global" name="# A test name that starts with a #" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1005: Comparing pointer to int and long (NULL can be either on various systems)" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1027" time="{duration}"/> @@ -13,6 +13,18 @@ loose text artifact <testcase classname="<exe-name>.global" name="#1238" time="{duration}"/> <testcase classname="<exe-name>.(Fixture_1245<int, int>)" name="#1245" time="{duration}"/> <testcase classname="<exe-name>.global" name="#1403" time="{duration}"/> + <testcase classname="<exe-name>.global" name="#1514: stderr/stdout is not captured in tests aborted by an exception" time="{duration}"> + <failure type="FAIL"> +1514 +Tricky.tests.cpp:<line number> + </failure> + <system-out> +This would not be caught previously + </system-out> + <system-err> +Nor would this + </system-err> + </testcase> <testcase classname="<exe-name>.global" name="#1548" time="{duration}"/> <testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}"> <error type="TEST_CASE"> @@ -998,11 +1010,13 @@ Exception.tests.cpp:<line number> <testcase classname="<exe-name>.global" name="xmlentitycheck/embedded xml: <test>it should be possible to embed xml characters, such as <, " or &, or even whole <xml>documents</xml> within an attribute</test>" time="{duration}"/> <testcase classname="<exe-name>.global" name="xmlentitycheck/encoded chars: these should all be encoded: &&&"""<<<&"<<&"" time="{duration}"/> <system-out> +This would not be caught previously A string sent directly to stdout Message from section one Message from section two </system-out> <system-err> +Nor would this A string sent directly to stderr A string sent to stderr via clog </system-err> diff --git a/projects/SelfTest/Baselines/xml.sw.approved.txt b/projects/SelfTest/Baselines/xml.sw.approved.txt index 60257c25..a1785047 100644 --- a/projects/SelfTest/Baselines/xml.sw.approved.txt +++ b/projects/SelfTest/Baselines/xml.sw.approved.txt @@ -142,6 +142,19 @@ </Expression> <OverallResult success="true"/> </TestCase> + <TestCase name="#1514: stderr/stdout is not captured in tests aborted by an exception" tags="[.][output-capture][regression]" filename="projects/<exe-name>/UsageTests/Tricky.tests.cpp" > + <Failure filename="projects/<exe-name>/UsageTests/Tricky.tests.cpp" > + 1514 + </Failure> + <OverallResult success="false"> + <StdOut> +This would not be caught previously + </StdOut> + <StdErr> +Nor would this + </StdErr> + </OverallResult> + </TestCase> <TestCase name="#1548" tags="[compilation]" filename="projects/<exe-name>/UsageTests/Compilation.tests.cpp" > <Expression success="true" type="REQUIRE" filename="projects/<exe-name>/UsageTests/Compilation.tests.cpp" > <Original> @@ -12959,7 +12972,7 @@ loose text artifact </Section> <OverallResult success="true"/> </TestCase> - <OverallResults successes="1244" failures="130" expectedFailures="21"/> + <OverallResults successes="1244" failures="131" expectedFailures="21"/> </Group> - <OverallResults successes="1244" failures="129" expectedFailures="21"/> + <OverallResults successes="1244" failures="130" expectedFailures="21"/> </Catch> diff --git a/projects/SelfTest/UsageTests/Tricky.tests.cpp b/projects/SelfTest/UsageTests/Tricky.tests.cpp index b0363d42..44863c49 100644 --- a/projects/SelfTest/UsageTests/Tricky.tests.cpp +++ b/projects/SelfTest/UsageTests/Tricky.tests.cpp @@ -19,6 +19,7 @@ #include <stdio.h> #include <sstream> +#include <iostream> /////////////////////////////////////////////////////////////////////////////// TEST_CASE @@ -426,3 +427,10 @@ TEST_CASE( "Bitfields can be captured (#1027)" ) { REQUIRE( y.v == 0 ); REQUIRE( 0 == y.v ); } + +TEST_CASE("#1514: stderr/stdout is not captured in tests aborted by an exception", "[output-capture][regression][.]") { + std::cout << "This would not be caught previously\n"; + std::clog << "Nor would this\n"; + // FAIL aborts the test by throwing a Catch exception + FAIL("1514"); +}