Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
diff options
context:
space:
mode:
authorMartin Hořeňovský2024-08-13 18:57:42 +0200
committerMartin Hořeňovský2024-08-13 19:13:03 +0200
commitfe483c056d9516d3550519c29fbf14a4d84d06d4 (patch)
treecbadf5902006a81a3abcc9753aa240c8ea792d95
parentb15158c1db35f4e191acc52c5b84ecfe1add5b38 (diff)
downloadcatch3-fe483c056d9516d3550519c29fbf14a4d84d06d4.tar.gz
Improve performance of JUnit reporter when handling passing assertions
This is done by no longer requiring all assertions to be seen by the JUnit reporter. Since the JUnit reporter never outputs all assertions, even with `-s`, the only difference from storing passing assertions in the `CumulativeReporterBase` was some bookkeeping in deciding which sections are relevant to the output. Given `TEST_CASE` that just runs many passing assertions, e.g. ``` TEST_CASE( "PERFORMANCE_TEST_CASE", "[.]" ) { for (size_t i = 0; i < 1000'000'000; ++i) { REQUIRE( i == i ); } } ``` the new JUnit reporter will finish in 5:47, using up ~7.7 MB of RAM. The old JUnit reporter would finish in 0:30, due to bad_alloc after using up 14.5 GB of RAM (the system has 16 GB total). If the total number of assertions is lowered to 10M, the old implementation uses up ~7.1 GB of RAM and finishes in 12 minutes. The new implementation still needs only ~7.7 MB of RAM, and finishes in 4 minutes. There is a slight downside in that the output is slightly different; the new implementation will include the `TEST_CASE` level section in output, even if it does not have its own assertion. In other words, given a `TEST_CASE` like this ``` TEST_CASE( "JsonWriter", "[JSON][JsonWriter]" ) { std::stringstream stream; SECTION( "Newly constructed JsonWriter does nothing" ) { Catch::JsonValueWriter writer{ stream }; REQUIRE( stream.str() == "" ); } SECTION( "Calling writeObject will create an empty pair of braces" ) { { auto writer = Catch::JsonValueWriter{ stream }.writeObject(); } REQUIRE( stream.str() == "{\n}" ); } } ``` the new implementation will output 3 `testcase` tags, 2 for the explicit `SECTION`s with tests, and 1 for the top level section. However, this can be worked-around if required, and the performance improvement is such that it is worth changing the current output, even if it needs to be fixed in the future. Closes #2897
-rw-r--r--src/catch2/reporters/catch_reporter_junit.cpp4
-rw-r--r--tests/SelfTest/Baselines/junit.sw.approved.txt162
-rw-r--r--tests/SelfTest/Baselines/junit.sw.multi.approved.txt162
3 files changed, 326 insertions, 2 deletions
diff --git a/src/catch2/reporters/catch_reporter_junit.cpp b/src/catch2/reporters/catch_reporter_junit.cpp
index 4589365c..27bdfe28 100644
--- a/src/catch2/reporters/catch_reporter_junit.cpp
+++ b/src/catch2/reporters/catch_reporter_junit.cpp
@@ -88,7 +88,7 @@ namespace Catch {
xml( m_stream )
{
m_preferences.shouldRedirectStdOut = true;
- m_preferences.shouldReportAllAssertions = true;
+ m_preferences.shouldReportAllAssertions = false;
m_shouldStoreSuccesfulAssertions = false;
}
@@ -198,7 +198,7 @@ namespace Catch {
if( !rootName.empty() )
name = rootName + '/' + name;
- if( sectionNode.hasAnyAssertions()
+ if ( sectionNode.stats.assertions.total() > 0
|| !sectionNode.stdOut.empty()
|| !sectionNode.stdErr.empty() ) {
XmlWriter::ScopedElement e = xml.scopedElement( "testcase" );
diff --git a/tests/SelfTest/Baselines/junit.sw.approved.txt b/tests/SelfTest/Baselines/junit.sw.approved.txt
index cfe9de63..710169c4 100644
--- a/tests/SelfTest/Baselines/junit.sw.approved.txt
+++ b/tests/SelfTest/Baselines/junit.sw.approved.txt
@@ -12,6 +12,7 @@
<testcase classname="<exe-name>.global" name="#1175 - Hidden Test" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1238" time="{duration}" status="run"/>
<testcase classname="<exe-name>.(Fixture_1245&lt;int, int>)" name="#1245" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="#1319: Sections can have description (even if it is not saved" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1319: Sections can have description (even if it is not saved/SectionName" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1403" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1455 - INFO and WARN can start with a linebreak" time="{duration}" status="run"/>
@@ -30,10 +31,12 @@ Nor would this
</testcase>
<testcase classname="<exe-name>.global" name="#1548" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1905 -- test spec parser properly clears internal state between compound tests" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="#1912 -- test spec parser handles escaping" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1912 -- test spec parser handles escaping/Various parentheses" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1912 -- test spec parser handles escaping/backslash in test name" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1913 - GENERATE inside a for loop should not keep recreating the generator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1913 - GENERATEs can share a line" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="#1938 - GENERATE after a section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - GENERATE after a section/A" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - GENERATE after a section/B" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#1938 - Section followed by flat generate" time="{duration}" status="run"/>
@@ -56,6 +59,7 @@ failure to init
at Generators.tests.cpp:<line number>
</error>
</testcase>
+ <testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<error type="TEST_CASE">
@@ -89,6 +93,7 @@ at Misc.tests.cpp:<line number>
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="#872" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported/Looped section 0" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported/Looped section 1" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="#961 -- Dynamically created sections should all be reported/Looped section 2" time="{duration}" status="run"/>
@@ -147,6 +152,7 @@ at Condition.tests.cpp:<line number>
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="'Not' checks that should succeed" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated/compare to true" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated/compare to false" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="(unimplemented) static bools can be evaluated/negation" time="{duration}" status="run"/>
@@ -323,6 +329,7 @@ with expansion:
at Class.tests.cpp:<line number>
</failure>
</testcase>
+ <testcase classname="<exe-name>.Persistent_Fixture" name="A TEST_CASE_PERSISTENT_FIXTURE based test run that succeeds" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Persistent_Fixture" name="A TEST_CASE_PERSISTENT_FIXTURE based test run that succeeds/First partial run" time="{duration}" status="run"/>
<testcase classname="<exe-name>.Persistent_Fixture" name="A TEST_CASE_PERSISTENT_FIXTURE based test run that succeeds/Second partial run" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A Template product test case - Foo&lt;float>" time="{duration}" status="run"/>
@@ -341,6 +348,7 @@ to infinity and beyond
at Misc.tests.cpp:<line number>
</failure>
</testcase>
+ <testcase classname="<exe-name>.global" name="A couple of nested sections followed by a failure/Outer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A couple of nested sections followed by a failure/Outer/Inner" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="A failing expression with a non streamable type is still captured" time="{duration}" status="run">
<failure message="&amp;o1 == &amp;o2" type="CHECK">
@@ -359,6 +367,7 @@ at Tricky.tests.cpp:<line number>
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="Absolute margin" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="An empty test with no assertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="An expression with side-effects should only be evaluated once" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="An unchecked exception reports the line of the last assertion" time="{duration}" status="run">
<error message="{Unknown expression after the reported line}">
@@ -376,6 +385,7 @@ at Exception.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Approximate comparisons with floats" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approximate comparisons with ints" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Approximate comparisons with mixed numeric types" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Arbitrary predicate matcher" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Arbitrary predicate matcher/Function pointer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Arbitrary predicate matcher/Lambdas + different type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Assertion macros support bit operators and bool conversions" time="{duration}" status="run"/>
@@ -383,30 +393,38 @@ at Exception.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Assertions then sections/A section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Assertions then sections/A section/Another section" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Assertions then sections/A section/Another other section" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Different argument ranges, same element type, default comparison" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Different argument ranges, same element type, custom comparison" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Different element type, custom comparisons" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Can handle type that requires ADL-found free function begin and end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Initialization with move only types" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Contains range matcher/Matching using matcher" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Basic use of the Empty range matcher" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Empty range matcher/Simple, std-provided containers" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Empty range matcher/Type with empty" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Basic use of the Empty range matcher/Type requires ADL found empty free function" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CAPTURE can deal with complex expressions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CAPTURE can deal with complex expressions involving commas" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CAPTURE parses string and character constants" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Capture and info messages" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Capture and info messages/Capture should stringify like assertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Capture and info messages/Info should NOT stringify the way assertions do" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="CaseInsensitiveEqualsTo is case insensitive" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CaseInsensitiveEqualsTo is case insensitive/Degenerate cases" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CaseInsensitiveEqualsTo is case insensitive/Plain comparisons" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="CaseInsensitiveLess is case insensitive" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CaseInsensitiveLess is case insensitive/Degenerate cases" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="CaseInsensitiveLess is case insensitive/Plain comparisons" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Character pretty printing" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Character pretty printing/Specifically escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Character pretty printing/General chars" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Character pretty printing/Low ASCII" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Clara::Arg supports single-arg parse the way Opt does" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Clara::Opt supports accept-many lambdas" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Clara::Opt supports accept-many lambdas/Parsing fails on multiple options without accept_many" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Clara::Opt supports accept-many lambdas/Parsing succeeds on multiple options with accept_many" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="ColourGuard behaviour" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="ColourGuard behaviour/ColourGuard is disengaged by default" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="ColourGuard behaviour/ColourGuard is engaged by op&lt;&lt;" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="ColourGuard behaviour/ColourGuard can be engaged explicitly" time="{duration}" status="run"/>
@@ -424,8 +442,10 @@ at Exception.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Comparisons between ints where one side is computed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Comparisons between unsigned ints and negative signed ints match c++ standard behaviour" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Comparisons with int literals don't warn when mixing signed/ unsigned" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Composed generic matchers shortcircuit" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Composed generic matchers shortcircuit/MatchAllOf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Composed generic matchers shortcircuit/MatchAnyOf" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Composed matchers shortcircuit" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Composed matchers shortcircuit/MatchAllOf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Composed matchers shortcircuit/MatchAnyOf" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Contains string matcher" time="{duration}" status="run">
@@ -445,6 +465,7 @@ with expansion:
at Matchers.tests.cpp:<line number>
</failure>
</testcase>
+ <testcase classname="<exe-name>.global" name="Copy and then generate a range" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Copy and then generate a range/from var and iterators" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Copy and then generate a range/From a temporary container" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Copy and then generate a range/Final validation" time="{duration}" status="run"/>
@@ -667,6 +688,7 @@ at Matchers.tests.cpp:<line number>
</testcase>
<testcase classname="<exe-name>.global" name="Exception matchers that succeed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Exception message can be matched" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Exception messages can be tested for" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Exception messages can be tested for/exact match" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Exception messages can be tested for/different case" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Exception messages can be tested for/wildcarded" time="{duration}" status="run"/>
@@ -712,6 +734,7 @@ at Message.tests.cpp:<line number>
</testcase>
<testcase classname="<exe-name>.global" name="Factorials are computed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Filter generator throws exception for empty generator" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Floating point matchers: double" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Relative" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Relative/Some subnormal values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Margin" time="{duration}" status="run"/>
@@ -719,6 +742,7 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Composed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/Constructor validation" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: double/IsNaN" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Floating point matchers: float" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Relative" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Relative/Some subnormal values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Margin" time="{duration}" status="run"/>
@@ -727,46 +751,64 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/Constructor validation" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Floating point matchers: float/IsNaN" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="GENERATE can combine literals and generators" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators -- adapters" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators -- adapters/Filtering by predicate" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Filtering by predicate/Basic usage" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Filtering by predicate/Throws if there are no matching values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Shortening a range" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators -- adapters/Transforming elements" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Transforming elements/Same type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Transforming elements/Different type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Transforming elements/Different deduced type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Repeating a generator" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces/Number of elements in source is divisible by chunk size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces/Number of elements in source is not divisible by chunk size" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces/Chunk size of zero" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- adapters/Chunking a generator into sized pieces/Throws on too small generators" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators -- simple" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- simple/one" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators -- simple/two" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Single value" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Preset values" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Generator combinator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Explicitly typed generator sequence" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Filter generator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Filter generator/Simple filtering" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Filter generator/Filter out multiple elements at the start and end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Filter generator/Throws on construction if it can't get initial element" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Take generator" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Take generator/Take less" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Take generator/Take more" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Map with explicit return type" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Map with deduced return type" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Repeat" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Repeat/Singular repeat" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Repeat/Actual repeat" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Range" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Range/Positive auto step" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive auto step/Integer" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Range/Negative auto step" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative auto step/Integer" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Integer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Integer/Exact" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Integer/Slightly over end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Integer/Slightly under end" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Floating Point" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Floating Point/Exact" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Floating Point/Slightly over end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Positive manual step/Floating Point/Slightly under end" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step/Integer" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step/Integer/Exact" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step/Integer/Slightly over end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Generators internals/Range/Negative manual step/Integer/Slightly under end" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Greater-than inequalities with different epsilons" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashers with different seed produce different hash with same test case" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashers with same seed produce same hash" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Hashing different test cases produces different result" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashing different test cases produces different result/Different test name" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashing different test cases produces different result/Different classname" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Hashing different test cases produces different result/Different tags" time="{duration}" status="run"/>
@@ -876,6 +918,7 @@ at Condition.tests.cpp:<line number>
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="Inequality checks that should succeed" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="JsonWriter" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Newly constructed JsonWriter does nothing" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Calling writeObject will create an empty pair of braces" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Calling writeObject with key will create an object to write the value" time="{duration}" status="run"/>
@@ -885,6 +928,7 @@ at Condition.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="JsonWriter/Moved from JsonObjectWriter shall not insert superfluous brace" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Moved from JsonArrayWriter shall not insert superfluous bracket" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter/Custom class shall be quoted" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/Quote in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/Backslash in a string is escaped" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="JsonWriter escapes charaters in strings properly/Forward slash in a string is **not** escaped" time="{duration}" status="run"/>
@@ -920,6 +964,9 @@ with expansion:
at Matchers.tests.cpp:<line number>
</failure>
</testcase>
+ <testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/1" time="{duration}" status="run">
+ <skipped message="TEST_CASE tagged with !mayfail"/>
+ </testcase>
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/1/A" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
@@ -927,6 +974,9 @@ FAILED:
at Condition.tests.cpp:<line number>
</failure>
</testcase>
+ <testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/2" time="{duration}" status="run">
+ <skipped message="TEST_CASE tagged with !mayfail"/>
+ </testcase>
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/2/A" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
@@ -934,6 +984,9 @@ FAILED:
at Condition.tests.cpp:<line number>
</failure>
</testcase>
+ <testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/1" time="{duration}" status="run">
+ <skipped message="TEST_CASE tagged with !mayfail"/>
+ </testcase>
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/1/B" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
@@ -941,6 +994,9 @@ FAILED:
at Condition.tests.cpp:<line number>
</failure>
</testcase>
+ <testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/2" time="{duration}" status="run">
+ <skipped message="TEST_CASE tagged with !mayfail"/>
+ </testcase>
<testcase classname="<exe-name>.global" name="Mayfail test case with nested sections/2/B" time="{duration}" status="run">
<skipped message="TEST_CASE tagged with !mayfail"/>
<failure type="FAIL">
@@ -1108,6 +1164,7 @@ at Condition.tests.cpp:<line number>
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="Ordering comparison checks that should succeed" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Our PCG implementation provides expected results for known seeds" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Our PCG implementation provides expected results for known seeds/Default seeded" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Our PCG implementation provides expected results for known seeds/Specific seed" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Output from all sections is reported/one" time="{duration}" status="run">
@@ -1125,9 +1182,11 @@ at Message.tests.cpp:<line number>
</failure>
</testcase>
<testcase classname="<exe-name>.global" name="Overloaded comma or address-of operators are not used" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Parse uints" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parse uints/proper inputs" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parse uints/Bad inputs" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsed tags are matched case insensitive" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/shard-count" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/Negative shard count reports error" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/Zero shard count reports error" time="{duration}" status="run"/>
@@ -1135,44 +1194,58 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/Negative shard index reports error" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing sharding-related cli flags/Shard index 0 is accepted" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing tags with non-alphabetical characters is pass-through" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Parsing warnings" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing warnings/NoAssertions" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing warnings/NoTests is no longer supported" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Parsing warnings/Combining multiple warnings" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Pointers can be compared to null" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Precision of floating point stringification can be set" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Precision of floating point stringification can be set/Floats" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Precision of floating point stringification can be set/Double" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Predicate matcher can accept const char*" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/empty args don't cause a crash" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/default - no arguments" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/test lists" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/test lists/Specify one test case using" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/test lists/Specify one test case exclusion using exclude:" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/test lists/Specify one test case exclusion using ~" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/-r/console" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/-r/xml" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/--reporter/junit" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/must match one of the available ones" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With output file" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/With Windows-like absolute path as output file" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/All with output files" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/Mixed output files and default output" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/reporter/Multiple reporters/cannot have multiple reporters with default output" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/debugger" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/debugger/-b" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/debugger/--break" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/abort" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/-a aborts after first failure" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/-x 2 aborts after two failures" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/-x must be numeric" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/wait-for-keypress" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/wait-for-keypress/Accepted options" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/abort/wait-for-keypress/invalid options are reported" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/nothrow" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/nothrow/-e" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/nothrow/--nothrow" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/output filename" time="{duration}" status="run"/>
<testcase classn