Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
path: root/tests/SelfTest/Baselines
diff options
context:
space:
mode:
authorBen Dunkin2021-07-11 12:46:05 -0700
committerMartin Hořeňovský2021-10-27 17:24:30 +0200
commit3087e19cc72ccb6caddc42ec2f39eebc56160754 (patch)
tree36d3adb84f1876a7136c169b219637343055a44d /tests/SelfTest/Baselines
parent6456ee8b01c6133e553b99da2f7420e7b20da7ec (diff)
downloadcatch3-3087e19cc72ccb6caddc42ec2f39eebc56160754.tar.gz
Allow test sharding for e.g. Bazel test sharding feature
This greatly simplifies running Catch2 tests in single binary in parallel from external test runners. Instead of having to shard the tests by tags/test names, an external test runner can now just ask for test shard 2 (out of X), and execute that in single process, without having to know what tests are actually in the shard. Note that sharding also applies to test listing, and happens after tests were ordered according to the `--order` feature.
Diffstat (limited to 'tests/SelfTest/Baselines')
-rw-r--r--tests/SelfTest/Baselines/compact.sw.approved.txt6
-rw-r--r--tests/SelfTest/Baselines/console.std.approved.txt2
-rw-r--r--tests/SelfTest/Baselines/console.sw.approved.txt57
-rw-r--r--tests/SelfTest/Baselines/junit.sw.approved.txt5
-rw-r--r--tests/SelfTest/Baselines/sonarqube.sw.approved.txt3
-rw-r--r--tests/SelfTest/Baselines/tap.sw.approved.txt14
-rw-r--r--tests/SelfTest/Baselines/xml.sw.approved.txt68
7 files changed, 150 insertions, 5 deletions
diff --git a/tests/SelfTest/Baselines/compact.sw.approved.txt b/tests/SelfTest/Baselines/compact.sw.approved.txt
index dfb08230..3f39422e 100644
--- a/tests/SelfTest/Baselines/compact.sw.approved.txt
+++ b/tests/SelfTest/Baselines/compact.sw.approved.txt
@@ -1273,6 +1273,12 @@ CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "--benchmark-no-ana
CmdLine.tests.cpp:<line number>: passed: config.benchmarkNoAnalysis for: true
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "--benchmark-warmup-time=10" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.benchmarkWarmupTime == 10 for: 10 == 10
+CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "--shard-count=8"}) for: {?}
+CmdLine.tests.cpp:<line number>: passed: config.shardCount == 8 for: 8 == 8
+CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "--shard-index=2"}) for: {?}
+CmdLine.tests.cpp:<line number>: passed: config.shardIndex == 2 for: 2 == 2
+CmdLine.tests.cpp:<line number>: passed: !result for: true
+CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring( "The shard count must be greater than 0" ) for: "The shard count must be greater than 0" contains: "The shard count must be greater than 0"
Misc.tests.cpp:<line number>: passed: std::tuple_size<TestType>::value >= 1 for: 3 >= 1
Misc.tests.cpp:<line number>: passed: std::tuple_size<TestType>::value >= 1 for: 2 >= 1
Misc.tests.cpp:<line number>: passed: std::tuple_size<TestType>::value >= 1 for: 1 >= 1
diff --git a/tests/SelfTest/Baselines/console.std.approved.txt b/tests/SelfTest/Baselines/console.std.approved.txt
index f7320bd7..af7fb09c 100644
--- a/tests/SelfTest/Baselines/console.std.approved.txt
+++ b/tests/SelfTest/Baselines/console.std.approved.txt
@@ -1427,5 +1427,5 @@ due to unexpected exception with message:
===============================================================================
test cases: 373 | 296 passed | 70 failed | 7 failed as expected
-assertions: 2115 | 1959 passed | 129 failed | 27 failed as expected
+assertions: 2121 | 1965 passed | 129 failed | 27 failed as expected
diff --git a/tests/SelfTest/Baselines/console.sw.approved.txt b/tests/SelfTest/Baselines/console.sw.approved.txt
index 695a30ef..189944cb 100644
--- a/tests/SelfTest/Baselines/console.sw.approved.txt
+++ b/tests/SelfTest/Baselines/console.sw.approved.txt
@@ -9391,6 +9391,61 @@ with expansion:
10 == 10
-------------------------------------------------------------------------------
+Process can be configured on command line
+ Sharding options
+ shard-count
+-------------------------------------------------------------------------------
+CmdLine.tests.cpp:<line number>
+...............................................................................
+
+CmdLine.tests.cpp:<line number>: PASSED:
+ CHECK( cli.parse({ "test", "--shard-count=8"}) )
+with expansion:
+ {?}
+
+CmdLine.tests.cpp:<line number>: PASSED:
+ REQUIRE( config.shardCount == 8 )
+with expansion:
+ 8 == 8
+
+-------------------------------------------------------------------------------
+Process can be configured on command line
+ Sharding options
+ shard-index
+-------------------------------------------------------------------------------
+CmdLine.tests.cpp:<line number>
+...............................................................................
+
+CmdLine.tests.cpp:<line number>: PASSED:
+ CHECK( cli.parse({ "test", "--shard-index=2"}) )
+with expansion:
+ {?}
+
+CmdLine.tests.cpp:<line number>: PASSED:
+ REQUIRE( config.shardIndex == 2 )
+with expansion:
+ 2 == 2
+
+-------------------------------------------------------------------------------
+Process can be configured on command line
+ Sharding options
+ Zero shard-count
+-------------------------------------------------------------------------------
+CmdLine.tests.cpp:<line number>
+...............................................................................
+
+CmdLine.tests.cpp:<line number>: PASSED:
+ CHECK( !result )
+with expansion:
+ true
+
+CmdLine.tests.cpp:<line number>: PASSED:
+ CHECK_THAT( result.errorMessage(), ContainsSubstring( "The shard count must be greater than 0" ) )
+with expansion:
+ "The shard count must be greater than 0" contains: "The shard count must be
+ greater than 0"
+
+-------------------------------------------------------------------------------
Product with differing arities - std::tuple<int, double, float>
-------------------------------------------------------------------------------
Misc.tests.cpp:<line number>
@@ -17040,5 +17095,5 @@ Misc.tests.cpp:<line number>: PASSED:
===============================================================================
test cases: 373 | 280 passed | 86 failed | 7 failed as expected
-assertions: 2132 | 1959 passed | 146 failed | 27 failed as expected
+assertions: 2138 | 1965 passed | 146 failed | 27 failed as expected
diff --git a/tests/SelfTest/Baselines/junit.sw.approved.txt b/tests/SelfTest/Baselines/junit.sw.approved.txt
index c2486391..ab55d93c 100644
--- a/tests/SelfTest/Baselines/junit.sw.approved.txt
+++ b/tests/SelfTest/Baselines/junit.sw.approved.txt
@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuitesloose text artifact
>
- <testsuite name="<exe-name>" errors="17" failures="129" tests="2132" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
+ <testsuite name="<exe-name>" errors="17" failures="129" tests="2138" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties>
<property name="random-seed" value="1"/>
<property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/>
@@ -1096,6 +1096,9 @@ Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/confidence-interval" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/no-analysis" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/warmup-time" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/Sharding options/shard-count" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/Sharding options/shard-index" time="{duration}" status="run"/>
+ <testcase classname="<exe-name>.global" name="Process can be configured on command line/Sharding options/Zero shard-count" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Product with differing arities - std::tuple&lt;int, double, float>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Product with differing arities - std::tuple&lt;int, double>" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Product with differing arities - std::tuple&lt;int>" time="{duration}" status="run"/>
diff --git a/tests/SelfTest/Baselines/sonarqube.sw.approved.txt b/tests/SelfTest/Baselines/sonarqube.sw.approved.txt
index 18bca507..4f5e0b12 100644
--- a/tests/SelfTest/Baselines/sonarqube.sw.approved.txt
+++ b/tests/SelfTest/Baselines/sonarqube.sw.approved.txt
@@ -75,6 +75,9 @@
<testCase name="Process can be configured on command line/Benchmark options/confidence-interval" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/no-analysis" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/warmup-time" duration="{duration}"/>
+ <testCase name="Process can be configured on command line/Sharding options/shard-count" duration="{duration}"/>
+ <testCase name="Process can be configured on command line/Sharding options/shard-index" duration="{duration}"/>
+ <testCase name="Process can be configured on command line/Sharding options/Zero shard-count" duration="{duration}"/>
<testCase name="Test with special, characters &quot;in name" duration="{duration}"/>
</file>
<file path="tests/<exe-name>/IntrospectiveTests/FloatingPoint.tests.cpp">
diff --git a/tests/SelfTest/Baselines/tap.sw.approved.txt b/tests/SelfTest/Baselines/tap.sw.approved.txt
index aad246f5..e5247b5f 100644
--- a/tests/SelfTest/Baselines/tap.sw.approved.txt
+++ b/tests/SelfTest/Baselines/tap.sw.approved.txt
@@ -2468,6 +2468,18 @@ ok {test-number} - config.benchmarkNoAnalysis for: true
ok {test-number} - cli.parse({ "test", "--benchmark-warmup-time=10" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.benchmarkWarmupTime == 10 for: 10 == 10
+# Process can be configured on command line
+ok {test-number} - cli.parse({ "test", "--shard-count=8"}) for: {?}
+# Process can be configured on command line
+ok {test-number} - config.shardCount == 8 for: 8 == 8
+# Process can be configured on command line
+ok {test-number} - cli.parse({ "test", "--shard-index=2"}) for: {?}
+# Process can be configured on command line
+ok {test-number} - config.shardIndex == 2 for: 2 == 2
+# Process can be configured on command line
+ok {test-number} - !result for: true
+# Process can be configured on command line
+ok {test-number} - result.errorMessage(), ContainsSubstring( "The shard count must be greater than 0" ) for: "The shard count must be greater than 0" contains: "The shard count must be greater than 0"
# Product with differing arities - std::tuple<int, double, float>
ok {test-number} - std::tuple_size<TestType>::value >= 1 for: 3 >= 1
# Product with differing arities - std::tuple<int, double>
@@ -4266,5 +4278,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} -
# xmlentitycheck
ok {test-number} -
-1..2132
+1..2138
diff --git a/tests/SelfTest/Baselines/xml.sw.approved.txt b/tests/SelfTest/Baselines/xml.sw.approved.txt
index 419941de..256e43fd 100644
--- a/tests/SelfTest/Baselines/xml.sw.approved.txt
+++ b/tests/SelfTest/Baselines/xml.sw.approved.txt
@@ -11469,6 +11469,72 @@ Nor would this
</Section>
<OverallResults successes="2" failures="0" expectedFailures="0"/>
</Section>
+ <Section name="Sharding options" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Section name="shard-count" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Original>
+ cli.parse({ "test", "--shard-count=8"})
+ </Original>
+ <Expanded>
+ {?}
+ </Expanded>
+ </Expression>
+ <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Original>
+ config.shardCount == 8
+ </Original>
+ <Expanded>
+ 8 == 8
+ </Expanded>
+ </Expression>
+ <OverallResults successes="2" failures="0" expectedFailures="0"/>
+ </Section>
+ <OverallResults successes="2" failures="0" expectedFailures="0"/>
+ </Section>
+ <Section name="Sharding options" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Section name="shard-index" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Original>
+ cli.parse({ "test", "--shard-index=2"})
+ </Original>
+ <Expanded>
+ {?}
+ </Expanded>
+ </Expression>
+ <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Original>
+ config.shardIndex == 2
+ </Original>
+ <Expanded>
+ 2 == 2
+ </Expanded>
+ </Expression>
+ <OverallResults successes="2" failures="0" expectedFailures="0"/>
+ </Section>
+ <OverallResults successes="2" failures="0" expectedFailures="0"/>
+ </Section>
+ <Section name="Sharding options" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Section name="Zero shard-count" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Expression success="true" type="CHECK" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Original>
+ !result
+ </Original>
+ <Expanded>
+ true
+ </Expanded>
+ </Expression>
+ <Expression success="true" type="CHECK_THAT" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+ <Original>
+ result.errorMessage(), ContainsSubstring( "The shard count must be greater than 0" )
+ </Original>
+ <Expanded>
+ "The shard count must be greater than 0" contains: "The shard count must be greater than 0"
+ </Expanded>
+ </Expression>
+ <OverallResults successes="2" failures="0" expectedFailures="0"/>
+ </Section>
+ <OverallResults successes="2" failures="0" expectedFailures="0"/>
+ </Section>
<OverallResult success="true"/>
</TestCase>
<TestCase name="Product with differing arities - std::tuple&lt;int, double, float>" tags="[product][template]" filename="tests/<exe-name>/UsageTests/Misc.tests.cpp" >
@@ -20030,6 +20096,6 @@ loose text artifact
</Section>
<OverallResult success="true"/>
</TestCase>
- <OverallResults successes="1959" failures="146" expectedFailures="27"/>
+ <OverallResults successes="1965" failures="146" expectedFailures="27"/>
<OverallResultsCases successes="280" failures="86" expectedFailures="7"/>
</Catch2TestRun>