Progmasoft Git

catch3

Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.

Clonegit clone https://git.progmasoft.com/catch3

Provide CTest a hint on test costs (and thus ordering)

When running tests in parallel, CTest runs the tests in decreasing
order of cost (time required), to get the largest speed up from
parallelism. However, the initial cost estimates for all tests are
0, and they are only updated after a test run. This works on a dev
machine, where the tests are ran over and over again, because
eventually the estimates become quite precise, but CI always does
a clean build with 0 estimates.

Because we have 2 slow tests, we want them to run first to avoid
losing parallelism. To do this, we provide them with a cost estimate
manually.
Commit
a3ffc20f57283178dab27d6cf0f1ff685ed6c4af
Author
Martin Hořeňovský ·
Parents
b86ab201541d

Files · Download .tar.gz

Changes

 tests/CMakeLists.txt            | 7 ++++++-
 tests/ExtraTests/CMakeLists.txt | 2 ++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 3c5e0612..4e6c98f2 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -187,7 +187,12 @@ set_tests_properties(FilteredSection-2 PROPERTIES FAIL_REGULAR_EXPRESSION "No te
 
 # AppVeyor has a Python 2.7 in path, but doesn't have .py files as autorunnable
 add_test(NAME ApprovalTests COMMAND ${PYTHON_EXECUTABLE} ${CATCH_DIR}/tools/scripts/approvalTests.py $<TARGET_FILE:SelfTest>)
-set_tests_properties(ApprovalTests PROPERTIES FAIL_REGULAR_EXPRESSION "Results differed")
+set_tests_properties(ApprovalTests
+  PROPERTIES
+    FAIL_REGULAR_EXPRESSION "Results differed"
+    COST 120 # We know that this is either the most, or second most,
+    # expensive test in the test suite, so we give it high estimate for CI runs
+)
 
 add_test(NAME RegressionCheck-1670 COMMAND $<TARGET_FILE:SelfTest> "#1670 regression check" -c A -r compact)
 set_tests_properties(RegressionCheck-1670 PROPERTIES PASS_REGULAR_EXPRESSION "Passed 1 test case with 2 assertions.")
diff --git a/tests/ExtraTests/CMakeLists.txt b/tests/ExtraTests/CMakeLists.txt
index b00ecf78..0001d940 100644
--- a/tests/ExtraTests/CMakeLists.txt
+++ b/tests/ExtraTests/CMakeLists.txt
@@ -128,6 +128,8 @@ set_tests_properties(
     BenchmarkingMacros
   PROPERTIES
     PASS_REGULAR_EXPRESSION "benchmark name                                  samples       iterations    estimated"
+    COST 80 # We know that this is either the most, or second most,
+   # expensive test in the test suite, so we give it high estimate for CI runs
 )
 
 # This test touches windows.h, so it should only be compiled under msvc