Progmasoft Git

catch3

Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.

Clonegit clone https://git.progmasoft.com/catch3

catch_discover_tests registers tests with --order decl

Catch2 defaults to random test execution order, which has to process
all tests present in the binary, even if they will be filtered down
to just 1 later. For the `SelfTest` binary, the overhead from this is
about 20% in Debug configuration and 5% in Release.

By registering the CTest tests with `--order decl`, we avoid this
overhead.
Commit
6534dbc3ef3a162111830d9b5d8a9092d378336d
Author
Martin Hořeňovský ·
Parents
b7f1a83ff0c3

Files · Download .tar.gz

Changes

 extras/CatchAddTests.cmake                            | 3 ++-
 tests/TestScripts/DiscoverTests/VerifyRegistration.py | 7 ++++---
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/extras/CatchAddTests.cmake b/extras/CatchAddTests.cmake
index f4840196..aba0d001 100644
--- a/extras/CatchAddTests.cmake
+++ b/extras/CatchAddTests.cmake
@@ -404,8 +404,9 @@ function(catch_discover_tests_impl)
     "${_TEST_EXECUTABLE}"
   )
   prepare_command_fragment(_args_reporter_fragment
-    ${extra_args}
+    "${extra_args}"
     "${reporter_arg}"
+    "--order decl"
   )
 
   # `set_tests_properties` calls are
diff --git a/tests/TestScripts/DiscoverTests/VerifyRegistration.py b/tests/TestScripts/DiscoverTests/VerifyRegistration.py
index d33006dd..e213bd3d 100644
--- a/tests/TestScripts/DiscoverTests/VerifyRegistration.py
+++ b/tests/TestScripts/DiscoverTests/VerifyRegistration.py
@@ -92,7 +92,7 @@ def get_test_names(build_path: str) -> List[TestInfo]:
         with open(fname, mode='r', encoding='utf-8') as file:
             test_listing = json.load(file)
 
-    assert test_listing['version'] == 2
+    assert test_listing['version'] == 2, test_listing['version']
 
     tests = []
     for test in test_listing['listings']['tests']:
@@ -129,10 +129,11 @@ def extract_tests_from_ctest(ctest_output) -> List[TestInfo]:
     test_infos = []
     for test in tests:
         test_command = test['command']
-        # First part of the command is the binary, second is the filter.
+        # First part of the command is the binary, second is the filter,
+        # third is the `--order decl` arg for faster startup.
         # If there are less, registration has failed. If there are more,
         # registration has changed and the script needs updating.
-        assert len(test_command) == 2
+        assert len(test_command) == 3, test_command
         test_name = test_command[1]
         labels = []
         for prop in test['properties']: