Progmasoft Git

catch3

Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.

Clonegit clone https://git.progmasoft.com/catch3

Decrease chance of false positive in random generator testing

Commit
0c223bb751ceb0c5d02e66ca2cb1d2a72bd3ba43
Author
Martin Hořeňovský ·
Parents
19ecad6f6815

Files · Download .tar.gz

Changes

 tests/SelfTest/IntrospectiveTests/GeneratorsImpl.tests.cpp | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/tests/SelfTest/IntrospectiveTests/GeneratorsImpl.tests.cpp b/tests/SelfTest/IntrospectiveTests/GeneratorsImpl.tests.cpp
index 81fb4ebc..81f5a811 100644
--- a/tests/SelfTest/IntrospectiveTests/GeneratorsImpl.tests.cpp
+++ b/tests/SelfTest/IntrospectiveTests/GeneratorsImpl.tests.cpp
@@ -341,8 +341,9 @@ TEST_CASE("Multiple random generators in one test case output different values",
             same += random1.get() == random2.get();
             random1.next(); random2.next();
         }
-        // 0.5% seems like a sane bound for random identical elements within 1000 runs
-        REQUIRE(same < 5);
+        // Because the previous low bound failed CI couple of times,
+        // we use a very high threshold of 20% before failure is reported.
+        REQUIRE(same < 200);
     }
     SECTION("Float") {
         auto random1 = Catch::Generators::random(0., 1000.);
@@ -352,7 +353,8 @@ TEST_CASE("Multiple random generators in one test case output different values",
             same += random1.get() == random2.get();
             random1.next(); random2.next();
         }
-        // 0.5% seems like a sane bound for random identical elements within 1000 runs
-        REQUIRE(same < 5);
+        // Because the previous low bound failed CI couple of times,
+        // we use a very high threshold of 20% before failure is reported.
+        REQUIRE(same < 200);
     }
 }