Progmasoft Git

catch3

Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.

Clonegit clone https://git.progmasoft.com/catch3

Add repeatability guarantees to faq.md

Commit
8beb74da8abd69c9cb083772edc230ea857b1229
Author
Martin Hořeňovský ·
Parents
e932bcf7a349

Files · Download .tar.gz

Changes

 docs/faq.md | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/docs/faq.md b/docs/faq.md
index c48944b7..0f303ee5 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -9,6 +9,8 @@
 [What is Catch2's API stability policy?](#what-is-catch2s-api-stability-policy)<br>
 [Does Catch2 support running tests in parallel?](#does-catch2-support-running-tests-in-parallel)<br>
 [Can I compile Catch2 into a dynamic library?](#can-i-compile-catch2-into-a-dynamic-library)<br>
+[What repeatability guarantees does Catch2 provide?](#what-repeatability-guarantees-does-catch2-provide)<br>
+
 
 ## How do I run global setup/teardown only if tests will be run?
 
@@ -69,6 +71,24 @@ library on platforms that default to public visibility, or with tooling
 support to force export Catch2's API.
 
 
+## What repeatability guarantees does Catch2 provide?
+
+There are two places where it is meaningful to talk about Catch2's
+repeatability guarantees without taking into account user-provided
+code. First one is in the test case shuffling, and the second one is
+the output from random generators.
+
+Test case shuffling is repeatable across different platforms since v2.12.0,
+and it is also generally repeatable across versions, but we might break
+it from time to time. E.g. we broke repeatability with previous versions
+in v2.13.4 so that test cases with similar names are shuffled better.
+
+Random generators currently rely on platform's stdlib, specifically
+the distributions from `<random>`. We thus provide no extra guarantee
+above what your platform does. **Important: `<random>`'s distributions
+are not specified to be repeatable across different platforms.**
+
+
 ---
 
 [Home](Readme.md#top)