Merge pull request #15 from Progmasoft/feature/catch3-runner

Replace Catch2's runner with the Catch3 runner

1 files changed, 26 insertions, 58 deletions

diff --git a/docs/usage-tips.md b/docs/usage-tips.md
index 0ed3d8d3..18e9cd11 100644
--- a/docs/usage-tips.md
+++ b/docs/usage-tips.md
@@ -9,77 +9,45 @@
 Your tests should be run in a manner roughly equivalent with:
 
 ```
-./tests --order rand --warn NoAssertions
+./tests --order=random --warn=no-assertions
 ```
 
-Notice that all the tests are run in a large batch, their relative order
-is randomized, and that you ask Catch2 to fail test whose leaf-path
-does not contain an assertion.
+The [Catch3 runner](progmasoft-runner.md#top) runs them in parallel worker
+processes, retries failures once alone to tell deterministic, flaky and
+order-dependent failures apart, and fails tests whose leaf path contains no
+assertion. Random order shakes out interference between tests that run in
+the same worker; when a test fails only after another, the runner names the
+other.
 
-The reason I recommend running all your tests in the same process is that
-this exposes your tests to interference from their runs. This can be both
-positive interference, where the changes in global state from previous
-test allow later tests to pass, but also negative interference, where
-changes in global state from previous test causes later tests to fail.
-
-In my experience, interference, especially destructive interference,
-usually comes from errors in the code under test, rather than the tests
-themselves. This means that by allowing interference to happen, our tests
-can find these issues. Obviously, to shake out interference coming from
-different orderings of tests, the test order also need to be shuffled
-between runs.
-
-However, running all tests in a single batch eventually becomes impractical
-as they will take too long to run, and you will want to run your tests
-in parallel.
+Interference, especially destructive interference, usually comes from
+errors in the code under test, rather than the tests themselves. Tests that
+share state outside the process, such as a file at a fixed path or a
+database, need `[!serial]` or `[!resource:NAME]`; the runner gives every
+test its own temporary directory.
 
 
 <a id="parallel-tests"></a>
 ## Running tests in parallel
 
-There are multiple ways of running tests in parallel, with various level
-of structure. If you are using CMake and CTest, then we provide a helper
-function [`catch_discover_tests`](cmake-integration.md#automatic-test-registration)
-that registers each Catch2 `TEST_CASE` as a single CTest test, which
-is then run in a separate process. This is an easy way to set up parallel
-tests if you are already using CMake & CTest to run your tests, but you
-will lose the advantage of running tests in batches.
-
-
-Catch2 also supports [splitting tests in a binary into multiple
-shards](command-line.md#test-sharding). This can be used by any test
-runner to run batches of tests in parallel. Do note that when selecting
-on the number of shards, you should have more shards than there are cores,
-to avoid issues with long-running tests getting accidentally grouped in
-the same shard, and causing long-tailed execution time.
-
-**Note that naively composing sharding and random ordering of tests will break.**
-
-Invoking Catch2 test executable like this
+The runner already runs the tests of a binary in parallel, one worker per
+hardware thread by default (`--jobs`). Across machines, split the run into
+[shards](progmasoft-runner.md#choosing-and-ordering-tests):
 
 ```text
-./tests --order rand --shard-index 0 --shard-count 3
-./tests --order rand --shard-index 1 --shard-count 3
-./tests --order rand --shard-index 2 --shard-count 3
+./tests --shard=0/3
+./tests --shard=1/3
+./tests --shard=2/3
 ```
 
-does not guarantee covering all tests inside the executable, because
-each invocation will have its own random seed, thus it will have its own
-random order of tests and thus the partitioning of tests into shards will
-be different as well.
-
-To do this properly, you need the individual shards to share the random
-seed, e.g.
-```text
-./tests --order rand --shard-index 0 --shard-count 3 --rng-seed 0xBEEF
-./tests --order rand --shard-index 1 --shard-count 3 --rng-seed 0xBEEF
-./tests --order rand --shard-index 2 --shard-count 3 --rng-seed 0xBEEF
-```
+A test goes to the shard a stable hash of its name chooses, so the shards
+cover every test exactly once whatever the order or seed of each run. With
+`--shard-durations=FILE`, a JSON report or history of an earlier run, the
+shards are balanced by duration instead.
 
-Catch2 actually provides a helper to automatically register multiple shards
-as CTest tests, with shared random seed that changes each CTest invocation.
-For details look at the documentation of
-[`CatchShardTests.cmake` CMake script](cmake-integration.md#catchshardtestscmake).
+If you use CMake and CTest, [`catch_discover_tests`](cmake-integration.md#automatic-test-registration)
+registers each `TEST_CASE` as a CTest test, and
+[`CatchShardTests.cmake`](cmake-integration.md#catchshardtestscmake)
+registers shards.
 
 
 ## Organizing tests into binaries