Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
path: root/docs
diff options
context:
space:
mode:
authorLeitwolf112026-10-23 23:25:41 +0300
committerLeitwolf112026-10-23 23:25:41 +0300
commit93d4e7dbebab867e0a7f59fc25308460a7a76280 (patch)
tree54c568a3063d49e21f039b6d38f40f51be7991f0 /docs
parent83c6fc3ac13abe83fa39f7b5f8559ccaf4412233 (diff)
downloadcatch3-93d4e7dbebab867e0a7f59fc25308460a7a76280.tar.gz
Replace Catch2's runner with the Catch3 runner
Catch3 keeps Catch2's API for writing tests (test cases, sections, assertions, matchers, generators, listeners, benchmarks) and replaces its session, command line and reporters with a runner of its own. The runner starts the test binary again as worker processes that take the next test when they are free. A crash, an exit or a hang ends one test, not the run. On top of that it adds: - retries that tell deterministic, flaky and order-dependent failures apart, and a bisection that names the test leaving state behind; - a history of verdicts and durations, kept under a file lock, that drives automatic quarantine, adaptive time limits, predictive ordering, incremental runs, failure replay and analytics; - selection by change, past failure or prediction, dependency closure, and sharding by a stable hash or by recorded durations; - tags for serial and isolated tests, shared resources, dependencies, time limits and retries; - a failure category, a signature, an analysis and a replay command for every failure; - console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome trace reports, and a versioned discovery listing (--list=json); - a temporary directory per test (TMPDIR, TMP, TEMP and TestTemporaryDirectory()); - Bazel's test environment: XML output file, test filter, sharding, seed, premature exit file and undeclared outputs. The command line is parsed by a hand-written, typed parser. Catch::Session remains as a thin facade over the runner for custom main functions. The result model in Progmasoft/Catch3/Results.hpp was rewritten around tests and attempts, and the JUnit writer now writes it. The property engine's example database is updated under a file lock with an atomic rename, so parallel workers can write it at the same time. Removed: Clara, the command-line and reporter-spec parsers, every reporter and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width, the JSON and XML writers, sharding in the test registry, approval tests and their baselines, Conan packaging, and the unused third_party/clara.hpp. Listeners keep working; reporters' list and skip events are gone. The Catch2 library is now compiled as C++20; code that includes only Catch2's headers still needs C++14. Duplicate test cases are still refused at start-up. Version 0.3.0. ExtraTests now turn exceptions off for clang-cl with MSVC-style options, which -fno-exceptions did not do.
Diffstat (limited to 'docs')
-rw-r--r--docs/Readme.md10
-rw-r--r--docs/ci-and-misc.md50
-rw-r--r--docs/cmake-integration.md21
-rw-r--r--docs/command-line.md785
-rw-r--r--docs/configuration.md41
-rw-r--r--docs/event-listeners.md21
-rw-r--r--docs/filtering-execution-path.md10
-rw-r--r--docs/own-main.md126
-rw-r--r--docs/progmasoft-catch3.md44
-rw-r--r--docs/progmasoft-roadmap.md44
-rw-r--r--docs/progmasoft-runner.md324
-rw-r--r--docs/reporter-events.md48
-rw-r--r--docs/reporters.md243
-rw-r--r--docs/skipping-passing-failing.md2
-rw-r--r--docs/tutorial.md2
-rw-r--r--docs/usage-tips.md84
-rw-r--r--docs/why-catch3.md18
17 files changed, 689 insertions, 1184 deletions
diff --git a/docs/Readme.md b/docs/Readme.md
index d8e9730e..5d21ef7f 100644
--- a/docs/Readme.md
+++ b/docs/Readme.md
@@ -1,9 +1,10 @@
<a id="top"></a>
# Catch3 documentation
-Catch3 combines the upstream Catch2-compatible runner and API with
-Progmasoft's C++20 extensions. Start with the [tutorial](tutorial.md#top),
-then read [what Catch3 adds](progmasoft-catch3.md),
+Catch3 keeps Catch2's API for writing tests, runs them with
+[its own runner](progmasoft-runner.md#top) and adds Progmasoft's C++20
+extensions. Start with the [tutorial](tutorial.md#top), then read
+[the runner](progmasoft-runner.md#top), [what Catch3 adds](progmasoft-catch3.md),
[native property testing](progmasoft-properties.md), [the roadmap](progmasoft-roadmap.md) and
[why the two API families remain distinct](why-catch3.md). The compatibility
pages below retain real `catch2/` include paths, `Catch2::` CMake targets,
@@ -24,7 +25,7 @@ engine, not a claim that Catch3 is an official Catch2 release.
* [Test cases and sections](test-cases-and-sections.md#top)
* [Test fixtures](test-fixtures.md#top)
* [Explicitly skipping, passing, and failing tests at runtime](skipping-passing-failing.md#top)
-* [Reporters (output customization)](reporters.md#top)
+* [Reports](reporters.md#top)
* [Event Listeners](event-listeners.md#top)
* [Data Generators (value parameterized tests)](generators.md#top)
* [Other macros](other-macros.md#top)
@@ -36,6 +37,7 @@ engine, not a claim that Catch3 is an official Catch2 release.
* [String Conversions](tostring.md#top)
**Running:**
+* [The Catch3 runner](progmasoft-runner.md#top)
* [Command line reference](command-line.md#top)
* [Running specific section/generator](filtering-execution-path.md#top)
diff --git a/docs/ci-and-misc.md b/docs/ci-and-misc.md
index 4fa033fd..9932ccbc 100644
--- a/docs/ci-and-misc.md
+++ b/docs/ci-and-misc.md
@@ -16,50 +16,12 @@ like Continuous Integration and 3rd party test runners.
## Continuous Integration systems
-Probably the most important aspect to using Catch with a build server is the use of different reporters. Catch comes bundled with three reporters that should cover the majority of build servers out there - although adding more for better integration with some is always a possibility (currently we also offer TeamCity, TAP, Automake and SonarQube reporters).
-
-Two of these reporters are built in (XML and JUnit) and the third (TeamCity) is included as a separate header. It's possible that the other two may be split out in the future too - as that would make the core of Catch smaller for those that don't need them.
-
-### XML Reporter
-```-r xml```
-
-The XML Reporter writes in an XML format that is specific to Catch.
-
-The advantage of this format is that it corresponds well to the way Catch works (especially the more unusual features, such as nested sections) and is a fully streaming format - that is it writes output as it goes, without having to store up all its results before it can start writing.
-
-The disadvantage is that, being specific to Catch, no existing build servers understand the format natively. It can be used as input to an XSLT transformation that could convert it to, say, HTML - although this loses the streaming advantage, of course.
-
-### JUnit Reporter
-```-r junit```
-
-The JUnit Reporter writes in an XML format that mimics the JUnit ANT schema.
-
-The advantage of this format is that the JUnit Ant schema is widely understood by most build servers and so can usually be consumed with no additional work.
-
-The disadvantage is that this schema was designed to correspond to how JUnit works - and there is a significant mismatch with how Catch works. Additionally the format is not streamable (because opening elements hold counts of failed and passing tests as attributes) - so the whole test run must complete before it can be written.
-
-
-### TeamCity Reporter
-```-r teamcity```
-
-The TeamCity Reporter writes TeamCity service messages to stdout. In order to be able to use this reporter an additional header must also be included.
-
-Being specific to TeamCity this is the best reporter to use with it - but it is completely unsuitable for any other purpose. It is a streaming format (it writes as it goes) - although test results don't appear in the TeamCity interface until the completion of a suite (usually the whole test run).
-
-### Automake Reporter
-```-r automake```
-
-The Automake Reporter writes out the [meta tags](https://www.gnu.org/software/automake/manual/html_node/Log-files-generation-and-test-results-recording.html#Log-files-generation-and-test-results-recording) expected by automake via `make check`.
-
-### TAP (Test Anything Protocol) Reporter
-```-r tap```
-
-Because of the incremental nature of Catch's test suites and ability to run specific tests, our implementation of TAP reporter writes out the number of tests in a suite last.
-
-### SonarQube Reporter
-```-r sonarqube```
-[SonarQube Generic Test Data](https://docs.sonarqube.org/latest/analysis/generic-test/) XML format for tests metrics.
-
+The [Catch3 runner](progmasoft-runner.md#top) writes the reports build
+servers read: `--junit=FILE` for JUnit XML, which most CI systems display,
+with flaky tests and retries in the `<flakyFailure>` and `<rerunFailure>`
+elements Jenkins and GitLab understand, and `--json=FILE` for everything
+else. Under `bazel test` the JUnit report goes to the file Bazel names. See
+[reports](reporters.md#top).
## Bazel test runner integration
diff --git a/docs/cmake-integration.md b/docs/cmake-integration.md
index bf68ae92..d3d45dfe 100644
--- a/docs/cmake-integration.md
+++ b/docs/cmake-integration.md
@@ -166,28 +166,27 @@ Note that this variable is only available in CTest.
* `REPORTER reporter`
-Use the specified reporter when running the test case. The reporter will
-be passed to the test runner as `--reporter reporter`.
+Write a report of each test: `junit` passes `--junit=<file>` and `json`
+passes `--json=<file>` to the test executable; `console` writes no file.
+`junit` and `json` need `OUTPUT_DIR`.
* `OUTPUT_DIR dir`
-If specified, the parameter is passed along as
-`--out dir/<test_name>` to test executable. The actual file name is the
-same as the test name. This should be used instead of
-`EXTRA_ARGS --out foo` to avoid race conditions writing the result output
-when using parallel test execution.
+The directory the reports of `REPORTER` go to, one file per test, named
+after the test. Use it instead of `EXTRA_ARGS --junit=foo` so that tests
+that run at the same time do not write the same file.
* `OUTPUT_PREFIX prefix`
May be used in conjunction with `OUTPUT_DIR`.
If specified, `prefix` is added to each output file name, like so
-`--out dir/prefix<test_name>`.
+`dir/prefix<test_name>`.
* `OUTPUT_SUFFIX suffix`
May be used in conjunction with `OUTPUT_DIR`.
If specified, `suffix` is added to each output file name, like so
-`--out dir/<test_name>suffix`. This can be used to add a file extension to
+`dir/<test_name>suffix`. This can be used to add a file extension to
the output file name e.g. ".xml".
* `DISCOVERY_MODE mode`
@@ -294,7 +293,7 @@ and the seed changes every invocation of CTest.
Currently there are 3 customization points for this script:
* SHARD_COUNT - number of shards to split target's tests into
- * REPORTER - reporter spec to use for tests
+ * REPORTER - report options for the shards, such as --junit=FILE
* TEST_SPEC - test spec used for filtering tests
Example usage:
@@ -304,13 +303,11 @@ include(CatchShardTests)
catch_add_sharded_tests(foo-tests
SHARD_COUNT 4
- REPORTER "xml::out=-"
TEST_SPEC "A"
)
catch_add_sharded_tests(tests
SHARD_COUNT 8
- REPORTER "xml::out=-"
TEST_SPEC "B"
)
```
diff --git a/docs/command-line.md b/docs/command-line.md
index 829eba5a..29abc11a 100644
--- a/docs/command-line.md
+++ b/docs/command-line.md
@@ -1,657 +1,146 @@
<a id="top"></a>
-
-> **Catch3 compatibility reference:** This page describes the bundled Catch2-compatible API. Names such as `Catch2`, `catch2/`, and `Catch2::Catch2WithMain` remain where they are actual compatibility interfaces. See [Catch3 additions](progmasoft-catch3.md) for Progmasoft APIs.
-
# Command line
**Contents**<br>
-[Specifying which tests to run](#specifying-which-tests-to-run)<br>
-[Choosing a reporter to use](#choosing-a-reporter-to-use)<br>
-[Breaking into the debugger](#breaking-into-the-debugger)<br>
-[Showing results for successful tests](#showing-results-for-successful-tests)<br>
-[Aborting after a certain number of failures](#aborting-after-a-certain-number-of-failures)<br>
-[Listing available tests, tags or reporters](#listing-available-tests-tags-or-reporters)<br>
-[Sending output to a file](#sending-output-to-a-file)<br>
-[Naming a test run](#naming-a-test-run)<br>
-[Eliding assertions expected to throw](#eliding-assertions-expected-to-throw)<br>
-[Make whitespace visible](#make-whitespace-visible)<br>
-[Warnings](#warnings)<br>
-[Reporting timings](#reporting-timings)<br>
-[Load test names to run from a file](#load-test-names-to-run-from-a-file)<br>
-[Specify the order test cases are run](#specify-the-order-test-cases-are-run)<br>
-[Specify a seed for the Random Number Generator](#specify-a-seed-for-the-random-number-generator)<br>
-[Identify framework and version according to the libIdentify standard](#identify-framework-and-version-according-to-the-libidentify-standard)<br>
-[Wait for key before continuing](#wait-for-key-before-continuing)<br>
-[Skip all benchmarks](#skip-all-benchmarks)<br>
-[Specify the number of benchmark samples to collect](#specify-the-number-of-benchmark-samples-to-collect)<br>
-[Specify the number of resamples for bootstrapping](#specify-the-number-of-resamples-for-bootstrapping)<br>
-[Specify the confidence-interval for bootstrapping](#specify-the-confidence-interval-for-bootstrapping)<br>
-[Disable statistical analysis of collected benchmark samples](#disable-statistical-analysis-of-collected-benchmark-samples)<br>
-[Specify the amount of time in milliseconds spent on warming up each test](#specify-the-amount-of-time-in-milliseconds-spent-on-warming-up-each-test)<br>
-[Usage](#usage)<br>
-[Specify the section/generator element to run](#specify-the-sectiongenerator-element-to-run)<br>
-[Filenames as tags](#filenames-as-tags)<br>
-[Override output colouring](#override-output-colouring)<br>
-[Test Sharding](#test-sharding)<br>
-[Allow running the binary without tests](#allow-running-the-binary-without-tests)<br>
-[Output verbosity](#output-verbosity)<br>
-[Create file to guard against silent early termination](#create-file-to-guard-against-silent-early-termination)<br>
-
-Catch works quite nicely without any command line options at all - but for those times when you want greater control the following options are available.
-Click one of the following links to take you straight to that option - or scroll on to browse the available options.
-
-<a href="#specifying-which-tests-to-run"> ` <test-spec> ...`</a><br />
-<a href="#usage"> ` -h, -?, --help`</a><br />
-<a href="#showing-results-for-successful-tests"> ` -s, --success`</a><br />
-<a href="#breaking-into-the-debugger"> ` -b, --break`</a><br />
-<a href="#eliding-assertions-expected-to-throw"> ` -e, --nothrow`</a><br />
-<a href="#invisibles"> ` -i, --invisibles`</a><br />
-<a href="#sending-output-to-a-file"> ` -o, --out`</a><br />
-<a href="#choosing-a-reporter-to-use"> ` -r, --reporter`</a><br />
-<a href="#naming-a-test-run"> ` -n, --name`</a><br />
-<a href="#aborting-after-a-certain-number-of-failures"> ` -a, --abort`</a><br />
-<a href="#aborting-after-a-certain-number-of-failures"> ` -x, --abortx`</a><br />
-<a href="#warnings"> ` -w, --warn`</a><br />
-<a href="#reporting-timings"> ` -d, --durations`</a><br />
-<a href="#input-file"> ` -f, --input-file`</a><br />
-<a href="#run-section"> ` -c, --section`</a><br />
-<a href="#path-filtering"> ` -g, --generator-index`</a><br />
-<a href="#path-filtering"> ` -p, --path-filter`</a><br />
-<a href="#filenames-as-tags"> ` -#, --filenames-as-tags`</a><br />
-
-
-</br>
-
-<a href="#listing-available-tests-tags-or-reporters"> ` --list-tests`</a><br />
-<a href="#listing-available-tests-tags-or-reporters"> ` --list-tags`</a><br />
-<a href="#listing-available-tests-tags-or-reporters"> ` --list-reporters`</a><br />
-<a href="#listing-available-tests-tags-or-reporters"> ` --list-listeners`</a><br />
-<a href="#order"> ` --order`</a><br />
-<a href="#rng-seed"> ` --rng-seed`</a><br />
-<a href="#libidentify"> ` --libidentify`</a><br />
-<a href="#wait-for-keypress"> ` --wait-for-keypress`</a><br />
-<a href="#skip-benchmarks"> ` --skip-benchmarks`</a><br />
-<a href="#benchmark-samples"> ` --benchmark-samples`</a><br />
-<a href="#benchmark-resamples"> ` --benchmark-resamples`</a><br />
-<a href="#benchmark-confidence-interval"> ` --benchmark-confidence-interval`</a><br />
-<a href="#benchmark-no-analysis"> ` --benchmark-no-analysis`</a><br />
-<a href="#benchmark-warmup-time"> ` --benchmark-warmup-time`</a><br />
-<a href="#colour-mode"> ` --colour-mode`</a><br />
-<a href="#test-sharding"> ` --shard-count`</a><br />
-<a href="#test-sharding"> ` --shard-index`</a><br />
-<a href=#no-tests-override> ` --allow-running-no-tests`</a><br />
-<a href=#output-verbosity> ` --verbosity`</a><br />
-
-</br>
-
-
-
-<a id="specifying-which-tests-to-run"></a>
-## Specifying which tests to run
-
-<pre>&lt;test-spec> ...</pre>
-
-By providing a test spec, you filter which tests will be run. If you call
-Catch2 without any test spec, then it will run all non-hidden test
-cases. A test case is hidden if it has the `[!benchmark]` tag, any tag
-with a dot at the start, e.g. `[.]` or `[.foo]`.
-
-There are three basic test specs that can then be combined into more
-complex specs:
-
- * Full test name, e.g. `"Test 1"`.
-
- This allows only test cases whose name is "Test 1".
-
- * Wildcarded test name, e.g. `"*Test"`, or `"Test*"`, or `"*Test*"`.
-
- This allows any test case whose name ends with, starts with, or contains
- in the middle the string "Test". Note that the wildcard can only be at
- the start or end.
-
- * Tag name, e.g. `[some-tag]`.
-
- This allows any test case tagged with "[some-tag]". Remember that some
- tags are special, e.g. those that start with "." or with "!".
-
-
-You can also combine the basic test specs to create more complex test
-specs. You can:
-
- * Concatenate specs to apply all of them, e.g. `[some-tag][other-tag]`.
-
- This allows test cases that are tagged with **both** "[some-tag]" **and**
- "[other-tag]". A test case with just "[some-tag]" will not pass the filter,
- nor will test case with just "[other-tag]".
-
- * Comma-join specs to apply any of them, e.g. `[some-tag],[other-tag]`.
-
- This allows test cases that are tagged with **either** "[some-tag]" **or**
- "[other-tag]". A test case with both will obviously also pass the filter.
-
- Note that commas take precedence over simple concatenation. This means
- that `[a][b],[c]` accepts tests that are tagged with either both "[a]" and
- "[b]", or tests that are tagged with just "[c]".
-
- * Negate the spec by prepending it with `~`, e.g. `~[some-tag]`.
-
- This rejects any test case that is tagged with "[some-tag]". Note that
- rejection takes precedence over other filters.
-
- Note that negations always binds to the following _basic_ test spec.
- This means that `~[foo][bar]` negates only the "[foo]" tag and not the
- "[bar]" tag.
-
-Note that when Catch2 is deciding whether to include a test, first it
-checks whether the test matches any negative filters. If it does,
-the test is rejected. After that, the behaviour depends on whether there
-are positive filters as well. If there are no positive filters, all
-remaining non-hidden tests are included. If there are positive filters,
-only tests that match the positive filters are included.
-
-You can also match test names with special characters by escaping them
-with a backslash (`"\"`), e.g. a test named `"Do A, then B"` is matched
-by `"Do A\, then B"` test spec. Backslash also escapes itself.
-
-
-### Examples
-
-Given these TEST_CASEs,
-```
-TEST_CASE("Test 1") {}
-
-TEST_CASE("Test 2", "[.foo]") {}
-
-TEST_CASE("Test 3", "[.bar]") {}
-
-TEST_CASE("Test 4", "[.][foo][bar]") {}
-```
-
-this is the result of these filters
-```
-./tests # Selects only the first test, others are hidden
-./tests "Test 1" # Selects only the first test, other do not match
-./tests ~"Test 1" # Selects no tests. Test 1 is rejected, other tests are hidden
-./tests "Test *" # Selects all tests.
-./tests [bar] # Selects tests 3 and 4. Other tests are not tagged [bar]
-./tests ~[foo] # Selects test 1, because it is the only non-hidden test without [foo] tag
-./tests [foo][bar] # Selects test 4.
-./tests [foo],[bar] # Selects tests 2, 3, 4.
-./tests ~[foo][bar] # Selects test 3. 2 and 4 are rejected due to having [foo] tag
-./tests ~"Test 2"[foo] # Selects test 4, because test 2 is explicitly rejected
-./tests [foo][bar],"Test 1" # Selects tests 1 and 4.
-./tests "Test 1*" # Selects test 1, wildcard can match zero characters
-```
-
-_Note: Using plain asterisk on a command line can cause issues with shell
-expansion. Make sure that the asterisk is passed to Catch2 and is not
-interpreted by the shell._
-
-
-<a id="choosing-a-reporter-to-use"></a>
-## Choosing a reporter to use
-
-<pre>-r, --reporter &lt;reporter[::key=value]*&gt;</pre>
-
-Reporters are how the output from Catch2 (results of assertions, tests,
-benchmarks and so on) is formatted and written out. The default reporter
-is called the "Console" reporter and is intended to provide relatively
-verbose and human-friendly output.
-
-Reporters are also individually configurable. To pass configuration options
-to the reporter, you append `::key=value` to the reporter specification
-as many times as you want, e.g. `--reporter xml::out=someFile.xml` or
-`--reporter custom::colour-mode=ansi::Xoption=2`.
-
-The keys must either be prefixed by "X", in which case they are not parsed
-by Catch2 and are only passed down to the reporter, or one of options
-hardcoded into Catch2. Currently there are 3 supported options:
-
-* ["out"](#sending-output-to-a-file)
-* ["colour-mode"](#colour-mode)
-* ["verbosity"](#output-verbosity)
-
-> Support for per-reporter verbosity option was added in Catch2 3.16.0
-
-_Note that the reporter might still check the X-prefixed options for
-validity, and throw an error if they are wrong._
-
-> Support for passing arguments to reporters through the `-r`, `--reporter` flag was introduced in Catch2 3.0.1
-
-There are multiple built-in reporters, you can see what they do by using the
-[`--list-reporters`](command-line.md#listing-available-tests-tags-or-reporters)
-flag. If you need a reporter providing custom format outside of the already
-provided ones, look at the ["write your own reporter" part of the reporter
-documentation](reporters.md#writing-your-own-reporter).
-
-This option may be passed multiple times to use multiple (different)
-reporters at the same time. See the [reporter documentation](reporters.md#multiple-reporters)
-for details on what the resulting behaviour is. Also note that at most one
-reporter can be provided without the output-file part of reporter spec.
-This reporter will use the "default" output destination, based on
-the [`-o`, `--out`](#sending-output-to-a-file) option.
-
-> Support for using multiple different reporters at the same time was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 3.0.1
-
-
-_Note: There is currently no way to escape `::` in the reporter spec,
-and thus the reporter names, or configuration keys and values, cannot
-contain `::`. As `::` in paths is relatively obscure (unlike ':'), we do
-not consider this an issue._
-
-
-<a id="breaking-into-the-debugger"></a>
-## Breaking into the debugger
-<pre>-b, --break</pre>
-
-Under most debuggers Catch2 is capable of automatically breaking on a test
-failure. This allows the user to see the current state of the test during
-failure.
-
-<a id="showing-results-for-successful-tests"></a>
-## Showing results for successful tests
-<pre>-s, --success</pre>
-
-Usually you only want to see reporting for failed tests. Sometimes it's useful to see *all* the output (especially when you don't trust that that test you just added worked first time!).
-To see successful, as well as failing, test results just pass this option. Note that each reporter may treat this option differently. The Junit reporter, for example, logs all results regardless.
-
-<a id="aborting-after-a-certain-number-of-failures"></a>
-## Aborting after a certain number of failures
-<pre>-a, --abort
--x, --abortx [&lt;failure threshold>]
-</pre>
-
-If a ```REQUIRE``` assertion fails the test case aborts, but subsequent test cases are still run.
-If a ```CHECK``` assertion fails even the current test case is not aborted.
-
-Sometimes this results in a flood of failure messages and you'd rather just see the first few. Specifying ```-a``` or ```--abort``` on its own will abort the whole test run on the first failed assertion of any kind. Use ```-x``` or ```--abortx``` followed by a number to abort after that number of assertion failures.
-
-<a id="listing-available-tests-tags-or-reporters"></a>
-## Listing available tests, tags or reporters
-```
---list-tests
---list-tags
---list-reporters
---list-listeners
-```
-
-> The `--list*` options became customizable through reporters in Catch2 3.0.1
-
-> The `--list-listeners` option was added in Catch2 3.0.1
-
-`--list-tests` lists all registered tests matching specified test spec.
-Usually this listing also includes tags, and potentially also other
-information, like source location, based on verbosity and reporter's design.
-
-`--list-tags` lists all tags from registered tests matching specified test
-spec. Usually this also includes number of tests cases they match and
-similar information.
-
-`--list-reporters` lists all available reporters and their descriptions.
-
-`--list-listeners` lists all registered listeners and their descriptions.
-
-The [`--verbosity` argument](#output-verbosity) modifies the level of detail provided by the default `--list*` options
-as follows:
-
-| Option | `normal` (default) | `quiet` | `high` |
-|--------------------|---------------------------------|---------------------|-----------------------------------------|
-| `--list-tests` | Test names and tags | Test names only | Same as `normal`, plus source code line |
-| `--list-tags` | Tags and counts | Tags only | Same as `normal` |
-| `--list-reporters` | Reporter names and descriptions | Reporter names only | Same as `normal` |
-| `--list-listeners` | Listener names and descriptions | Listener names only | Same as `normal` |
-
-
-<a id="sending-output-to-a-file"></a>
-## Sending output to a file
-<pre>-o, --out &lt;filename&gt;
-</pre>
-
-Use this option to send all output to a file, instead of stdout. You can
-use `-` as the filename to explicitly send the output to stdout (this is
-useful e.g. when using multiple reporters).
-
-> Support for `-` as the filename was introduced in Catch2 3.0.1
-
-Filenames starting with "%" (percent symbol) are reserved by Catch2 for
-meta purposes, e.g. using `%debug` as the filename opens stream that
-writes to platform specific debugging/logging mechanism.
-
-Catch2 currently recognizes 3 meta streams:
-
-* `%debug` - writes to platform specific debugging/logging output
-* `%stdout` - writes to stdout
-* `%stderr` - writes to stderr
-
-> Support for `%stdout` and `%stderr` was introduced in Catch2 3.0.1
-
-
-<a id="naming-a-test-run"></a>
-## Naming a test run
-<pre>-n, --name &lt;name for test run></pre>
-
-If a name is supplied it will be used by the reporter to provide an overall name for the test run. This can be useful if you are sending to a file, for example, and need to distinguish different test runs - either from different Catch executables or runs of the same executable with different options. If not supplied the name is defaulted to the name of the executable.
-
-<a id="eliding-assertions-expected-to-throw"></a>
-## Eliding assertions expected to throw
-<pre>-e, --nothrow</pre>
-
-Skips all assertions that test that an exception is thrown, e.g. ```REQUIRE_THROWS```.
-
-These can be a nuisance in certain debugging environments that may break when exceptions are thrown (while this is usually optional for handled exceptions, it can be useful to have enabled if you are trying to track down something unexpected).
-
-Sometimes exceptions are expected outside of one of the assertions that tests for them (perhaps thrown and caught within the code-under-test). The whole test case can be skipped when using ```-e``` by marking it with the ```[!throws]``` tag.
-
-When running with this option any throw checking assertions are skipped so as not to contribute additional noise. Be careful if this affects the behaviour of subsequent tests.
-
-<a id="invisibles"></a>
-## Make whitespace visible
-<pre>-i, --invisibles</pre>
-
-If a string comparison fails due to differences in whitespace - especially leading or trailing whitespace - it can be hard to see what's going on.
-This option transforms tabs and newline characters into ```\t``` and ```\n``` respectively when printing.
-
-<a id="warnings"></a>
-## Warnings
-<pre>-w, --warn &lt;warning name></pre>
-
-You can think of Catch2's warnings as the equivalent of `-Werror` (`/WX`)
-flag for C++ compilers. It turns some suspicious occurrences, like a section
-without assertions, into errors. Because these might be intended, warnings
-are not enabled by default, but user can opt in.
-
-You can enable multiple warnings at the same time.
-
-There are currently two warnings implemented:
-
-```
- NoAssertions // Fail test case / leaf section if no assertions
- // (e.g. `REQUIRE`) is encountered.
- UnmatchedTestSpec // Fail test run if any of the CLI test specs did
- // not match any tests.
- InfiniteGenerators // Fail if GENERATE would run infinitely
-```
-
-> `UnmatchedTestSpec` was introduced in Catch2 3.0.1.
-
-> `InfiniteGenerators` was introduced in Catch2 3.13.0
-
-
-<a id="reporting-timings"></a>
-## Reporting timings
-<pre>-d, --durations &lt;yes/no></pre>
-
-When set to ```yes``` Catch will report the duration of each test case, in seconds with millisecond precision. Note that it does this regardless of whether a test case passes or fails. Note, also, the certain reporters (e.g. Junit) always report test case durations regardless of this option being set or not.
-
-<pre>-D, --min-duration &lt;value></pre>
-
-> `--min-duration` was [introduced](https://github.com/catchorg/Catch2/pull/1910) in Catch2 2.13.0
-
-When set, Catch will report the duration of each test case that took more
-than &lt;value> seconds, in seconds with millisecond precision. This option is overridden by both
-`-d yes` and `-d no`, so that either all durations are reported, or none
-are.
-
-
-<a id="input-file"></a>
-## Load test names to run from a file
-<pre>-f, --input-file &lt;filename></pre>
-
-Provide the name of a file that contains a list of test case names,
-one per line. Blank lines are skipped.
-
-A useful way to generate an initial instance of this file is to combine
-the [`--list-tests`](#listing-available-tests-tags-or-reporters) flag with
-the [`--verbosity quiet`](#output-verbosity) option. You can also
-use test specs to filter this list down to what you want first.
-
-
-<a id="order"></a>
-## Specify the order test cases are run
-<pre>--order &lt;decl|lex|rand&gt;</pre>
-
-Test cases are ordered one of three ways:
-
-### decl
-Declaration order.
-
-Tests in the same translation unit are sorted using their declaration orders,
-different TUs are sorted in an implementation (linking) dependent order.
-
-
-### lex
-Lexicographic order.
-
-Tests are sorted by their name, their tags are ignored.
-
-
-### rand
-Randomized order. The default order.
-
-> Randomized order has been made default in Catch2 3.9.0
-
-The order is dependent on Catch2's random seed (see
-[`--rng-seed`](#rng-seed)), and is subset invariant. What this means
-is that as long as the random seed is fixed, running only some tests
-(e.g. via tag) does not change their relative order.
-
-> The subset stability was introduced in Catch2 v2.12.0
-
-Since the random order was made subset stable, we promise that given
-the same random seed, the order of test cases will be the same across
-different platforms, as long as the tests were compiled against identical
-version of Catch2. We reserve the right to change the relative order
-of tests cases between Catch2 versions, but it is unlikely to happen often.
-
-
-<a id="rng-seed"></a>
-## Specify a seed for the Random Number Generator
-<pre>--rng-seed &lt;'time'|'random-device'|number&gt;</pre>
-
-Sets the seed for random number generators used by Catch2. These are used
-e.g. to shuffle tests when user asks for tests to be in random order.
-
-Using `time` as the argument asks Catch2 generate the seed through call
-to `std::time(nullptr)`. This provides very weak randomness and multiple
-runs of the binary can generate the same seed if they are started close
-to each other.
-
-Using `random-device` asks for `std::random_device` to be used instead.