Progmasoft Git

catch3

Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.

Clonegit clone https://git.progmasoft.com/catch3

Rewrite commandline test spec docs

Closes #2738
Commit
2a19ae16b87547f152f262e05a55c7b2b969807b
Author
Martin Hořeňovský ·
Parents
f24d39e42b3b

Files · Download .tar.gz

Changes

 docs/command-line.md | 109 +++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 84 insertions(+), 25 deletions(-)

diff --git a/docs/command-line.md b/docs/command-line.md
index a15a2131..77d6a727 100644
--- a/docs/command-line.md
+++ b/docs/command-line.md
@@ -85,43 +85,102 @@ Click one of the following links to take you straight to that option - or scroll
 
 <pre>&lt;test-spec> ...</pre>
 
-Test cases, wildcarded test cases, tags and tag expressions are all passed directly as arguments. Tags are distinguished by being enclosed in square brackets.
+By providing a test spec, you filter which tests will be run. If you call
+Catch2 without any test spec, then it will run all non-hidden test
+cases. A test case is hidden if it has the `[!benchmark]` tag, any tag
+with a dot at the start, e.g. `[.]` or `[.foo]`.
 
-If no test specs are supplied then all test cases, except "hidden" tests, are run.
-A test is hidden by giving it any tag starting with (or just) a period (```.```) - or, in the deprecated case, tagged ```[hide]``` or given name starting with `'./'`. To specify hidden tests from the command line ```[.]``` or ```[hide]``` can be used *regardless of how they were declared*.
+There are three basic test specs that can then be combined into more
+complex specs:
 
-Specs must be enclosed in quotes if they contain spaces. If they do not contain spaces the quotes are optional.
+  * Full test name, e.g. `"Test 1"`.
 
-Wildcards consist of the `*` character at the beginning and/or end of test case names and can substitute for any number of any characters (including none).
+This allows only test cases whose name is "Test 1".
 
-Test specs are case insensitive.
+  * Wildcarded test name, e.g. `"*Test"`, or `"Test*"`, or `"*Test*"`.
 
-If a spec is prefixed with `exclude:` or the `~` character then the pattern matches an exclusion. This means that tests matching the pattern are excluded from the set - even if a prior inclusion spec included them. Subsequent inclusion specs will take precedence, however.
-Inclusions and exclusions are evaluated in left-to-right order.
+This allows any test case whose name ends with, starts with, or contains
+in the middle the string "Test". Note that the wildcard can only be at
+the start or end.
 
-Test case examples:
+  * Tag name, e.g. `[some-tag]`.
 
+This allows any test case tagged with "[some-tag]". Remember that some
+tags are special, e.g. those that start with "." or with "!".
+
+
+You can also combine the basic test specs to create more complex test
+specs. You can
+
+  * Concatenate specs to apply all of them, e.g. `[some-tag][other-tag]`.
+
+This allows test cases that are tagged with **both** "[some-tag]" **and**
+"[other-tag]". A test case with just "[some-tag]" will not pass the filter,
+nor will test case with just "[other-tag]".
+
+  * Comma-join specs to apply any of them, e.g. `[some-tag],[other-tag]`.
+
+This allows test cases that are tagged with **either** "[some-tag]" **or**
+"[other-tag]". A test case with both will obviously also pass the filter.
+
+Note that commas take precendence over simple concatenation. This means
+that `[a][b],[c]` accepts tests that are tagged with either both "[a]" and
+"[b]", or tests that are tagged with just "[c]".
+
+  * Negate the spec by prepending it with `~`, e.g. `~[some-tag]`.
+
+This rejects any test case that is tagged with "[some-tag]". Note that
+rejection takes precedence over other filters.
+
+Note that negations always binds to the following _basic_ test spec.
+This means that `~[foo][bar]` negates only the "[foo]" tag and not the
+"[bar]" tag.
+
+Note that when Catch2 is deciding whether to include a test, first it
+checks whether the test matches any negative filters. If it does,
+the test is rejected. After that, the behaviour depends on whether there
+are positive filters as well. If there are no positive filters, all
+remaining non-hidden tests are included. If there are positive filters,
+only tests that match the positive filters are included.
+
+You can also match test names with special characters by escaping them
+with a backslash (`"\"`), e.g. a test named `"Do A, then B"` is matched
+by "Do A\, then B" test spec. Backslash also escapes itself.
+
+
+### Examples
+
+Given these TEST_CASEs,
 ```
-thisTestOnly            Matches the test case called, 'thisTestOnly'
-"this test only"        Matches the test case called, 'this test only'
-these*                  Matches all cases starting with 'these'
-exclude:notThis         Matches all tests except, 'notThis'
-~notThis                Matches all tests except, 'notThis'
-~*private*              Matches all tests except those that contain 'private'
-a* ~ab* abc             Matches all tests that start with 'a', except those that
-                        start with 'ab', except 'abc', which is included
-~[tag1]                 Matches all tests except those tagged with '[tag1]'
--# [#somefile]          Matches all tests from the file 'somefile.cpp'
+TEST_CASE("Test 1") {}
+
+TEST_CASE("Test 2", "[.foo]") {}
+
+TEST_CASE("Test 3", "[.bar]") {}
+
+TEST_CASE("Test 4", "[.][foo][bar]") {}
 ```
 
-Names within square brackets are interpreted as tags.
-A series of tags form an AND expression whereas a comma-separated sequence forms an OR expression. e.g.:
+this is the result of these filters
+```
+./tests                      # Selects only the first test, others are hidden
+./tests "Test 1"             # Selects only the first test, other do not match
+./tests ~"Test 1"            # Selects no tests. Test 1 is rejected, other tests are hidden
+./tests "Test *"             # Selects all tests.
+./tests [bar]                # Selects tests 3 and 4. Other tests are not tagged [bar]
+./tests ~[foo]               # Selects test 1, because it is the only non-hidden test without [foo] tag
+./tests [foo][bar]           # Selects test 4.
+./tests [foo],[bar]          # Selects tests 2, 3, 4.
+./tests ~[foo][bar]          # Selects test 3. 2 and 4 are rejected due to having [foo] tag
+./tests ~"Test 2"[foo]       # Selects test 4, because test 2 is explicitly rejected
+./tests [foo][bar],"Test 1"  # Selects tests 1 and 4.
+./tests "Test 1*"            # Selects test 1, wildcard can match zero characters
+```
 
-<pre>[one][two],[three]</pre>
-This matches all tests tagged `[one]` and `[two]`, as well as all tests tagged `[three]`
+_Note: Using plain asterisk on a command line can cause issues with shell
+expansion. Make sure that the asterisk is passed to Catch2 and is not
+interpreted by the shell._
 
-Test names containing special characters, such as `,` or `[` can specify them on the command line using `\`.
-`\` also escapes itself.
 
 <a id="choosing-a-reporter-to-use"></a>
 ## Choosing a reporter to use