<feed xmlns='http://www.w3.org/2005/Atom'>
<title>catch3/docs/command-line.md, branch feature/per-test-coverage</title>
<subtitle>Progmasoft Catch3: an independently maintained C++17 test framework with a parallel runner, property checks, snapshots and structured results.</subtitle>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/'/>
<entry>
<title>Keep per-test coverage with --coverage (roadmap step 4)</title>
<updated>2026-11-13T04:46:08+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-11-13T04:46:08+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=17248837a78462596809f902dd47cf113a39670b'/>
<id>17248837a78462596809f902dd47cf113a39670b</id>
<content type='text'>
--coverage=DIR runs every attempt in a process of its own and keeps the
coverage of each test, for binaries built with LLVM source-based
coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov
(GCC: --coverage).

How it works:

- The scheduler gives each attempt a fresh worker, as --isolate does,
  and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the
  process writes at exit is then the coverage of that one test, in
  DIR/profiles/t&lt;index&gt;-a&lt;attempt&gt;.profraw or a .gcov directory. Each
  runtime reads its own variable, so no symbol of either runtime is
  looked up: weak references would not pull them out of their static
  archives, and Mach-O does not allow undefined weak symbols.
- A process that runs other tests first (bisection) writes beside the
  profiles, never into one. The attempt records its profile, also in
  the JSON results.
- The profiles of an earlier run are removed when the run starts,
  because gcov adds to the .gcda files it finds.
- After the run, CoverageReport reads the last profile of every test:
  llvm-profdata and llvm-cov (PATH, xcrun on macOS, or
  CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or
  CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is
  DIR/coverage.json, the files, lines and branches each test reached,
  and DIR/coverage.lcov for the whole run. Without the tools the
  profiles stay and the run says so.
- --coverage with --in-process or --break is refused.

Tests: option parsing, LCOV reading and writing, gcov JSON, and an
integration test on CoverageFixture, a binary CMake builds with the
coverage flags of Clang or GCC: two tests reach different lines, each
map holds only its own, the LCOV report holds both, and a stale profile
of an earlier run is gone. Locally all 84 CTest tests pass with Clang,
and the coverage tests pass with GCC (built with
CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as
errors, before and after this change).

CI: the Linux jobs install LLVM's coverage tools, a new job checks the
gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map
into a failure on Linux and macOS instead of a skip. MSVC has no
source-based coverage, so its builds skip those tests; the roadmap
lists step 4 as in progress until Windows with clang-cl is tested.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
--coverage=DIR runs every attempt in a process of its own and keeps the
coverage of each test, for binaries built with LLVM source-based
coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov
(GCC: --coverage).

How it works:

- The scheduler gives each attempt a fresh worker, as --isolate does,
  and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the
  process writes at exit is then the coverage of that one test, in
  DIR/profiles/t&lt;index&gt;-a&lt;attempt&gt;.profraw or a .gcov directory. Each
  runtime reads its own variable, so no symbol of either runtime is
  looked up: weak references would not pull them out of their static
  archives, and Mach-O does not allow undefined weak symbols.
- A process that runs other tests first (bisection) writes beside the
  profiles, never into one. The attempt records its profile, also in
  the JSON results.
- The profiles of an earlier run are removed when the run starts,
  because gcov adds to the .gcda files it finds.
- After the run, CoverageReport reads the last profile of every test:
  llvm-profdata and llvm-cov (PATH, xcrun on macOS, or
  CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or
  CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is
  DIR/coverage.json, the files, lines and branches each test reached,
  and DIR/coverage.lcov for the whole run. Without the tools the
  profiles stay and the run says so.
- --coverage with --in-process or --break is refused.

Tests: option parsing, LCOV reading and writing, gcov JSON, and an
integration test on CoverageFixture, a binary CMake builds with the
coverage flags of Clang or GCC: two tests reach different lines, each
map holds only its own, the LCOV report holds both, and a stale profile
of an earlier run is gone. Locally all 84 CTest tests pass with Clang,
and the coverage tests pass with GCC (built with
CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as
errors, before and after this change).

CI: the Linux jobs install LLVM's coverage tools, a new job checks the
gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map
into a failure on Linux and macOS instead of a skip. MSVC has no
source-based coverage, so its builds skip those tests; the roadmap
lists step 4 as in progress until Windows with clang-cl is tested.
</pre>
</div>
</content>
</entry>
<entry>
<title>Withhold secret-looking variables from the processes the runner starts</title>
<updated>2026-11-05T15:30:13+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-11-05T15:30:13+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=89b4769c35986444a0297ad0806a584c7181c85c'/>
<id>89b4769c35986444a0297ad0806a584c7181c85c</id>
<content type='text'>
Workers and the other processes the runner starts copied its whole
environment, tokens and passwords included. ChildProcess::Start now
leaves out every variable whose name contains, in any case, TOKEN,
SECRET, PASSWORD, PASSWD, CREDENTIAL, API_KEY, APIKEY, PRIVATE_KEY or
ACCESS_KEY (IsSecretVariable), unless ChildOptions::keptEnvironment
names it; names compare without case on Windows.

--pass-env=NAME, repeatable, and "pass-env" in catch3.json keep a
variable for the tests. The runner itself still starts with the whole
environment, so code that runs before the first worker sees it;
docs/progmasoft-runner.md says so.

A runner test starts the shell's environment listing with a secret and
a plain variable set, sees only the plain one, and sees the secret once
it is kept.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Workers and the other processes the runner starts copied its whole
environment, tokens and passwords included. ChildProcess::Start now
leaves out every variable whose name contains, in any case, TOKEN,
SECRET, PASSWORD, PASSWD, CREDENTIAL, API_KEY, APIKEY, PRIVATE_KEY or
ACCESS_KEY (IsSecretVariable), unless ChildOptions::keptEnvironment
names it; names compare without case on Windows.

--pass-env=NAME, repeatable, and "pass-env" in catch3.json keep a
variable for the tests. The runner itself still starts with the whole
environment, so code that runs before the first worker sees it;
docs/progmasoft-runner.md says so.

A runner test starts the shell's environment listing with a secret and
a plain variable set, sees only the plain one, and sees the secret once
it is kept.
</pre>
</div>
</content>
</entry>
<entry>
<title>Read the options of a project from catch3.json</title>
<updated>2026-10-29T16:22:21+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-29T16:22:21+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=f41a9c3c01b4df1100e24a63987716cb4dbf28f2'/>
<id>f41a9c3c01b4df1100e24a63987716cb4dbf28f2</id>
<content type='text'>
A project can now keep the options it always wants in a catch3.json
instead of repeating them in every CTest, Bazel or CI command line.

Where it is found: the runner looks in the working directory and in each
directory above it, up to the root of the Git repository that holds the
working directory, and reads the nearest file. Outside a repository it
looks only in the working directory, so a stray file in a home directory
never applies. --config=FILE names a file instead (relative to the
working directory) and --no-config reads none; the last of the two wins.
Workers, --help and --version read no file, so a broken file never stands
in the way of help, and workers keep taking everything from their runner.

What it says: strict JSON with "version": 1 required and "$schema"
allowed for editors. The keys are the long options of the command line,
and each value becomes the arguments it stands for and goes through the
same parser, so there is one set of checks and one set of messages; the
file and the key are named in front of them. Values are typed: switches
take true (false for the negated adaptive-timeout and bisect), counts
take whole numbers, timeout and the confidence interval take numbers,
retries and seed take a number or a word, tests and warn take arrays,
verbosity takes quiet, normal or verbose, and history takes a path or
false. Paths are relative to the directory of the file. An unknown key is
an error with a suggestion when it is close to a known one; a key that
names an option for one run (shard, section, list, replay, ...) is told
that it belongs on the command line. Repeated keys are refused.

Precedence: the file's options are applied first and the command line on
top, so the command line wins; test specs on the command line replace the
file's tests instead of widening them with "or". --verbose names the file
that was read.

Options gains --config and --no-config, a ParseOptions overload that
starts from given options, and a public EditDistance for the suggestions.
The new ProjectConfig module holds the search, the key table and
ParseCommandLine, which Runner now calls in place of ParseOptions.

Tests cover the search (nearest file, the repository boundary, outside a
repository), every key, words and defaults, relative and absolute paths,
the messages for wrong files, precedence, --config, --no-config and the
modes that read no file. A fuzzer reads arbitrary file text and checks
that the worker arguments of the result parse again. The command line
reference, the runner guide and the release notes describe the file; the
amalgamated sources are regenerated.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A project can now keep the options it always wants in a catch3.json
instead of repeating them in every CTest, Bazel or CI command line.

Where it is found: the runner looks in the working directory and in each
directory above it, up to the root of the Git repository that holds the
working directory, and reads the nearest file. Outside a repository it
looks only in the working directory, so a stray file in a home directory
never applies. --config=FILE names a file instead (relative to the
working directory) and --no-config reads none; the last of the two wins.
Workers, --help and --version read no file, so a broken file never stands
in the way of help, and workers keep taking everything from their runner.

What it says: strict JSON with "version": 1 required and "$schema"
allowed for editors. The keys are the long options of the command line,
and each value becomes the arguments it stands for and goes through the
same parser, so there is one set of checks and one set of messages; the
file and the key are named in front of them. Values are typed: switches
take true (false for the negated adaptive-timeout and bisect), counts
take whole numbers, timeout and the confidence interval take numbers,
retries and seed take a number or a word, tests and warn take arrays,
verbosity takes quiet, normal or verbose, and history takes a path or
false. Paths are relative to the directory of the file. An unknown key is
an error with a suggestion when it is close to a known one; a key that
names an option for one run (shard, section, list, replay, ...) is told
that it belongs on the command line. Repeated keys are refused.

Precedence: the file's options are applied first and the command line on
top, so the command line wins; test specs on the command line replace the
file's tests instead of widening them with "or". --verbose names the file
that was read.

Options gains --config and --no-config, a ParseOptions overload that
starts from given options, and a public EditDistance for the suggestions.
The new ProjectConfig module holds the search, the key table and
ParseCommandLine, which Runner now calls in place of ParseOptions.

Tests cover the search (nearest file, the repository boundary, outside a
repository), every key, words and defaults, relative and absolute paths,
the messages for wrong files, precedence, --config, --no-config and the
modes that read no file. A fuzzer reads arbitrary file text and checks
that the worker arguments of the result parse again. The command line
reference, the runner guide and the release notes describe the file; the
amalgamated sources are regenerated.
</pre>
</div>
</content>
</entry>
<entry>
<title>Replace Catch2's runner with the Catch3 runner</title>
<updated>2026-10-23T20:25:41+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-10-23T20:25:41+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=93d4e7dbebab867e0a7f59fc25308460a7a76280'/>
<id>93d4e7dbebab867e0a7f59fc25308460a7a76280</id>
<content type='text'>
Catch3 keeps Catch2's API for writing tests (test cases, sections,
assertions, matchers, generators, listeners, benchmarks) and replaces its
session, command line and reporters with a runner of its own.

The runner starts the test binary again as worker processes that take the
next test when they are free. A crash, an exit or a hang ends one test, not
the run. On top of that it adds:

- retries that tell deterministic, flaky and order-dependent failures
  apart, and a bisection that names the test leaving state behind;
- a history of verdicts and durations, kept under a file lock, that drives
  automatic quarantine, adaptive time limits, predictive ordering,
  incremental runs, failure replay and analytics;
- selection by change, past failure or prediction, dependency closure,
  and sharding by a stable hash or by recorded durations;
- tags for serial and isolated tests, shared resources, dependencies,
  time limits and retries;
- a failure category, a signature, an analysis and a replay command for
  every failure;
- console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome
  trace reports, and a versioned discovery listing (--list=json);
- a temporary directory per test (TMPDIR, TMP, TEMP and
  TestTemporaryDirectory());
- Bazel's test environment: XML output file, test filter, sharding, seed,
  premature exit file and undeclared outputs.

The command line is parsed by a hand-written, typed parser. Catch::Session
remains as a thin facade over the runner for custom main functions.

The result model in Progmasoft/Catch3/Results.hpp was rewritten around
tests and attempts, and the JUnit writer now writes it. The property
engine's example database is updated under a file lock with an atomic
rename, so parallel workers can write it at the same time.

Removed: Clara, the command-line and reporter-spec parsers, every reporter
and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width,
the JSON and XML writers, sharding in the test registry, approval tests and
their baselines, Conan packaging, and the unused third_party/clara.hpp.
Listeners keep working; reporters' list and skip events are gone.

The Catch2 library is now compiled as C++20; code that includes only
Catch2's headers still needs C++14. Duplicate test cases are still refused
at start-up. Version 0.3.0.

ExtraTests now turn exceptions off for clang-cl with MSVC-style options,
which -fno-exceptions did not do.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Catch3 keeps Catch2's API for writing tests (test cases, sections,
assertions, matchers, generators, listeners, benchmarks) and replaces its
session, command line and reporters with a runner of its own.

The runner starts the test binary again as worker processes that take the
next test when they are free. A crash, an exit or a hang ends one test, not
the run. On top of that it adds:

- retries that tell deterministic, flaky and order-dependent failures
  apart, and a bisection that names the test leaving state behind;
- a history of verdicts and durations, kept under a file lock, that drives
  automatic quarantine, adaptive time limits, predictive ordering,
  incremental runs, failure replay and analytics;
- selection by change, past failure or prediction, dependency closure,
  and sharding by a stable hash or by recorded durations;
- tags for serial and isolated tests, shared resources, dependencies,
  time limits and retries;
- a failure category, a signature, an analysis and a replay command for
  every failure;
- console, JUnit XML (with flakyFailure and rerunFailure), JSON and Chrome
  trace reports, and a versioned discovery listing (--list=json);
- a temporary directory per test (TMPDIR, TMP, TEMP and
  TestTemporaryDirectory());
- Bazel's test environment: XML output file, test filter, sharding, seed,
  premature exit file and undeclared outputs.

The command line is parsed by a hand-written, typed parser. Catch::Session
remains as a thin facade over the runner for custom main functions.

The result model in Progmasoft/Catch3/Results.hpp was rewritten around
tests and attempts, and the JUnit writer now writes it. The property
engine's example database is updated under a file lock with an atomic
rename, so parallel workers can write it at the same time.

Removed: Clara, the command-line and reporter-spec parsers, every reporter
and CATCH_REGISTER_REPORTER, listing, text flow, console colour and width,
the JSON and XML writers, sharding in the test registry, approval tests and
their baselines, Conan packaging, and the unused third_party/clara.hpp.
Listeners keep working; reporters' list and skip events are gone.

The Catch2 library is now compiled as C++20; code that includes only
Catch2's headers still needs C++14. Duplicate test cases are still refused
at start-up. Version 0.3.0.

ExtraTests now turn exceptions off for clang-cl with MSVC-style options,
which -fno-exceptions did not do.
</pre>
</div>
</content>
</entry>
<entry>
<title>Complete Catch3 documentation and Windows CI: distinguish compatibility docs, correct fork integration, fix Conan CRT and Xmake test entry point</title>
<updated>2026-09-24T21:36:48+00:00</updated>
<author>
<name>Leitwolf11</name>
</author>
<published>2026-09-24T21:36:48+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=c3a325d284ab2487f472f3a8b928c0b12a200979'/>
<id>c3a325d284ab2487f472f3a8b928c0b12a200979</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>v3.16.0</title>
<updated>2026-08-25T07:32:22+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-25T07:32:22+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=317ac1ed4c0bb6e6b91eafc817e05c488feffcb3'/>
<id>317ac1ed4c0bb6e6b91eafc817e05c488feffcb3</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Make verbosity per-reporter</title>
<updated>2026-08-07T18:57:42+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-08-01T14:52:08+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=a0eba50e9d56c743aff1de7a50d02a6e01e73cdc'/>
<id>a0eba50e9d56c743aff1de7a50d02a6e01e73cdc</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix bad ToC link in docs/command-line.md</title>
<updated>2026-07-24T21:52:44+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-07-24T16:29:28+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=dcbb2d5d84560ad729e296a0cf567c77856859e1'/>
<id>dcbb2d5d84560ad729e296a0cf567c77856859e1</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Add quiet verbosity variant to the default listener and tag listing</title>
<updated>2026-02-16T21:52:39+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-02-16T21:51:27+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=a404f37cec56e6ff161db9b26cfb1e77de3bc1e0'/>
<id>a404f37cec56e6ff161db9b26cfb1e77de3bc1e0</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>v3.13.0</title>
<updated>2026-02-15T21:55:48+00:00</updated>
<author>
<name>Martin Hořeňovský</name>
</author>
<published>2026-02-15T21:55:48+00:00</published>
<link rel='alternate' type='text/html' href='https://git.progmasoft.com/catch3/commit/?id=29c9844f688acb27c87338c39cd186ebfe41aa19'/>
<id>29c9844f688acb27c87338c39cd186ebfe41aa19</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
