Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
path: root/.github
diff options
context:
space:
mode:
authorLeitwolf112026-11-13 07:46:08 +0300
committerLeitwolf112026-11-13 07:46:08 +0300
commit17248837a78462596809f902dd47cf113a39670b (patch)
tree666d16ddc0a69acb805a5f32f17c8502208349e7 /.github
parent43dabc9adb45b808ae24173ff4633774aa71b2ee (diff)
downloadcatch3-feature/per-test-coverage.tar.gz
Keep per-test coverage with --coverage (roadmap step 4)feature/per-test-coverage
--coverage=DIR runs every attempt in a process of its own and keeps the coverage of each test, for binaries built with LLVM source-based coverage (Clang: -fprofile-instr-generate -fcoverage-mapping) or gcov (GCC: --coverage). How it works: - The scheduler gives each attempt a fresh worker, as --isolate does, and sets LLVM_PROFILE_FILE and GCOV_PREFIX for it: the profile the process writes at exit is then the coverage of that one test, in DIR/profiles/t<index>-a<attempt>.profraw or a .gcov directory. Each runtime reads its own variable, so no symbol of either runtime is looked up: weak references would not pull them out of their static archives, and Mach-O does not allow undefined weak symbols. - A process that runs other tests first (bisection) writes beside the profiles, never into one. The attempt records its profile, also in the JSON results. - The profiles of an earlier run are removed when the run starts, because gcov adds to the .gcda files it finds. - After the run, CoverageReport reads the last profile of every test: llvm-profdata and llvm-cov (PATH, xcrun on macOS, or CATCH3_LLVM_PROFDATA / CATCH3_LLVM_COV) export LCOV, gcov (or CATCH3_GCOV) writes JSON next to a copy of the .gcno. The result is DIR/coverage.json, the files, lines and branches each test reached, and DIR/coverage.lcov for the whole run. Without the tools the profiles stay and the run says so. - --coverage with --in-process or --break is refused. Tests: option parsing, LCOV reading and writing, gcov JSON, and an integration test on CoverageFixture, a binary CMake builds with the coverage flags of Clang or GCC: two tests reach different lines, each map holds only its own, the LCOV report holds both, and a stale profile of an earlier run is gone. Locally all 84 CTest tests pass with Clang, and the coverage tests pass with GCC (built with CATCH3_ENABLE_WERROR=OFF: Catch3 does not build with GCC's warnings as errors, before and after this change). CI: the Linux jobs install LLVM's coverage tools, a new job checks the gcov path with GCC, and CATCH3_COVERAGE_REQUIRE_MAP turns a missing map into a failure on Linux and macOS instead of a skip. MSVC has no source-based coverage, so its builds skip those tests; the roadmap lists step 4 as in progress until Windows with clang-cl is tested.
Diffstat (limited to '.github')
-rw-r--r--.github/workflows/linux-builds.yml52
-rw-r--r--.github/workflows/mac-builds.yml2
2 files changed, 48 insertions, 6 deletions
diff --git a/.github/workflows/linux-builds.yml b/.github/workflows/linux-builds.yml
index a9c38c66..02ee8d67 100644
--- a/.github/workflows/linux-builds.yml
+++ b/.github/workflows/linux-builds.yml
@@ -32,10 +32,10 @@ jobs:
with:
persist-credentials: false
- - name: Install Clang, CMake and Ninja
+ - name: Install Clang, LLVM's coverage tools, CMake and Ninja
run: |
sudo apt-get update
- sudo apt-get install -y clang cmake ninja-build python3
+ sudo apt-get install -y clang llvm cmake ninja-build python3
- name: Configure
run: |
@@ -51,6 +51,8 @@ jobs:
- name: Test
run: ctest --test-dir build -j
+ env:
+ CATCH3_COVERAGE_REQUIRE_MAP: 1
- name: Install, and find the installed package
run: |
@@ -75,10 +77,10 @@ jobs:
with:
persist-credentials: false
- - name: Install Clang, CMake and Ninja
+ - name: Install Clang, LLVM's coverage tools, CMake and Ninja
run: |
sudo apt-get update
- sudo apt-get install -y clang cmake ninja-build python3
+ sudo apt-get install -y clang llvm cmake ninja-build python3
- name: Configure with the sanitizers
env:
@@ -111,8 +113,8 @@ jobs:
steps:
# Git comes first: without it the checkout falls back to an archive of
# the repository, which is not a checkout the tests can ask questions of.
- - name: Install Git, Clang, CMake and Ninja
- run: dnf install -y git clang gcc-c++ cmake ninja-build python3
+ - name: Install Git, Clang, LLVM's coverage tools, CMake and Ninja
+ run: dnf install -y git clang llvm gcc-c++ cmake ninja-build python3
- uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
with:
@@ -132,3 +134,41 @@ jobs:
- name: Test
run: ctest --test-dir build -j
+ env:
+ CATCH3_COVERAGE_REQUIRE_MAP: 1
+
+ # Catch3 is built with Clang everywhere; GCC's stricter warnings are not
+ # errors here. The job checks per-test coverage in gcov's format, which
+ # only GCC writes.
+ gcc-coverage:
+ name: Ubuntu 26.04, GCC, per-test coverage with gcov
+ runs-on: ubuntu-26.04
+ timeout-minutes: 30
+
+ steps:
+ - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6.1.0
+ with:
+ persist-credentials: false
+
+ - name: Install GCC, CMake and Ninja
+ run: |
+ sudo apt-get update
+ sudo apt-get install -y g++ cmake ninja-build python3
+
+ - name: Configure
+ run: |
+ cmake --preset basic-tests -GNinja \
+ -DCMAKE_BUILD_TYPE=Debug \
+ -DCMAKE_CXX_COMPILER=g++ \
+ -DCMAKE_CXX_STANDARD=17 \
+ -DCATCH3_ENABLE_WERROR=OFF
+
+ - name: Build
+ run: cmake --build build --target ProgmasoftSelfTest
+
+ - name: Test per-test coverage
+ env:
+ CATCH3_RUNNER_FIXTURE: build/tests/RunnerFixture
+ CATCH3_COVERAGE_FIXTURE: build/tests/CoverageFixture
+ CATCH3_COVERAGE_REQUIRE_MAP: 1
+ run: build/tests/ProgmasoftSelfTest --no-history "[coverage]"
diff --git a/.github/workflows/mac-builds.yml b/.github/workflows/mac-builds.yml
index c115bf00..4044db9b 100644
--- a/.github/workflows/mac-builds.yml
+++ b/.github/workflows/mac-builds.yml
@@ -44,3 +44,5 @@ jobs:
- name: Test
run: ctest --test-dir build -j
+ env:
+ CATCH3_COVERAGE_REQUIRE_MAP: 1