diff options
| author | Leitwolf11 | 2026-10-18 20:00:39 +0300 |
|---|---|---|
| committer | Leitwolf11 | 2026-10-18 20:00:39 +0300 |
| commit | 8d1fb4807edc7920fbecd0554c9c61625827e19c (patch) | |
| tree | dd3d7e1d4f48ca27140a2bd5f970817ad02ae35a /Guide/TESTING.md | |
| download | progmasoft-cpp-8d1fb4807edc7920fbecd0554c9c61625827e19c.tar.gz | |
Write the first revision of Progmasoft C++ Style
Progmasoft C++ Style is a style guide for C++20, written for anyone
who wants a complete rule set to adopt. It writes down the conventions
the Progmasoft C++ projects already follow (naming, the clang-format
and clang-tidy profiles, exception-free error handling, Doxygen) and
decides what they left open.
Thirteen chapters under Guide/:
1. Principles: what the guide optimizes for and how to decide a case
it does not cover.
2. Formatting: the full .clang-format profile, the reason for each
choice, and the rules the tool cannot apply.
3. Naming: casing for every kind of name, acronyms, choosing words,
namespaces, files.
4. Files and headers: extensions, #pragma once, includes, what goes in
a header, the order inside a class, file size, vendored code.
5. Language: functions, auto, initialization, const and constexpr,
casts, integers, strings, enumerations, templates and concepts,
lambdas, control flow, and the features not used.
6. Types and classes: struct or class, final, construction, the rule of
zero, inheritance, operators, strong types, optional and variant.
7. Errors: no exceptions by default, Expected and Error, invariants,
fatal defects, destructors, diagnostics.
8. Ownership and memory: owners and borrowers, views and lifetimes,
RAII, containers and deterministic iteration, allocation, undefined
behavior.
9. Comments and documentation: Doxygen for public declarations,
comments that explain why, TODOs that name an issue.
10. C interfaces: C11 headers, prefixes, opaque types, ownership and
failure across the boundary, stability.
11. Concurrency: sharing less, threads, locks, thread-safety
annotations, atomics, statics.
12. Testing: unit tests, fuzzing, sanitizers, benchmarks, coverage.
13. Tooling: the compiler flags, the full .clang-tidy profile with the
reason for every disabled check, other checks, version pinning.
The guide prescribes no test, fuzzing or benchmark framework and no
error library; it prescribes how they are used. ADOPTING.md explains
how a project points to the guide, copies the profiles and records its
deviations. CONTRIBUTING.md explains how to propose a change.
The text is licensed under CC BY 4.0. The configuration and code
examples are dedicated to the public domain under CC0 1.0, so projects
can copy them under their own license.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Diffstat (limited to 'Guide/TESTING.md')
| -rw-r--r-- | Guide/TESTING.md | 94 |
1 files changed, 94 insertions, 0 deletions
diff --git a/Guide/TESTING.md b/Guide/TESTING.md new file mode 100644 index 0000000..f35013f --- /dev/null +++ b/Guide/TESTING.md @@ -0,0 +1,94 @@ +<!-- SPDX-FileCopyrightText: 2026 Progmasoft <support@progmasoft.com> --> +<!-- SPDX-License-Identifier: CC-BY-4.0 --> + +# 12. Testing + +A change that alters behavior comes with a test that fails without it. A bug fix comes with the test that reproduces the +bug. Tests are code: they follow this guide, they are reviewed, and a flaky test is a bug. + +## 12.1 Unit tests + +- **Tests live next to the component they test**, in a `Tests/` directory of that component, not in one tree for the + whole repository. The test then moves, and is found, with its code. +- **One test file per unit under test**: `ControlFlowGraphTests.cpp` for `ControlFlowGraph.hpp`. +- **A test name is a sentence that states the behavior**: `"DenseIdMap keeps zero and the maximum id as distinct keys"`. + When it fails, the name alone says what broke. +- **Arrange, act, assert**, separated by blank lines. One behavior per test; several assertions about that one behavior + are fine. +- **Test the contract, not the implementation**: inputs and observable results, boundary values (zero, one, the maximum, + one past it), the empty case, and every documented failure. +- **No shared mutable state between tests**, and no dependence on their order. A test creates what it needs, in a + temporary directory when it touches files. +- **Deterministic**: no wall clock, no unseeded random numbers, no network. Property-based tests use a fixed or a + recorded seed, so a failure can be reproduced. +- **Fast**: a unit test runs in milliseconds. A slow scenario is a separate, labeled test target. + +The guide does not prescribe a test framework. Catch2, [Catch3](https://github.com/Progmasoft/catch3), GoogleTest, +doctest and others all fit; choose one per project and use it everywhere in that project. The example below uses the +Catch2-style API: + +```cpp +TEST_CASE("BitSet union keeps the bits of both operands") +{ + auto left = llvm::cantFail(BitSet::Create(8U)); + auto right = llvm::cantFail(BitSet::Create(8U)); + left.Set(1U); + right.Set(6U); + + // errorToBool consumes the error, so a failure is reported, not leaked. + REQUIRE_FALSE(llvm::errorToBool(left.UnionWith(right))); + + CHECK(left.Test(1U)); + CHECK(left.Test(6U)); + CHECK(left.Count() == 2U); +} +``` + +Whatever the framework, use its two kinds of assertion for what they are for: one that stops a test whose further checks +would make no sense (`REQUIRE`, `ASSERT_*`), and one that reports and continues (`CHECK`, `EXPECT_*`), so one run shows +every failed expectation. + +## 12.2 Fuzzing + +Every parser of untrusted input, every decoder of a binary format and every round trip (encode, then decode) has a fuzz +target. + +- **One target per entry point**, built with AddressSanitizer and UndefinedBehaviorSanitizer. Write it against the + `LLVMFuzzerTestOneInput` entry point, which libFuzzer, AFL++ and honggfuzz all accept, so the engine can be chosen + or changed without touching the target. +- **A seed corpus** of small valid inputs checked into the repository, and a dictionary of the format's keywords and + magic numbers when it has them. +- **Bounded runs in CI** on every change, and longer campaigns on a schedule. A crash found by a fuzzer becomes a + regression test with the minimized input. +- **An oracle beyond "it did not crash"**: a round trip compares the result with the input; a verifier and a + transformation are checked against each other. + +## 12.3 Sanitizers + +CI builds and runs the tests under: + +| Sanitizer | Finds | +| --- | --- | +| AddressSanitizer | out-of-bounds access, use after free, double free, leaks (with LeakSanitizer) | +| UndefinedBehaviorSanitizer | signed overflow, invalid shifts, misaligned access, invalid enum and bool values, null dereference | +| ThreadSanitizer | data races and lock-order inversions, for every test that runs threads | + +UBSan runs with `-fno-sanitize-recover=all`, so the first finding fails the test instead of printing and continuing. +MemorySanitizer may be added where all dependencies can be built with it. + +## 12.4 Benchmarks + +Performance claims are measured. A change that is made for speed comes with a benchmark, and its pull request shows the +numbers before and after. + +- **One benchmark binary per component**, next to its tests, written with the project's benchmark library (Google + Benchmark, nanobench, the benchmarking support of the test framework, or another). +- **Benchmark realistic sizes**, and keep the setup outside the timed region. +- **Compare like with like**: the same machine, an optimized build, several repetitions, and the variance reported along + with the mean. +- A benchmark whose fixture is invalid fails; it must not time a broken workload. + +## 12.5 Coverage + +Coverage is measured in CI and reported per component. It shows what is not tested; it is not a goal in itself, and a +line that is executed is not thereby tested. Review the uncovered branches of a change rather than chasing a percentage. |