Add new SKIP macro for skipping tests at runtime (#2360)

* Add new SKIP macro for skipping tests at runtime

This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.

* Don't show "skipped assertions" in console/compact reporters

Also extend skip tests to cover a few more use cases.

* Return exit code 4 if all test cases are skipped

* Use LightGrey for the skip colour

This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.

* Add support for explicit skips in all reporters

* --allow-running-no-tests also allows all tests to be skipped

* Add docs for SKIP macro, deprecate IEventListener::skipTest

Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>

1 files changed, 73 insertions, 0 deletions

diff --git a/tests/SelfTest/UsageTests/Skip.tests.cpp b/tests/SelfTest/UsageTests/Skip.tests.cpp
new file mode 100644
index 00000000..6bd4189b
--- /dev/null
+++ b/tests/SelfTest/UsageTests/Skip.tests.cpp
@@ -0,0 +1,73 @@
+
+//              Copyright Catch2 Authors
+// Distributed under the Boost Software License, Version 1.0.
+//   (See accompanying file LICENSE.txt or copy at
+//        https://www.boost.org/LICENSE_1_0.txt)
+
+// SPDX-License-Identifier: BSL-1.0
+
+#include <catch2/catch_test_macros.hpp>
+#include <catch2/generators/catch_generators_range.hpp>
+
+#include <iostream>
+
+TEST_CASE( "tests can be skipped dynamically at runtime", "[skipping]" ) {
+    SKIP();
+    FAIL( "this is not reached" );
+}
+
+TEST_CASE( "skipped tests can optionally provide a reason", "[skipping]" ) {
+    const int answer = 43;
+    SKIP( "skipping because answer = " << answer );
+    FAIL( "this is not reached" );
+}
+
+TEST_CASE( "sections can be skipped dynamically at runtime", "[skipping]" ) {
+    SECTION( "not skipped" ) { SUCCEED(); }
+    SECTION( "skipped" ) { SKIP(); }
+    SECTION( "also not skipped" ) { SUCCEED(); }
+}
+
+TEST_CASE( "nested sections can be skipped dynamically at runtime",
+           "[skipping]" ) {
+    SECTION( "A" ) { std::cout << "a"; }
+    SECTION( "B" ) {
+        SECTION( "B1" ) { std::cout << "b1"; }
+        SECTION( "B2" ) { SKIP(); }
+    }
+    std::cout << "!\n";
+}
+
+TEST_CASE( "dynamic skipping works with generators", "[skipping]" ) {
+    const int answer = GENERATE( 41, 42, 43 );
+    if ( answer != 42 ) { SKIP( "skipping because answer = " << answer ); }
+    SUCCEED();
+}
+
+TEST_CASE( "failed assertions before SKIP cause test case to fail",
+           "[skipping][!shouldfail]" ) {
+    CHECK( 3 == 4 );
+    SKIP();
+}
+
+TEST_CASE( "a succeeding test can still be skipped",
+           "[skipping][!shouldfail]" ) {
+    SUCCEED();
+    SKIP();
+}
+
+TEST_CASE( "failing in some unskipped sections causes entire test case to fail",
+           "[skipping][!shouldfail]" ) {
+    SECTION( "skipped" ) { SKIP(); }
+    SECTION( "not skipped" ) { FAIL(); }
+}
+
+TEST_CASE( "failing for some generator values causes entire test case to fail",
+           "[skipping][!shouldfail]" ) {
+    int i = GENERATE( 1, 2, 3, 4 );
+    if ( i % 2 == 0 ) {
+        SKIP();
+    } else {
+        FAIL();
+    }
+}