Add new SKIP macro for skipping tests at runtime (#2360)

* Add new SKIP macro for skipping tests at runtime

This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.

* Don't show "skipped assertions" in console/compact reporters

Also extend skip tests to cover a few more use cases.

* Return exit code 4 if all test cases are skipped

* Use LightGrey for the skip colour

This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.

* Add support for explicit skips in all reporters

* --allow-running-no-tests also allows all tests to be skipped

* Add docs for SKIP macro, deprecate IEventListener::skipTest

Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>

47 files changed, 3723 insertions, 2172 deletions

diff --git a/docs/Readme.md b/docs/Readme.md
index 52eb64a5..126ee1be 100644
--- a/docs/Readme.md
+++ b/docs/Readme.md
@@ -11,6 +11,7 @@ Once you're up and running consider the following reference material.
 * [Logging macros](logging.md#top)
 * [Test cases and sections](test-cases-and-sections.md#top)
 * [Test fixtures](test-fixtures.md#top)
+* [Skipping tests at runtime](skipping.md#top)
 * [Reporters (output customization)](reporters.md#top)
 * [Event Listeners](event-listeners.md#top)
 * [Data Generators (value parameterized tests)](generators.md#top)
diff --git a/docs/command-line.md b/docs/command-line.md
index 6fc99f59..e7a448b1 100644
--- a/docs/command-line.md
+++ b/docs/command-line.md
@@ -561,10 +561,10 @@ processes, as is done with the [Bazel test sharding](https://docs.bazel.build/ve
 
 > Introduced in Catch2 3.0.1.
 
-By default, Catch2 test binaries return non-0 exit code if no tests were
-run, e.g. if the binary was compiled with no tests, or the provided test
-spec matched no tests. This flag overrides that, so a test run with no
-tests still returns 0.
+By default, Catch2 test binaries return non-0 exit code if no tests were run,
+e.g. if the binary was compiled with no tests, the provided test spec matched no
+tests, or all tests [were skipped at runtime](skipping.md#top). This flag
+overrides that, so a test run with no tests still returns 0.
 
 ## Output verbosity
 ```
diff --git a/docs/deprecations.md b/docs/deprecations.md
index 2c9bf551..5926f2c5 100644
--- a/docs/deprecations.md
+++ b/docs/deprecations.md
@@ -26,6 +26,15 @@ to accurately probe the environment for this information so the flag
 where it will export `BAZEL_TEST=1` for purposes like the above. Catch2
 will now instead inspect the environment instead of relying on build configuration.
 
+### `IEventLister::skipTest( TestCaseInfo const& testInfo )`
+
+This event (including implementations in derived classes such as `ReporterBase`)
+is deprecated and will be removed in the next major release. It is currently
+invoked for all test cases that are not going to be executed due to the test run
+being aborted (when using `--abort` or `--abortx`). It is however
+**NOT** invoked for test cases that are [explicitly skipped using the `SKIP`
+macro](skipping.md#top).
+
 ---
 
 [Home](Readme.md#top)
diff --git a/docs/skipping.md b/docs/skipping.md
new file mode 100644
index 00000000..edd2a1a1
--- /dev/null
+++ b/docs/skipping.md
@@ -0,0 +1,72 @@
+<a id="top"></a>
+# Skipping Test Cases at Runtime
+
+> [Introduced](https://github.com/catchorg/Catch2/pull/2360) in Catch2 X.Y.Z.
+
+In some situations it may not be possible to meaningfully execute a test case, for example when the system under test is missing certain hardware capabilities.
+If the required conditions can only be determined at runtime, it often doesn't make sense to consider such a test case as either passed or failed, because it simply can not run at all.
+To properly express such scenarios, Catch2 allows to explicitly _skip_ test cases, using the `SKIP` macro:
+
+**SKIP(** _message expression_ **)**
+
+Example usage:
+
+```c++
+TEST_CASE("copy files between drives") {
+    if(getNumberOfHardDrives() < 2) {
+        SKIP("at least two hard drives required");
+    }
+    // ...
+}
+```
+
+This test case is then reported as _skipped_ instead of _passed_ or _failed_.
+
+The `SKIP` macro behaves similarly to an explicit [`FAIL`](logging.md#top), in that it is the last expression that will be executed:
+
+```c++
+TEST_CASE("my test") {
+    printf("foo");
+    SKIP();
+    printf("bar"); // not printed
+}
+```
+
+However a failed assertion _before_ a `SKIP` still causes the entire test case to fail:
+
+```c++
+TEST_CASE("failing test") {
+    CHECK(1 == 2);
+    SKIP();
+}
+```
+
+## Interaction with Sections and Generators
+
+Sections, nested sections as well as test cases with [generators](generators.md#top) can all be individually skipped, with the rest executing as usual:
+
+```c++
+TEST_CASE("complex test case") {
+  int value = GENERATE(2, 4, 6);
+  SECTION("a") {
+    SECTION("a1") { CHECK(value < 8); }
+    SECTION("a2") {
+      if (value == 4) {
+        SKIP();
+      }
+      CHECK(value % 2 == 0);
+    }
+  }
+}
+```
+
+This test case will report 5 passing assertions; one for each of the three values in section `a1`, as well as one for each in `a2`, except for when `value == 4`.
+
+Note that as soon as one section is skipped, the entire test case will be reported as _skipped_ (unless there is a failing assertion, in which case it will be reported as _failed_ instead).
+
+If all test cases in a run are skipped, Catch2 returns a non-zero exit code by default.
+This can be overridden using the [--allow-running-no-tests](command-line.md#no-tests-override) flag.
+
+---
+
+[Home](Readme.md#top)
diff --git a/src/catch2/catch_session.cpp b/src/catch2/catch_session.cpp
index 128f21d9..43465f0c 100644
--- a/src/catch2/catch_session.cpp
+++ b/src/catch2/catch_session.cpp
@@ -341,6 +341,12 @@ namespace Catch {
                 return 2;
             }
 
+            if ( totals.testCases.total() > 0 &&
+                 totals.testCases.total() == totals.testCases.skipped
+                && !m_config->zeroTestsCountAsSuccess() ) {
+                return 4;
+            }
+
             // Note that on unices only the lower 8 bits are usually used, clamping
             // the return value to 255 prevents false negative when some multiple
             // of 256 tests has failed
diff --git a/src/catch2/catch_test_macros.hpp b/src/catch2/catch_test_macros.hpp
index cce2852f..1088afbe 100644
--- a/src/catch2/catch_test_macros.hpp
+++ b/src/catch2/catch_test_macros.hpp
@@ -49,6 +49,7 @@
   #define CATCH_FAIL( ... ) INTERNAL_CATCH_MSG( "CATCH_FAIL", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::Normal, __VA_ARGS__ )
   #define CATCH_FAIL_CHECK( ... ) INTERNAL_CATCH_MSG( "CATCH_FAIL_CHECK", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
   #define CATCH_SUCCEED( ... ) INTERNAL_CATCH_MSG( "CATCH_SUCCEED", Catch::ResultWas::Ok, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
+  #define CATCH_SKIP( ... ) INTERNAL_CATCH_MSG( "SKIP", Catch::ResultWas::ExplicitSkip, Catch::ResultDisposition::Normal, __VA_ARGS__ )
 
 
   #if !defined(CATCH_CONFIG_RUNTIME_STATIC_REQUIRE)
@@ -102,6 +103,7 @@
   #define CATCH_FAIL( ... ) (void)(0)
   #define CATCH_FAIL_CHECK( ... ) (void)(0)
   #define CATCH_SUCCEED( ... ) (void)(0)
+  #define CATCH_SKIP( ... ) (void)(0)
 
   #define CATCH_STATIC_REQUIRE( ... )       (void)(0)
   #define CATCH_STATIC_REQUIRE_FALSE( ... ) (void)(0)
@@ -146,6 +148,7 @@
   #define FAIL( ... ) INTERNAL_CATCH_MSG( "FAIL", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::Normal, __VA_ARGS__ )
   #define FAIL_CHECK( ... ) INTERNAL_CATCH_MSG( "FAIL_CHECK", Catch::ResultWas::ExplicitFailure, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
   #define SUCCEED( ... ) INTERNAL_CATCH_MSG( "SUCCEED", Catch::ResultWas::Ok, Catch::ResultDisposition::ContinueOnFailure, __VA_ARGS__ )
+  #define SKIP( ... ) INTERNAL_CATCH_MSG( "SKIP", Catch::ResultWas::ExplicitSkip, Catch::ResultDisposition::Normal, __VA_ARGS__ )
 
 
   #if !defined(CATCH_CONFIG_RUNTIME_STATIC_REQUIRE)
@@ -198,6 +201,7 @@
   #define FAIL( ... ) (void)(0)
   #define FAIL_CHECK( ... ) (void)(0)
   #define SUCCEED( ... ) (void)(0)
+  #define SKIP( ... ) (void)(0)
 
   #define STATIC_REQUIRE( ... )       (void)(0)
   #define STATIC_REQUIRE_FALSE( ... ) (void)(0)
diff --git a/src/catch2/catch_totals.cpp b/src/catch2/catch_totals.cpp
index a3e2b384..bd1954fb 100644
--- a/src/catch2/catch_totals.cpp
+++ b/src/catch2/catch_totals.cpp
@@ -14,6 +14,7 @@ namespace Catch {
         diff.passed = passed - other.passed;
         diff.failed = failed - other.failed;
         diff.failedButOk = failedButOk - other.failedButOk;
+        diff.skipped = skipped - other.skipped;
         return diff;
     }
 
@@ -21,14 +22,15 @@ namespace Catch {
         passed += other.passed;
         failed += other.failed;
         failedButOk += other.failedButOk;
+        skipped += other.skipped;
         return *this;
     }
 
     std::uint64_t Counts::total() const {
-        return passed + failed + failedButOk;
+        return passed + failed + failedButOk + skipped;
     }
     bool Counts::allPassed() const {
-        return failed == 0 && failedButOk == 0;
+        return failed == 0 && failedButOk == 0 && skipped == 0;
     }
     bool Counts::allOk() const {
         return failed == 0;
@@ -53,6 +55,8 @@ namespace Catch {
             ++diff.testCases.failed;
         else if( diff.assertions.failedButOk > 0 )
             ++diff.testCases.failedButOk;
+        else if ( diff.assertions.skipped > 0 )
+            ++ diff.testCases.skipped;
         else
             ++diff.testCases.passed;
         return diff;
diff --git a/src/catch2/catch_totals.hpp b/src/catch2/catch_totals.hpp
index 8dd360c6..386392c9 100644
--- a/src/catch2/catch_totals.hpp
+++ b/src/catch2/catch_totals.hpp
@@ -23,6 +23,7 @@ namespace Catch {
         std::uint64_t passed = 0;
         std::uint64_t failed = 0;
         std::uint64_t failedButOk = 0;
+        std::uint64_t skipped = 0;
     };
 
     struct Totals {
diff --git a/src/catch2/interfaces/catch_interfaces_reporter.hpp b/src/catch2/interfaces/catch_interfaces_reporter.hpp
index 5f286363..da0112e3 100644
--- a/src/catch2/interfaces/catch_interfaces_reporter.hpp
+++ b/src/catch2/interfaces/catch_interfaces_reporter.hpp
@@ -242,7 +242,12 @@ namespace Catch {
          */
         virtual void testRunEnded( TestRunStats const& testRunStats ) = 0;
 
-        //! Called with test cases that are skipped due to the test run aborting
+        /**
+         * Called with test cases that are skipped due to the test run aborting.
+         * NOT called for test cases that are explicitly skipped using the `SKIP` macro.
+         *
+         * Deprecated - will be removed in the next major release.
+         */
         virtual void skipTest( TestCaseInfo const& testInfo ) = 0;
 
         //! Called if a fatal error (signal/structured exception) occured
diff --git a/src/catch2/internal/catch_assertion_handler.cpp b/src/catch2/internal/catch_assertion_handler.cpp
index f051314c..0b14e0bb 100644
--- a/src/catch2/internal/catch_assertion_handler.cpp
+++ b/src/catch2/internal/catch_assertion_handler.cpp
@@ -50,6 +50,13 @@ namespace Catch {
         if (m_reaction.shouldThrow) {
             throw_test_failure_exception();
         }
+        if ( m_reaction.shouldSkip ) {
+#if !defined( CATCH_CONFIG_DISABLE_EXCEPTIONS )
+            throw Catch::TestSkipException();
+#else
+            CATCH_ERROR( "Explicitly skipping tests during runtime requires exceptions" );
+#endif
+        }
     }
     void AssertionHandler::setCompleted() {
         m_completed = true;
diff --git a/src/catch2/internal/catch_assertion_handler.hpp b/src/catch2/internal/catch_assertion_handler.hpp
index 36b55243..ae7776d8 100644
--- a/src/catch2/internal/catch_assertion_handler.hpp
+++ b/src/catch2/internal/catch_assertion_handler.hpp
@@ -22,6 +22,7 @@ namespace Catch {
     struct AssertionReaction {
         bool shouldDebugBreak = false;
         bool shouldThrow = false;
+        bool shouldSkip = false;
     };
 
     class AssertionHandler {
diff --git a/src/catch2/internal/catch_console_colour.hpp b/src/catch2/internal/catch_console_colour.hpp
index 9aa6a163..d9144315 100644
--- a/src/catch2/internal/catch_console_colour.hpp
+++ b/src/catch2/internal/catch_console_colour.hpp
@@ -47,6 +47,7 @@ namespace Catch {
 
             Error = BrightRed,
             Success = Green,
+            Skip = LightGrey,
 
             OriginalExpression = Cyan,
             ReconstructedExpression = BrightYellow,
diff --git a/src/catch2/internal/catch_exception_translator_registry.cpp b/src/catch2/internal/catch_exception_translator_registry.cpp
index 2a240a9b..0645c6ce 100644
--- a/src/catch2/internal/catch_exception_translator_registry.cpp
+++ b/src/catch2/internal/catch_exception_translator_registry.cpp
@@ -44,6 +44,9 @@ namespace Catch {
         catch( TestFailureException& ) {
             std::rethrow_exception(std::current_exception());
         }
+        catch( TestSkipException& ) {
+            std::rethrow_exception(std::current_exception());
+        }
         catch( std::exception const& ex ) {
             return ex.what();
         }
diff --git a/src/catch2/internal/catch_result_type.hpp b/src/catch2/internal/catch_result_type.hpp
index faf0683d..e66afaff 100644
--- a/src/catch2/internal/catch_result_type.hpp
+++ b/src/catch2/internal/catch_result_type.hpp
@@ -16,6 +16,8 @@ namespace Catch {
         Ok = 0,
         Info = 1,
         Warning = 2,
+        // TODO: Should explicit skip be considered "not OK" (cf. isOk)? I.e., should it have the failure bit?
+        ExplicitSkip = 4,
 
         FailureBit = 0x10,
 
diff --git a/src/catch2/internal/catch_run_context.cpp b/src/catch2/internal/catch_run_context.cpp
index d2e8fb8c..e1b81d0b 100644
--- a/src/catch2/internal/catch_run_context.cpp
+++ b/src/catch2/internal/catch_run_context.cpp
@@ -270,6 +270,9 @@ namespace Catch {
         if (result.getResultType() == ResultWas::Ok) {
             m_totals.assertions.passed++;
             m_lastAssertionPassed = true;
+        } else if (result.getResultType() == ResultWas::ExplicitSkip) {
+            m_totals.assertions.skipped++;
+            m_lastAssertionPassed = true;
         } else if (!result.succeeded()) {
             m_lastAssertionPassed = false;
             if (result.isOk()) {
@@ -475,6 +478,8 @@ namespace Catch {
             duration = timer.getElapsedSeconds();
         } CATCH_CATCH_ANON (TestFailureException&) {
             // This just means the test was aborted due to failure
+        } CATCH_CATCH_ANON (TestSkipException&) {
+            // This just means the test was explicitly skipped
         } CATCH_CATCH_ALL {
             // Under CATCH_CONFIG_FAST_COMPILE, unexpected exceptions under REQUIRE assertions
             // are reported without translation at the point of origin.
@@ -571,8 +576,13 @@ namespace Catch {
         data.message = static_cast<std::string>(message);
         AssertionResult assertionResult{ m_lastAssertionInfo, data };
         assertionEnded( assertionResult );
-        if( !assertionResult.isOk() )
+        if ( !assertionResult.isOk() ) {
             populateReaction( reaction );
+        } else if ( resultType == ResultWas::ExplicitSkip ) {
+            // TODO: Need to handle this explicitly, as ExplicitSkip is
+            // considered "OK"
+            reaction.shouldSkip = true;
+        }
     }
     void RunContext::handleUnexpectedExceptionNotThrown(
             AssertionInfo const& info,
diff --git a/src/catch2/internal/catch_test_failure_exception.hpp b/src/catch2/internal/catch_test_failure_exception.hpp
index 810a81c9..13c5fc08 100644
--- a/src/catch2/internal/catch_test_failure_exception.hpp
+++ b/src/catch2/internal/catch_test_failure_exception.hpp
@@ -20,6 +20,9 @@ namespace Catch {
      */
     [[noreturn]] void throw_test_failure_exception();
 
+    //! Used to signal that the remainder of a test should be skipped
+    struct TestSkipException{};
+
 } // namespace Catch
 
 #endif // CATCH_TEST_FAILURE_EXCEPTION_HPP_INCLUDED
diff --git a/src/catch2/reporters/catch_reporter_automake.cpp b/src/catch2/reporters/catch_reporter_automake.cpp
index 0660d092..993b594b 100644
--- a/src/catch2/reporters/catch_reporter_automake.cpp
+++ b/src/catch2/reporters/catch_reporter_automake.cpp
@@ -17,7 +17,9 @@ namespace Catch {
     void AutomakeReporter::testCaseEnded(TestCaseStats const& _testCaseStats) {
         // Possible values to emit are PASS, XFAIL, SKIP, FAIL, XPASS and ERROR.
         m_stream << ":test-result: ";
-        if (_testCaseStats.totals.assertions.allPassed()) {
+        if ( _testCaseStats.totals.testCases.skipped > 0 ) {
+            m_stream << "SKIP";
+        } else if (_testCaseStats.totals.assertions.allPassed()) {
             m_stream << "PASS";
         } else if (_testCaseStats.totals.assertions.allOk()) {
             m_stream << "XFAIL";
diff --git a/src/catch2/reporters/catch_reporter_compact.cpp b/src/catch2/reporters/catch_reporter_compact.cpp
index d8088457..643626ea 100644
--- a/src/catch2/reporters/catch_reporter_compact.cpp
+++ b/src/catch2/reporters/catch_reporter_compact.cpp
@@ -105,6 +105,11 @@ public:
             printIssue("explicitly");
             printRemainingMessages(Colour::None);
             break;
+        case ResultWas::ExplicitSkip:
+            printResultType(Colour::Skip, "skipped"_sr);
+            printMessage();
+            printRemainingMessages();
+            break;
             // These cases are here to prevent compiler warnings
         case ResultWas::Unknown:
         case ResultWas::FailureBit:
@@ -220,7 +225,7 @@ private:
 
             // Drop out if result was successful and we're not printing those
             if( !m_config->includeSuccessfulResults() && result.isOk() ) {
-                if( result.getResultType() != ResultWas::Warning )
+                if( result.getResultType() != ResultWas::Warning && result.getResultType() != ResultWas::ExplicitSkip )
                     return;
                 printInfoMessages = false;
             }
diff --git a/src/catch2/reporters/catch_reporter_console.cpp b/src/catch2/reporters/catch_reporter_console.cpp
index 0edb121e..b394d2be 100644
--- a/src/catch2/reporters/catch_reporter_console.cpp
+++ b/src/catch2/reporters/catch_reporter_console.cpp
@@ -111,6 +111,14 @@ public:
             if (_stats.infoMessages.size() > 1)
                 messageLabel = "explicitly with messages";
             break;
+        case ResultWas::ExplicitSkip:
+            colour = Colour::Skip;
+            passOrFail = "SKIPPED"_sr;
+            if (_stats.infoMessages.size() == 1)
+                messageLabel = "explicitly with message";
+            if (_stats.infoMessages.size() > 1)
+                messageLabel = "explicitly with messages";
+            break;
             // These cases are here to prevent compiler warnings
         case ResultWas::Unknown:
         case ResultWas::FailureBit:
@@ -185,13 +193,16 @@ std::size_t makeRatio( std::uint64_t number, std::uint64_t total ) {
     return (ratio == 0 && number > 0) ? 1 : static_cast<std::size_t>(ratio);
 }
 
-std::size_t& findMax( std::size_t& i, std::size_t& j, std::size_t& k ) {
-    if (i > j && i > k)
+std::size_t&
+findMax( std::size_t& i, std::size_t& j, std::size_t& k, std::size_t& l ) {
+    if (i > j && i > k && i > l)
         return i;
-    else if (j > k)
+    else if (j > k && j > l)
         return j;
-    else
+    else if (k > l)
         return k;
+    else
+        return l;
 }
 
 enum class Justification { Left, Right };
@@ -400,7 +411,8 @@ void ConsoleReporter::assertionEnded(AssertionStats const& _assertionStats) {
     bool includeResults = m_config->includeSuccessfulResults() || !result.isOk();