diff options
| author | Martin Hořeňovský | 2022-04-04 21:07:04 +0200 |
|---|---|---|
| committer | Martin Hořeňovský | 2022-04-07 12:50:08 +0200 |
| commit | 423e1d2ebbd75bb92cb486f83f2c5800b4b9689e (patch) | |
| tree | 3cfff7f75517b1771ab43c7439ba195a77dcec31 | |
| parent | 3c06bcb374aaebf91ad1d63aba655dee66cf7b91 (diff) | |
| download | catch3-423e1d2ebbd75bb92cb486f83f2c5800b4b9689e.tar.gz | |
Integrate the new reporter specs into Catch2
This means that the CLI interface now uses the new key-value oriented
reporter spec, the common reporter base creates the colour implementation
based on the reporter-specific configuration, and it also stores the
custom configuration options for each reporter instance.
Closes #339 as it allows per-reporter forcing of ansi colour codes.
29 files changed, 395 insertions, 448 deletions
diff --git a/docs/command-line.md b/docs/command-line.md index 29e6a365..829c156d 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -123,22 +123,33 @@ Test names containing special characters, such as `,` or `[` can specify them on <a id="choosing-a-reporter-to-use"></a> ## Choosing a reporter to use -<pre>-r, --reporter <reporter[::output-file]></pre> - -> Support for providing output-file through the `-r`, `--reporter` flag was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z +<pre>-r, --reporter <reporter[::key=value]*></pre> Reporters are how the output from Catch2 (results of assertions, tests, benchmarks and so on) is formatted and written out. The default reporter is called the "Console" reporter and is intended to provide relatively verbose and human-friendly output. +Reporters are also individually configurable. To pass configuration options +to the reporter, you append `::key=value` to the reporter specification +as many times as you want, e.g. `--reporter xml::out=someFile.xml`. + +The keys must either be prefixed by "X", in which case they are not parsed +by Catch2 and are only passed down to the reporter, or one of options +hardcoded into Catch2. Currently there are only 2, +["out"](#sending-output-to-a-file), and ["colour-mode"](#colour-mode). + +_Note that the reporter might still check the X-prefixed options for +validity, and throw an error if they are wrong._ + +> Support for passing arguments to reporters through the `-r`, `--reporter` flag was introduced in Catch2 X.Y.Z + There are multiple built-in reporters, you can see what they do by using the [`--list-reporter`](command-line.md#listing-available-tests-tags-or-reporters) flag. If you need a reporter providing custom format outside of the already provided ones, look at the ["write your own reporter" part of the reporter documentation](reporters.md#writing-your-own-reporter). - This option may be passed multiple times to use multiple (different) reporters at the same time. See the [reporter documentation](reporters.md#multiple-reporters) for details on what the resulting behaviour is. Also note that at most one @@ -148,13 +159,12 @@ the [`-o`, `--out`](#sending-output-to-a-file) option. > Support for using multiple different reporters at the same time was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z -As with the `--out` option, using `-` for the output file name sends the -output to stdout. _Note: There is currently no way to escape `::` in the reporter spec, -and thus reporter/file names with `::` in them will not work properly. -As `::` in paths is relatively obscure (unlike `:`), we do not consider -this an issue._ +and thus the reporter names, or configuration keys and values, cannot +contain `::`. As `::` in paths is relatively obscure (unlike ':'), we do +not consider this an issue._ + <a id="breaking-into-the-debugger"></a> ## Breaking into the debugger diff --git a/docs/reporters.md b/docs/reporters.md index ab6f6453..5f86b02e 100644 --- a/docs/reporters.md +++ b/docs/reporters.md @@ -43,9 +43,14 @@ them write into different destinations. The two main uses of this are Specifying multiple reporter looks like this: ``` ---reporter console::- --reporter JUnit::result-junit.xml +--reporter JUnit::out=result-junit.xml --reporter console::out=-::colour-mode=ansi ``` +This tells Catch2 to use two reporters, `JUnit` reporter that writes +its machine-readable XML output to file `result-junit.xml`, and the +`console` reporter that writes its user-friendly output to stdout and +uses ANSI colour codes for colouring the output. + Using multiple reporters (or one reporter and one-or-more [event listeners](event-listener.md#top)) can have surprisingly complex semantics when using customization points provided to reporters by Catch2, namely @@ -162,6 +167,26 @@ Currently there are two customization options: format includes passing assertions even without the `-s` flag. +### Per-reporter configuration + +> Per-reporter configuration was introduced in Catch2 X.Y.Z + +Catch2 supports some configuration to happen per reporter. The configuration +options fall into one of two categories: + +* Catch2-recognized options +* Reporter-specific options + +The former is a small set of universal options that Catch2 handles for +the reporters, e.g. output file or console colour mode. The latter are +options that the reporters have to handle themselves, but the keys and +values can be arbitrary strings, as long as they don't contain `::`. This +allows writing reporters that can be significantly customized at runtime. + +Reporter-specific options always have to be prefixed with "X" (large +letter X). + + ### Other expected functionality of a reporter When writing a custom reporter, there are few more things that you should diff --git a/src/catch2/catch_config.cpp b/src/catch2/catch_config.cpp index 4c8b3319..75a92883 100644 --- a/src/catch2/catch_config.cpp +++ b/src/catch2/catch_config.cpp @@ -33,17 +33,6 @@ namespace Catch { } // unnamed namespace } // namespace Detail - std::ostream& operator<<( std::ostream& os, - ConfigData::ReporterAndFile const& reporter ) { - os << "{ " << reporter.reporterName << ", "; - if ( reporter.outputFileName ) { - os << *reporter.outputFileName; - } else { - os << "<default-output>"; - } - return os << " }"; - } - Config::Config( ConfigData const& data ): m_data( data ) { // We need to trim filter specs to avoid trouble with superfluous @@ -76,14 +65,14 @@ namespace Catch { #else "console", #endif - {} + {}, {}, {} } ); } bool defaultOutputUsed = false; m_reporterStreams.reserve( m_data.reporterSpecifications.size() ); - for ( auto const& reporterAndFile : m_data.reporterSpecifications ) { - if ( reporterAndFile.outputFileName.none() ) { + for ( auto const& reporterSpec : m_data.reporterSpecifications ) { + if ( reporterSpec.outputFile().none() ) { CATCH_ENFORCE( !defaultOutputUsed, "Internal error: cannot use default output for " "multiple reporters" ); @@ -93,7 +82,7 @@ namespace Catch { openStream( data.defaultOutputFilename ) ); } else { m_reporterStreams.push_back( - openStream( *reporterAndFile.outputFileName ) ); + openStream( *reporterSpec.outputFile() ) ); } } } @@ -108,7 +97,7 @@ namespace Catch { std::vector<std::string> const& Config::getTestsOrTags() const { return m_data.testsOrTags; } std::vector<std::string> const& Config::getSectionsToRun() const { return m_data.sectionsToRun; } - std::vector<ConfigData::ReporterAndFile> const& Config::getReportersAndOutputFiles() const { + std::vector<ReporterSpec> const& Config::getReporterSpecs() const { return m_data.reporterSpecifications; } diff --git a/src/catch2/catch_config.hpp b/src/catch2/catch_config.hpp index 91ce4e64..324d634c 100644 --- a/src/catch2/catch_config.hpp +++ b/src/catch2/catch_config.hpp @@ -14,6 +14,7 @@ #include <catch2/internal/catch_unique_ptr.hpp> #include <catch2/internal/catch_optional.hpp> #include <catch2/internal/catch_random_seed_generation.hpp> +#include <catch2/internal/catch_reporter_spec_parser.hpp> #include <iosfwd> #include <vector> @@ -24,17 +25,6 @@ namespace Catch { class IStream; struct ConfigData { - struct ReporterAndFile { - std::string reporterName; - - // If none, the output goes to the default output. - Optional<std::string> outputFileName; - - friend bool operator==(ReporterAndFile const& lhs, ReporterAndFile const& rhs) { - return lhs.reporterName == rhs.reporterName && lhs.outputFileName == rhs.outputFileName; - } - friend std::ostream& operator<<(std::ostream &os, ReporterAndFile const& reporter); - }; bool listTests = false; bool listTags = false; @@ -72,7 +62,7 @@ namespace Catch { std::string defaultOutputFilename; std::string name; std::string processName; - std::vector<ReporterAndFile> reporterSpecifications; + std::vector<ReporterSpec> reporterSpecifications; std::vector<std::string> testsOrTags; std::vector<std::string> sectionsToRun; @@ -90,7 +80,7 @@ namespace Catch { bool listTags() const; bool listReporters() const; - std::vector<ConfigData::ReporterAndFile> const& getReportersAndOutputFiles() const; + std::vector<ReporterSpec> const& getReporterSpecs() const; IStream const* getReporterOutputStream(std::size_t reporterIdx) const; std::vector<std::string> const& getTestsOrTags() const override; diff --git a/src/catch2/catch_session.cpp b/src/catch2/catch_session.cpp index 6af52685..74731cd0 100644 --- a/src/catch2/catch_session.cpp +++ b/src/catch2/catch_session.cpp @@ -41,11 +41,18 @@ namespace Catch { return reporter; } - IStreamingReporterPtr makeReporter(Config const* config) { + IStreamingReporterPtr prepareReporters(Config const* config) { if (Catch::getRegistryHub().getReporterRegistry().getListeners().empty() - && config->getReportersAndOutputFiles().size() == 1) { + && config->getReporterSpecs().size() == 1) { + auto const& spec = config->getReporterSpecs()[0]; auto stream = config->getReporterOutputStream(0); - return createReporter(config->getReportersAndOutputFiles()[0].reporterName, ReporterConfig(config, stream)); + return createReporter( + config->getReporterSpecs()[0].name(), + ReporterConfig( + config, + stream, + spec.colourMode().valueOr( config->colourMode() ), + spec.customOptions() ) ); } auto multi = Detail::make_unique<MultiReporter>(config); @@ -56,9 +63,15 @@ namespace Catch { } std::size_t reporterIdx = 0; - for (auto const& reporterAndFile : config->getReportersAndOutputFiles()) { + for (auto const& reporterSpec : config->getReporterSpecs()) { auto stream = config->getReporterOutputStream(reporterIdx); - multi->addReporter(createReporter(reporterAndFile.reporterName, ReporterConfig(config, stream))); + multi->addReporter( createReporter( + reporterSpec.name(), + ReporterConfig( config, + stream, + reporterSpec.colourMode().valueOr( + config->colourMode() ), + reporterSpec.customOptions() ) ) ); reporterIdx++; } @@ -304,7 +317,7 @@ namespace Catch { getCurrentMutableContext().setConfig(m_config.get()); // Create reporter(s) so we can route listings through them - auto reporter = makeReporter(m_config.get()); + auto reporter = prepareReporters(m_config.get()); auto const& invalidSpecs = m_config->testSpec().getInvalidSpecs(); if ( !invalidSpecs.empty() ) { diff --git a/src/catch2/interfaces/catch_interfaces_reporter.cpp b/src/catch2/interfaces/catch_interfaces_reporter.cpp index 5f5d274a..585d4365 100644 --- a/src/catch2/interfaces/catch_interfaces_reporter.cpp +++ b/src/catch2/interfaces/catch_interfaces_reporter.cpp @@ -14,17 +14,31 @@ #include <catch2/internal/catch_string_manip.hpp> #include <catch2/catch_test_case_info.hpp> #include <catch2/reporters/catch_reporter_helpers.hpp> +#include <catch2/internal/catch_move_and_forward.hpp> #include <algorithm> #include <iomanip> namespace Catch { - ReporterConfig::ReporterConfig( IConfig const* _fullConfig, IStream const* _stream ) - : m_stream( _stream ), m_fullConfig( _fullConfig ) {} + ReporterConfig::ReporterConfig( + IConfig const* _fullConfig, + IStream const* _stream, + ColourMode colourMode, + std::map<std::string, std::string> customOptions ): + m_stream( _stream ), + m_fullConfig( _fullConfig ), + m_colourMode( colourMode ), + m_customOptions( CATCH_MOVE( customOptions ) ) {} IStream const* ReporterConfig::stream() const { return m_stream; } IConfig const * ReporterConfig::fullConfig() const { return m_fullConfig; } + ColourMode ReporterConfig::colourMode() const { return m_colourMode; } + + std::map<std::string, std::string> const& + ReporterConfig::customOptions() const { + return m_customOptions; + } AssertionStats::AssertionStats( AssertionResult const& _assertionResult, std::vector<MessageInfo> const& _infoMessages, diff --git a/src/catch2/interfaces/catch_interfaces_reporter.hpp b/src/catch2/interfaces/catch_interfaces_reporter.hpp index 53d5ae60..9c367fe9 100644 --- a/src/catch2/interfaces/catch_interfaces_reporter.hpp +++ b/src/catch2/interfaces/catch_interfaces_reporter.hpp @@ -19,6 +19,7 @@ #include <catch2/benchmark/catch_outlier_classification.hpp> +#include <map> #include <string> #include <vector> #include <iosfwd> @@ -31,16 +32,24 @@ namespace Catch { class TestCaseHandle; struct IConfig; class IStream; + enum class ColourMode : std::uint8_t; struct ReporterConfig { - ReporterConfig( IConfig const* _fullConfig, IStream const* _stream ); + ReporterConfig( IConfig const* _fullConfig, + IStream const* _stream, + ColourMode colourMode, + std::map<std::string, std::string> customOptions ); IStream const* stream() const; IConfig const* fullConfig() const; + ColourMode colourMode() const; + std::map<std::string, std::string> const& customOptions() const; private: IStream const* m_stream; IConfig const* m_fullConfig; + ColourMode m_colourMode; + std::map<std::string, std::string> m_customOptions; }; struct TestRunInfo { diff --git a/src/catch2/internal/catch_commandline.cpp b/src/catch2/internal/catch_commandline.cpp index 5095f1db..0033a92f 100644 --- a/src/catch2/internal/catch_commandline.cpp +++ b/src/catch2/internal/catch_commandline.cpp @@ -141,57 +141,42 @@ namespace Catch { return ParserResult::runtimeError( "Unrecognised verbosity, '" + verbosity + '\'' ); return ParserResult::ok( ParseResultType::Matched ); }; - auto const setReporter = [&]( std::string const& reporterSpec ) { - if ( reporterSpec.empty() ) { + auto const setReporter = [&]( std::string const& userReporterSpec ) { + if ( userReporterSpec.empty() ) { return ParserResult::runtimeError( "Received empty reporter spec." ); } - // Exactly one of the reporters may be specified without an output - // file, in which case it defaults to the output specified by "-o" - // (or standard output). - static constexpr auto separator = "::"; - static constexpr size_t separatorSize = 2; - auto fileNameSeparatorPos = reporterSpec.find( separator ); - const bool containsFileName = fileNameSeparatorPos != reporterSpec.npos; - if ( containsFileName ) { - auto nextSeparatorPos = reporterSpec.find( - separator, fileNameSeparatorPos + separatorSize ); - if ( nextSeparatorPos != reporterSpec.npos ) { - return ParserResult::runtimeError( - "Too many separators in reporter spec '" + reporterSpec + '\'' ); - } + Optional<ReporterSpec> parsed = + parseReporterSpec( userReporterSpec ); + if ( !parsed ) { + return ParserResult::runtimeError( + "Could not parse reporter spec '" + userReporterSpec + + "'" ); } - std::string reporterName; - Optional<std::string> outputFileName; - reporterName = reporterSpec.substr( 0, fileNameSeparatorPos ); - if ( reporterName.empty() ) { - return ParserResult::runtimeError( "Reporter name cannot be empty." ); - } - - if ( containsFileName ) { - outputFileName = reporterSpec.substr( - fileNameSeparatorPos + separatorSize, reporterSpec.size() ); - } + auto const& reporterSpec = *parsed; IReporterRegistry::FactoryMap const& factories = getRegistryHub().getReporterRegistry().getFactories(); - auto result = factories.find( reporterName ); + auto result = factories.find( reporterSpec.name() ); - if( result == factories.end() ) - return ParserResult::runtimeError( "Unrecognized reporter, '" + reporterName + "'. Check available with --list-reporters" ); - if( containsFileName && outputFileName->empty() ) - return ParserResult::runtimeError( "Reporter '" + reporterName + "' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." ); + if ( result == factories.end() ) { + return ParserResult::runtimeError( + "Unrecognized reporter, '" + reporterSpec.name() + + "'. Check available with --list-reporters" ); + } - config.reporterSpecifications.push_back({ std::move(reporterName), std::move(outputFileName) }); - // It would be enough to check this only once at the very end, but there is - // not a place where we could call this check, so do it every time it could fail. - // For valid inputs, this is still called at most once. - if (!containsFileName) { + const bool hadOutputFile = reporterSpec.outputFile().some(); + config.reporterSpecifications.push_back( CATCH_MOVE( *parsed ) ); + // It would be enough to check this only once at the very end, but + // there is not a place where we could call this check, so do it + // every time it could fail. For valid inputs, this is still called + // at most once. + if (!hadOutputFile) { int n_reporters_without_file = 0; for (auto const& spec : config.reporterSpecifications) { - if (spec.outputFileName.none()) { + if (spec.outputFile().none()) { n_reporters_without_file++; } } diff --git a/src/catch2/reporters/catch_reporter_common_base.cpp b/src/catch2/reporters/catch_reporter_common_base.cpp index 85092db8..aa2ab70a 100644 --- a/src/catch2/reporters/catch_reporter_common_base.cpp +++ b/src/catch2/reporters/catch_reporter_common_base.cpp @@ -18,7 +18,9 @@ namespace Catch { IEventListener( config.fullConfig() ), m_wrapped_stream( config.stream() ), m_stream( m_wrapped_stream->stream() ), - m_colour( makeColourImpl( m_config->colourMode(), m_wrapped_stream ) ) {} + m_colour( makeColourImpl( config.colourMode(), m_wrapped_stream ) ), + m_customOptions( config.customOptions() ) + {} ReporterBase::~ReporterBase() = default; diff --git a/src/catch2/reporters/catch_reporter_common_base.hpp b/src/catch2/reporters/catch_reporter_common_base.hpp index ba7ab116..e110763b 100644 --- a/src/catch2/reporters/catch_reporter_common_base.hpp +++ b/src/catch2/reporters/catch_reporter_common_base.hpp @@ -10,6 +10,9 @@ #include <catch2/interfaces/catch_interfaces_reporter.hpp> +#include <map> +#include <string> + namespace Catch { class ColourImpl; @@ -30,7 +33,10 @@ namespace Catch { //! Cached output stream from `m_wrapped_stream` to reduce //! number of indirect calls needed to write output. std::ostream& m_stream; + //! Colour implementation this reporter was configured for Detail::unique_ptr<ColourImpl> m_colour; + //! The custom reporter options user passed down to the reporter + std::map<std::string, std::string> m_customOptions; public: ReporterBase( ReporterConfig const& config ); diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index a4d01dc9..46294c3e 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -435,8 +435,8 @@ if (NOT MEMORYCHECK_COMMAND) add_test(NAME "MultiReporter::CapturingReportersDontPropagateStdOut" COMMAND $<TARGET_FILE:SelfTest> "Sends stuff to stdout and stderr" - --reporter xml::${_NullFile} - --reporter junit::${_NullFile} + --reporter xml::out=${_NullFile} + --reporter junit::out=${_NullFile} ) set_tests_properties("MultiReporter::CapturingReportersDontPropagateStdOut" PROPERTIES @@ -447,8 +447,8 @@ endif() add_test(NAME "MultiReporter::NonCapturingReportersPropagateStdout" COMMAND $<TARGET_FILE:SelfTest> "Sends stuff to stdout and stderr" - --reporter xml::${_NullFile} - --reporter console::${_NullFile} + --reporter xml::out=${_NullFile} + --reporter console::out=${_NullFile} ) set_tests_properties("MultiReporter::NonCapturingReportersPropagateStdout" PROPERTIES @@ -468,13 +468,34 @@ set_tests_properties("Outputs::DashAsOutLocationSendsOutputToStdout" add_test(NAME "Reporters::DashAsLocationInReporterSpecSendsOutputToStdout" COMMAND $<TARGET_FILE:SelfTest> "Factorials are computed" - --reporter console::- + --reporter console::out=- ) set_tests_properties("Reporters::DashAsLocationInReporterSpecSendsOutputToStdout" PROPERTIES PASS_REGULAR_EXPRESSION "All tests passed \\(5 assertions in 1 test case\\)" ) +add_test(NAME "Reporters::ReporterSpecificColourOverridesDefaultColour" + COMMAND + $<TARGET_FILE:SelfTest> "Factorials are computed" + --reporter console::colour-mode=ansi + --colour-mode none +) +set_tests_properties("Reporters::ReporterSpecificColourOverridesDefaultColour" + PROPERTIES + PASS_REGULAR_EXPRESSION "\\[1\;32mAll tests passed" +) + +add_test(NAME "Reporters::UnrecognizedOptionInSpecCausesError" + COMMAND + $<TARGET_FILE:SelfTest> "Factorials are computed" + --reporter console::bad-option=ansi +) +set_tests_properties("Reporters::UnrecognizedOptionInSpecCausesError" + PROPERTIES + WILL_FAIL ON +) + add_test(NAME "Colours::ColourModeCanBeExplicitlySetToAnsi" COMMAND $<TARGET_FILE:SelfTest> "Factorials are computed" diff --git a/tests/ExtraTests/CMakeLists.txt b/tests/ExtraTests/CMakeLists.txt index 8e6b3094..1a400454 100644 --- a/tests/ExtraTests/CMakeLists.txt +++ b/tests/ExtraTests/CMakeLists.txt @@ -272,8 +272,8 @@ if (NOT MEMORYCHECK_COMMAND) add_test( NAME MultiReporter::NoncapturingListenerDoesntCauseStdoutPassThrough COMMAND ListenerStdoutCaptureInMultireporter - --reporter xml::${_NullFile} - --reporter junit::${_NullFile} + --reporter xml::out=${_NullFile} + --reporter junit::out=${_NullFile} ) set_tests_properties( @@ -290,8 +290,8 @@ target_link_libraries(ListenerCanAskForCapturedStdout PRIVATE Catch2::Catch2With add_test( NAME MultiReporter::CapturingListenerCausesStdoutCapture COMMAND ListenerCanAskForCapturedStdout - --reporter compact::${_NullFile} - --reporter console::${_NullFile} + --reporter compact::out=${_NullFile} + --reporter console::out=${_NullFile} ) set_tests_properties( MultiReporter::CapturingListenerCausesStdoutCapture @@ -317,7 +317,7 @@ add_test( NAME MultiReporter::PreferencesForPassingAssertionsIsRespected COMMAND ReporterPreferencesForPassingAssertionsIsRespected --reporter test-reporter - --reporter console::${_NullFile} + --reporter console::out=${_NullFile} ) set_tests_properties( MultiReporter::PreferencesForPassingAssertionsIsRespected @@ -339,6 +339,20 @@ set_tests_properties( FAIL_REGULAR_EXPRESSION "X28 - ERROR" ) +add_executable(CustomArgumentsForReporters ${TESTS_DIR}/X29-CustomArgumentsForReporters.cpp) +target_link_libraries(CustomArgumentsForReporters PRIVATE Catch2::Catch2WithMain) +add_test( + NAME CustomArgumentsForReporters + COMMAND CustomArgumentsForReporters + --reporter "test-reporter::Xa b=c 1::Xz:e = 1234" +) +set_tests_properties( + CustomArgumentsForReporters + PROPERTIES + PASS_REGULAR_EXPRESSION "Xa b=c 1::Xz:e = 1234" +) + + add_executable(DuplicatedTestCases-SameNameAndTags ${TESTS_DIR}/X31-DuplicatedTestCases.cpp) target_link_libraries(DuplicatedTestCases-SameNameAndTags PRIVATE Catch2::Catch2WithMain) add_test( diff --git a/tests/ExtraTests/X29-CustomArgumentsForReporters.cpp b/tests/ExtraTests/X29-CustomArgumentsForReporters.cpp new file mode 100644 index 00000000..9721775d --- /dev/null +++ b/tests/ExtraTests/X29-CustomArgumentsForReporters.cpp @@ -0,0 +1,59 @@ + +// Copyright Catch2 Authors +// Distributed under the |