diff options
| author | Martin Hořeňovský | 2022-01-01 16:19:31 +0100 |
|---|---|---|
| committer | GitHub | 2022-01-01 16:19:31 +0100 |
| commit | f2f585b4788b61ed202e78b73b105f099d9cce7b (patch) | |
| tree | 174f72ba09e22e564a571f96c21a175a68d85103 | |
| parent | 6b55f5d780c5d8e935720f9d7d8ed28a3ba8617f (diff) | |
| parent | b5547f2ef6be18f70d2e9638d8f1eb0d8d42d2ba (diff) | |
| download | catch3-f2f585b4788b61ed202e78b73b105f099d9cce7b.tar.gz | |
Merge pull request #2183 from mjerabek/dev-multiple-reporters
Add support for multiple reporters
42 files changed, 52355 insertions, 220 deletions
diff --git a/docs/command-line.md b/docs/command-line.md index b03e7e3c..c1568891 100644 --- a/docs/command-line.md +++ b/docs/command-line.md @@ -123,7 +123,9 @@ Test names containing special characters, such as `,` or `[` can specify them on <a id="choosing-a-reporter-to-use"></a> ## Choosing a reporter to use -<pre>-r, --reporter <reporter></pre> +<pre>-r, --reporter <reporter[::output-file]></pre> + +> Support for providing output-file through the `-r`, `--reporter` flag was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z A reporter is an object that formats and structures the output of running tests, and potentially summarises the results. By default a console reporter is used that writes, IDE friendly, textual output. Catch comes bundled with some alternative reporters, but more can be added in client code.<br /> The bundled reporters are: @@ -136,6 +138,15 @@ The bundled reporters are: The JUnit reporter is an xml format that follows the structure of the JUnit XML Report ANT task, as consumed by a number of third-party tools, including Continuous Integration servers such as Jenkins. If not otherwise needed, the standard XML reporter is preferred as this is a streaming reporter, whereas the Junit reporter needs to hold all its results until the end so it can write the overall results into attributes of the root node. +This option may be passed multiple times to use multiple (different) reporters at the same time. See [Reporters](reporters.md#multiple-reporters) for details. + +As with the `--out` flag, `-` means writing to stdout. + +_Note: There is currently no way to escape `::` in the reporter spec, +and thus reporter/file names with `::` in them will not work properly. +As `::` in paths is relatively obscure (unlike `:`), we do not consider +this an issue._ + <a id="breaking-into-the-debugger"></a> ## Breaking into the debugger <pre>-b, --break</pre> @@ -178,11 +189,16 @@ If one or more test-specs have been supplied too then only the matching tests wi <a id="sending-output-to-a-file"></a> ## Sending output to a file -<pre>-o, --out <filename> +<pre>-o, --out <filename> </pre> Use this option to send all output to a file. By default output is sent to stdout (note that uses of stdout and stderr *from within test cases* are redirected and included in the report - so even stderr will effectively end up on stdout). +Using `-` as the filename sends the output to stdout. + +> Support for `-` as the filename was introduced in Catch2 X.Y.Z + + <a id="naming-a-test-run"></a> ## Naming a test run <pre>-n, --name <name for test run></pre> @@ -414,7 +430,7 @@ There are some limitations of this feature to be aware of: - Code outside of sections being skipped will still be executed - e.g. any set-up code in the TEST_CASE before the start of the first section.</br> - At time of writing, wildcards are not supported in section names. -- If you specify a section without narrowing to a test case first then all test cases will be executed +- If you specify a section without narrowing to a test case first then all test cases will be executed (but only matching sections within them). @@ -422,7 +438,7 @@ start of the first section.</br> ## Filenames as tags <pre>-#, --filenames-as-tags</pre> -When this option is used then every test is given an additional tag which is formed of the unqualified +When this option is used then every test is given an additional tag which is formed of the unqualified filename it is found in, with any extension stripped, prefixed with the `#` character. So, for example, tests within the file `~\Dev\MyProject\Ferrets.cpp` would be tagged `[#Ferrets]`. diff --git a/docs/reporters.md b/docs/reporters.md index a33e55bf..9552154b 100644 --- a/docs/reporters.md +++ b/docs/reporters.md @@ -19,13 +19,13 @@ There are four reporters built in to the single include: * `console` writes as lines of text, formatted to a typical terminal width, with colours if a capable terminal is detected. * `compact` similar to `console` but optimised for minimal output - each entry on one line * `junit` writes xml that corresponds to Ant's [junitreport](http://help.catchsoftware.com/display/ET/JUnit+Format) target. Useful for build systems that understand Junit. -Because of the way the junit format is structured the run must complete before anything is written. +Because of the way the junit format is structured the run must complete before anything is written. * `xml` writes an xml format tailored to Catch. Unlike `junit` this is a streaming format so results are delivered progressively. There are a few additional reporters, for specific build systems, in the Catch repository (in `include\reporters`) which you can `#include` in your project if you would like to make use of them. Do this in one source file - the same one you have `CATCH_CONFIG_MAIN` or `CATCH_CONFIG_RUNNER`. -* `teamcity` writes the native, streaming, format that [TeamCity](https://www.jetbrains.com/teamcity/) understands. +* `teamcity` writes the native, streaming, format that [TeamCity](https://www.jetbrains.com/teamcity/) understands. Use this when building as part of a TeamCity build to see results as they happen ([code example](../examples/207-Rpt-TeamCityReporter.cpp)). * `tap` writes in the TAP ([Test Anything Protocol](https://en.wikipedia.org/wiki/Test_Anything_Protocol)) format. * `automake` writes in a format that correspond to [automake .trs](https://www.gnu.org/software/automake/manual/html_node/Log-files-generation-and-test-results-recording.html) files @@ -35,6 +35,19 @@ You see what reporters are available from the command line by running with `--li By default all these reports are written to stdout, but can be redirected to a file with [`-o` or `--out`](command-line.md#sending-output-to-a-file) +<a id="multiple-reporters"></a> +## Using multiple reporters + +> Support for having multiple parallel reporters was [introduced](https://github.com/catchorg/Catch2/pull/2183) in Catch2 X.Y.Z + +Multiple reporters may be used at the same time, e.g. to save a machine-readable output to a file but still print the human-readable output to the console: +``` +-r console -r xml::result.xml -r junit::result-junit.xml +``` + +The output file name is given after the reporter name, delimited by a colon. If omitted, it defaults to the file name specified by `-o` (or stdout). Only one reporter may use the default output. + + ## Writing your own reporter You can write your own custom reporter and register it with Catch. diff --git a/src/catch2/catch_config.cpp b/src/catch2/catch_config.cpp index da9457da..c507b1b6 100644 --- a/src/catch2/catch_config.cpp +++ b/src/catch2/catch_config.cpp @@ -13,13 +13,39 @@ #include <catch2/internal/catch_test_spec_parser.hpp> #include <catch2/interfaces/catch_interfaces_tag_alias_registry.hpp> +#include <ostream> namespace Catch { + namespace Detail { + namespace { + class RDBufStream : public IStream { + mutable std::ostream m_os; + + public: + //! The streambuf `sb` must outlive the constructed object. + RDBufStream( std::streambuf* sb ): m_os( sb ) {} + ~RDBufStream() override = default; + + public: // IStream + std::ostream& stream() const override { return m_os; } + }; + } // unnamed namespace + } // namespace Detail + + std::ostream& operator<<( std::ostream& os, + ConfigData::ReporterAndFile const& reporter ) { + os << "{ " << reporter.reporterName << ", "; + if ( reporter.outputFileName ) { + os << *reporter.outputFileName; + } else { + os << "<default-output>"; + } + return os << " }"; + } - Config::Config( ConfigData const& data ) - : m_data( data ), - m_stream( Catch::makeStream(m_data.outputFilename) ) - { + Config::Config( ConfigData const& data ): + m_data( data ), + m_defaultStream( openStream( data.defaultOutputFilename ) ) { // We need to trim filter specs to avoid trouble with superfluous // whitespace (esp. important for bdd macros, as those are manually // aligned with whitespace). @@ -39,24 +65,37 @@ namespace Catch { } } m_testSpec = parser.testSpec(); + + m_reporterStreams.reserve( m_data.reporterSpecifications.size() ); + for ( auto const& reporterAndFile : m_data.reporterSpecifications ) { + if ( reporterAndFile.outputFileName.none() ) { + m_reporterStreams.emplace_back( new Detail::RDBufStream( + m_defaultStream->stream().rdbuf() ) ); + } else { + m_reporterStreams.emplace_back( + openStream( *reporterAndFile.outputFileName ) ); + } + } } Config::~Config() = default; - std::string const& Config::getFilename() const { - return m_data.outputFilename ; - } - bool Config::listTests() const { return m_data.listTests; } bool Config::listTags() const { return m_data.listTags; } bool Config::listReporters() const { return m_data.listReporters; } - std::string const& Config::getReporterName() const { return m_data.reporterName; } - std::vector<std::string> const& Config::getTestsOrTags() const { return m_data.testsOrTags; } std::vector<std::string> const& Config::getSectionsToRun() const { return m_data.sectionsToRun; } + std::vector<ConfigData::ReporterAndFile> const& Config::getReportersAndOutputFiles() const { + return m_data.reporterSpecifications; + } + + std::ostream& Config::getReporterOutputStream(std::size_t reporterIdx) const { + return m_reporterStreams.at(reporterIdx)->stream(); + } + TestSpec const& Config::testSpec() const { return m_testSpec; } bool Config::hasTestFilters() const { return m_hasTestFilters; } @@ -64,8 +103,8 @@ namespace Catch { // IConfig interface bool Config::allowThrows() const { return !m_data.noThrow; } - std::ostream& Config::stream() const { return m_stream->stream(); } - StringRef Config::name() const { return m_data.name.empty() ? m_data.processName : m_data.name; } + std::ostream& Config::defaultStream() const { return m_defaultStream->stream(); } + StringRef Config::name() const { return m_data.name.empty() ? m_data.processName : m_data.name; } bool Config::includeSuccessfulResults() const { return m_data.showSuccessfulTests; } bool Config::warnAboutMissingAssertions() const { return !!( m_data.warnings & WarnAbout::NoAssertions ); @@ -92,4 +131,8 @@ namespace Catch { unsigned int Config::benchmarkResamples() const { return m_data.benchmarkResamples; } std::chrono::milliseconds Config::benchmarkWarmupTime() const { return std::chrono::milliseconds(m_data.benchmarkWarmupTime); } + Detail::unique_ptr<IStream const> Config::openStream(std::string const& outputFileName) { + return Catch::makeStream(outputFileName); + } + } // end namespace Catch diff --git a/src/catch2/catch_config.hpp b/src/catch2/catch_config.hpp index 2b881f6b..1aaa9828 100644 --- a/src/catch2/catch_config.hpp +++ b/src/catch2/catch_config.hpp @@ -14,6 +14,7 @@ #include <catch2/internal/catch_optional.hpp> #include <catch2/internal/catch_random_seed_generation.hpp> +#include <iosfwd> #include <vector> #include <string> @@ -22,6 +23,18 @@ namespace Catch { struct IStream; struct ConfigData { + struct ReporterAndFile { + std::string reporterName; + + // If none, the output goes to the default output. + Optional<std::string> outputFileName; + + friend bool operator==(ReporterAndFile const& lhs, ReporterAndFile const& rhs) { + return lhs.reporterName == rhs.reporterName && lhs.outputFileName == rhs.outputFileName; + } + friend std::ostream& operator<<(std::ostream &os, ReporterAndFile const& reporter); + }; + bool listTests = false; bool listTags = false; bool listReporters = false; @@ -55,13 +68,17 @@ namespace Catch { UseColour useColour = UseColour::Auto; WaitForKeypress::When waitForKeypress = WaitForKeypress::Never; - std::string outputFilename; + std::string defaultOutputFilename; std::string name; std::string processName; #ifndef CATCH_CONFIG_DEFAULT_REPORTER #define CATCH_CONFIG_DEFAULT_REPORTER "console" #endif - std::string reporterName = CATCH_CONFIG_DEFAULT_REPORTER; + std::vector<ReporterAndFile> reporterSpecifications = { + {CATCH_CONFIG_DEFAULT_REPORTER, {}} + }; + // Internal: used as parser state + bool _nonDefaultReporterSpecifications = false; #undef CATCH_CONFIG_DEFAULT_REPORTER std::vector<std::string> testsOrTags; @@ -76,13 +93,12 @@ namespace Catch { Config( ConfigData const& data ); ~Config() override; // = default in the cpp file - std::string const& getFilename() const; - bool listTests() const; bool listTags() const; bool listReporters() const; - std::string const& getReporterName() const; + std::vector<ConfigData::ReporterAndFile> const& getReportersAndOutputFiles() const; + std::ostream& getReporterOutputStream(std::size_t reporterIdx) const; std::vector<std::string> const& getTestsOrTags() const override; std::vector<std::string> const& getSectionsToRun() const override; @@ -94,7 +110,7 @@ namespace Catch { // IConfig interface bool allowThrows() const override; - std::ostream& stream() const override; + std::ostream& defaultStream() const override; StringRef name() const override; bool includeSuccessfulResults() const override; bool warnAboutMissingAssertions() const override; @@ -118,13 +134,14 @@ namespace Catch { std::chrono::milliseconds benchmarkWarmupTime() const override; private: + Detail::unique_ptr<IStream const> openStream(std::string const& outputFileName); ConfigData m_data; - Detail::unique_ptr<IStream const> m_stream; + Detail::unique_ptr<IStream const> m_defaultStream; + std::vector<Detail::unique_ptr<IStream const>> m_reporterStreams; TestSpec m_testSpec; bool m_hasTestFilters = false; }; - } // end namespace Catch #endif // CATCH_CONFIG_HPP_INCLUDED diff --git a/src/catch2/catch_session.cpp b/src/catch2/catch_session.cpp index 38111fe8..2325bf2f 100644 --- a/src/catch2/catch_session.cpp +++ b/src/catch2/catch_session.cpp @@ -34,7 +34,7 @@ namespace Catch { namespace { const int MaxExitCode = 255; - IStreamingReporterPtr createReporter(std::string const& reporterName, IConfig const* config) { + IStreamingReporterPtr createReporter(std::string const& reporterName, ReporterConfig const& config) { auto reporter = Catch::getRegistryHub().getReporterRegistry().create(reporterName, config); CATCH_ENFORCE(reporter, "No reporter registered with name: '" << reporterName << '\''); @@ -42,16 +42,26 @@ namespace Catch { } IStreamingReporterPtr makeReporter(Config const* config) { - if (Catch::getRegistryHub().getReporterRegistry().getListeners().empty()) { - return createReporter(config->getReporterName(), config); + if (Catch::getRegistryHub().getReporterRegistry().getListeners().empty() + && config->getReportersAndOutputFiles().size() == 1) { + auto& stream = config->getReporterOutputStream(0); + return createReporter(config->getReportersAndOutputFiles()[0].reporterName, ReporterConfig(config, stream)); } auto multi = Detail::make_unique<ListeningReporter>(config); + auto const& listeners = Catch::getRegistryHub().getReporterRegistry().getListeners(); for (auto const& listener : listeners) { - multi->addListener(listener->create(Catch::ReporterConfig(config))); + multi->addListener(listener->create(Catch::ReporterConfig(config, config->defaultStream()))); + } + + std::size_t reporterIdx = 0; + for (auto const& reporterAndFile : config->getReportersAndOutputFiles()) { + auto& stream = config->getReporterOutputStream(reporterIdx); + multi->addReporter(createReporter(reporterAndFile.reporterName, ReporterConfig(config, stream))); + reporterIdx++; } - multi->addReporter(createReporter(config->getReporterName(), config)); + return multi; } diff --git a/src/catch2/interfaces/catch_interfaces_config.hpp b/src/catch2/interfaces/catch_interfaces_config.hpp index b8da0df0..4b2bd365 100644 --- a/src/catch2/interfaces/catch_interfaces_config.hpp +++ b/src/catch2/interfaces/catch_interfaces_config.hpp @@ -61,7 +61,7 @@ namespace Catch { virtual ~IConfig(); virtual bool allowThrows() const = 0; - virtual std::ostream& stream() const = 0; + virtual std::ostream& defaultStream() const = 0; virtual StringRef name() const = 0; virtual bool includeSuccessfulResults() const = 0; virtual bool shouldDebugBreak() const = 0; diff --git a/src/catch2/interfaces/catch_interfaces_reporter.cpp b/src/catch2/interfaces/catch_interfaces_reporter.cpp index 89a86f59..5439254c 100644 --- a/src/catch2/interfaces/catch_interfaces_reporter.cpp +++ b/src/catch2/interfaces/catch_interfaces_reporter.cpp @@ -20,9 +20,6 @@ namespace Catch { - ReporterConfig::ReporterConfig( IConfig const* _fullConfig ) - : m_stream( &_fullConfig->stream() ), m_fullConfig( _fullConfig ) {} - ReporterConfig::ReporterConfig( IConfig const* _fullConfig, std::ostream& _stream ) : m_stream( &_stream ), m_fullConfig( _fullConfig ) {} diff --git a/src/catch2/interfaces/catch_interfaces_reporter.hpp b/src/catch2/interfaces/catch_interfaces_reporter.hpp index 5d3e85a3..432bc4e5 100644 --- a/src/catch2/interfaces/catch_interfaces_reporter.hpp +++ b/src/catch2/interfaces/catch_interfaces_reporter.hpp @@ -32,8 +32,6 @@ namespace Catch { struct IConfig; struct ReporterConfig { - explicit ReporterConfig( IConfig const* _fullConfig ); - ReporterConfig( IConfig const* _fullConfig, std::ostream& _stream ); std::ostream& stream() const; diff --git a/src/catch2/interfaces/catch_interfaces_reporter_registry.hpp b/src/catch2/interfaces/catch_interfaces_reporter_registry.hpp index 57c652e7..582c39b4 100644 --- a/src/catch2/interfaces/catch_interfaces_reporter_registry.hpp +++ b/src/catch2/interfaces/catch_interfaces_reporter_registry.hpp @@ -23,13 +23,14 @@ namespace Catch { using IStreamingReporterPtr = Detail::unique_ptr<IStreamingReporter>; struct IReporterFactory; using IReporterFactoryPtr = Detail::unique_ptr<IReporterFactory>; + struct ReporterConfig; struct IReporterRegistry { using FactoryMap = std::map<std::string, IReporterFactoryPtr, Detail::CaseInsensitiveLess>; using Listeners = std::vector<IReporterFactoryPtr>; virtual ~IReporterRegistry(); // = default - virtual IStreamingReporterPtr create( std::string const& name, IConfig const* config ) const = 0; + virtual IStreamingReporterPtr create( std::string const& name, ReporterConfig const& config ) const = 0; virtual FactoryMap const& getFactories() const = 0; virtual Listeners const& getListeners() const = 0; }; diff --git a/src/catch2/internal/catch_commandline.cpp b/src/catch2/internal/catch_commandline.cpp index de27e899..ca0e1c20 100644 --- a/src/catch2/internal/catch_commandline.cpp +++ b/src/catch2/internal/catch_commandline.cpp @@ -14,6 +14,7 @@ #include <catch2/interfaces/catch_interfaces_reporter_registry.hpp> #include <catch2/interfaces/catch_interfaces_reporter.hpp> +#include <algorithm> #include <fstream> #include <string> @@ -133,15 +134,72 @@ namespace Catch { return ParserResult::runtimeError( "Unrecognised verbosity, '" + verbosity + '\'' ); return ParserResult::ok( ParseResultType::Matched ); }; - auto const setReporter = [&]( std::string const& reporter ) { + auto const setReporter = [&]( std::string const& reporterSpec ) { + if ( reporterSpec.empty() ) { + return ParserResult::runtimeError( "Received empty reporter spec." ); + } + IReporterRegistry::FactoryMap const& factories = getRegistryHub().getReporterRegistry().getFactories(); - auto result = factories.find( reporter ); + // clear the default reporter + if (!config._nonDefaultReporterSpecifications) { + config.reporterSpecifications.clear(); + config._nonDefaultReporterSpecifications = true; + } + + + // Exactly one of the reporters may be specified without an output + // file, in which case it defaults to the output specified by "-o" + // (or standard output). + static constexpr auto separator = "::"; + static constexpr size_t separatorSize = 2; + auto fileNameSeparatorPos = reporterSpec.find( separator ); + const bool containsFileName = fileNameSeparatorPos != reporterSpec.npos; + if ( containsFileName ) { + auto nextSeparatorPos = reporterSpec.find( + separator, fileNameSeparatorPos + separatorSize ); + if ( nextSeparatorPos != reporterSpec.npos ) { + return ParserResult::runtimeError( + "Too many separators in reporter spec '" + reporterSpec + '\'' ); + } + } + + std::string reporterName; + Optional<std::string> outputFileName; + reporterName = repo |