Read-only mirrors. Development, issues and pull requests are on GitHub.

summaryrefslogtreecommitdiffhomepage
path: root/tests/SelfTest
diff options
context:
space:
mode:
authorMartin Jeřábek2021-02-06 20:12:07 +0100
committerMartin Hořeňovský2022-01-01 14:02:23 +0100
commitccd67b293d7595a0cf2dc6c4bfb2017fc18c1f14 (patch)
treeba5f166d1cf867ad97617656083354fbddfc5d9f /tests/SelfTest
parent6b55f5d780c5d8e935720f9d7d8ed28a3ba8617f (diff)
downloadcatch3-ccd67b293d7595a0cf2dc6c4bfb2017fc18c1f14.tar.gz
Add support for multiple parallel reporters
This requires a bunch of different changes across the reporter subsystem. * We need to handle multiple reporters and their differing preferences in `ListeningReporter`, e.g. what to do when we mix reporters that capture and don't capture stdout. * We need to change how the reporter is given output and how we parse reporter's output destination from CLI. * Approval tests need to handle multireporter option
Diffstat (limited to 'tests/SelfTest')
-rw-r--r--tests/SelfTest/Baselines/automake.sw.multi.approved.txt380
-rw-r--r--tests/SelfTest/Baselines/compact.sw.approved.txt47
-rw-r--r--tests/SelfTest/Baselines/compact.sw.multi.approved.txt2480
-rw-r--r--tests/SelfTest/Baselines/console.std.approved.txt2
-rw-r--r--tests/SelfTest/Baselines/console.sw.approved.txt179
-rw-r--r--tests/SelfTest/Baselines/console.sw.multi.approved.txt17650
-rw-r--r--tests/SelfTest/Baselines/default.sw.multi.approved.txt8
-rw-r--r--tests/SelfTest/Baselines/junit.sw.approved.txt9
-rw-r--r--tests/SelfTest/Baselines/junit.sw.multi.approved.txt1902
-rw-r--r--tests/SelfTest/Baselines/sonarqube.sw.approved.txt7
-rw-r--r--tests/SelfTest/Baselines/sonarqube.sw.multi.approved.txt1920
-rw-r--r--tests/SelfTest/Baselines/tap.sw.approved.txt46
-rw-r--r--tests/SelfTest/Baselines/tap.sw.multi.approved.txt4416
-rw-r--r--tests/SelfTest/Baselines/teamcity.sw.multi.approved.txt922
-rw-r--r--tests/SelfTest/Baselines/xml.sw.approved.txt219
-rw-r--r--tests/SelfTest/Baselines/xml.sw.multi.approved.txt20738
-rw-r--r--tests/SelfTest/IntrospectiveTests/CmdLine.tests.cpp64
17 files changed, 50906 insertions, 83 deletions
diff --git a/tests/SelfTest/Baselines/automake.sw.multi.approved.txt b/tests/SelfTest/Baselines/automake.sw.multi.approved.txt
new file mode 100644
index 00000000..328fcf50
--- /dev/null
+++ b/tests/SelfTest/Baselines/automake.sw.multi.approved.txt
@@ -0,0 +1,380 @@
+:test-result: PASS # A test name that starts with a #
+:test-result: PASS #1005: Comparing pointer to int and long (NULL can be either on various systems)
+:test-result: PASS #1027: Bitfields can be captured
+:test-result: PASS #1147
+:test-result: PASS #1175 - Hidden Test
+:test-result: PASS #1238
+:test-result: PASS #1245
+:test-result: PASS #1319: Sections can have description (even if it is not saved
+:test-result: PASS #1403
+:test-result: FAIL #1455 - INFO and WARN can start with a linebreak
+:test-result: FAIL #1514: stderr/stdout is not captured in tests aborted by an exception
+:test-result: PASS #1548
+:test-result: PASS #1905 -- test spec parser properly clears internal state between compound tests
+:test-result: PASS #1912 -- test spec parser handles escaping
+:test-result: PASS #1913 - GENERATE inside a for loop should not keep recreating the generator
+:test-result: PASS #1913 - GENERATEs can share a line
+:test-result: PASS #1938 - GENERATE after a section
+:test-result: PASS #1938 - Section followed by flat generate
+:test-result: PASS #1938 - flat generate
+:test-result: PASS #1938 - mixed sections and generates
+:test-result: PASS #1938 - nested generate
+:test-result: PASS #1954 - 7 arg template test case sig compiles - 1, 1, 1, 1, 1, 0, 0
+:test-result: PASS #1954 - 7 arg template test case sig compiles - 5, 1, 1, 1, 1, 0, 0
+:test-result: PASS #1954 - 7 arg template test case sig compiles - 5, 3, 1, 1, 1, 0, 0
+:test-result: PASS #2152 - ULP checks between differently signed values were wrong - double
+:test-result: PASS #2152 - ULP checks between differently signed values were wrong - float
+:test-result: XFAIL #748 - captures with unexpected exceptions
+:test-result: PASS #809
+:test-result: PASS #833
+:test-result: XFAIL #835 -- errno should not be touched by Catch
+:test-result: PASS #872
+:test-result: PASS #961 -- Dynamically created sections should all be reported
+:test-result: FAIL 'Not' checks that should fail
+:test-result: PASS 'Not' checks that should succeed
+:test-result: PASS (unimplemented) static bools can be evaluated
+:test-result: PASS 3x3x3 ints
+:test-result: FAIL A METHOD_AS_TEST_CASE based test run that fails
+:test-result: PASS A METHOD_AS_TEST_CASE based test run that succeeds
+:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo<float>
+:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - Template_Foo<int>
+:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector<float>
+:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that fails - std::vector<int>
+:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo<float>
+:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - Template_Foo<int>
+:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector<float>
+:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD based test run that succeeds - std::vector<int>
+:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2<float, 6>
+:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - Template_Foo_2<int, 2>
+:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array<float, 6>
+:test-result: FAIL A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that fails - std::array<int, 2>
+:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2<float,6>
+:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - Template_Foo_2<int,2>
+:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array<float,6>
+:test-result: PASS A TEMPLATE_PRODUCT_TEST_CASE_METHOD_SIG based test run that succeeds - std::array<int,2>
+:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD based test run that fails - double
+:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD based test run that fails - float
+:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD based test run that fails - int
+:test-result: PASS A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - double
+:test-result: PASS A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - float
+:test-result: PASS A TEMPLATE_TEST_CASE_METHOD based test run that succeeds - int
+:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 1
+:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 3
+:test-result: FAIL A TEMPLATE_TEST_CASE_METHOD_SIG based test run that fails - 6
+:test-result: PASS A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 1
+:test-result: PASS A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 3
+:test-result: PASS A TEMPLATE_TEST_CASE_METHOD_SIG based test run that succeeds - 6
+:test-result: FAIL A TEST_CASE_METHOD based test run that fails
+:test-result: PASS A TEST_CASE_METHOD based test run that succeeds
+:test-result: PASS A Template product test case - Foo<float>
+:test-result: PASS A Template product test case - Foo<int>
+:test-result: PASS A Template product test case - std::vector<float>
+:test-result: PASS A Template product test case - std::vector<int>
+:test-result: PASS A Template product test case with array signature - Bar<float, 42>
+:test-result: PASS A Template product test case with array signature - Bar<int, 9>
+:test-result: PASS A Template product test case with array signature - std::array<float, 42>
+:test-result: PASS A Template product test case with array signature - std::array<int, 9>
+:test-result: PASS A comparison that uses literals instead of the normal constructor
+:test-result: FAIL A couple of nested sections followed by a failure
+:test-result: FAIL A failing expression with a non streamable type is still captured
+:test-result: PASS Absolute margin
+:test-result: FAIL An empty test with no assertions
+:test-result: PASS An expression with side-effects should only be evaluated once
+:test-result: FAIL An unchecked exception reports the line of the last assertion
+:test-result: PASS Anonymous test case 1
+:test-result: PASS Approx setters validate their arguments
+:test-result: PASS Approx with exactly-representable margin
+:test-result: PASS Approximate PI
+:test-result: PASS Approximate comparisons with different epsilons
+:test-result: PASS Approximate comparisons with floats
+:test-result: PASS Approximate comparisons with ints
+:test-result: PASS Approximate comparisons with mixed numeric types
+:test-result: PASS Arbitrary predicate matcher
+:test-result: PASS Assertion macros support bit operators and bool conversions
+:test-result: PASS Assertions then sections
+:test-result: PASS Basic use of the Contains range matcher
+:test-result: PASS Basic use of the Empty range matcher
+:test-result: PASS CAPTURE can deal with complex expressions
+:test-result: PASS CAPTURE can deal with complex expressions involving commas
+:test-result: PASS CAPTURE parses string and character constants
+:test-result: PASS Capture and info messages
+:test-result: PASS CaseInsensitiveEqualsTo is case insensitive
+:test-result: PASS CaseInsensitiveLess is case insensitive
+:test-result: PASS Character pretty printing
+:test-result: PASS Clara::Arg supports single-arg parse the way Opt does
+:test-result: PASS Clara::Opt supports accept-many lambdas
+:test-result: PASS Combining MatchAllOfGeneric does not nest
+:test-result: PASS Combining MatchAnyOfGeneric does not nest
+:test-result: PASS Combining MatchNotOfGeneric does not nest
+:test-result: PASS Combining concrete matchers does not use templated matchers
+:test-result: PASS Combining only templated matchers
+:test-result: PASS Combining templated and concrete matchers
+:test-result: PASS Combining templated matchers
+:test-result: PASS Commas in various macros are allowed
+:test-result: PASS Comparing function pointers
+:test-result: PASS Comparison ops
+:test-result: PASS Comparison with explicitly convertible types
+:test-result: PASS Comparisons between ints where one side is computed
+:test-result: PASS Comparisons between unsigned ints and negative signed ints match c++ standard behaviour
+:test-result: PASS Comparisons with int literals don't warn when mixing signed/ unsigned
+:test-result: PASS Composed generic matchers shortcircuit
+:test-result: PASS Composed matchers shortcircuit
+:test-result: FAIL Contains string matcher
+:test-result: PASS Copy and then generate a range
+:test-result: FAIL Custom exceptions can be translated when testing for nothrow
+:test-result: FAIL Custom exceptions can be translated when testing for throwing as something else
+:test-result: FAIL Custom std-exceptions can be custom translated
+:test-result: PASS Default scale is invisible to comparison
+:test-result: PASS Directly creating an EnumInfo
+:test-result: PASS Empty tag is not allowed
+:test-result: FAIL EndsWith string matcher
+:test-result: PASS Enums can quickly have stringification enabled using REGISTER_ENUM
+:test-result: PASS Enums in namespaces can quickly have stringification enabled using REGISTER_ENUM
+:test-result: PASS Epsilon only applies to Approx's value
+:test-result: XFAIL Equality checks that should fail
+:test-result: PASS Equality checks that should succeed
+:test-result: PASS Equals
+:test-result: FAIL Equals string matcher
+:test-result: PASS Exception as a value (e.g. in REQUIRE_THROWS_MATCHES) can be stringified
+:test-result: FAIL Exception matchers that fail
+:test-result: PASS Exception matchers that succeed
+:test-result: PASS Exception messages can be tested for
+:test-result: PASS Exceptions matchers
+:test-result: FAIL Expected exceptions that don't throw or unexpected exceptions fail the test
+:test-result: FAIL FAIL aborts the test
+:test-result: FAIL FAIL does not require an argument
+:test-result: FAIL FAIL_CHECK does not abort the test
+:test-result: PASS Factorials are computed
+:test-result: PASS Floating point matchers: double
+:test-result: PASS Floating point matchers: float
+:test-result: PASS Generators -- adapters
+:test-result: PASS Generators -- simple
+:test-result: PASS Generators internals
+:test-result: PASS Greater-than inequalities with different epsilons
+:test-result: FAIL INFO and WARN do not abort tests
+:test-result: FAIL INFO gets logged on failure
+:test-result: FAIL INFO gets logged on failure, even if captured before successful assertions
+:test-result: FAIL INFO is reset for each loop
+:test-result: XFAIL Inequality checks that should fail
+:test-result: PASS Inequality checks that should succeed
+:test-result: PASS Lambdas in assertions
+:test-result: PASS Less-than inequalities with different epsilons
+:test-result: PASS ManuallyRegistered
+:test-result: PASS Matchers can be (AllOf) composed with the && operator
+:test-result: PASS Matchers can be (AnyOf) composed with the || operator
+:test-result: PASS Matchers can be composed with both && and ||
+:test-result: FAIL Matchers can be composed with both && and || - failing
+:test-result: PASS Matchers can be negated (Not) with the ! operator
+:test-result: FAIL Matchers can be negated (Not) with the ! operator - failing
+:test-result: XFAIL Mayfail test case with nested sections
+:test-result: FAIL Mismatching exception messages failing the test
+:test-result: PASS Nested generators and captured variables
+:test-result: FAIL Nice descriptive name
+:test-result: FAIL Non-std exceptions can be translated
+:test-result: PASS Objects that evaluated in boolean contexts can be checked
+:test-result: PASS Optionally static assertions
+:test-result: FAIL Ordering comparison checks that should fail
+:test-result: PASS Ordering comparison checks that should succeed
+:test-result: PASS Our PCG implementation provides expected results for known seeds
+:test-result: FAIL Output from all sections is reported
+:test-result: PASS Overloaded comma or address-of operators are not used
+:test-result: PASS Parse test names and tags
+:test-result: PASS Parsed tags are matched case insensitive
+:test-result: PASS Parsing sharding-related cli flags
+:test-result: PASS Parsing tags with non-alphabetical characters is pass-through
+:test-result: PASS Parsing warnings
+:test-result: PASS Pointers can be compared to null
+:test-result: PASS Precision of floating point stringification can be set
+:test-result: PASS Predicate matcher can accept const char*
+:test-result: PASS Process can be configured on command line
+:test-result: PASS Product with differing arities - std::tuple<int, double, float>
+:test-result: PASS Product with differing arities - std::tuple<int, double>
+:test-result: PASS Product with differing arities - std::tuple<int>
+:test-result: PASS Random seed generation accepts known methods
+:test-result: PASS Random seed generation reports unknown methods
+:test-result: PASS Range type with sentinel
+:test-result: FAIL Reconstruction should be based on stringification: #914
+:test-result: FAIL Regex string matcher
+:test-result: PASS Regression test #1
+:test-result: PASS Reporter's write listings to provided stream
+:test-result: PASS Reproducer for #2309 - a very long description past 80 chars (default console width) with a late colon : blablabla
+:test-result: PASS SUCCEED counts as a test pass
+:test-result: PASS SUCCEED does not require an argument
+:test-result: PASS Scenario: BDD tests requiring Fixtures to provide commonly-accessed data or methods
+:test-result: PASS Scenario: Do that thing with the thing
+:test-result: PASS Scenario: This is a really long scenario name to see how the list command deals with wrapping
+:test-result: PASS Scenario: Vector resizing affects size and capacity
+:test-result: FAIL Sends stuff to stdout and stderr
+:test-result: PASS Some simple comparisons between doubles
+:test-result: FAIL Standard output from all sections is reported
+:test-result: FAIL StartsWith string matcher
+:test-result: PASS Static arrays are convertible to string
+:test-result: PASS String matchers
+:test-result: PASS StringRef
+:test-result: PASS StringRef at compilation time
+:test-result: PASS Stringifying char arrays with statically known sizes - char
+:test-result: PASS Stringifying char arrays with statically known sizes - signed char
+:test-result: PASS Stringifying char arrays with statically known sizes - unsigned char
+:test-result: PASS Stringifying std::chrono::duration helpers
+:test-result: PASS Stringifying std::chrono::duration with weird ratios
+:test-result: PASS Stringifying std::chrono::time_point<system_clock>
+:test-result: FAIL Tabs and newlines show in output
+:test-result: PASS Tag alias can be registered against tag patterns
+:test-result: PASS Tags with spaces and non-alphanumerical characters are accepted
+:test-result: PASS Template test case method with test types specified inside std::tuple - MyTypes - 0
+:test-result: PASS Template test case method with test types specified inside std::tuple - MyTypes - 1
+:test-result: PASS Template test case method with test types specified inside std::tuple - MyTypes - 2
+:test-result: PASS Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 0
+:test-result: PASS Template test case with test types specified inside non-copyable and non-movable std::tuple - NonCopyableAndNonMovableTypes - 1
+:test-result: PASS Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 0
+:test-result: PASS Template test case with test types specified inside non-default-constructible std::tuple - MyNonDefaultConstructibleTypes - 1
+:test-result: PASS Template test case with test types specified inside std::tuple - MyTypes - 0
+:test-result: PASS Template test case with test types specified inside std::tuple - MyTypes - 1
+:test-result: PASS Template test case with test types specified inside std::tuple - MyTypes - 2
+:test-result: PASS TemplateTest: vectors can be sized and resized - float
+:test-result: PASS TemplateTest: vectors can be sized and resized - int
+:test-result: PASS TemplateTest: vectors can be sized and resized - std::string
+:test-result: PASS TemplateTest: vectors can be sized and resized - std::tuple<int,float>
+:test-result: PASS TemplateTestSig: vectors can be sized and resized - (std::tuple<int, float>), 6
+:test-result: PASS TemplateTestSig: vectors can be sized and resized - float,4
+:test-result: PASS TemplateTestSig: vectors can be sized and resized - int,5
+:test-result: PASS TemplateTestSig: vectors can be sized and resized - std::string,15
+:test-result: PASS Test case with identical tags keeps just one
+:test-result: PASS Test case with one argument
+:test-result: PASS Test enum bit values
+:test-result: PASS Test with special, characters "in name
+:test-result: PASS Testing checked-if
+:test-result: XFAIL Testing checked-if 2
+:test-result: XFAIL Testing checked-if 3
+:test-result: FAIL The NO_FAIL macro reports a failure but does not fail the test
+:test-result: PASS The default listing implementation write to provided stream
+:test-result: FAIL This test 'should' fail but doesn't
+:test-result: FAIL Thrown string literals are translated
+:test-result: PASS Tracker
+:test-result: PASS Trim strings
+:test-result: FAIL Unexpected exceptions can be translated
+:test-result: PASS Upcasting special member functions
+:test-result: PASS Usage of AllMatch range matcher
+:test-result: PASS Usage of AnyMatch range matcher
+:test-result: PASS Usage of NoneMatch range matcher
+:test-result: PASS Usage of the SizeIs range matcher
+:test-result: PASS Use a custom approx
+:test-result: PASS Variadic macros
+:test-result: PASS Vector Approx matcher
+:test-result: FAIL Vector Approx matcher -- failing
+:test-result: PASS Vector matchers
+:test-result: FAIL Vector matchers that fail
+:test-result: PASS When checked exceptions are thrown they can be expected or unexpected
+:test-result: FAIL When unchecked exceptions are thrown directly they are always failures
+:test-result: FAIL When unchecked exceptions are thrown during a CHECK the test should continue
+:test-result: FAIL When unchecked exceptions are thrown during a REQUIRE the test should abort fail
+:test-result: FAIL When unchecked exceptions are thrown from functions they are always failures
+:test-result: FAIL When unchecked exceptions are thrown from sections they are always failures
+:test-result: FAIL When unchecked exceptions are thrown, but caught, they do not affect the test
+:test-result: PASS X/level/0/a
+:test-result: PASS X/level/0/b
+:test-result: PASS X/level/1/a
+:test-result: PASS X/level/1/b
+:test-result: PASS XmlEncode
+:test-result: PASS XmlWriter writes boolean attributes as true/false
+:test-result: PASS analyse no analysis
+:test-result: PASS array<int, N> -> toString
+:test-result: PASS benchmark function call
+:test-result: PASS boolean member
+:test-result: PASS checkedElse
+:test-result: FAIL checkedElse, failing
+:test-result: PASS checkedIf
+:test-result: FAIL checkedIf, failing
+:test-result: PASS classify_outliers
+:test-result: PASS comparisons between const int variables
+:test-result: PASS comparisons between int variables
+:test-result: PASS convertToBits
+:test-result: PASS empty tags are not allowed
+:test-result: PASS erfc_inv
+:test-result: PASS estimate_clock_resolution
+:test-result: PASS even more nested SECTION tests
+:test-result: FAIL first tag
+:test-result: FAIL has printf
+:test-result: PASS is_unary_function
+:test-result: FAIL just failure
+:test-result: FAIL just failure after unscoped info
+:test-result: FAIL just info
+:test-result: FAIL just unscoped info
+:test-result: PASS long long
+:test-result: FAIL looped SECTION tests
+:test-result: FAIL looped tests
+:test-result: PASS make_unique reimplementation
+:test-result: PASS mean
+:test-result: PASS measure
+:test-result: FAIL mix info, unscoped info and warning
+:test-result: FAIL more nested SECTION tests
+:test-result: PASS nested SECTION tests
+:test-result: PASS non streamable - with conv. op
+:test-result: PASS non-copyable objects
+:test-result: PASS normal_cdf
+:test-result: PASS normal_quantile
+:test-result: PASS not allowed
+:test-result: FAIL not prints unscoped info from previous failures
+:test-result: PASS null strings
+:test-result: PASS null_ptr
+:test-result: PASS pair<pair<int,const char *,pair<std::string,int> > -> toString
+:test-result: PASS parseEnums
+:test-result: PASS pointer to class
+:test-result: PASS print unscoped info if passing unscoped info is printed
+:test-result: FAIL prints unscoped info on failure
+:test-result: FAIL prints unscoped info only for the first assertion
+:test-result: PASS random SECTION tests
+:test-result: PASS replaceInPlace
+:test-result: PASS resolution
+:test-result: PASS run_for_at_least, chronometer
+:test-result: PASS run_for_at_least, int
+:test-result: FAIL second tag
+:test-result: FAIL send a single char to INFO
+:test-result: FAIL sends information to INFO
+:test-result: PASS shortened hide tags are split apart
+:test-result: PASS splitString
+:test-result: FAIL stacks unscoped info in loops
+:test-result: PASS startsWith
+:test-result: PASS std::map is convertible string
+:test-result: PASS std::pair<int,const std::string> -> toString
+:test-result: PASS std::pair<int,std::string> -> toString
+:test-result: PASS std::set is convertible string
+:test-result: PASS std::vector<std::pair<std::string,int> > -> toString
+:test-result: PASS stringify ranges
+:test-result: PASS stringify( has_maker )
+:test-result: PASS stringify( has_maker_and_operator )
+:test-result: PASS stringify( has_neither )
+:test-result: PASS stringify( has_operator )
+:test-result: PASS stringify( has_template_operator )
+:test-result: PASS stringify( vectors<has_maker> )
+:test-result: PASS stringify( vectors<has_maker_and_operator> )
+:test-result: PASS stringify( vectors<has_operator> )
+:test-result: PASS strlen3
+:test-result: PASS tables
+:test-result: PASS tags with dots in later positions are not parsed as hidden
+:test-result: FAIL thrown std::strings are translated
+:test-result: PASS toString on const wchar_t const pointer returns the string contents
+:test-result: PASS toString on const wchar_t pointer returns the string contents
+:test-result: PASS toString on wchar_t const pointer returns the string contents
+:test-result: PASS toString on wchar_t returns the string contents
+:test-result: PASS toString(enum class w/operator<<)
+:test-result: PASS toString(enum class)
+:test-result: PASS toString(enum w/operator<<)
+:test-result: PASS toString(enum)
+:test-result: PASS tuple<>
+:test-result: PASS tuple<float,int>
+:test-result: PASS tuple<int>
+:test-result: PASS tuple<0,int,const char *>
+:test-result: PASS tuple<string,string>
+:test-result: PASS tuple<tuple<int>,tuple<>,float>
+:test-result: PASS uniform samples
+:test-result: PASS unique_ptr reimplementation: basic functionality
+:test-result: PASS vec<vec<string,alloc>> -> toString
+:test-result: PASS vector<bool> -> toString
+:test-result: PASS vector<int,allocator> -> toString
+:test-result: PASS vector<int> -> toString
+:test-result: PASS vector<string> -> toString
+:test-result: PASS vectors can be sized and resized
+:test-result: PASS warmup
+:test-result: PASS weighted_average_quantile
+:test-result: PASS xmlentitycheck
diff --git a/tests/SelfTest/Baselines/compact.sw.approved.txt b/tests/SelfTest/Baselines/compact.sw.approved.txt
index f7aaa5b3..517b4d93 100644
--- a/tests/SelfTest/Baselines/compact.sw.approved.txt
+++ b/tests/SelfTest/Baselines/compact.sw.approved.txt
@@ -1250,7 +1250,9 @@ CmdLine.tests.cpp:<line number>: passed: config.processName == "test" for: "test
CmdLine.tests.cpp:<line number>: passed: config.shouldDebugBreak == false for: false == false
CmdLine.tests.cpp:<line number>: passed: config.abortAfter == -1 for: -1 == -1
CmdLine.tests.cpp:<line number>: passed: config.noThrow == false for: false == false
-CmdLine.tests.cpp:<line number>: passed: config.reporterName == "console" for: "console" == "console"
+CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == std::vector<Catch::ConfigData::ReporterAndFile>{ {"console", {}} } for: { { console, <default-output> } }
+==
+{ { console, <default-output> } }
CmdLine.tests.cpp:<line number>: passed: !(cfg.hasTestFilters()) for: !false
CmdLine.tests.cpp:<line number>: passed: result for: {?}
CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
@@ -1264,15 +1266,40 @@ CmdLine.tests.cpp:<line number>: passed: result for: {?}
CmdLine.tests.cpp:<line number>: passed: cfg.hasTestFilters() for: true
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("test1")) == false for: false == false
CmdLine.tests.cpp:<line number>: passed: cfg.testSpec().matches(*fakeTestCase("alwaysIncluded")) for: true
-CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-r", "console"}) for: {?}
-CmdLine.tests.cpp:<line number>: passed: config.reporterName == "console" for: "console" == "console"
-CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-r", "xml"}) for: {?}
-CmdLine.tests.cpp:<line number>: passed: config.reporterName == "xml" for: "xml" == "xml"
-CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "--reporter", "junit"}) for: {?}
-CmdLine.tests.cpp:<line number>: passed: config.reporterName == "junit" for: "junit" == "junit"
-CmdLine.tests.cpp:<line number>: passed: !(cli.parse({ "test", "-r", "xml", "-r", "junit" })) for: !{?}
+CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", {}} } for: { { console, <default-output> } }
+==
+{ { console, <default-output> } } with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", {}} } for: { { xml, <default-output> } }
+==
+{ { xml, <default-output> } } with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"junit", {}} } for: { { junit, <default-output> } }
+==
+{ { junit, <default-output> } } with 1 message: 'result.errorMessage() := ""'
CmdLine.tests.cpp:<line number>: passed: !result for: true
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Unrecognized reporter") for: "Unrecognized reporter, 'unsupported'. Check available with --list-reporters" contains: "Unrecognized reporter"
+CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "out.txt"s} } for: { { console, out.txt } }
+==
+{ { console, out.txt } } with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: result for: {?} with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"console", "C:\\Temp\\out.txt"s} } for: { { console, C:\Temp\out.txt } }
+==
+{ { console, C:\Temp\out.txt } } with 1 message: 'result.errorMessage() := ""'
+CmdLine.tests.cpp:<line number>: passed: !result for: true
+CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("empty filename") for: "Reporter 'console' has empty filename specified as its output. Supply a filename or remove the colons to use the default output." contains: "empty filename"
+CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "junit::output-junit.xml" }) for: {?}
+CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"junit", "output-junit.xml"s} } for: { { xml, output.xml }, { junit, output-junit.xml } }
+==
+{ { xml, output.xml }, { junit, output-junit.xml } }
+CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "-r", "xml::output.xml", "-r", "console" }) for: {?}
+CmdLine.tests.cpp:<line number>: passed: config.reporterSpecifications == vec_ReporterAndFile{ {"xml", "output.xml"s}, {"console", {}} } for: { { xml, output.xml }, { console, <default-output> } }
+==
+{ { xml, output.xml }, { console, <default-output> } }
+CmdLine.tests.cpp:<line number>: passed: !result for: true
+CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Only one reporter may have unspecified output file.") for: "Only one reporter may have unspecified output file." contains: "Only one reporter may have unspecified output file."
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-b"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.shouldDebugBreak == true for: true == true
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "--break"}) for: {?}
@@ -1298,9 +1325,9 @@ CmdLine.tests.cpp:<line number>: passed: config.noThrow for: true
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "--nothrow"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.noThrow for: true
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-o", "filename.ext"}) for: {?}
-CmdLine.tests.cpp:<line number>: passed: config.outputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
+CmdLine.tests.cpp:<line number>: passed: config.defaultOutputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "--out", "filename.ext"}) for: {?}
-CmdLine.tests.cpp:<line number>: passed: config.outputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
+CmdLine.tests.cpp:<line number>: passed: config.defaultOutputFilename == "filename.ext" for: "filename.ext" == "filename.ext"
CmdLine.tests.cpp:<line number>: passed: cli.parse({"test", "-abe"}) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.abortAfter == 1 for: 1 == 1
CmdLine.tests.cpp:<line number>: passed: config.shouldDebugBreak for: true
diff --git a/tests/SelfTest/Baselines/compact.sw.multi.approved.txt b/tests/SelfTest/Baselines/compact.sw.multi.approved.txt
new file mode 100644
index 00000000..7196e831
--- /dev/null
+++ b/tests/SelfTest/Baselines/compact.sw.multi.approved.txt
@@ -0,0 +1,2480 @@
+Misc.tests.cpp:<line number>: passed: with 1 message: 'yay'
+Decomposition.tests.cpp:<line number>: passed: fptr == 0 for: 0 == 0
+Decomposition.tests.cpp:<line number>: passed: fptr == 0l for: 0 == 0
+Compilation.tests.cpp:<line number>: passed: y.v == 0 for: 0 == 0
+Compilation.tests.cpp:<line number>: passed: 0 == y.v for: 0 == 0
+Compilation.tests.cpp:<line number>: passed: t1 == t2 for: {?} == {?}
+Compilation.tests.cpp:<line number>: passed: t1 != t2 for: {?} != {?}
+Compilation.tests.cpp:<line number>: passed: t1 < t2 for: {?} < {?}
+Compilation.tests.cpp:<line number>: passed: t1 > t2 for: {?} > {?}
+Compilation.tests.cpp:<line number>: passed: t1 <= t2 for: {?} <= {?}
+Compilation.tests.cpp:<line number>: passed: t1 >= t2 for: {?} >= {?}
+Misc.tests.cpp:<line number>: passed:
+Compilation.tests.cpp:<line number>: passed: std::memcmp(uarr, "123", sizeof(uarr)) == 0 for: 0 == 0 with 2 messages: 'uarr := "123"' and 'sarr := "456"'
+Compilation.tests.cpp:<line number>: passed: std::memcmp(sarr, "456", sizeof(sarr)) == 0 for: 0 == 0 with 2 messages: 'uarr := "123"' and 'sarr := "456"'
+Compilation.tests.cpp:<line number>: passed:
+Compilation.tests.cpp:<line number>: passed:
+Compilation.tests.cpp:<line number>: passed: h1 == h2 for: [1403 helper] == [1403 helper]
+Message.tests.cpp:<line number>: warning: '
+This info message starts with a linebreak' with 1 message: '
+This warning message starts with a linebreak'
+Tricky.tests.cpp:<line number>: failed: explicitly with 1 message: '1514'
+Compilation.tests.cpp:<line number>: passed: std::is_same<TypeList<int>, TypeList<int>>::value for: true
+CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase("spec . char")) for: true
+CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase("spec , char")) for: true
+CmdLine.tests.cpp:<line number>: passed: !(spec.matches(*fakeTestCase(R"(spec \, char)"))) for: !false
+CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase(R"(spec {a} char)")) for: true
+CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase(R"(spec [a] char)")) for: true
+CmdLine.tests.cpp:<line number>: passed: !(spec.matches(*fakeTestCase("differs but has similar tag", "[a]"))) for: !false
+CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase(R"(spec \ char)")) for: true
+Generators.tests.cpp:<line number>: passed: counter < 7 for: 3 < 7
+Generators.tests.cpp:<line number>: passed: counter < 7 for: 6 < 7
+Generators.tests.cpp:<line number>: passed: i != j for: 1 != 3
+Generators.tests.cpp:<line number>: passed: i != j for: 1 != 4
+Generators.tests.cpp:<line number>: passed: i != j for: 2 != 3
+Generators.tests.cpp:<line number>: passed: i != j for: 2 != 4
+PartTracker.tests.cpp:<line number>: passed: with 1 message: 'A'
+PartTracker.tests.cpp:<line number>: passed: m for: 1
+PartTracker.tests.cpp:<line number>: passed: m for: 2
+PartTracker.tests.cpp:<line number&g