This release mostly focusses on an overhaul of how testthat works with conditions (i.e. errors, warnings and messages). There are relatively few user-facing changes, although you should now see more informative backtraces from errors and failures.
Unexpected errors are now printed with a simplified backtrace.
expect_message() now call
conditionMessage() to get the condition message. This generic makes it possible to generate messages at print-time rather than signal-time.
expect_error() gets a better warning message when you test for a custom error class with
Existence of restarts is first checked before invokation. This makes it possible to signal warnings or messages with a different condition signaller (#874).
ListReporter now tracks expectations and errors, even when they occur outside of tests. This ensures that
stop_on_failure matches the results displayed by the reporter (#936).
You can silence warnings about untested error classes by implementing a method for
is_uninformative_warning(). This method should be lazily registered, e.g. with
vctrs::s3_register(). This is useful for introducing an experimental error class without encouraging users to depend on the class in their tests.
Expectations can now be explicitly subclassed with
new_expectation(). This constructor follows our new conventions for S3 classes and takes an optional subclass and optional attributes.
Unquoted inputs no longer potentially generate multiple test messages (#929).
verify_output() gains a
unicode parameter to turn on or off the use of Unicode characters by the cli package. It is disabled by default to prevent the tests from failing on platforms like Windows that don’t support UTF-8 (which could be your contributors’ or your CI machines).
verify_output() now correctly handles multi-line condition messages.
verify_output() now adds spacing after condition messages, consistent with the spacing added after normal output.
verify_output() has a new syntax for inserting headers in output files: insert a
"# Header" string (starting with
# as in Markdown) to add a header to a set of outputs.
test_rd()and add a couple of tests to hopefully detect the problem earlier in the future.
skip_if_translated() now tests for translation of a specific message. This is more robust than the previous approach because translation happens message-by-message, not necessarily for the entire session (#879) (and in general, it’s impossible to determine what language R is currently using).
expect_known_value() gains a new serialisation
version argument, defaulting to 2. Prevents the
.rds files created to hold reference objects from making a package appear to require R >= 3.5 (#888 @jennybc).
expect_mapequal(x, y) checks that
y have the same names, and the same value associated with each name (i.e. they compare the values of the vector standardising the order of the names) (#863).
All expectations give clearer error messages if you forget the
expected arguments (#743).
expect_error() now returns the error object as documentated (#724). It also now warns if you’re using a classed expectation and you’re not using the
class argument. This is good practice as it decouples the error object (which tends to be stable) from its rendering to the user (which tends to be fragile) (#816).
expect_output() gains a
width argument, allowing you to control the output width. This does not inherit from
getOption("width"), ensuring that tests return the same results regardless of environment (#805).
expect_setequal() now works with more vector types (including lists), because it uses
%in%, rather than
sort(). It also warns if the inputs are named, as this suggests that your mental model of how
expect_setequal() works is wrong (#750).
Reporter documentation has been considerably improved (#657).
CheckReporter, used by R CMD check, now includes a count of warnings.
ListReporter now records number of passed tests and original results in new columns (#675).
ProgressReporter, the default reporter, now:
Automatically generates a context from the file name. We no longer recommend the use of
context() and instead encourage you to delete it, allowing the context to be autogenerated from the file name.
Uses a 3 character wide column to display test successes, so up to 999 successful tests can be displayed without changing the alignment (#712).
reporter$end_reporter() is now only called when testing completes successfully. This ensures that you don’t get unnecessary output when the test fails partway through (#727).
testthat_example() make it easy to access new test files bundled with the package. These are used in various examples to make it easier to understand how to use the package.
expectation objects now contain the failure message, even when successful (#836)
run_testthat_tests C entrypoint is registered more robustly.
“Can’t mock functions in base packages”: You can no longer use
with_mock() to mock functions in base packages, because this no longer works in R-devel due to changes with the byte code compiler. I recommend using mockery or mockr instead.
If you see one of these errors, check the number, order, and names of arguments to the expectation.
“Failure: (unknown)”. The last release mistakenly failed to test bare expectations not wrapped inside
test_that(). If you see “(unknown)” in a failure message, this is a failing expectation that you previously weren’t seeing. As well as fixing the failure, please also wrap inside a
test_that() with an informative name.
“Error: the argument has already been evaluated”: the way in which expectations now need create labels has changed, which caused a couple of failures with unusual usage when combined with
Map(). Avoid these functions in favour of for loops. I also recommend reading the section below on quasiquotation support in order to create more informative failure messages.
skip_if_translated() skips tests if you’re running in an locale where translations are likely to occur (#565). Use this to avoid spurious failures when checking the text of error messages in non-English locales.
We have identified a useful family of expectations that compares the results of an expression to a known good value stored in a file. They are designed to be use in conjunction with git so that you can see what precisely has changed, and revert it if needed.
expect_output_file(), which has been soft-deprecated. It now defaults to
update = TRUE and warn, rather than failing on the first run. It gains a
expect_equal_to_reference(), which has been soft-deprecated. It gains an update argument defaulting to
TRUE. This changes behaviour from the previous version, and soft-deprecated
update = FALSE.
expect_known_failure() stored and compares the failure message from an expectation. It’s a useful regression test when developing informative failure messges for your own expectations.
All expectations can now use unquoting (#626). This makes it much easier to generate informative failure messages when running tests in a for loop.
For example take this test:
When it fails, you’ll see the message
Error: `f(i)` not equal to `i * 10`. That’s hard to diagnose because you don’t know which iteration caused the problem!
If you unquote the values using
!!, you get the failure message
`f(4L)` not equal to 40.. This is much easier to diagnose! See
?quasi_label() for more details.
(Note that this is not tidy evaluation per se, but is closely related. At this time you can not unquote quosures.)
Two new prefixes are recognised in the
test/ directory. Files starting with
setup are run before tests (but unlike
helpers are not run in
devtools::load_all()). Files starting with
teardown are run after all tests are completed (#589).
is_testing() allows you to tell if your code is being run inside a testing environment (#631). Rather than taking a run-time dependency on testthat you may want to inline the function into your own package:
It’s frequently useful to combine with
A new default reporter,
ReporterProgress, produces more aesthetically pleasing output and makes the most important information available upfront (#529). You can return to the previous default by setting
options(testthat.default_reporter = "summary").
Output colours have been tweaked to be consistent with clang: warnings are now in magenta, and skips in blue.
LocationReporter which just prints the location of every expectation. This is useful for locating segfaults and C/C++ breakpoints (#551).
SummaryReporter recieved a number of smaller tweaks
Aborts testing as soon the limit given by the option
testthat.summary.max_reports (default 10) is reached (#520).
testthat.summary.omit_dots = TRUE hides the progress dots speeding up tests by a small amount (#502).
Bring back random praise and encouragement which I accidentally dropped (#478).
testthat.default_check_reporter, defaults to
"check". Continuous Integration system can set this option before evaluating package test sources in order to direct test result details to known location.
All reporters now accept a
file argument on initialization. If provided, reporters will write the test results to that path. This output destination can also be controlled with the option
testthat.output_file (#635, @nealrichardson).
testthatnow understands and makes use of the package routine registration mechanism required by CRAN with R >= 3.4.0.
Better reporting for deeply nested failures, limiting the stack trace to the first and last 10 entries (#474).
Bare expectations notify the reporter once again. This is achieved by running all tests inside
test_code() by default (#427, #498). This behaviour can be overridden by setting
wrap = FALSE in
test_dir() and friends (#586).
hash parameter to enable switching to faster, time-stamp-based modification detection (#598, @katrinleinweber).
auto_test_package() works correctly on windows (#465).
expect_match() now accepts explicit
fixed arguments, and adapts the failure message to the value of
fixed. This also affects other expectations that forward to
show_failures() simply prints a failure if it occurs. This makes it easier to show failures in examples.
std::string()argument, to avoid build errors on Solaris.
Properly scoped use + compilation of C++ unit testing code using Catch to
clang only, as Catch includes code that does not strictly conform to the C++98 standard. (@kevinushey)
Fixed an out-of-bounds memory access when routing Catch output through
Improved overriding of calls to
exit() within Catch, to ensure compatibility with GCC 6.0. (@krlmlr)
expectation() function now expects an expectation type (one of “success”, “failure”, “error”, “skip”, “warning”) as first argument. If you’re creating your own expectations, you’ll need to use
expect() instead (#437).
The expectation system got a thorough overhaul (#217). This primarily makes it easier to add new expectations in the future, but also included a thorough review of the documentation, ensuring that related expectations are documented together, and have evocative names.
One useful change is that most expectations invisibly return the input
object. This makes it possible to chain together expectations with magrittr:
(And to make this style even easier, testthat now re-exports the pipe, #412).
The exception to this rule are the expectations that evaluate ( i.e. for messages, warnings, errors, output etc), which invisibly return
NULL. These functions are now more consistent: using
NA will cause a failure if there is a errors/warnings/mesages/output (i.e. they’re not missing), and will
NULL fail if there aren’t any errors/warnings/mesages/output. This previously didn’t work for
expect_output() (#323), and the error messages were confusing with
expect_error(..., NA) (#342, @nealrichardson + @krlmlr, #317).
Another change is that
expect_output() now requires you to explicitly print the output if you want to test a print method:
expect_output("a", "a") will fail,
expect_output(print("a"), "a") will succeed.
There are six new expectations:
expect_type() checks the type of the object (#316),
expect_s3_class() tests that an object is S3 with given class,
expect_s4_class() tests that an object is S4 with given class (#373). I recommend using these more specific expectations instead of the more general
expect_length() checks that an object has expected length.
A number of older features have been deprecated:
Now all expectations are also conditions, and R’s condition system is used to signal failures and successes (#360, @krlmlr). All known conditions (currently, “error”, “warning”, “message”, “failure”, and “success”) are converted to expectations using the new
as.expectation(). This allows third-party test packages (such as
assertive) to seamlessly establish
testthat compatibility by issuing custom error conditions (e.g.,
structure(list(message = "Error message"), class = c("customError", "error", "condition"))) and then implementing
assertthat package contains an example.
The reporters system class has been considerably refactored to make existing reporters simpler and to make it easier to write new reporters. There are two main changes:
Reporters classes are now R6 classes instead of Reference Classes.
Each callbacks receive the full context:
add_results()is passed context and test as well as the expectation.
test_end()both get the context and test.
context_end()get the context.
Warnings are now captured and reported in most reporters.
The default summary reporter lists all warnings (#310), and all skipped tests (@krlmlr, #343). New option
testthat.summary.max_reports limits the number of reports printed by the summary reporter. The default is 15 (@krlmlr, #354).
MinimalReporter correct labels errors with E and failures with F (#311).
capture_warnings() selectively capture function output. These are used in
expect_warning() to allow other types out output to percolate up (#410).
source_file() exports the function testthat uses to load files from disk.
Comparing integers to non-numbers doesn’t raise errors anymore, and falls back to string comparison if objects have different lengths. Complex numbers are compared using the same routine (#309, @krlmlr).
compare.character() received another overhaul. This should improve behaviour of edge cases, and provides a strong foundation for further work. Added
compare.POSIXt() for better reporting of datetime differences.
Handle skipped tests in the TAP reporter (#262).
Praise gets more diverse thanks to the praise package, and you’ll now get random encouragment if your tests don’t pass.
Use tests in
inst/tests is formally deprecated. Please move them into
tests/testthat instead (#231).
comparison() constructure now checks its arguments are the correct type and length. This bugs a bug where tests failed with an error like “values must be length 1, but FUN(X[]) result is length 2” (#279).
Failure locations are now formated as R error locations.
library_if_available() has been removed.
test_check() have an added
... argument that allows filtering of test files using, e.g., Perl-style regular expressions,or
fixed character filtering. Arguments in
... are passed to
test_check() uses a new reporter specifically designed for
R CMD check. It displays a summary at the end of the tests, designed to be <13 lines long so test failures in
R CMD check display something more useful. This will hopefully stop BDR from calling testthat a “test obfuscation suite” (#201).
compare() is now documented and exported. Added a numeric method so when long numeric vectors don’t match you’ll see some examples of where the problem is (#177). The line spacing in
compare.character() was tweaked.
expect_that(a, equals(b)) style of testing has been soft-deprecated. It will keep working, but it’s no longer demonstrated any where, and new expectations will only be available in
expect_equal(a, b) style. (#172)
Once again, testthat suppresses messages and warnings in tests (#189)
succeed() expectation always succeeds.
colourise() was removed. (Colour is still supported, via the
All equality expectations are now documented together (#173); all matching expectations are also documented together.
?describe() for further information and examples.
It’s now possible to
skip() a test with an informative message - this is useful when tests are only available under certain conditions, as when not on CRAN, or when an internet connection is available (#141).
skip_on_cran() allows you to skip tests when run on CRAN. To take advantage of this code, you’ll need either to use devtools, or run
Sys.setenv(NOT_CRAN = "true"))
with_mock() makes it easy to temporarily replace functions defined in packages. This is useful for testing code that relies on functions that are slow, have unintended side effects or access resources that may not be available when testing (#159, @krlmlr).
compare.character() fixed, as reported by Georgi Boshnakov.
Random praise is renabled by default (again!) (#164).
testthat 0.8 comes with a new recommended structure for storing your tests. To better meet CRAN recommended practices, testthat now recommend that you to put your tests in
tests/testthat, instead of
inst/tests (this makes it possible for users to choose whether or not to install tests). With this new structure, you’ll need to use
test_check() instead of
test_packages() in the test file (usually
tests/testthat.R) that runs all testthat unit tests.
The other big improvement to usability comes from @kforner, who contributed code to allow the default results (i.e. those produced by
SummaryReporter) to include source references so you can see exactly where failures occured.
MultiReporter, which combines several reporters into one. (Thanks to @kforner)
ListReporter, which captures all test results with their file, context, test and elapsed time.
test_check now use the
ListReporter to invisibly return a summary of the tests as a data frame. (Thanks to @kforner)
TeamCityReporter to produce output compatible with the TeamCity continuous integration environment. (Thanks to @windelinckx)
fixed two bugs with source_dir(): it did not look for the source scripts at the right place, and it did not use its
expect_equal() to compare strings, the default output for failure provides a lot more information, which should hopefully help make finding string mismatches easier.
SummaryReporter has a
max_reports option to limit the number of detailed failure reports to show. (Thanks to @crowding)
Tracebacks will now also contain information about where the functions came from (where that information is available).
expect_match now pass additional arguments on to
grepl so that you can use
fixed = TRUE,
perl = TRUE or
ignore.case = TRUE to control details of the match.
expect_match now correctly fails to match NULL. (#100)
expect_error also pass … on to
grepl, so that you can use
fixed = TRUE,
perl = TRUE or
ignore.case = TRUE
Ignore attributes in
make_expectation works for more types of input (#52)
Now works better with evaluate 0.4.3.
TapReporter to produce output compatible with the “test anything protocol”. Contributed by Dan Keshet.
auto_test would identify the wrong files as having changed. (Thanks to Peter Meilstrup)
SummaryReporter: still return informative messages even if no tests defined (just bare expectations). (Fixes #31)
Improvements to reference classes (Thanks to John Chambers)
Bug fixes for when nothing was generated in
shows_message. (Thanks to Bernd Bischl)
make_expectation function to programmatically generate an equality expectation. (Fixes #24)
SummaryReporter: You don’t get praise until you have some tests.
methods rather than requiring it so that testthat works when run from
auto_test now normalises paths to enable better identification of file changes, and fixes bug in instantiating new reporter object.
mutatr classes have been replaced with ReferenceClasses.
Better documentation for short-hand expectations.
test_package gain new
filter argument which allows you to restrict which tests are run.
autotest correctly loads code and executes tests in same environment
contexts are never closed before they are opened, and always closed at the end of file
fixed small bug in
test_dir where each test was not given its own environment
expect_* short cut functions gain a label argument, thanks to Steve Lianoglou
all expectations now have a shortcut form, so instead of expect_that(a, is_identical_to(b)) you can do expect_identical(a, b)
new shows_message and gives_warning expectations to test warnings and messages
expect_that, equals, is_identical_to and is_equivalent to now have additional label argument which allows you to control the appearance of the text used for the expected object (for expect_that) and actual object (for all other functions) in failure messages. This is useful when you have loops that run tests as otherwise all the variable names are identical, and it’s difficult to tell which iteration caused the failure.
executing bare tests gives nicer output
all expectations now give more information on failure to make it easier to track down the problem.
test_file and test_dir now run in code in separate environment to avoid pollution of global environment. They also temporary change the working directory so tests can use relative paths.
test_package makes it easier to run all tests in an installed package. Code run in this manner has access to non-exported functions and objects. If any errors or failures occur, test_package will throw an error, making it suitable for use with R CMD check.
colourise also works in screen terminal
equals expectation provides more information about failure
expect_that has extra info argument to allow you to pass in any extra information you’d like included in the message - this is very helpful if you’re using a loop to run tests
is_equivalent_to: new expectation that tests for equality ignoring attributes
library_if_available now works! (thanks to report and fix from Felix Andrews)
specify larger width and join pieces back together whenever deparse used (thanks to report and fix from Felix Andrews)
test_dir now looks for any files starting with test (not test- as before)