testthat 3.0.0 introduces the idea of an “edition” of testthat. An edition is a bundle of behaviours that you have to explicitly choose to use, allowing us to make otherwise backward incompatible changes. This is particularly important for testthat since it has a very large number of packages that use it (almost 5,000 at last count). Choosing to use the 3rd edition allows you to use our latest recommendations for ongoing and new work, while historical packages continue to use the old behaviour.

(We don’t anticipate creating new editions very often, and they’ll always be matched with major version, i.e. if there’s another edition, it’ll be the fourth edition and will come with testthat 4.0.0.)

This vignette shows you how to activate the 3rd edition, introduces the main features, and discusses common challenges when upgrading a package. If you have a problem that this vignette doesn’t cover, please let me know, as it’s likely that the problem also affects others.

library(testthat)
local_edition(3)

Activating

The usual way to activate the 3rd edition is to add a line to your DESCRIPTION:

Config/testthat/edition: 3

This will activate the 3rd edition for every test in your package.

You can also control the edition used for indvidual tests with testthat::local_edition():

test_that("I can use the 3rd edition", {
  local_edition(3)
  expect_true(TRUE)
})
#> Test passed 🥇

This is also useful if you’ve switched to the 3rd edition and have a couple of tests that fail. You can use local_edition(2) to revert back to the old behaviour, giving you some breathing room to figure out the underlying issue.

test_that("I want to use the 2nd edition", {
  local_edition(2)
  expect_true(TRUE)
})
#> Test passed 🌈

Changes

There are three major changes in the 3rd edition:

  • A number of outdated functions are now deprecated, so you’ll be warned about them every time you run your tests (but they won’t cause R CMD check to fail).

  • testthat no longer silently swallows messages; you now need to deliberately handle them.

  • expect_equal() and expect_identical() now use the waldo package instead of identical() and all.equal(). This makes them more consistent and provides an enhanced display of differences when a test fails.

Deprecations

A number of outdated functions have been deprecated. Most of these functions have not been recommended for a number of years, but before the introduction of the edition idea, I didn’t have a good way of preventing people from using them without breaking a lot of code on CRAN.

Fixing these deprecation warnings should be straightforward.

Messages

For reasons that I can no longer remember, testthat silently ignores all messages. This is inconsistent with other types of output, so as of the 3rd edition, they now bubble up to your test results. You’ll have to explicit ignore them with supressMesssages(), or if they’re important, test for their presence with expect_message().

waldo

Probably the the biggest day-to-day difference (and the biggest reason to upgrade!) is the use of waldo::compare() inside of expect_equal() and expect_identical(). The goal of waldo is to find and concisely describe the difference between a pair of R objects, and it’s designed specifically to help you figure out what’s gone wrong in your unit tests.

f1 <- factor(letters[1:3])
f2 <- ordered(letters[1:3], levels = letters[1:4])

local_edition(2)
expect_equal(f1, f2)
#> Error: `f1` not equal to `f2`.
#> Attributes: < Component "class": Lengths (1, 2) differ (string compare on first 1) >
#> Attributes: < Component "class": 1 string mismatch >
#> Attributes: < Component "levels": Lengths (3, 4) differ (string compare on first 3) >

local_edition(3)
expect_equal(f1, f2)
#> Error: `f1` (`actual`) not equal to `f2` (`expected`).
#> 
#> `class(actual)`:             "factor"
#> `class(expected)`: "ordered" "factor"
#> 
#> `levels(actual)`:   "a" "b" "c"    
#> `levels(expected)`: "a" "b" "c" "d"

waldo looks even better in your console because it carefully uses colours to help highlight the differences.

The use of waldo also makes precise the difference between expect_equal() and expect_identical(): expect_equal() sets tolerance so that waldo will ignore small numerical differences arising from floating point computation. Otherwise the functions are identical (HA HA).

This change is likely to result in the most work during an upgrade, because waldo can give slightly different results to both identical() and all.equal() in moderately common situations. I believe on the whole the differences are meaningful and useful, so you’ll need to handle them by tweaking your tests. The following changes are most likely to affect you:

  • expect_equal() previously ignored the environments of formulas and functions. This is most like to arise if you are testing models. It’s worth thinking about what the correct values should be, but if that is to annoying you can opt out of the comparison with ignore_function_env or ignore_formula_env.

  • expect_equal() used a combination of all.equal() and a home-grown testthat::compare() which unfortunately used a slightly different definition of tolerance. Now expect_equal() always uses the same defintion of tolerance everywhere, which may require tweaks to your exising tolerance values.

  • expect_equal() previously ignored timezone differences when one object had the current timezone set implicitly (with "") and the other had it set explictly:

    dt1 <- dt2 <- ISOdatetime(2020, 1, 2, 3, 4, 0)
    attr(dt1, "tzone") <- ""
    attr(dt2, "tzone") <- Sys.timezone()
    
    local_edition(2)
    expect_equal(dt1, dt2)
    
    local_edition(3)
    expect_equal(dt1, dt2)
    #> Error: `dt1` (`actual`) not equal to `dt2` (`expected`).
    #> 
    #> `attr(actual, 'tzone')`:   ""   
    #> `attr(expected, 'tzone')`: "UTC"