This vignette shows you how to write your own expectations. Custom
expectations allow you to extend testthat to meet your own specialized
testing needs, creating new expect_*
functions that work
exactly the same way as the built-ins. Custom expectations are
particularly useful if you want to produce expectations tailored for
domain-specific data structures, combine multiple checks into a single
expectation, or create more actionable feedback when an expectation
fails. You can use them within your package by putting them in a helper
file, or share them with others by exporting them from your package.
In this vignette, you’ll learn about the three-part structure of expectations, how to test your custom expectations, see a few examples, and, if you’re writing a lot of expectations, learn how to reduce repeated code.
Expectation basics
An expectation has three main parts, as illustrated by
expect_length()
:
expect_length <- function(object, n) {
# 1. Capture object and label
act <- quasi_label(rlang::enquo(object))
# 2. Check if expectations are violated
act_n <- length(act$val)
if (act_n != n) {
msg <- c(
sprintf("Expected %s to have length %i.", act$lab, n),
sprintf("Actual length: %i.", act_n)
)
return(fail(msg))
}
# 3. Pass when expectations are met
pass(act$val)
}
The first step in any expectation is to use
quasi_label()
to capture a “labeled value”, i.e., a list
that contains both the value ($val
) for testing and a label
($lab
) used to make failure messages as informative as
possible. This is a pattern that exists for fairly esoteric reasons; you
don’t need to understand it, just copy and paste it.
Next you need to check each way that object
could
violate the expectation. In this case, there’s only one check, but in
more complicated cases there can be multiple checks. In most cases, it’s
easier to check for violations one by one, using early returns to
fail()
. This makes it easier to write informative failure
messages that first describe what was expected and then what was
actually seen.
Note that you need to use return(fail())
here. If you
don’t, your expectation might end up failing multiple times or both
failing and succeeding. You won’t see these problems when interactively
testing your expectation, but forgetting to return()
can
lead to incorrect fail and pass counts in typical usage. In the next
section, you’ll learn how to test your expectation to avoid this
issue.
Finally, if the object is as expected, call pass()
with
act$val
. This is good practice because expectation
functions are called primarily for their side-effects (triggering a
failure), and returning the value allows expectations to be piped
together:
test_that("mtcars is a 13 row data frame", {
mtcars |>
expect_type("list") |>
expect_s3_class("data.frame") |>
expect_length(11)
})
#> Test passed with 3 successes 🥇.
Testing your expectations
Once you’ve written your expectation, you need to test it: expectations are functions that can have bugs, just like any other function, and it’s really important that they generate actionable failure messages. Luckily testthat comes with three expectations designed specifically to test expectations:
-
expect_success()
checks that your expectation emits exactly one success and zero failures. -
expect_failure()
checks that your expectation emits exactly one failure and zero successes. -
expect_snapshot_failure()
captures the failure message in a snapshot, making it easier to review whether it’s useful.
The first two expectations are particularly important because they ensure that your expectation always reports either a single success or a single failure. If it doesn’t, the end user is going to get confusing results in their test suite reports.
test_that("expect_length works as expected", {
x <- 1:10
expect_success(expect_length(x, 10))
expect_failure(expect_length(x, 11))
})
#> Test passed with 2 successes 🎊.
test_that("expect_length gives useful feedback", {
x <- 1:10
expect_snapshot_failure(expect_length(x, 11))
})
#> ── Warning: expect_length gives useful feedback ───────────────────────
#> Adding new snapshot:
#> Code
#> expect_length(x, 11)
#> Condition
#> Error:
#> ! Expected `x` to have length 11.
#> Actual length: 10.
#> Test passed with 1 success 🌈.
Examples
The following sections show you a few more variations, loosely based on existing testthat expectations. These expectations were picked to show how you can generate actionable failures in slightly more complex situations.
expect_vector_length()
Let’s make expect_length()
a bit more strict by also
checking that the input is a vector. R is a bit unusual in that it gives
a length to pretty much every object, and you can imagine not wanting
code like the following to succeed, because it’s likely that the user
passed the wrong object to the test.
expect_length(mean, 1)
To do this we’ll add an extra check that the input is either an atomic vector or a list:
expect_vector_length <- function(object, n) {
act <- quasi_label(rlang::enquo(object))
# It's non-trivial to check if an object is a vector in base R so we
# use an rlang helper
if (!rlang::is_vector(act$val)) {
msg <- c(
sprintf("Expected %s to be a vector", act$lab),
sprintf("Actual type: %s", typeof(act$val))
)
return(fail(msg))
}
act_n <- length(act$val)
if (act_n != n) {
msg <- c(
sprintf("Expected %s to have length %i.", act$lab, n),
sprintf("Actual length: %i.", act_n)
)
return(fail(msg))
}
pass(act$val)
}
expect_vector_length(mean, 1)
#> Error: Expected `mean` to be a vector
#> Actual type: closure
expect_vector_length(mtcars, 15)
#> Error: Expected `mtcars` to have length 15.
#> Actual length: 11.
expect_s3_class()
Or imagine you’re checking to see if an object inherits from an S3 class. R has a lot of different OO systems, and you want your failure messages to be as informative as possible, so before checking that the class matches, you probably want to check that the object is from the correct OO family.
expect_s3_class <- function(object, class) {
if (!rlang::is_string(class)) {
rlang::abort("`class` must be a string.")
}
act <- quasi_label(rlang::enquo(object))
if (!is.object(act$val)) {
msg <- sprintf("Expected %s to be an object.", act$lab)
return(fail(msg))
}
if (isS4(act$val)) {
msg <- c(
sprintf("Expected %s to be an S3 object.", act$lab),
"Actual OO type: S4"
)
return(fail(msg))
}
if (!inherits(act$val, class)) {
msg <- c(
sprintf("Expected %s to inherit from %s.", act$lab, class),
sprintf("Actual class: %s", class(act$val))
)
return(fail(msg))
}
pass(act$val)
}
x1 <- 1:10
TestClass <- methods::setClass("Test", contains = "integer")
x2 <- TestClass()
x3 <- factor()
expect_s3_class(x1, "integer")
#> Error: Expected `x1` to be an object.
expect_s3_class(x2, "integer")
#> Error: Expected `x2` to be an S3 object.
#> Actual OO type: S4
expect_s3_class(x3, "integer")
#> Error: Expected `x3` to inherit from integer.
#> Actual class: factor
expect_s3_class(x3, "factor")
Note the variety of error messages. We always print what was expected, and where possible, also display what was actually received:
- When
object
isn’t an object, we can only say what we expected. - When
object
is an S4 object, we can report that. - When
inherits()
isFALSE
, we provide the actual class, since that’s most informative.
The general principle is to tailor error messages to what the user can act on based on what you know about the input.
Also note that I check that the class
argument is a
string. If it’s not a string, I throw an error. This is not a test
failure; the user is calling the function incorrectly. In general, you
should check the type of all arguments that affect the operation and
error if they’re not what you expect.
expect_s3_class(x1, 1)
#> Error in `expect_s3_class()`:
#> ! `class` must be a string.
Optional class
A common pattern in testthat’s own expectations it to use arguments
to control the level of detail in the test. Here it would be nice if we
check that an object is an S3 object without checking for a specific
class. I think we could do that by renaming
expect_s3_class()
to expect_s3_object()
. Now
expect_s3_object(x)
would verify that x
is an
S3 object, and expect_s3_object(x, class = "foo")
to verify
that x
is an S3 object with the given class. The
implementation of this is straightforward: we also allow
class
to be NULL
and then only verify
inheritance when non-NULL
.
expect_s3_object <- function(object, class = NULL) {
if (!rlang::is_string(class) && is.null(class)) {
rlang::abort("`class` must be a string or NULL.")
}
act <- quasi_label(rlang::enquo(object))
if (!is.object(act$val)) {
msg <- sprintf("Expected %s to be an object.", act$lab)
return(fail(msg))
}
if (isS4(act$val)) {
msg <- c(
sprintf("Expected %s to be an S3 object.", act$lab),
"Actual OO type: S4"
)
return(fail(msg))
}
if (!is.null(class) && !inherits(act$val, class)) {
msg <- c(
sprintf("Expected %s to inherit from %s.", act$lab, class),
sprintf("Actual class: %s", class(act$val))
)
return(fail(msg))
}
pass(act$val)
}
Repeated code
As you write more expectations, you might discover repeated code that
you want to extract into a helper. Unfortunately, creating 100% correct
helper functions is not straightforward in testthat because
fail()
captures the calling environment in order to give
useful tracebacks, and testthat’s own expectations don’t expose this as
an argument. Fortunately, getting this right is not critical (you’ll
just get a slightly suboptimal traceback in the case of failure), so we
don’t recommend bothering in most cases. We document it here, however,
because it’s important to get it right in testthat itself.
The key challenge is that fail()
captures a
trace_env
, which should be the execution environment of the
expectation. This usually works because the default value of
trace_env
is caller_env()
. But when you
introduce a helper, you’ll need to explicitly pass it along:
expect_length_ <- function(act, n, trace_env = caller_env()) {
act_n <- length(act$val)
if (act_n != n) {
msg <- sprintf("%s has length %i, not length %i.", act$lab, act_n, n)
return(fail(msg, trace_env = trace_env))
}
pass(act$val)
}
expect_length <- function(object, n) {
act <- quasi_label(rlang::enquo(object))
expect_length_(act, n)
}
A few recommendations:
- The helper shouldn’t be user-facing, so we give it a
_
suffix to make that clear. - It’s typically easiest for a helper to take the labeled value
produced by
quasi_label()
. - Your helper should usually call both
fail()
andpass()
and be returned from the wrapping expectation.
Again, you’re probably not writing so many expectations that it makes sense for you to go to this effort, but it is important for testthat to get it right.