Skip to content

Custom expectations

Source: vignettes/custom-expectation.Rmd
custom-expectation.Rmd

This vignette shows you how to write your own expectations. Custom expectations allow you to extend testthat to meet your own specialized testing needs, creating new expect_* functions that work exactly the same way as the built-ins. Custom expectations are particularly useful if you want to produce expectations tailored for domain-specific data structures, combine multiple checks into a single expectation, or create more actionable feedback when an expectation fails. You can use them within your package by putting them in a helper file, or share them with others by exporting them from your package.

In this vignette, you’ll learn about the three-part structure of expectations, how to test your custom expectations, see a few examples, and, if you’re writing a lot of expectations, learn how to reduce repeated code.

Expectation basics

An expectation has three main parts, as illustrated by expect_length():

expect_length <- function(object, n) {  
  # 1. Capture object and label
  act <- quasi_label(rlang::enquo(object))

  # 2. Check if expectations are violated
  act_n <- length(act$val)
  if (act_n != n) {
    msg <- c(
      sprintf("Expected %s to have length %i.", act$lab, n),
      sprintf("Actual length: %i.", act_n)
    )
    return(fail(msg))
  }
  
  # 3. Pass when expectations are met
  pass(act$val)
}

The first step in any expectation is to use quasi_label() to capture a “labeled value”, i.e., a list that contains both the value ($val) for testing and a label ($lab) used to make failure messages as informative as possible. This is a pattern that exists for fairly esoteric reasons; you don’t need to understand it, just copy and paste it.

Next you need to check each way that object could violate the expectation. In this case, there’s only one check, but in more complicated cases there can be multiple checks. In most cases, it’s easier to check for violations one by one, using early returns to fail(). This makes it easier to write informative failure messages that first describe what was expected and then what was actually seen.

Note that you need to use return(fail()) here. If you don’t, your expectation might end up failing multiple times or both failing and succeeding. You won’t see these problems when interactively testing your expectation, but forgetting to return() can lead to incorrect fail and pass counts in typical usage. In the next section, you’ll learn how to test your expectation to avoid this issue.

Finally, if the object is as expected, call pass() with act$val. This is good practice because expectation functions are called primarily for their side-effects (triggering a failure), and returning the value allows expectations to be piped together:

test_that("mtcars is a 13 row data frame", {
  mtcars |>
    expect_type("list") |>
    expect_s3_class("data.frame") |> 
    expect_length(11)
})
#> Test passed with 3 successes 🥇.

Testing your expectations

Once you’ve written your expectation, you need to test it: expectations are functions that can have bugs, just like any other function, and it’s really important that they generate actionable failure messages. Luckily testthat comes with three expectations designed specifically to test expectations:

  • expect_success() checks that your expectation emits exactly one success and zero failures.
  • expect_failure() checks that your expectation emits exactly one failure and zero successes.
  • expect_snapshot_failure() captures the failure message in a snapshot, making it easier to review whether it’s useful.

The first two expectations are particularly important because they ensure that your expectation always reports either a single success or a single failure. If it doesn’t, the end user is going to get confusing results in their test suite reports.

test_that("expect_length works as expected", {
  x <- 1:10
  expect_success(expect_length(x, 10))
  expect_failure(expect_length(x, 11))
})
#> Test passed with 2 successes 🎊.

test_that("expect_length gives useful feedback", {
  x <- 1:10
  expect_snapshot_failure(expect_length(x, 11))
})
#> ── Warning: expect_length gives useful feedback ───────────────────────
#> Adding new snapshot:
#> Code
#>   expect_length(x, 11)
#> Condition
#>   Error:
#>   ! Expected `x` to have length 11.
#>   Actual length: 10.
#> Test passed with 1 success 🌈.

Examples

The following sections show you a few more variations, loosely based on existing testthat expectations. These expectations were picked to show how you can generate actionable failures in slightly more complex situations.

expect_vector_length()

Let’s make expect_length() a bit more strict by also checking that the input is a vector. R is a bit unusual in that it gives a length to pretty much every object, and you can imagine not wanting code like the following to succeed, because it’s likely that the user passed the wrong object to the test.

expect_length(mean, 1)

To do this we’ll add an extra check that the input is either an atomic vector or a list:

expect_vector_length <- function(object, n) {  
  act <- quasi_label(rlang::enquo(object))

  # It's non-trivial to check if an object is a vector in base R so we
  # use an rlang helper
  if (!rlang::is_vector(act$val)) {
    msg <- c(
      sprintf("Expected %s to be a vector", act$lab),
      sprintf("Actual type: %s", typeof(act$val))
    )
    return(fail(msg))
  }

  act_n <- length(act$val)
  if (act_n != n) {
    msg <- c(
      sprintf("Expected %s to have length %i.", act$lab, n),
      sprintf("Actual length: %i.", act_n)
    )
    return(fail(msg))
  }
  
  pass(act$val)
}
expect_vector_length(mean, 1)
#> Error: Expected `mean` to be a vector
#> Actual type: closure
expect_vector_length(mtcars, 15)
#> Error: Expected `mtcars` to have length 15.
#> Actual length: 11.

expect_s3_class()

Or imagine you’re checking to see if an object inherits from an S3 class. R has a lot of different OO systems, and you want your failure messages to be as informative as possible, so before checking that the class matches, you probably want to check that the object is from the correct OO family.

expect_s3_class <- function(object, class) {
  if (!rlang::is_string(class)) {
    rlang::abort("`class` must be a string.")
  }

  act <- quasi_label(rlang::enquo(object))

  if (!is.object(act$val)) {
    msg <- sprintf("Expected %s to be an object.", act$lab)
    return(fail(msg))
  }

  if (isS4(act$val)) {
    msg <- c(
      sprintf("Expected %s to be an S3 object.", act$lab),
      "Actual OO type: S4"
    )
    return(fail(msg))
  }

  if (!inherits(act$val, class)) {
    msg <- c(
      sprintf("Expected %s to inherit from %s.", act$lab, class),
      sprintf("Actual class: %s", class(act$val))
    )
    return(fail(msg))
  }

  pass(act$val)
}
x1 <- 1:10
TestClass <- methods::setClass("Test", contains = "integer")
x2 <- TestClass()
x3 <- factor()

expect_s3_class(x1, "integer")
#> Error: Expected `x1` to be an object.
expect_s3_class(x2, "integer")
#> Error: Expected `x2` to be an S3 object.
#> Actual OO type: S4
expect_s3_class(x3, "integer")
#> Error: Expected `x3` to inherit from integer.
#> Actual class: factor
expect_s3_class(x3, "factor")

Note the variety of error messages. We always print what was expected, and where possible, also display what was actually received:

  • When object isn’t an object, we can only say what we expected.
  • When object is an S4 object, we can report that.
  • When inherits() is FALSE, we provide the actual class, since that’s most informative.

The general principle is to tailor error messages to what the user can act on based on what you know about the input.

Also note that I check that the class argument is a string. If it’s not a string, I throw an error. This is not a test failure; the user is calling the function incorrectly. In general, you should check the type of all arguments that affect the operation and error if they’re not what you expect.

expect_s3_class(x1, 1)
#> Error in `expect_s3_class()`:
#> ! `class` must be a string.

Optional class

A common pattern in testthat’s own expectations it to use arguments to control the level of detail in the test. Here it would be nice if we check that an object is an S3 object without checking for a specific class. I think we could do that by renaming expect_s3_class() to expect_s3_object(). Now expect_s3_object(x) would verify that x is an S3 object, and expect_s3_object(x, class = "foo") to verify that x is an S3 object with the given class. The implementation of this is straightforward: we also allow class to be NULL and then only verify inheritance when non-NULL.

expect_s3_object <- function(object, class = NULL) {
  if (!rlang::is_string(class) && is.null(class)) {
    rlang::abort("`class` must be a string or NULL.")
  }

  act <- quasi_label(rlang::enquo(object))

  if (!is.object(act$val)) {
    msg <- sprintf("Expected %s to be an object.", act$lab)
    return(fail(msg))
  }

  if (isS4(act$val)) {
    msg <- c(
      sprintf("Expected %s to be an S3 object.", act$lab),
      "Actual OO type: S4"
    )
    return(fail(msg))
  }

  if (!is.null(class) && !inherits(act$val, class)) {
    msg <- c(
      sprintf("Expected %s to inherit from %s.", act$lab, class),
      sprintf("Actual class: %s", class(act$val))
    )
    return(fail(msg))
  }

  pass(act$val)
}

Repeated code

As you write more expectations, you might discover repeated code that you want to extract into a helper. Unfortunately, creating 100% correct helper functions is not straightforward in testthat because fail() captures the calling environment in order to give useful tracebacks, and testthat’s own expectations don’t expose this as an argument. Fortunately, getting this right is not critical (you’ll just get a slightly suboptimal traceback in the case of failure), so we don’t recommend bothering in most cases. We document it here, however, because it’s important to get it right in testthat itself.

The key challenge is that fail() captures a trace_env, which should be the execution environment of the expectation. This usually works because the default value of trace_env is caller_env(). But when you introduce a helper, you’ll need to explicitly pass it along:

expect_length_ <- function(act, n, trace_env = caller_env()) {
  act_n <- length(act$val)
  if (act_n != n) {
    msg <- sprintf("%s has length %i, not length %i.", act$lab, act_n, n)
    return(fail(msg, trace_env = trace_env))
  }

  pass(act$val)
}

expect_length <- function(object, n) {  
  act <- quasi_label(rlang::enquo(object))
  expect_length_(act, n)
}

A few recommendations:

  • The helper shouldn’t be user-facing, so we give it a _ suffix to make that clear.
  • It’s typically easiest for a helper to take the labeled value produced by quasi_label().
  • Your helper should usually call both fail() and pass() and be returned from the wrapping expectation.

Again, you’re probably not writing so many expectations that it makes sense for you to go to this effort, but it is important for testthat to get it right.