Customizing test scripts
The test recorder is the easiest way to create test scripts, but it is not the only way. You can create and edit test scripts manually.
A test script has this basic structure: first, there is an initialization, then the tests, and finally the tests are wrapped up.
In the initialization, the script creates a new ShinyDriver object and tells it what name to use for this set of tests.
# Initialize a ShinyDriver object using the app in the test script's parent
# directory
app <- ShinyDriver$new("..")
app$snapshotInit("mytest")Next, it defines some interactions with the application and takes snapshots.
app$setInputs(checkGroup = c("1", "2"))
app$setInputs(checkGroup = c("1", "2", "3"))
app$setInputs(action = "click")
app$snapshot()
app$setInputs(action = "click")
app$snapshot()For customizing a script, the second portion – the interactions and
snapshots – is the part you will want to modify. For snapshot-based
testing, there are two methods that are used:
app$setInputs() and app$snapshot().
Setting inputs with app$setInputs()
With app$setInputs(), you provide the name of one or
more inputs and corresponding values to set them to. Consider this set
of directives:
app$setInputs(checkGroup = c("1", "2"))
app$setInputs(checkGroup = c("1", "2", "3"))
app$setInputs(action = "click")Notice that we set the value of checkGroup two times in
a row. When we recorded this test script, it started with the value
"1", and then we checked the "2" and
"3" boxes. The recorded script set the value to
c("1", "2"), and then c("1", "2", "3"). The
c("1", "2") value was simply an intermediate step.
It’s possible to simplify and speed up the tests by dropping the intermediate step, which leaves us with this:
app$setInputs(checkGroup = c("1", "2", "3"))
app$setInputs(action = "click")And it’s also possible to set action in the same call,
resulting in this:
app$setInputs(
checkGroup = c("1", "2", "3"),
action = "click"
)This will set the values of inputs simultaneously, which will make the tests run faster.
This is because, when app$setInputs() is called, it
normally returns control and moves on to the next step only after the
server sends a response to the client.
The reason it waits for a response is so that a subsequent call to
app$snapshot() will be sure to capture the updated output
values. If app$setInputs() did not wait for a update, then,
if the output update did not happen very quickly, a snapshot might
capture the state of the application before the outputs are updated.
Because app$setInputs() waits for an update each time,
it is faster to set multiple inputs in a single call to
app$setInputs() than it is to have multiple calls to
app$setInputs().
Note: In versions of Shiny before 1.0.4, calls to
app$setInputs() which did not result in an output value
change would timeout, and print a message about setting a longer timeout
or calling setInputs(wait_ = FALSE, values_ = FALSE). This
is because those versions of Shiny did not send a response when no
output values changed. As of Shiny 1.0.4, when in testing mode, Shiny
always sends a response to input changes, even if no output values have
changed, so this message should no longer appear.
Taking snapshots with app$snapshot()
There are two ways to use app$snapshot(). The simplest
way is to call it with no arguments:
app$snapshot()The first time this is called in a test script, it will record all
input, output, and exported values from the application, in a file
called 001.json. The next call will save the values in
002.json, and so on.
Each time you call app$snapshot(), it will also save a
screen shot of the web browser, in a file
001.png, 002.png, and so on. These screen
shots are useful for debugging your tests and inspecting what they’re
doing. You can tell it to not take screen shots, to save space and make
the tests run slightly faster, in the initialization step, with:
app$snapshotInit("mytest", screenshot = FALSE)If you want to disable screenshots for a single snapshot, you can use:
app$snapshot(screenshot = FALSE)If you want more targeted tests, you can snapshot specific items with
the items argument. For example, to capture the value of
just the outputs named "a" and "b", you would
call:
The value passed to items is a named list, where the
output is a character vector with the names of outputs to
snapshot. You could also capture specific inputs or exports:
Finally, if you want to snapshot all outputs but no inputs or
exports, you can simply set output to
TRUE:
app$snapshot(items = list(output = TRUE))The same can be used to snapshot all inputs and/or all exports. To capture all outputs and exports, but no inputs:
app$snapshot(items = list(output = TRUE, export = TRUE))Exported values
In some cases, it’s useful to snapshot some bits of internal state of an application – state that’s not reflected directly in the inputs or outputs. This can be done by exporting values.
Consider this toy example where an application has an internal reactive expression that would be convenient to capture.
shinyApp(
fluidPage(
numericInput("x", "x", 4),
numericInput("y", "y", 10),
numericInput("z", "z", 100),
verbatimTextOutput("result", placeholder = TRUE)
),
function(input, output, session) {
xy <- reactive(input$x * input$y)
yz <- reactive(input$y + input$z)
output$result <- renderText({
xy() / yz()
})
exportTestValues(xy = xy(), yz = yz())
}
)Notice this line:
exportTestValues(xy = xy(), yz = yz())This creates exported values called xy and
yz. When a test snapshot is taken, it evaluates the
reactive expression and inserts their values in the snapshot:
{
"input": {
"x": 4,
"y": 10,
"z": 100
},
"output": {
"result": "0.3636364"
},
"export": {
"xy": 40,
"yz": 110
}
}Note that the expression (i.e. xy() and
yz()) is saved exportTestValues() is called,
not the specific values. Generally, you should keep these expressions as
simple as possible so that test failures yield informative outputs. Note
that any value here has to be serializable to JSON, so it’s not suitable
for testing complex objects.
Adding delays
In some cases, you may need to wait for some amount of time between
steps. You can do this by adding Sys.sleep() in your
script. For example:
Sys.sleep(0.5)If you are waiting for some reactives to be calculated, please read
more in the Waiting for an input
(or output) value section.
Controlling randomness
If your application uses randomly-generated values (by calling
functions like rnorm, runif,
sample directly or indirectly), then, in normal operation,
it will produce different results on each run. Because Shinytest works
by comparing the current state of an application to a previous state,
these random changes will cause test failures.
To make such applications exactly repeatable, you can set the random seed. This can be done by specifying a random seed in the recorder. The seed can be any integer value.
If your test script has already been created, you can set the random
seed by editing the test script so that the
ShinyDriver$new() call has a value for seed.
Moreover, if you need these tests to pass on multiple versions of R with
different RNGkind() defaults (e.g., 3.5 and 3.6), you may
want to set a fixed RNGkind() across your tests (perhaps
via RNGversion("3.5.0")). The shinytest
package does this in its own automated tests via
a testthat helper.
app <- ShinyDriver$new("../", seed = 4323)In many cases, instead of setting the random seed, it is more elegant to use an output preprocessor.
Widgets
Tabs
With tabbed views, such as tabsetPanel() and
navbarPage(), in order for shinytest to keep track of which
tab is currently selected, they must have an id. For
example:
tabsetPanel(id = "tabs", ....)or
navbarPage(id = "tabs", ....)Uploading files
If you record a file upload event to a fileInput, you
should uncheck Run test script on exit in the recorder,
for reasons explained below. The test script will have a line like
this:
app$uploadFile(file1 = "mtcars.csv")Notice that the filename mtcars.csv appears in the script without a path. This is because the test recorder does not know where the file came from; it only knows the name of the file.
Before you run the test script, you must copy the file to the tests/ directory. See the tests/ subdirectory of this app for an example.
After copying the file to that directory, run testApp()
as usual.
Debugging test scripts
If you need to debug a test script, you can run line-by-line from the
R console. However, you likely will have modify the first line. It
normally will refer to the Shiny application the parent directory
(".."):
app <- ShinyDriver$new("..")When the test is run the usual way, with testApp(), it
will be run with the test directory as the working directory. However,
when you run the tests from the command line, you generally will have a
different working directory. To run the tests from a different
directory, you will have to pass in the path to the application. It can
be a relative path, for example:
app <- ShinyDriver$new("path/to/app")The rest of the test script can be run unchanged.
Screenshots
As you step through the script, you can inspect the state of the
application in a few different ways. One is to view a screenshot. You
should not to call app$snapshot() to get a screenshot,
because it will increment the snapshot counter and shift the numbers of
snapshots that really are part of the tests (e.g., snapshot 003 would
become 004). Instead you can do this:
app$takeScreenshot()This will display the screenshot as if it were a plot. (In RStudio, it will show in the Viewer pane.) You can inspect the screenshot to see the state of the application.
Getting input, output, and export values
It can also be useful to get the current input, output, and export
values. As with screenshots, this is something that
app$snapshot() does, but we don’t want to call that
function because increments the snapshot counter.
To fetch all values, you can call app$getAllValues().
This returns a list, which you can inspect with the str()
function. It may look something like this:
vals <- app$getAllValues()
str(vals)
#> List of 3
#> $ input :List of 4
#> ..$ action :Classes 'integer', 'shinyActionButtonValue' int 0
#> ..$ checkbox : logi TRUE
#> ..$ checkGroup: chr "1"
#> ..$ text : chr "Enter text..."
#> $ output:List of 12
#> ..$ actionOut : chr "[1] 0\nattr(,\"class\")\n[1] \"integer\" #> \"shinyActionButtonValue\""
#> ..$ checkboxOut : chr "[1] TRUE"
#> ..$ checkGroupOut: chr "[1] \"1\""
#> ..$ textOut : chr "[1] \"Enter text...\""
#> $ export: Named list()The same data is returned (invisibly) from each call to
app$setInput(), so you can also look at the return value
from those function calls to get the same result.
The values retrieved this way can be used for expectation-based testing. For example, if you are using the testthat package for testing you could do something like:
vals <- app$getAllValues()
# Another option: save values when setting input values
# vals <- app$setInputs(checkbox = TRUE)
expect_identical(vals$output$checkboxOut, "[1] TRUE")Waiting for an input (or output)
value
In most situations, when an input value is set,
app$setInputs will wait up to 3 seconds (default) for the
next value to be sent to the server. If, however, you have an
application that does not immediately send the value you are interested
in, you will need to insert at app$waitForValue function
call. This is readily useful for htmlwidgets or anything that involves
javascript code for values to be set. This method can also be used to
determine when input (or output) values have
been set after the initial output values have been set or for when
reactivity is done in two or more stages.
For example, when checking a checkbox that adds dynamic UI, such as the old faithful Shiny example, three round trips from the browser to the Shiny server will need to occur.
- Checking the checkbox
- Putting the slider and plot placeholder into the app’s UI
- Once the slider is recognized by the browser, it will send information back to Shiny causing the plot to be rendered and placed in the browser
If we were to only use app$setInput("chkbx", TRUE) and
immediately take a snapshot, we would most likely miss the correct plot
value.
To dig into this further, we can look at this situation’s example code:
app.R:
library(shiny)
ui <- fluidPage(
checkboxInput("chkbx", "Display Graph?", FALSE),
uiOutput("dynamic_output")
)
server <- function(input, output) {
output$dynamic_output <- renderUI({
if(!input$chkbx) {
return(NULL)
}
tagList(
sliderInput(inputId = "bins", label = "Number of bins:", min = 1, max = 50, value = 30),
plotOutput(outputId = "distPlot")
)
})
# will not execute until `input$bins` is available
# `input$bins` is not available until `chkbx` is checked
output$distPlot <- renderPlot({
# artificially slow the plot
Sys.sleep(5)
x <- faithful$waiting
bins <- seq(min(x), max(x), length.out = input$bins + 1)
hist(x, breaks = bins, col = "#75AADB", border = "white",
xlab = "Waiting time to next eruption (in mins)",
main = "Histogram of waiting times")
})
}
shinyApp(ui = ui, server = server)Typically we would add Sys.sleep() function calls to
slow down shinytest testing to hope that the Shiny
application is in a stable state.
./tests/mytest-sleep-bad.R:
app <- ShinyDriver$new("../")
app$snapshotInit("mytest")
# Check the box to display the slider and plot
app$setInputs('chkbx', TRUE)
# Hope that the plot appears within 7 seconds
Sys.sleep(7)
app$snapshot()
app$setInputs(bins = 40)
# Hope the newer plot appears within 7 seconds
Sys.sleep(7)
app$snapshot()The above pattern could fail if executed on a slower machine. As of
shinytest v1.4.0, we can wait for input (or
output) values using app$waitForValue. The
pattern of sleeping and hoping can now be replaced with
app$waitForValue.
./tests/mytest-waitForValue-good.R:
app <- ShinyDriver$new("../")
app$snapshotInit("mytest")
# Check the box to display the slider and plot
app$setInputs(chkbx = TRUE)
# Wait until `input$bins` is not `NULL`
app$waitForValue("bins", ignore = list(NULL))
# Wait until `output$distPlot` is not `NULL`
# (Store the retrieved plot value for later comparison)
priorPlotValue <- app$waitForValue("distPlot", iotype = "output", ignore = list(NULL))
app$snapshot()
# Change slider value to 40
app$setInputs(bins = 40, wait_ = FALSE, values_=FALSE)
# Wait until plot does not equal the plot with 30 bins
app$waitForValue("distPlot", iotype = "output", ignore = list(priorPlotValue))
app$snapshot()Note: The app$waitForValue("bins", ignore = list(NULL))
is not necessarily needed as output$distPlot depends upon
input$bins.
Dealing with dynamic data
If your application uses a data source that changes over time, then a snapshot taken yesterday may not match a snapshot taken today, even if the app itself hasn’t changed. Dynamic data inherently poses a challenge for snapshot-based testing.
This problem can be avoided by detecting when the application is being tested, and in that case use a static data set instead. To do the detection, you can do something like the following:
Limitations
Inputs without input bindings
Most input components in Shiny set their values by communicating through something called an input binding. Shinytest works well with input values that are set via input bindings.
However, there are some components that set input values without
using an input binding. These include some htmlwidgets, such as DT and plotly, as well as
Shiny’s built-in plot interactions with renderPlot().
As of shinytest 1.3.0.9001, inputs without a binding can be recorded and replayed, although with some limitations. (Prior to that version, inputs without a binding could not be recorded or replated at all.)
To enable recording of these inputs in the recorder, you must enable the option labeled Save inputs that do not have a binding. The resulting test script code will look something like this:
app$setInputs(table_rows_selected = 1, allowInputNoBinding_ = TRUE)
app$setInputs(table_row_last_clicked = 1, allowInputNoBinding_ = TRUE)Note: If you don’t enable that option, you will see this:
# Input 'table_rows_selected' was set, but doesn't have an input binding.
# Input 'table_row_last_clicked' was set, but doesn't have an input binding.When the test script is replayed, the headless browser will set these inputs to the specified values and send them to the server running Shiny in R. However, the browser will not be able to tell the input object to do the exact same behaviors. For example, with DT, when a row is selected, the mouse click event in the browser triggers the DataTable to highlight the row in the browser and also set the input value for Shiny. When shinytest plays the script, it can only do the latter part, setting the input value. The result is that when a screenshot is taken, it won’t have highlighted rows. For components that have internal state that is updated in response to user interaction, that internal state will not be updated when the test script is played. In some cases, this may mean that when the script is played, the behavior when the script is played will not be the same as when a user actually interacts with the application.
If the input component sets multiple input values (without bindings)
in response to a single user event, it may make sense to coalesce them
into a single setInputs call, such as this:
app$setInputs(table_rows_selected = 1,
table_row_last_clicked = 1,
allowInputNoBinding_ = TRUE)Next
Learn about using shinytest with continuous integration.