Tutorial (intermediate): Randomising item administration

Overview

This tutorial addresses the common task of randomising the order in which items are administered to participants. Such techniques can be useful for avoiding order effects.

In the simplest randomisation scheme, every participant receives the same set of items, and the order of these items is randomly permuted for each participant. However, many experiments require more sophisticated randomisation schemes, for example:

• Choosing only short questions if the participant says they have a train to catch;
• Using the participant’s initials as the seed for the randomisation algorithm;
• Ensuring that each item is administered the same number of times across participants.

Simple randomisation

Suppose we wish to administer the three survey items, each with a yes or no response:

1. Are you afraid of dogs?
2. Are you afraid of birds?
3. Are you afraid of heights?

We can code these three items as a tibble:¹

library(psychTestR)
library(purrr)
library(tibble)

items <- tribble(
  ~ id,       ~ prompt,
  "dogs",    "Are you afraid of dogs?",
  "birds",   "Are you afraid of birds?",
  "heights", "Are you afraid of heights?"
)
items

## # A tibble: 3 × 2
##   id      prompt                    
##   <chr>   <chr>                     
## 1 dogs    Are you afraid of dogs?   
## 2 birds   Are you afraid of birds?  
## 3 heights Are you afraid of heights?

There are three items in total, corresponding to the three rows of items. First, let’s imagine that we want each participant to receive these three items in a random order, with this random order freshly determined for each participant. We can achieve this using the psychTestR function randomise_at_run_time.

First, we construct a list of test elements corresponding to our three questions:

item_pages <- pmap(items, function(id, prompt) {
  NAFC_page(
    label = id,
    prompt = prompt, 
    choices = c("Yes", "No")
  )
})

Note that we used the pmap function from the purrr package to iterate over the rows of items, mapping each row to a list element. We then simply pass this list of pages to randomise_at_run_time:

timeline <- join(
  randomise_at_run_time(label = "item_order",
                        logic = item_pages),
  elt_save_results_to_disk(complete = TRUE),
  final_page("You completed the test.")
)
make_test(timeline)

This code should launch the test, complete with randomisation.² Results will be saved in the output/results directory. Login to the admin panel using the password demo, download the results as a csv file, and observe how the order of test elements varies between test runs, and how the order for a given participant is saved as the variable results.item_order.

Complex randomisation

Arbitrary randomisation schemes can be implemented using the order_at_run_time function. For example, suppose that we wish to randomly choose between two item orders, 1-2-3 or 3-2-1. We can implement this as follows:

randomised <- order_at_run_time(
  label = "item_order",
  logic = item_pages,
  get_order <- function(...) {
    if (sample(2, 1) == 1) 1:3 else 3:1
  }
)

timeline <- join(
  randomised,
  elt_save_results_to_disk(complete = TRUE),
  final_page("You completed the test.")
)
make_test(timeline)

---

1. A tibble is a nicely formatted version of a data frame, provide by the tibble package and exported by the tidyverse package.↩︎

2. In RStudio, this means running the code with the ‘Run’ button, not the ‘Source’ button. You can use ‘Source’ but then you have to wrap the code in shiny::runApp.↩︎