Skip to contents

Backtest

Source: R/backtest.R
backtest.Rd

Compute historical nowcasts for given time period

Usage

backtest(
  ncast,
  start_date = infer_start_date(ncast),
  end_date = infer_end_date(ncast, start_date = start_date),
  stride = 1,
  subsample = NULL,
  dates_to_test = generate_nowcast_dates(start_date, end_date, ncast, stride, subsample =
    subsample),
  retrain = 1,
  quantiles = c(0.025, 0.05, 0.25, 0.5, 0.75, 0.95, 0.975),
  min_horizon = 0,
  model_name = NULL,
  refresh = 250 * rlang::is_interactive(),
  ...
)

Arguments

ncast

A nowcaster object generated by nowcast()

start_date

An object of datatype Date indicating the date at which to start the historical nowcasting (first 'now' date to nowcast).

end_date

An object of datatype Date indicating the date at which to end the historical nowcasting (last 'now' date to nowcast).

stride

Integer variable indicating the number of time steps between two consecutive nowcasts. Default 1 means nowcasting for each date between start and end,

subsample

Either NULL or an integer specifying how large of a subsample from the dates_to_test to use for the backtest. If NULL (default) then it uses all of the dates.

dates_to_test

Specific values of moments now to test the nowcast. If dates_to_test is given this overrides start_date, end_date and stride.

retrain

Integer variable indicating the number of iterations for which to retrain the model. Default 1 means retraining the model for each nowcast. NOT IMPLEMENTED YET - currently will be retrained at each time step.

quantiles

list of quantiles between 0 and 1 defining which quantile estimates will be returned

min_horizon

the minimum horizon (value <= 0) to keep from each nowcast estimates (e.g. min_horizon=-5 means estimates of up to previous 5 time steps from now will be kept)

model_name

A model name that will be used to identify the results of the backtest with the given parameters in a subsequent call to backtest_metrics()

refresh

Refresh parameter for rstan::sampling()

...

Additional arguments to pass to update.nowcaster()

Value

A tibble with as many rows as dates_to_test are given. Each row represents a different now with the following columns:

  • now: The date nowcasted

  • true_date: The column corresponding to the true_date (has the true_date's name).

  • mean: The mean estimate of the nowcast.

  • sd: The standard deviation estimate of the nowcast.

  • median: The median estimate of the nowcast.

  • x%: Quantile columns corresponding to the estimated xth quantile.

  • Strata_unified: A column containing the strata modeled.

  • observed: The value actually obnserved for that date

  • model: The name of the fitted model

Examples

# Load the data
data(denguedat)

# Run a nowcast with very few iterations
# change to method = "sampling" when working and remove the iter = 10 (or set to iter = 2000)
now <- as.Date("1990-10-01")
ncast <- nowcast(denguedat, "onset_week", "report_week", now = now,
  method = "optimization", seed = 2495624, iter = 10)

# Run a backtest for the model checking the model fit for two dates:
btest <- backtest(ncast, dates_to_test = c(as.Date("1990-06-11"), as.Date("1990-06-18")))

#The following examples are slow:
if (FALSE) { # \dontrun{
# Run a backtest for the model automatically selecting 3 possible dates at random
btest2 <- backtest(ncast, subsample = 3)

# Run a backtest for the model using all possible dates
btest3 <- backtest(ncast)
} # }