Create a tbl_now object
tbl_now.Rd![[Experimental]](figures/lifecycle-experimental.svg)
Surveillance data arrives late. A case that happened on Monday may only reach the surveillance system on Thursday, so counts for the most recent days always look artificially low. Nowcasting corrects that artifact: it estimates how many cases have already happened but have not been reported yet.
To do that, a model needs two dates for every case – when it happened
(event_date) and when it was reported (report_date) – together with
the date you are standing on (now). tbl_now() takes an ordinary
data.frame and records which of its columns play those roles, so you only
have to say it once.
The result still behaves like a tibble: dplyr verbs, $, [ and
ggplot2 keep working, and every tbl.now function knows where to find the
dates without being told again.
Usage
tbl_now(
data,
event_date = NULL,
report_date = NULL,
delay = NULL,
strata = NULL,
covariates = NULL,
case_count = NULL,
is_censored_report = NULL,
validation_date = NULL,
validation_type = NULL,
validation_units = units,
validation_levels = NULL,
is_censored_validation = NULL,
now = NULL,
event_units = units,
report_units = units,
units = "auto",
data_type = "auto",
t_effects = character(0),
verbose = TRUE,
force = FALSE,
warn_non_uniqueness = TRUE,
align_weeks = FALSE,
...
)Arguments
- data
A
data.frameortibbleto be converted.- event_date
tidy-select name of the column containing the event date. Optional when
delayis provided together withreport_date; the event date will be computed asreport_date - delay.- report_date
tidy-select name of the column containing the report date. Optional when
delayis provided together withevent_date; the report date will be computed asevent_date + delay.- delay
(optional) tidy-select or
NULL(default). Name of a numeric column containing the delay (inevent_units) betweenevent_dateandreport_date. When provided with only one ofevent_dateorreport_date, the missing date is reconstructed from the known date and the delay. Requires units to be known (either specified viaevent_unitsor inferrable from the provided date column). Every value must be a whole number of those units: a fractional delay is an error, not a rounding, because a calendar has no half-days and bending the value silently moves the reconstructed date.- strata
(optional) tidy-select or
NULL(default). Name of different variables (column names) in strata. Strata correspond to variables that are of interest by themselves. For example if it is of interest to generate nowcasts by gender thengenderis astrata.- covariates
(optional) tidy-select or
NULL(default). Name of different variables (column names) that influence the nowcast but are not strata. For example precipitation might influence a dengue nowcast but in general it is not of interest to generate nowcasts by precipitation levels.- case_count
(optional) tidy-select or
NULLName of the column with the case counts ifdata_typeis "count-incidence" or "count-cumulative".- is_censored_report
(optional) tidy-select or
NULL(default). The name of a column containing eitherTRUEorFALSEindicating whether thereport_dateis correctly specified or corresponds to abatchand thus is censored. In other words, if thereport_dateis accurately measured setis_censored_report = FALSEbut if thereport_datecorresponds to an error and is only an upper bound of the real report date setis_censored_report = TRUE.- validation_date
(optional) tidy-select column holding a third date: the day the report was resolved. Influenza is the picture to keep in mind – symptoms begin (the event), the patient sees a doctor (the report), and days later a swab comes back. The assumed timeline is
event_date <= report_date <= validation_date <= now. LeaveNULL(the default) for the usual two-date object. Seeadd_validation_date().- validation_type
(optional) tidy-select column saying what the resolution was:
"confirmed","retracted"(it was reported, but it is not a case after all),"pending"orNA."pending"means reported and still waiting, so it carries no validation date – which is a different thing from a result that was never recorded (NA). A validation date with no type warns rather than guessing, because a date alone cannot say whether the case was confirmed or retracted.- validation_units
(optional) Character. Either
"auto"(default),"days","weeks","months","years"or"numeric"– the grid the validation date lives on, resolved the same way asreport_units.- validation_levels
(optional)
NULL(default) or a named character vector translating the labels invalidation_typeinto the canonical outcomes, for data that was not recorded in English:c(confirmado = "confirmed", retractado = "retracted", pendiente = "pending"). The names are the labels in your data, the values are the canonical ones. The column is rewritten to the canonical values and the dictionary is kept as an attribute, readable withget_validation_levels(). Only"confirmed","retracted","pending"andNAare ever stored.- is_censored_validation
(optional) tidy-select or
NULL(default). The validation-axis counterpart ofis_censored_report: the name of a logical column marking rows whose validation delay is a bound rather than a measurement. Requires avalidation_date. See censor_validation_delays_above().- now
(optional) Date or
NULL(default). The date that is considered thenowof the nowcast. If nonowis given then the function automatically uses the lastevent_date.- event_units
(optional) Character. Either "auto" (default), "days", "weeks", "months", "years" or "numeric". Defaults to
units.- report_units
(optional) Character. Either "auto" (default), "days", "weeks", "months", "years" or "numeric". Defaults to
units.- units
(optional) Character. Either
"auto"(default),"days","weeks","months","years"or"numeric". The default forevent_units,report_unitsandvalidation_units: say it once instead of three times. Any of the three that you give explicitly wins overunits, sounits = "days", report_units = "weeks"reads a daily event date against a weekly report date.- data_type
(optional) Character. Either "auto", "linelist" or "count-incidence" or "count-cumulative". See section below for an explanation on data types.
- t_effects
(optional) Either
NULL(default), atemporal_effects()object or a character vector with the names of the columns containing the temporal effects.- verbose
(optional) Logical. Whether to throw a message. Default =
TRUE.- force
(optional) Logical. Whether to force computation overwriting pre-existing variables. Default =
FALSE.- warn_non_uniqueness
(optional) Logical. Whether to throw a warning if data has multiple observations for same event and report date (conditional on covariates and strata)
- align_weeks
(optional) Logical. If both event and report units are weeks and
align_weeks = TRUEit ensures that all weeks start in a Sunday so that week differences and.delaysare all integer.- ...
Additional metadata to be stored as attributes on the object. Use this for provenance you want to travel with the data –
data_source,citation,population– and read it back withtbl_now_attributes().Because anything unmatched lands here, a misspelled argument name would otherwise be accepted in silence. Names close enough to a real argument to be a typo (
case_colforcase_count,stataforstrata) warn instead; the warning is safe to ignore if the name really was metadata.
Value
An object of class tbl_now: the input data as a tibble, carrying
extra attributes that record which columns hold the event date, report date,
strata, covariates and so on, plus the now of the nowcast. List them with
tbl_now_attributes().
Details
The minimum you must supply is event_date and report_date (or one of them
plus a delay column, from which the other is reconstructed). Everything else
is optional and can be added later with add_strata(), add_covariates(),
add_validation_date() and the rest of the add() family.
Once the object exists the usual path is summary() to see
what is in the data, diagnose() to see what is wrong with it,
autoplot() to look at it, and run_nowcast() to fit a
model. vignette("tbl.now") walks through that path end to end.
Attributes
The following attributes are part of a tbl_now and are validated
by the validate_tbl_now() function:
- event_date
Name of the column refering to the event of interest.
- report_date
Name of the column refering to when the event of interest was reported.
- strata
Names of the columns corresponding to the strata (for modelling).
- covariates
Names of the columns corresponding to covariates (for modelling).
- case_count
Column containing the number of observations for that moment if
data_typeiscount-incidenceorcount-cumulative.- temporal_effects
Names of the columns refering to the temporal effects.
- now
Date of the
nowfor a nowcast.- is_censored_report
Column indicating whether the measurement is noisy (only upper bound) or not.
- event_units
Either
days,weeks,months,yearsornumeric. Corresponds to the units ofevent_date- report_units
Either
days,weeks,months,yearsornumeric. Corresponds to the units ofreport_date- data_type
Either
linelist,count-incidenceorcount-cumulativedepending on whether it is linelist data or count data with incidence (each report date's incidence) or cumulative (overall known cases at report date)- validation_date
Name of the column with the (optional) third date: when the report was resolved.
- validation_type
Name of the column saying what that resolution was (
"confirmed","retracted","pending").- validation_units
Units of
validation_date, resolved likereport_units.- validation_levels
The (optional) dictionary translating the labels in
validation_typeinto the canonical outcomes.- is_censored_validation
Column indicating whether the validation delay is only a bound (the validation-axis counterpart of
is_censored_report).- computed_temporal_effect_cols
Names of the temporal-effect columns that have actually been materialised in the data by
compute_temporal_effects().
You can list all tbl_now related attributes in a specific tbl_now with tbl_now_attributes().
Data types
The following data-types are admitted at tbl_now objects.
Linelist
Each row is an individual that was reported at report_date as happening at event_date.
df <- data.frame(
patient = 1:6,
event_date = c(rep(as.Date("2020/09/12"), 3),
rep(as.Date("2020/09/13"), 3)),
report_date = c(as.Date("2020/09/12"),
as.Date("2020/09/13"),
as.Date("2020/09/14"),
as.Date("2020/09/13"),
as.Date("2020/09/14"),
as.Date("2020/09/15")))
print(df)
#> patient event_date report_date
#> 1 1 2020-09-12 2020-09-12
#> 2 2 2020-09-12 2020-09-13
#> 3 3 2020-09-12 2020-09-14
#> 4 4 2020-09-13 2020-09-13
#> 5 5 2020-09-13 2020-09-14
#> 6 6 2020-09-13 2020-09-15Count-incidence
Each report_date-event_date combination contains the total number of
cases observed exactly at report_date for event_date.
df <- data.frame(
n = c(7, 1, 9, 5, 0, 2),
event_date = c(rep(as.Date("2020/09/12"), 3),
rep(as.Date("2020/09/13"), 3)),
report_date = c(as.Date("2020/09/12"),
as.Date("2020/09/13"),
as.Date("2020/09/14"),
as.Date("2020/09/13"),
as.Date("2020/09/14"),
as.Date("2020/09/15")))
print(df)
#> n event_date report_date
#> 1 7 2020-09-12 2020-09-12
#> 2 1 2020-09-12 2020-09-13
#> 3 9 2020-09-12 2020-09-14
#> 4 5 2020-09-13 2020-09-13
#> 5 0 2020-09-13 2020-09-14
#> 6 2 2020-09-13 2020-09-15Count-cumulative
Each report_date-event_date combination contains the total number of
cases observed up until report_date for event_date. The most recent
report_date contains the best estimation of cases happening at event_date.
df <- data.frame(
n = c(1,5, 8, 2, 2, 4),
event_date = c(rep(as.Date("2020/09/12"), 3),
rep(as.Date("2020/09/13"), 3)),
report_date = c(as.Date("2020/09/12"),
as.Date("2020/09/13"),
as.Date("2020/09/14"),
as.Date("2020/09/13"),
as.Date("2020/09/14"),
as.Date("2020/09/15")))
print(df)
#> n event_date report_date
#> 1 1 2020-09-12 2020-09-12
#> 2 5 2020-09-12 2020-09-13
#> 3 8 2020-09-12 2020-09-14
#> 4 2 2020-09-13 2020-09-13
#> 5 2 2020-09-13 2020-09-14
#> 6 4 2020-09-13 2020-09-15The to_count() function allows you to easily convert from between different
data-types.
See also
as_tbl_now() to convert an object created by another nowcasting package;
to_count() to move between linelist and aggregated count data;
tbl_now_attributes() to list what the object recorded;
validate_tbl_now() and diagnose() to check it;
summary() to describe it;
autoplot() to plot it;
run_nowcast() to fit a nowcast.
Examples
# `denguedat` is a linelist: one row per dengue case, with the week symptoms
## began (`onset_week`) and the week the case reached the surveillance system
## (`report_week`).
data(denguedat)
head(denguedat)
#> onset_week report_week gender
#> 1 1990-01-01 1990-01-01 Male
#> 2 1990-01-01 1990-01-01 Female
#> 3 1990-01-01 1990-01-01 Female
#> 4 1990-01-01 1990-01-08 Female
#> 5 1990-01-01 1990-01-08 Male
#> 6 1990-01-01 1990-01-15 Female
# Tell tbl.now which column plays which role. `now` defaults to the last
# event date seen in the data.
ndata <- tbl_now(denguedat,
event_date = onset_week,
report_date = report_week,
strata = gender
)
#> ℹ Identified data as <linelist-data> where each observation is a test.
# Printing reports back the roles it recorded, and the `now` it chose.
ndata
#> # A tibble: 52,987 × 6
#> # Data type: "linelist"
#> # Frequency: Event: `weeks` | Report: `weeks`
#> onset_week report_week gender .event_num .report_num .delay
#> <date> <date> <chr> <dbl> <dbl> <dbl>
#> [event_date] [report_date] [strata] [...] [...] [...]
#> 1 1990-01-01 1990-01-01 Male 0 0 0
#> 2 1990-01-01 1990-01-01 Female 0 0 0
#> 3 1990-01-01 1990-01-01 Female 0 0 0
#> 4 1990-01-01 1990-01-08 Female 0 1 1
#> 5 1990-01-01 1990-01-08 Male 0 1 1
#> 6 1990-01-01 1990-01-15 Female 0 2 2
#> 7 1990-01-01 1990-01-15 Female 0 2 2
#> 8 1990-01-01 1990-01-15 Female 0 2 2
#> 9 1990-01-01 1990-01-22 Female 0 3 3
#> 10 1990-01-01 1990-01-08 Female 0 1 1
#> # ────────────────────────────────────────────────────────────────────────────────
#> # Now: 2010-12-20 | Event date: "onset_week" | Report date: "report_week"
#> # Strata: "gender"
#> # ────────────────────────────────────────────────────────────────────────────────
#> # ℹ 52,977 more rows
# A `tbl_now` is still a tibble, so ordinary manipulation works ...
ndata$newcolumn <- "something"
ndata[1:10, ]
#> # A tibble: 10 × 7
#> # Data type: "linelist"
#> # Frequency: Event: `weeks` | Report: `weeks`
#> onset_week report_week gender .event_num .report_num .delay newcolumn
#> <date> <date> <chr> <dbl> <dbl> <dbl> <chr>
#> [event_date] [report_date] [strata] [...] [...] [...] [...]
#> 1 1990-01-01 1990-01-01 Male 0 0 0 something
#> 2 1990-01-01 1990-01-01 Female 0 0 0 something
#> 3 1990-01-01 1990-01-01 Female 0 0 0 something
#> 4 1990-01-01 1990-01-08 Female 0 1 1 something
#> 5 1990-01-01 1990-01-08 Male 0 1 1 something
#> 6 1990-01-01 1990-01-15 Female 0 2 2 something
#> 7 1990-01-01 1990-01-15 Female 0 2 2 something
#> 8 1990-01-01 1990-01-15 Female 0 2 2 something
#> 9 1990-01-01 1990-01-22 Female 0 3 3 something
#> 10 1990-01-01 1990-01-08 Female 0 1 1 something
#> # ────────────────────────────────────────────────────────────────────────────────
#> # Now: 2010-12-20 | Event date: "onset_week" | Report date: "report_week"
#> # Strata: "gender"
#> # ────────────────────────────────────────────────────────────────────────────────
# ... including dplyr verbs.
ndata |>
dplyr::filter(report_week <= as.Date("1991-01-02"))
#> # A tibble: 1,981 × 7
#> # Data type: "linelist"
#> # Frequency: Event: `weeks` | Report: `weeks`
#> onset_week report_week gender .event_num .report_num .delay newcolumn
#> <date> <date> <chr> <dbl> <dbl> <dbl> <chr>
#> [event_date] [report_date] [strata] [...] [...] [...] [...]
#> 1 1990-01-01 1990-01-01 Male 0 0 0 something
#> 2 1990-01-01 1990-01-01 Female 0 0 0 something
#> 3 1990-01-01 1990-01-01 Female 0 0 0 something
#> 4 1990-01-01 1990-01-08 Female 0 1 1 something
#> 5 1990-01-01 1990-01-08 Male 0 1 1 something
#> 6 1990-01-01 1990-01-15 Female 0 2 2 something
#> 7 1990-01-01 1990-01-15 Female 0 2 2 something
#> 8 1990-01-01 1990-01-15 Female 0 2 2 something
#> 9 1990-01-01 1990-01-22 Female 0 3 3 something
#> 10 1990-01-01 1990-01-08 Female 0 1 1 something
#> # ────────────────────────────────────────────────────────────────────────────────
#> # Now: 2010-12-20 | Event date: "onset_week" | Report date: "report_week"
#> # Strata: "gender"
#> # ────────────────────────────────────────────────────────────────────────────────
#> # ℹ 1,971 more rows
# Dropping a strata column simply forgets that stratum.
ndata |> dplyr::select(-gender)
#> # A tibble: 52,987 × 6
#> # Data type: "linelist"
#> # Frequency: Event: `weeks` | Report: `weeks`
#> onset_week report_week .event_num .report_num .delay newcolumn
#> <date> <date> <dbl> <dbl> <dbl> <chr>
#> [event_date] [report_date] [...] [...] [...] [...]
#> 1 1990-01-01 1990-01-01 0 0 0 something
#> 2 1990-01-01 1990-01-01 0 0 0 something
#> 3 1990-01-01 1990-01-01 0 0 0 something
#> 4 1990-01-01 1990-01-08 0 1 1 something
#> 5 1990-01-01 1990-01-08 0 1 1 something
#> 6 1990-01-01 1990-01-15 0 2 2 something
#> 7 1990-01-01 1990-01-15 0 2 2 something
#> 8 1990-01-01 1990-01-15 0 2 2 something
#> 9 1990-01-01 1990-01-22 0 3 3 something
#> 10 1990-01-01 1990-01-08 0 1 1 something
#> # ────────────────────────────────────────────────────────────────────────────────
#> # Now: 2010-12-20 | Event date: "onset_week" | Report date: "report_week"
#> # ────────────────────────────────────────────────────────────────────────────────
#> # ℹ 52,977 more rows
# But dropping a column the class depends on demotes the object back to a
## plain tibble (with a warning): without an event date it can no longer
# describe a nowcast.
suppressWarnings(
ndata |> dplyr::select(-onset_week)
)
#> # A tibble: 52,987 × 6
#> report_week gender .event_num .report_num .delay newcolumn
#> <date> <chr> <dbl> <dbl> <dbl> <chr>
#> 1 1990-01-01 Male 0 0 0 something
#> 2 1990-01-01 Female 0 0 0 something
#> 3 1990-01-01 Female 0 0 0 something
#> 4 1990-01-08 Female 0 1 1 something
#> 5 1990-01-08 Male 0 1 1 something
#> 6 1990-01-15 Female 0 2 2 something
#> 7 1990-01-15 Female 0 2 2 something
#> 8 1990-01-15 Female 0 2 2 something
#> 9 1990-01-22 Female 0 3 3 something
#> 10 1990-01-08 Female 0 1 1 something
#> # ℹ 52,977 more rows