Compare two sampling designs on one frame

Resolves both designs on the same register, without drawing, and reports where they agree and where they part. The allocation level asks whether the two designs form the same units and pools at every stage and give each pool the same take. The probability level asks whether they give every unit the same first-order inclusion probability.

Usage

compare_designs(
  x,
  y,
  frame,
  ...,
  key,
  level = c("allocation", "probabilities"),
  atol = sqrt(.Machine$double.eps),
  rtol = 0,
  allow_approximate = FALSE
)

Arguments

x, y: Two sampling_design objects, each with a complete draw() on every stage. A design restored with read_design() is one. An executed sample is refused.
frame: The register both designs are resolved on, a data frame.
...: Must be empty. Arguments after it are matched by exact name.
key: A bare column of frame identifying the population unit. It must be unique and have no missing values.
level: What to compare, "allocation" (default) or "probabilities".
atol, rtol: Absolute and relative tolerances. Two values a and b agree when abs(a - b) <= atol + rtol * max(abs(a), abs(b)). They apply to expected pool sizes and to probabilities. Requested takes are compared exactly.
allow_approximate: Passed to exante_probabilities() at the probability level. Both designs must then have the same probability tier.

Value

An object of class samplyr_design_comparison, a list with

verdict: "agree", "differ" or "unresolved".
level: the level compared.
probability_quality: at the probability level, the tier both designs share, "exact" or "approximate", and NA otherwise. With "approximate", what agrees is the targets of the methods, which they meet only to an approximation, and the print says so.
checks: a tibble with one row per check, with stage (NA for a check of the whole design), check, status ("agree", "differ", "unresolved" or "not_run") and detail.
pools: at the allocation level, one row per pool the two designs share, with stage, pool_key, N, n_target_x, n_target_y, n_expected_x, n_expected_y and agree.
units: at the probability level, one row per register row, with the key column, probability_x, probability_y, abs_diff, rel_diff and agree.
specification: two rows per stage the designs both have, with stage, aspect ("method" or "mos"), x, y and status. The status is "same", "differ", or for two measures of size in different columns, "proportional" when one is a constant multiple of the other on every row of frame. It does not enter the verdict.

The verdict is "differ" when any check differs, "unresolved" when none differs but one could not run, and "agree" otherwise.

Details

The allocation level ignores the selection method and the measure of size. A design written with the default srswor can therefore stand for a requirement of a protocol, such as five districts in every region, and be compared with a design that draws the districts with probability proportional to size. The probability level compares the designs as they would select.

At either level, the method and the measure of size of each stage are listed side by side, outside the verdict. Two designs can agree on their probabilities with different methods, and a requirement written with the default method differs from the design that meets it by construction, so the list describes the two specifications without judging them.

Allocation

Pools are matched by the frame rows they hold, never by the names of the stratification variables. stratify_by(province, urban_rural) and stratify_by(urban_rural) under a province parent form the same pools, and cluster_by(region, province) and cluster_by(province) form the same units when province ids are nested in regions. When the units of a stage differ, its pools and takes are not compared. When the pools differ, the check names the pools of x that y splits or merges.

Two designs on the same frame see the same pools, so a comparison cannot find a category the frame lacks. A province without urban EAs forms one pool in both designs, and they agree. execute() and validate_frame() report such a parent with samplyr_message_absent_strata when the requirement names its strata in an explicit n table.

Probabilities

Units with probability zero in one design and positive in the other are a coverage difference and are counted on their own. A relative difference is computed only where both probabilities are positive. A verdict of agreement says that the first-order inclusion probabilities agree on this frame. It says nothing about joint inclusion probabilities, variances or the selection algorithm, which can differ between designs that agree.

What is not compared

A check that cannot run is reported as "unresolved" with the reason, never as agreement: designs with different numbers of stages, a stage below a with-replacement stage, methods whose probabilities are unknown, mixed probability tiers, and a stage that sets on_capped = "error" on a frame with short pools.

Examples

frame <- data.frame(
  region = rep(c("North", "South", "East"), each = 40),
  district = rep(sprintf("d%02d", 1:12), each = 10),
  size = rep(c(5, 9, 4, 7, 6, 8, 3, 9, 5, 6, 7, 4), each = 10),
  household = 1:120
)

# A protocol asks for two districts in every region. A sketch design
# states that requirement in the grammar, with the default method.
requirement <- sampling_design() |>
  stratify_by(region) |>
  cluster_by(district) |>
  draw(n = 2)

# The design as written draws districts with PPS.
written <- sampling_design() |>
  stratify_by(region) |>
  cluster_by(district) |>
  draw(n = 2, method = "pps_brewer", mos = size)
compare_designs(requirement, written, frame, key = household)
#> ── Design comparison ───────────────────────────────────────────────────────────
#> Allocations agree on this frame.
#> 
#> ✔ Stage 1 units: agree. 12 units.
#> ✔ Stage 1 pools: agree. 3 pools.
#> ✔ Stage 1 targets: agree. Every pool takes the same. x takes 6 units in all, y 6.
#> 
#> Methods and measures of size, outside the verdict:
#> ℹ Stage 1 method: srswor in x, pps_brewer in y.
#> ℹ Stage 1 measure of size: none in x, size in y.

# A design that shares four districts across the regions departs from it.
shared <- sampling_design() |>
  stratify_by(region, alloc = "equal") |>
  cluster_by(district) |>
  draw(n = 4, method = "pps_brewer", mos = size)
compare_designs(requirement, shared, frame, key = household)
#> ── Design comparison ───────────────────────────────────────────────────────────
#> Allocations differ on this frame.
#> 
#> ✔ Stage 1 units: agree. 12 units.
#> ✔ Stage 1 pools: agree. 3 pools.
#> ✖ Stage 1 targets: differ. 2 of 3 pools differ ("North", "South"). x takes 6 units in all, y 4.
#> 
#> Methods and measures of size, outside the verdict:
#> ℹ Stage 1 method: srswor in x, pps_brewer in y.
#> ℹ Stage 1 measure of size: none in x, size in y.

# A design and its own file give the same probabilities.
path <- tempfile(fileext = ".json")
write_design(written, path)
compare_designs(
  written, read_design(path), frame,
  key = household, level = "probabilities"
)
#> ── Design comparison ───────────────────────────────────────────────────────────
#> First-order inclusion probabilities agree on this frame.
#> 
#> ✔ Probabilities: agree. 120 units, 0 differ. Largest absolute difference 0, largest relative difference 0.
#> 
#> Methods and measures of size, outside the verdict:
#> ✔ Stage 1 method: pps_brewer in both.
#> ✔ Stage 1 measure of size: size in both.