Power analysis for difference-in-differences designs

Compute sample size, power, or minimum detectable effect (MDE) for a two-group, two-period difference-in-differences (DiD) contrast.

Usage

power_did(treat, ...)

# Default S3 method
power_did(
  treat,
  control,
  ...,
  outcome = c("mean", "prop"),
  var = NULL,
  sd = NULL,
  effect = NULL,
  n = NULL,
  power = 0.8,
  alpha = 0.05,
  N = Inf,
  deff = 1,
  resp_rate = 1,
  alternative = c("two.sided", "one.sided"),
  ratio = 1,
  overlap = 0,
  overlap_cor = 0,
  plan = NULL
)

Arguments

treat: Numeric length-2 vector for treated group outcomes: c(baseline, endline).
...: Additional arguments passed to methods. Unused arguments are rejected.
control: Numeric length-2 vector for control group outcomes: c(baseline, endline).
outcome: Outcome scale: "mean" (default) or "prop".
var: Outcome variance. Applies to outcome = "mean" only. Under outcome = "prop" the cell variances follow from treat and control, so supplying it is an error rather than a silent no-op. Length 1: common variance for all four cells. Length 2: group-specific variances c(var_treat, var_control), assumed equal across waves. Length 4: cell-specific variances in order c(var_treat_baseline, var_treat_endline, var_control_baseline, var_control_endline).
sd: Outcome standard deviation, an alternative spelling of var taking the same lengths. Supply at most one of var or sd.
effect: Absolute DiD effect size to detect (> 0). Defaults to the contrast implied by treat and control, |(treat[2] - treat[1]) - (control[2] - control[1])|, so it only needs to be supplied to plan for an effect other than the one those paths describe. Supplying a value that disagrees with them warns. Leave NULL with both n and power given to solve for the MDE.
n: Per-arm sample size per wave. Scalar (equal treated/control) or length-2 vector c(n_treat, n_control). Leave NULL to solve for n.
power: Target power, in (0, 1). When solving for sample size or MDE, it must exceed alpha, the power at zero effect. Leave NULL to solve for power.
alpha: Significance level, default 0.05.
N: Population size for finite-population correction. A scalar applies to both arms. Use a length-2 vector c(N_treat, N_control) for arm-specific population sizes. Inf (default) disables FPC.
deff: Design effect multiplier (> 0).
resp_rate: Expected response rate, in (0, 1]. Default 1 (no adjustment). The required sample size is inflated by 1 / resp_rate.
alternative: Character: "two.sided" (default) or "one.sided". Use "two.sided" when the intervention could plausibly move the outcome in either direction (the usual default). Use "one.sided" when only one direction is of interest, which requires a smaller sample for the same power.
ratio: Allocation ratio n_treat / n_control (default 1). Used only when solving for n (n = NULL).
overlap: Panel overlap fraction in [0, 1] within each arm across baseline and endline.
overlap_cor: Correlation between baseline and endline outcomes within overlapping units, in [0, 1].
plan: Optional svyplan() object providing design defaults.

Value

A svyplan_power object with components:

n: Per-arm sample size (scalar or length-2).
power: Achieved power.
effect: DiD effect size.
type: "did_prop" or "did_mean".
solved: Which quantity was solved for ("n", "power", or "mde").
params: List of input parameters.

Details

treat and control already determine the contrast, so effect is optional: leave n or power as NULL to solve for it. Leaving both n and power supplied while effect is NULL solves for the MDE instead. Exactly one quantity must remain unknown.

The DiD estimand is (treat_endline - treat_baseline) - (control_endline - control_baseline).

Per-arm variance of the before-after change accounts for panel overlap:

$$V_{\text{arm}} = V_{\text{pre}} + V_{\text{post}} - 2 \cdot \text{overlap} \cdot \rho \cdot \sqrt{V_{\text{pre}} \cdot V_{\text{post}}}$$

A finite arm population does not scale that quantity by an ordinary $1 - n/N$. The overlap covariance carries a single $1/N$ rather than the arm's marginal factor, so each arm splits into a per-unit part and a census part that are different expressions whenever the overlap is partial:

$$V_{\text{arm}}^{\text{census}} = V_{\text{pre}} + V_{\text{post}} - 2 \rho \sqrt{V_{\text{pre}} \cdot V_{\text{post}}}$$

the same expression with the overlap fraction set to one. The DiD test statistic variance is then

$$V_d = \text{deff} \sum_{a \in \{t, c\}} \left( \frac{V_{a}}{n_{a}} - \frac{V_{a}^{\text{census}}}{N_{a}} \right)$$

with the census part dropped for an infinite $N_a$. The two expressions coincide only at the extremes. At overlap = 1 the census part equals the per-unit numerator and the whole collapses to $V_a(1/n_a - 1/N_a)$. At overlap = 0 the two occasions are independent populations, the census part loses its cross term as well, and the result reduces to the classical flat-variance formula. At a partial overlap it is neither, which is why the correction is written as a subtracted census term rather than as a factor on $V_a$.

What `treat` and `control` are used for

Both paths always supply the contrast. Whether they also supply the variance depends on outcome: for "prop" the four cell variances are $p(1-p)$ at each path value, multiplied by $N/(N-1)$ for a finite arm population, while for "mean" the variance comes entirely from var and the paths are used for the contrast alone.

Because the contrast is derived, effect is redundant with the paths and is only needed to plan for a different effect than they describe, for example a conservative target below the change a pilot observed. Doing so warns, since the printed result then shows an effect that the displayed paths do not produce.

Normal approximation

Critical values and power come from the standard normal, with the variance treated as known and no degrees-of-freedom correction, as in power_mean() and power_prop(). The cluster count, not the unit count, is what governs the accuracy of that approximation for a clustered DiD design: deff inflates the variance but does not model the loss of degrees of freedom, so a design with few clusters per arm is optimistic here by more than the unit count suggests.

The df argument that n_prop(), n_mean() and n_alloc() accept has no counterpart here, and its absence is a decision rather than an omission. There the quantile is the half-width of a confidence interval and a t quantile substitutes for a normal one directly. Here it is a normal deviate for an alternative, and a t-based power calculation is a different procedure. Passing df is an error that says so.

References

Valliant, R., Dever, J. A., & Kreuter, F. (2018). Practical Tools for Designing and Weighting Survey Samples (2nd ed.). Springer. Chapter 4.

Examples

# DiD sample size for means. The effect is the contrast the two paths
# describe: (55 - 50) - (52 - 50) = 3.
power_did(
  treat = c(50, 55), control = c(50, 52),
  outcome = "mean", var = 100
)
#> Power analysis for DiD means (solved for sample size)
#> n = 349 (per group), power = 0.800, effect = 3.0000
#> (treat = (50.000, 55.000), control = (50.000, 52.000), alpha = 0.05, deff = 1, var = (100.00, 100.00, 100.00, 100.00))

# DiD power for proportions
power_did(
  treat = c(0.30, 0.36), control = c(0.30, 0.33),
  outcome = "prop", effect = 0.03, n = 800, power = NULL
)
#> Power analysis for DiD proportions (solved for power)
#> n = 800 (per group), power = 0.149, effect = 0.0300
#> (treat = (0.300, 0.360), control = (0.300, 0.330), alpha = 0.05, deff = 1)

# MDE for means with n = 500
power_did(
  treat = c(50, 55), control = c(50, 52),
  outcome = "mean", var = 100, effect = NULL, n = 500
)
#> Power analysis for DiD means (solved for minimum detectable effect)
#> n = 500 (per group), power = 0.800, effect = 2.5058
#> (treat = (50.000, 55.000), control = (50.000, 52.000), alpha = 0.05, deff = 1, var = (100.00, 100.00, 100.00, 100.00))

# Panel overlap reduces required n
power_did(
  treat = c(0.50, 0.55), control = c(0.50, 0.48),
  outcome = "prop", effect = 0.07, overlap = 0.5, overlap_cor = 0.6
)
#> Power analysis for DiD proportions (solved for sample size)
#> n = 1119 (per group), power = 0.800, effect = 0.0700
#> (treat = (0.500, 0.550), control = (0.500, 0.480), alpha = 0.05, deff = 1, overlap = 0.50, overlap_cor = 0.60)