Power Analysis and Sample Size

Why simulation is needed

In a \(t\)-test, power is determined by effect size, \(n\), and \(\alpha\): a closed-form solution exists.

In a multilevel design, power depends on:

• the fixed effect of interest
• variance components at each level
• number of groups
• number of observations per group
• correlation structure of the random effects

No single formula captures all of this.

The simulation approach

1. Specify a hypothetical model with the planned design and expected parameters.
2. Simulate many datasets from this model.
3. For each dataset, fit the model and test the effect of interest.
4. Estimate power as the proportion of significant tests.

The simr package implements this workflow.

Key functions

makeLmer() constructs a hypothetical lmer model with specified parameters.

powerSim() estimates power by simulation.

extend() modifies the sample size to explore the power curve.

Specifying the hypothetical model

Use the sleep deprivation study as a template. Plan a new study with 10 days of measurement: what is the power to detect a slope of 10 ms/day?

fake_data <- expand_grid(subject = 1:10, days = 0:9)

fixed_b <- c(250, 10)        # intercept, slope
V <- matrix(c(625, 0, 0, 25), nrow = 2)  # tau_0=25, tau_1=5, rho=0
s <- 25                       # residual SD

Constructing the model object

fake_model <- makeLmer(
  y ~ days + (days | subject),
  data    = fake_data,
  fixef   = fixed_b,
  VarCorr = V,
  sigma   = s
)
summary(fake_model)$coefficients

Estimating power

powerSim(fake_model, test = fixed("days"), nsim = 1000)

The output reports estimated power with a 95% confidence interval. Use nsim = 100 for exploration; nsim = 1000 or more for a reportable estimate.

Varying sample size

Scale up the sample to find the \(n\) needed for target power (80% or 90%).

fake_20 <- extend(fake_model, along = "subject", n = 20)
powerSim(fake_20, test = fixed("days"), nsim = 1000)

fake_30 <- extend(fake_model, along = "subject", n = 30)
powerSim(fake_30, test = fixed("days"), nsim = 1000)

Running this across a range of sample sizes and plotting the results gives a power curve.

Two-condition design

Add a between-subjects condition to the model.

fake_data2 <- expand_grid(subject = 1:10, days = 0:9, x = c("A", "B"))
fixed_b2   <- c(250, 10, 5, 2)  # intercept, days, xB, days:xB

fake_model2 <- makeLmer(
  y ~ x * days + (days | subject),
  data    = fake_data2,
  fixef   = fixed_b2,
  VarCorr = V,
  sigma   = s
)

The effect of primary interest is the interaction xB:days (differential slowing by condition).

Sensitivity to variance components

Power in multilevel designs is sensitive to the variance components as well as the fixed effects.

If between-subject variability in slopes (\(\tau_1\)) is large, the fixed slope is harder to detect. Explore this by varying \(\tau_1\) in V and re-running powerSim.

This sensitivity analysis reveals which assumptions drive the power estimate.

Practical advice

Key inputs to a power analysis:

1. The minimum effect size that would be scientifically meaningful.
2. The variance components, from pilot data or prior literature.
3. The planned design: number of groups and observations per group.

Power is often more sensitive to the number of groups than to the number of observations per group. The degrees of freedom for testing fixed effects depend primarily on the number of independent units at the highest relevant level.