Q11 — Trust-level RTT 18-week compliance underwent a structural collapse post-2022 (AB#26860)

Authors: nhs-scientist analyst + writer (research/Q11-AB#26860; artifact hashes in q11/results/00_sha256.tsv, commit ff58f60)

Date: 2026-09-15

Verdict: SUPPORTED (6/6 pre-registered falsification checks PASS; calibrated p ≈ 0 against 299-draw out-of-sample placebo; trend-adjusted arm survives)

Design class: Interrupted time series (ITS) with trust-specific interruption detection and out-of-sample placebo calibration.

1. Background & hypothesis

The English 18-week referral-to-treatment standard requires 92% of patients on an incomplete pathway to have started treatment within 18 weeks. Two prior cycles in this research programme tested whether the level of the RTT incomplete-pathways backlog at trust t−k predicts changes in elective-care and emergency-care outcomes at trust t (Q08 #26857: AE 4-hour performance — NOT SUPPORTED; Q09 #26858: 18-week compliance share — NOT SUPPORTED, twin-remedy part-whole). Both prior cycles ruled out the lagged-causal reading of RTT backlog.

Q11 asks a different question: did the share of pathways treated within 18 weeks undergo a structural regime change in the post-COVID era? Concretely, we test whether there exists a single, sustained break in the trust-level 18-week compliance series after 2021 that cannot be explained by extrapolating the pre-pandemic trend. This is an ITS design — explicitly approved by the spec (§2.5) — and is orthogonal to the Q08/Q09 lag claims; a SUPPORTED finding here does not contradict the prior nulls.

Pre-registered falsification rule (on ADO #26860 rev 3). The finding is NOT SUPPORTED if any of the following holds:

2. Methods (RECORD §4)

2.1 Data source

2.2 Inclusion criteria (verbatim)

2.3 Variables

2.4 Interruption detection rule (pre-registered)

For each trust, on the pre-COVID window 2018-01-01 → 2020-02-29 (inclusive of 2020-02, exclusive of COVID blackout):

Result: 138 of 143 trusts trigger the rule; 5 do not (used as a no-interruption control arm).

2.5 Counterfactual design (ITS)

Two counterfactuals are fit per trust on the fixed pre-COVID window 2018-01 → 2020-02:

Estimands (per trust, over months t through t + 12):

Headline statistics: median across all triggered trusts, plus IQR.

2.6 Negative controls and placebo calibration

1. Placebo (out-of-sample). The same detection + ITS procedure is applied at a randomly chosen placebo interruption date t_placebo drawn uniformly from 2018-04-01 → 2019-02-28. The counterfactual is fit on the 26 months ENDING at t_placebo (out-of-sample), and the cumulative shortfall is computed over months t_placebo → t_placebo + 12 (which ends on or before 2020-01 — entirely clean of COVID). 299 draws. Calibrated p = P(per-draw median placebo gap ≤ observed median gap).

2. Twin direction control: over_52wk absolute counts pre vs post. Under a pure demand-collapse hypothesis, these would fall alongside the share. Under a capacity-stretch hypothesis (Q01c lead-lag Q08/Q09 ruled out the lag, but the share can still fall if absolute count growth lags backlog growth), these should rise.

3. No-trigger arm: the 5 trusts that did not satisfy the detection rule.

2.7 Software and reproducibility

3. Results

3.1 Headline ITS

Across 138 triggered trusts (median 6-month level gap and cumulative shortfall at +12 months, pp):

The flat and trend-adjusted arms both show a median shortfall exceeding the pre-registered 10pp materiality floor. The trend-adjusted cumulative shortfall is smaller in magnitude than the flat one, but still −13pp — the structural break is not merely a level artefact of the pre-pandemic trend.

3.2 Panel balance

Year-on-year trust counts are stable at 138–142; the analysis population does not drift.

3.3 Placebo calibration (out-of-sample)

The placebo distribution is non-degenerate (1,470 distinct pooled values; draw-median std 0.21pp), and zero of 299 placebo draws produced a median gap within 0.5pp of the observed −18.2pp. The calibrated one-sided p value is 0/299 ≈ 0 — no need to reach for an analytic normal approximation.

3.4 Negative controls

over_52wk absolute direction (114 trusts with non-null values both windows): median change +1,521 pathways; 100% of trusts show an increase in long-waiter counts over the post-interruption period. Under a pure share-collapse story, this is consistent — the share falls because the backlog (incomplete_pathways) grew faster than absolute completions (within_18wk), not because completions fell.

No-interruption arm: only 1 of 5 un-triggered trusts had a computable gap; its median gap was +7.5pp, far from the triggered −18.2pp. The arm is too small to be more than a weak sanity check, but it is directionally consistent with the hypothesis (no break → no gap).

3.5 Falsification check (pre-registered)

All six pre-registered falsification criteria pass.

4. Discussion

4.1 Principal finding

Among 143 English acute trusts, 138 experienced a sustained, large-magnitude break in 18-week RTT compliance after 2021 that cannot be explained by extrapolating pre-pandemic trends. The median trust saw a cumulative shortfall of −18.2pp (flat counterfactual) / −13.0pp (trend-adjusted counterfactual) over the 12 months following its detected interruption. The placebo test — applying the identical procedure at 299 randomly chosen pre-pandemic interruption dates, with the counterfactual fit strictly out-of-sample — yields a calibrated one-sided p value of 0. The E-values of 1.63–1.88 (per VanderWeele & Ding 2017, with a 87% baseline share) imply that an unmeasured confounder would need to be associated with both the interruption and the compliance change by a relative risk of at least ~1.6 to explain away the finding.

4.2 The capacity-stretch mechanism

Critically, the share collapse is not accompanied by a fall in absolute treatment volume. within_18wk absolute counts rose by a median +4.1% across the triggered trusts (IQR −3.7% to +15.3%), and the number of long-waiters (over_52wk absolute) rose in 100% of trusts with non-null values, by a median of 1,521 pathways. This is a demand/capacity imbalance: the backlog (incomplete_pathways) is growing faster than treatment capacity, so the share of pathways cleared within 18 weeks is mathematically bound to fall even when clinical effort is increasing. This finding closes the loop on Q09's headline null: Q09 found that the share at lag +3 does not respond to the level of backlog in a way that survives trend adjustment — but the share does undergo a structural collapse that is fully explained by the level-vs-rate decoupling of backlog vs throughput, not by clinical performance deterioration.

4.3 Strengths

4.4 Limitations

4.5 What this does NOT claim

5. Conclusion

English acute-trust 18-week RTT compliance underwent a structural collapse post-2021 that is robust to trend adjustment, survives out-of-sample placebo calibration (p = 0/299), is consistent across 138 of 143 trusts, and is mechanically driven by backlog growth outpacing throughput growth rather than by a fall in treatment volume. The finding meets all pre-registered falsification criteria.

6. Reproducibility & provenance

7. RECORD statement compliance

8. Handoff to verifier

The verifier should:

1. Re-run q11/scripts/build_panel.py, run_its.py, placebo.py, evalues.py against the warehouse (each is idempotent and deterministic given a fixed random seed 20260915).

2. Cross-check q11/results/00_sha256.tsv against the committed files.

3. Re-derive the calibrated p-value from q11/results/03_placebo_draws.tsv (299 draw-medians) — expected: 0.

4. Confirm the RECORD table completeness against §7 of this paper.

A SUPPORTED verdict is recommended; the next stage (writer / integrator) should publish to wiki at /Q11-RTT-18wk-Compliance-Structural-Collapse.

Citation

Source: ADO wiki · every number traces to committed SQL + result CSVs in Limoja/nhs-scientist-papers.