FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better40 / 280 · beu-06c5c076c9404dba

A Practical Introduction to Regression Discontinuity Designs: Extensions

Statistics · 2301.08958v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Practical Introduction to Regression Discontinuity Designs: Extensions

1 Introduction

…rence, graphical presentation, and falsification for various RD design settings. In Foundations , we focused exclusively on the canonical Sharp RD design, presenting a practical discussion of the methods developed by Hahn, Todd, and van der Klaauw (2001) , Lee (2008) , McCrary (2008) , Calonico, Cattaneo, and Titiunik (2014b) ; Calonico, Cattaneo, and Titiunik (2015a) , Calonico, Cattaneo, and Farrell (2018) ; Calonico, Cattaneo, and Farrell (2020) ; Calonico, Cattaneo, and Farrell (2022) , Calonico, Cattaneo, Farrell, and Titiunik (2019) , and Cattaneo, Jansson, and Ma (2020) , among others.

…y-based approach in other settings. Second, employing both continuity-based and local randomization approaches,

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs attempt to continue a sentence about extending RD designs beyond the canonical Sharp RD. The draft ends with 'employing both continuity-based and local randomization approaches,' which grammatically requires a continuation (the sentence is incomplete with a hanging comma). However, both outputs provide completion actions with text that would follow logically. Output A: 'we extend the canonical Sharp RD design to a variety of settings that relax its three defining features.' - This is somewhat abrupt. The draft mentions 'continuity-based and local randomization approaches,' but Output A jumps to 'the canonical Sharp RD design' and its 'three defining features' without connecting to the dual approaches just mentioned. The source mentions three features of the canonical design (continuous score, one cutoff, perfect compliance), but this feels like a non-sequitur from the immediate context. Output B: 'we illustrate how the core ideas of the canonical Sharp RD design extend to more complex settings.' - This flows better. 'Core ideas' is more general and encompasses both 'continuity-based and local randomization approaches' mentioned in the draft. 'More complex settings' naturally follows from discussing multiple approaches. The verb 'illustrate' also fits an academic paper's purpose better than the bald 'extend' in Output A. Output B maintains better coherence with the immediate antecedent text by keeping the focus on how the dual approaches apply broadly, rather than abruptly introducing a specific three-feature taxonomy that wasn't signaled in the draft.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

we extend the canonical Sharp RD design to a variety of settings that relax its three defining features.

Passages supplied to the Evidence version

Card 1 · Regression Discontinuity Designs

Both Multi-Score and Multi-Cutoff RD designs offer a rich class of RD treatment effects. For a given treatment assignment boundary curve (in the Multi-Score RD design) or each cutoff value (in the Multi-Cutoff RD design), a Sharp or Fuzzy RD treatment effect can be defined, using either the continui…

Read full passage excerpt

Both Multi-Score and Multi-Cutoff RD designs offer a rich class of RD treatment effects. For a given treatment assignment boundary curve (in the Multi-Score RD design) or each cutoff value (in the Multi-Cutoff RD design), a Sharp or Fuzzy RD treatment effect can be defined, using either the continuity or local randomization frameworks. Furthermore, it is common practice to also normalize and pool the data along the treatment assignment boundary curve or the multiple cutoff values in order to consider a single, pooled RD treatment effect. Keele and Titiunik (2015) and Cattaneo, Keele, Titiunik, and Vazquez-Bare (2016) discuss a causal link between the normalized-and-pooled RD treatment effect and the RD treatment effects along the treatment assignment curve or at multiple cutoffs.

Card 2 · A Practical Introduction to Regression Discontinuity Designs

In this Element and its accompanying second Element, A Practical Introduction to Regression Discontinuity Designs: Extensions, Matias Cattaneo, Nicolás Idrobo, and Rocıìo Titiunik provide an accessible and practical guide for the analysis and interpretation of regression discontinuity (RD) designs t…

Read full passage excerpt

In this Element and its accompanying second Element, A Practical Introduction to Regression Discontinuity Designs: Extensions, Matias Cattaneo, Nicolás Idrobo, and Rocıìo Titiunik provide an accessible and practical guide for the analysis and interpretation of regression discontinuity (RD) designs that encourages the use of a common set of practices and facilitates the accumulation of RD-based empirical evidence. In this Element, the authors discuss the foundations of the canonical Sharp RD design, which has the following features: (i) the score is continuously distributed and has only one dimension, (ii) there is only one cutoff, and (iii) compliance with the treatment assignment is perfect. In the second Element, the authors discuss practical and conceptual extensions to this basic RD setup.

Card 3 · Empirical Likelihood Covariate Adjustment for Regression Discontinuity Designs

This paper proposes a versatile covariate adjustment method that directly incorporates covariate balance in regression discontinuity (RD) designs. The new empirical entropy balancing method reweights the standard local polynomial RD estimator by using the entropy balancing weights that minimize the…

Read full passage excerpt

This paper proposes a versatile covariate adjustment method that directly incorporates covariate balance in regression discontinuity (RD) designs. The new empirical entropy balancing method reweights the standard local polynomial RD estimator by using the entropy balancing weights that minimize the Kullback--Leibler divergence from the uniform weights while satisfying the covariate balance constraints. Our estimator can be formulated as an empirical likelihood estimator that efficiently incorporates the information from the covariate balance condition as correctly specified over-identifying moment restrictions, and thus has an asymptotic variance no larger than that of the standard estimator without covariates. We demystify the asymptotic efficiency gain of Calonico, Cattaneo, Farrell, and Titiunik (2019)'s regression-based covariate-adjusted estimator, as their estimator has the same asymptotic variance as ours. Further efficiency improvement from balancing over sieve spaces is possib …

Card 4 · Optimal bandwidth choice for robust bias-corrected inference in regression discontinuity designs

Summary Modern empirical work in regression discontinuity (RD) designs often employs local polynomial estimation and inference with a mean square error (MSE) optimal bandwidth choice. This bandwidth yields an MSE-optimal RD treatment effect estimator, but is by construction invalid for inference. Ro…

Read full passage excerpt

Summary Modern empirical work in regression discontinuity (RD) designs often employs local polynomial estimation and inference with a mean square error (MSE) optimal bandwidth choice. This bandwidth yields an MSE-optimal RD treatment effect estimator, but is by construction invalid for inference. Robust bias-corrected (RBC) inference methods are valid when using the MSE-optimal bandwidth, but we show that they yield suboptimal confidence intervals in terms of coverage error. We establish valid coverage error expansions for RBC confidence interval estimators and use these results to propose new inference-optimal bandwidth choices for forming these intervals. We find that the standard MSE-optimal bandwidth for the RD point estimator is too large when the goal is to construct RBC confidence intervals with the smaller coverage error rate. We further optimize the constant terms behind the coverage error to derive new optimal choices for the auxiliary bandwidth required for RBC inference. Ou …

Card 5 · Evaluating the Causal Effect of University Grants on Student Dropout: Evidence from a Regression Discontinuity Design Using Principal Stratification

Regression discontinuity (RD) designs are often interpreted as local randomized experiments: a RD design can be considered as a randomized experiment for units with a realized value of a so-called forcing variable falling around a pre-fixed threshold. Motivated by the evaluation of Italian universit…

Read full passage excerpt

Regression discontinuity (RD) designs are often interpreted as local randomized experiments: a RD design can be considered as a randomized experiment for units with a realized value of a so-called forcing variable falling around a pre-fixed threshold. Motivated by the evaluation of Italian university grants, we consider a fuzzy RD design where the receipt of the treatment is based on both eligibility criteria and a voluntary application status. Resting on the fact that grant application and grant receipt statuses are post-assignment (post-eligibility) intermediate variables, we use the principal stratification framework to define causal estimands within the Rubin Causal Model. We propose a probabilistic formulation of the assignment mechanism underlying RD designs, by re-formulating the Stable Unit Treatment Value Assumption (SUTVA) and making an explicit local overlap assumption for a subpopulation around the threshold. A local randomization assumption is invoked instead of more stand …

Model-reported card use

Card 2 · terminology

Affected output: the canonical Sharp RD design

Model explanation: Supplied the field-native term 'canonical Sharp RD design' used in the continuation.

Card 2 · direction

Affected output: we extend the canonical Sharp RD design to a variety of settings that relax its three defining features.

Model explanation: Established that the second Element's move is to present extensions of the basic RD setup, guiding the continuation toward naming that extension move.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

we illustrate how the core ideas of the canonical Sharp RD design extend to more complex settings.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Both outputs are grammatically coherent continuations that preserve the draft's meaning. However, Output A is more precise and substantive. The draft mentions 'continuity-based and local randomization approaches' and the sources discuss extensions to canonical RD designs. Output A specifically names 'the canonical Sharp RD design' and its 'three defining features'—directly grounded in Source 2 ('A Practical Introduction to Regression Discontinuity Designs'), which explicitly lists '(i) the score is continuously distributed and has only one dimension, (ii) there is only one cutoff, and (iii) compliance with the treatment assignment is perfect.' Output A's 'extend...to a variety of settings that relax its three defining features' precisely matches the source's framing of extensions. Output B's 'illustrate how the core ideas...extend to more complex settings' is vaguer—'core ideas' and 'more complex settings' are less specific than Output A's explicit reference to the three features. Output A requires less editing to be publication-ready because it provides concrete, source-grounded content rather than a general statement.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.