FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better180 / 280 · beu-acd7d2d7610955d5

A Practical Introduction to Regression Discontinuity Designs: Extensions

Statistics · 2301.08958v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Practical Introduction to Regression Discontinuity Designs: Extensions

1 Introduction

…Section 3 with a discussion of the Fuzzy RD design where, in contrast to the Sharp RD design, compliance with the treatment assignment is imperfect: some units above the cutoff fail to take the treatment despite being assigned to take it, and/or some units below the cutoff take the treatment despite being assigned to the untreated condition. We define several parameters of interest that can be recovered under noncompliance, and discuss how to employ both continuity-based and local randomization approaches for analysis. We also discuss how to perform falsification analysis under noncompliance.

…llecting data on all students in a college campus results in a dataset where hundreds or thousands of students have the same GPA value. In the RD design, the existence of such “mass points” in the score variable sometimes requires using alternative methods, as the standard continuity-based methods discussed in Foundations are no longer generally applicable without modifications. We discuss when and why continuity-based methods may be inadequate to analyze RD designs with discrete scores, and also describe how the local randomization approach can be a useful alternative framework for analysis.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs attempt to continue the manuscript by introducing a new section on RD designs with multiple scores. The draft ends Section 3 (Fuzzy RD) and has a blank paragraph before a discussion of mass points/discrete scores. The task is to fill this blank with one coherent academic paragraph. Output A introduces "Section 4" on multiple scores, but this creates a numbering problem: the visible text after the blank discusses mass points/discrete scores, which would logically be Section 4 (or 4.1), making multiple scores Section 5. Output A's "Section 4" directly contradicts the manuscript structure implied by the draft. Output B introduces "Section 5" on multiple scores, which correctly accounts for the fact that the mass points discussion (visible after the blank) would occupy Section 4, making multiple scores Section 5. This shows better fit to the manuscript's actual structure. Additionally, Output B provides slightly more substantive content about "additional assumptions required" and "multidimensional score space," which is more precise academically than Output A's more generic description. Both are otherwise reasonable continuations, but Output B's section numbering is accurate while Output A's is materially wrong for this manuscript.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

In Section 4 we consider RD designs with multiple scores, where treatment assignment depends on more than one variable and the standard single-score framework no longer applies directly. We describe how to define and interpret treatment effects in this setting, and how estimation and inference proceed when the forcing variable is multidimensional.

Passages supplied to the Evidence version

Card 1 · Selecting Subpopulations for Causal Inference in Regression Discontinuity Designs

The Brazil Bolsa Familia (BF) program is a conditional cash transfer program aimed to reduce short-term poverty by direct cash transfers and to fight long-term poverty by increasing human capital among poor Brazilian people. Eligibility for Bolsa Familia benefits depends on a cutoff rule, which clas…

Read full passage excerpt

The Brazil Bolsa Familia (BF) program is a conditional cash transfer program aimed to reduce short-term poverty by direct cash transfers and to fight long-term poverty by increasing human capital among poor Brazilian people. Eligibility for Bolsa Familia benefits depends on a cutoff rule, which classifies the BF study as a regression discontinuity (RD) design. Extracting causal information from RD studies is challenging. Following Li et al (2015) and Branson and Mealli (2019), we formally describe the BF RD design as a local randomized experiment within the potential outcome approach. Under this framework, causal effects can be identified and estimated on a subpopulation where a local overlap assumption, a local SUTVA and a local ignorability assumption hold. We first discuss the potential advantages of this framework over local regression methods based on continuity assumptions, which concern the definition of the causal estimands, the design and the analysis of the study, and the int …

Card 2 · Evaluating the Causal Effect of University Grants on Student Dropout: Evidence from a Regression Discontinuity Design Using Principal Stratification

Regression discontinuity (RD) designs are often interpreted as local randomized experiments: a RD design can be considered as a randomized experiment for units with a realized value of a so-called forcing variable falling around a pre-fixed threshold. Motivated by the evaluation of Italian universit…

Read full passage excerpt

Regression discontinuity (RD) designs are often interpreted as local randomized experiments: a RD design can be considered as a randomized experiment for units with a realized value of a so-called forcing variable falling around a pre-fixed threshold. Motivated by the evaluation of Italian university grants, we consider a fuzzy RD design where the receipt of the treatment is based on both eligibility criteria and a voluntary application status. Resting on the fact that grant application and grant receipt statuses are post-assignment (post-eligibility) intermediate variables, we use the principal stratification framework to define causal estimands within the Rubin Causal Model. We propose a probabilistic formulation of the assignment mechanism underlying RD designs, by re-formulating the Stable Unit Treatment Value Assumption (SUTVA) and making an explicit local overlap assumption for a subpopulation around the threshold. A local randomization assumption is invoked instead of more stand …

Card 3 · Regression Discontinuity Designs

The regression discontinuity (RD) design is one of the most widely used nonexperimental methods for causal inference and program evaluation. Over the last two decades, statistical and econometric methods for RD analysis have expanded and matured, and there is now a large number of methodological res…

Read full passage excerpt

The regression discontinuity (RD) design is one of the most widely used nonexperimental methods for causal inference and program evaluation. Over the last two decades, statistical and econometric methods for RD analysis have expanded and matured, and there is now a large number of methodological results for RD identification, estimation, inference, and validation. We offer a curated review of this methodological literature organized around the two most popular frameworks for the analysis and interpretation of RD designs: the continuity framework and the local randomization framework. For each framework, we discuss three main topics: ( a) designs and parameters, focusing on different types of RD settings and treatment effects of interest; ( b) estimation and inference, presenting the most popular methods based on local polynomial regression and methods for the analysis of experiments, as well as refinements, extensions, and alternatives; and ( c) validation and falsification, summarizin …

Card 4 · The Local Randomization Framework for Regression Discontinuity Designs: A Review and Some Extensions

In this work, we extend the local randomization framework to allow for any assignment mechanism, such as Bernoulli trials and block randomization, where propensity scores are allowed to differ. We also develop exact randomization tests for covariate balance to test whether a particular assignment me…

Read full passage excerpt

In this work, we extend the local randomization framework to allow for any assignment mechanism, such as Bernoulli trials and block randomization, where propensity scores are allowed to differ. We also develop exact randomization tests for covariate balance to test whether a particular assignment mechanism holds within an RDD. These tests allow for multiple testing corrections and can be used to select the largest window around the cutoff where a particular assignment mechanism is most plausible. Then, an analysis using this assignment mechanism can be conducted within that window. We apply our methodology to a fuzzy RDD that assesses the effects of financial aid on college dropout rates in Italy.

Model-reported card use

The model did not report that any card materially affected this completion.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

In Section 5 we consider RD designs with multiple scores, where treatment assignment depends on more than one running variable and the standard single-score framework no longer applies directly. We describe the additional assumptions required to define a treatment effect in this setting and discuss how estimation and inference proceed when the cutoff is defined over a multidimensional score space.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

This case does not have both with-card continuation text and a model-reported use record. It remains among all 280 tests but does not enter the reported-use comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.