FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better32 / 280 · beu-ebf86eb708352779

A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances

Statistics · 2405.03910v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances

1 Introduction

…ld standard” in program evaluation, but this body of work has revealed many subtle aspects of randomized experiments that may have been previously unappreciated. This article provides an overview of some of these topics, primarily focused on stratification, regression adjustment, and cluster randomization, although we also provide short discussions on a broad range of other topics at the end of the article. We also provide a companion R package sreg , which is designed to facilitate inference in stratified (as well as potentially clustered) randomized experiments: see Trifonov et al. (2025) .

…w in Section 3 by providing a comparison of these two sampling frameworks. Specifically, we introduce both frameworks in the context of an analysis of the difference-in-means estimator for the average treatment effect in a completely randomized experiment. There, we argue that the super-population framework approximates a finite-population framework in which a negligible fraction of the finite population is sampled; we argue further, however, that results derived in a super-population framework may provide useful methods for inference in a finite population even outside of this limiting case.

We then turn our attention in Section 4 to the impact of stratification on subsequent inferences drawn from randomized experiments. More specifically, instead of assigning treatment status according to complete

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs complete the sentence grammatically and coherently. Output A is preferable because it more precisely captures the contrast the draft sets up. The draft says 'instead of assigning treatment status according to complete' — the key contrast is between complete randomization and stratified randomization. Output A explicitly names this contrast ('complete randomization, we consider designs in which treatment assignment is stratified according to baseline covariates'), making the logical structure transparent. Output B describes the mechanics of stratification ('partition...into strata and then assign treatment within each stratum') but does so without explicitly naming what is being contrasted with — 'complete randomization' is cut off and not completed as a phrase. The reader must infer that 'complete randomization' was the intended object. Output A's 'complete randomization' properly completes the prepositional phrase 'according to complete randomization' and makes the contrast explicit. Both are factually accurate, but Output A better serves the argumentative move of the sentence.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

randomization, we consider designs in which treatment assignment is stratified according to baseline covariates.

Passages supplied to the Evidence version

Card 1 · Inference for Cluster Randomized Experiments with Non-ignorable Cluster Sizes

This paper considers the problem of inference in cluster randomized experiments when cluster sizes are non-ignorable. Here, by a cluster randomized experiment, we mean one in which treatment is assigned at the cluster level. By non-ignorable cluster sizes, we refer to the possibility that the treatm…

Read full passage excerpt

This paper considers the problem of inference in cluster randomized experiments when cluster sizes are non-ignorable. Here, by a cluster randomized experiment, we mean one in which treatment is assigned at the cluster level. By non-ignorable cluster sizes, we refer to the possibility that the treatment effects may depend non-trivially on the cluster sizes. We frame our analysis in a super-population framework in which cluster sizes are random. In this way, our analysis departs from earlier analyses of cluster randomized experiments in which cluster sizes are treated as non-random. We distinguish between two different parameters of interest: the equally-weighted cluster-level average treatment effect, and the size-weighted cluster-level average treatment effect. For each parameter, we provide methods for inference in an asymptotic framework where the number of clusters tends to infinity and treatment is assigned using a covariate-adaptive stratified randomization procedure. We additiona …

Card 2 · Sharp Variance Estimator and Causal Bootstrap in Stratified Randomized Experiments

Randomized experiments are the gold standard for estimating treatment effects, and randomization serves as a reasoned basis for inference. In widely used stratified randomized experiments, randomization-based finite-population asymptotic theory enables valid inference for the average treatment effec…

Read full passage excerpt

Randomized experiments are the gold standard for estimating treatment effects, and randomization serves as a reasoned basis for inference. In widely used stratified randomized experiments, randomization-based finite-population asymptotic theory enables valid inference for the average treatment effect, relying on normal approximation and a Neyman-type conservative variance estimator. However, when the sample size is small or the outcomes are skewed, the Neyman-type variance estimator may become overly conservative, and the normal approximation can fail. To address these issues, we propose a sharp variance estimator and two causal bootstrap methods to more accurately approximate the sampling distribution of the weighted difference-in-means estimator in stratified randomized experiments. The first causal bootstrap procedure is based on rank-preserving imputation, and we prove its second-order refinement over normal approximation. The second causal bootstrap procedure is based on constant- …

Card 3 · Sharp bounds for variance of treatment effect estimators in the finite population in the presence of covariates

Most of the literature assumes that the observations are sampled from an infinite super population ( Hirano et al., 2003 ; Imbens, 2004 ; Belloni et al., 2014 ; Chan et al., 2016 ) . In some cases, the infinite super population seems contrived if we are interested in evaluating the treatment effect…

Read full passage excerpt

Most of the literature assumes that the observations are sampled from an infinite super population ( Hirano et al., 2003 ; Imbens, 2004 ; Belloni et al., 2014 ; Chan et al., 2016 ) . In some cases, the infinite super population seems contrived if we are interested in evaluating the treatment effect for a particular finite population ( Li and Ding, 2017 ) . In randomized experiments, the finite population framework has been widely used in data analysis since Neyman and Iwaszkiewicz (1935) . This framework views all potential outcomes of a finite population as fixed; estimation and inference are based on randomization solely ( Imbens and Rosenbaum, 2005 ; Nolen and Hudgens, 2011 ) . However, in the completely randomized experiment, a fundamental problem in the finite population is that the variance of the widely-used difference-in-means estimator is unidentifiable.

Card 4 · Covariate-adjusted Fisher randomization tests for the average treatment effect

Fisher's randomization test (FRT) delivers exact $p$-values under the strong null hypothesis of no treatment effect on any units whatsoever and allows for flexible covariate adjustment to improve the power. Of interest is whether the procedure could also be valid for testing the weak null hypothesis…

Read full passage excerpt

Fisher's randomization test (FRT) delivers exact $p$-values under the strong null hypothesis of no treatment effect on any units whatsoever and allows for flexible covariate adjustment to improve the power. Of interest is whether the procedure could also be valid for testing the weak null hypothesis of zero average treatment effect. Towards this end, we evaluate two general strategies for FRT with covariate-adjusted test statistics: that based on the residuals from an outcome model with only the covariates, and that based on the output from an outcome model with both the treatment and the covariates. Based on theory and simulation, we recommend using the ordinary least squares (OLS) fit of the observed outcome on the treatment, centered covariates, and their interactions for covariate adjustment, and conducting FRT with the robust $t$-value of the treatment as the test statistic. The resulting FRT is finite-sample exact for the strong null hypothesis, asymptotically valid for the weak …

Card 5 · Sharp bounds for variance of treatment effect estimators in the finite population in the presence of covariates

Most of the literature assumes that the observations are sampled from an infinite super population ( Hirano et al., 2003 ; Imbens, 2004 ; Belloni et al., 2014 ; Chan et al., 2016 ) . In some cases, the infinite super population seems contrived if we are interested in evaluating the treatment effect…

Read full passage excerpt

Most of the literature assumes that the observations are sampled from an infinite super population ( Hirano et al., 2003 ; Imbens, 2004 ; Belloni et al., 2014 ; Chan et al., 2016 ) . In some cases, the infinite super population seems contrived if we are interested in evaluating the treatment effect for a particular finite population ( Li and Ding, 2017 ) . In randomized experiments, the finite population framework has been widely used in data analysis since Neyman and Iwaszkiewicz (1935) . This framework views all potential outcomes of a finite population as fixed; estimation and inference are based on randomization solely ( Imbens and Rosenbaum, 2005 ; Nolen and Hudgens, 2011 ) . However, in the completely randomized experiment, a fundamental problem in the finite population is that the variance of the widely-used difference-in-means estimator is unidentifiable.

Model-reported card use

The model did not report that any card materially affected this completion.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

randomization, we first partition the experimental units into strata and then assign treatment within each stratum.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

This case does not have both with-card continuation text and a model-reported use record. It remains among all 280 tests but does not enter the reported-use comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.