FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better33 / 280 · beu-008126236acaf005

A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances

Statistics · 2405.03910v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances

1 Introduction

…he average treatment effect by reducing what we refer to as the ex-post (as opposed to ex-ante) bias of the estimator. Here, the ex-post bias of the estimator is used to describe its behavior conditional on treatment status rather than unconditionally. Section 4.2 develops the implications of this increase in precision for inferences about the average treatment effect more formally for the special case of stratified block randomization with finitely many “large” strata; Section 4.2.1 describes how this analysis changes when strata are “small,” for instance in the case of matched pair designs.

… Section 5 we consider the use of baseline covariates (including ones that are possibly not used in the assignment of treatment status) to further improve the precision of estimators of the average treatment effect through regression adjustment. We explain how naïve regression adjustment may increase rather than decrease the precision of estimators of the average treatment effect, and how a more careful use of the covariates can ensure an improvement in (asymptotic) precision. Our discussion includes linear adjustments (Section 5.1 ) as well as more general forms of adjustment (Section 5.2 ).

Finally, Section 6 extends our discussion to cluster randomized experiments, i.e., randomized experiments in which the unit of randomization is a cluster. Such

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Both outputs successfully complete the sentence by defining cluster randomized experiments and giving concrete examples. Output B is superior because it adds the crucial methodological point that 'outcomes within a cluster are typically correlated'—this is the fundamental statistical reason why cluster randomization requires special treatment and why it merits a dedicated section. This addition makes the sentence more informative for academic readers by explaining not just what cluster designs are, but why they matter methodologically. Output A stops at the definition and examples without signaling the analytical challenge that Section 6 will presumably address. The extra clause in B is concise, accurate, and directly relevant to the paper's likely focus on estimation and inference methods.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

designs arise when treatment is assigned at the level of groups, such as schools, villages, or clinics, rather than to individuals, so that outcomes within a cluster are typically correlated.

Passages supplied to the Evidence version

Card 1 · Covariate Adjustment in Stratified Experiments

This paper studies covariate adjusted estimation of the average treatment effect in stratified experiments. We work in a general framework that includes matched tuples designs, coarse stratification, and complete randomization as special cases. Regression adjustment with treatment-covariate interact…

Read full passage excerpt

This paper studies covariate adjusted estimation of the average treatment effect in stratified experiments. We work in a general framework that includes matched tuples designs, coarse stratification, and complete randomization as special cases. Regression adjustment with treatment-covariate interactions is known to weakly improve efficiency for completely randomized designs. By contrast, we show that for stratified designs such regression estimators are generically inefficient, potentially even increasing estimator variance relative to the unadjusted benchmark. Motivated by this result, we derive the asymptotically optimal linear covariate adjustment for a given stratification. We construct several feasible estimators that implement this efficient adjustment in large samples. In the special case of matched pairs, for example, the regression including treatment, covariates, and pair fixed effects is asymptotically optimal. We also provide novel asymptotically exact inference methods tha …

Card 2 · Regression-adjusted average treatment effect estimates in stratified randomized experiments

Researchers often use linear regression to analyse randomized experiments to improve treatment effect estimation by adjusting for imbalances of covariates in the treatment and control groups. Our work offers a randomization-based inference framework for regression adjustment in stratified randomized…

Read full passage excerpt

Researchers often use linear regression to analyse randomized experiments to improve treatment effect estimation by adjusting for imbalances of covariates in the treatment and control groups. Our work offers a randomization-based inference framework for regression adjustment in stratified randomized experiments. Under mild conditions, we re-establish the finite population central limit theorem for a stratified experiment. We prove that both the stratified difference-in-means and the regression-adjusted average treatment effect estimators are consistent and asymptotically normal. The asymptotic variance of the latter is no greater and is typically lesser than that of the former. We also provide conservative variance estimators to construct large-sample confidence intervals for the average treatment effect.

Card 3 · Improving Estimation Efficiency via Regression-Adjustment in Covariate-Adaptive Randomizations with Imperfect Compliance

We investigate how to improve efficiency using regression adjustments with covariates in covariate-adaptive randomizations (CARs) with imperfect subject compliance. Our regression-adjusted estimators, which are based on the doubly robust moment for local average treatment effects, are consistent and…

Read full passage excerpt

We investigate how to improve efficiency using regression adjustments with covariates in covariate-adaptive randomizations (CARs) with imperfect subject compliance. Our regression-adjusted estimators, which are based on the doubly robust moment for local average treatment effects, are consistent and asymptotically normal even with heterogeneous probabilities of assignment and misspecified regression adjustments. We propose an optimal but potentially misspecified linear adjustment and its further improvement via a nonlinear adjustment, both of which lead to more efficient estimators than the one without adjustments. We also provide conditions for nonparametric and regularized adjustments to achieve the semiparametric efficiency bound under CARs.

Card 4 · Blocking, rerandomization, and regression adjustment in randomized experiments with high-dimensional covariates

This study is aimed at enhancing the estimation and inference efficiencies of the average treatment effect in randomized experiments when many baseline covariates are available. We propose novel methods to combine blocking, rerandomization, and regression adjustment using the Lasso. Under mild condi…

Read full passage excerpt

This study is aimed at enhancing the estimation and inference efficiencies of the average treatment effect in randomized experiments when many baseline covariates are available. We propose novel methods to combine blocking, rerandomization, and regression adjustment using the Lasso. Under mild conditions, we obtain the asymptotic distributions of the Lasso-adjusted average treatment effect estimators when blocking or rerandomization or both are implemented in the design stage. We demonstrate that the proposed Lasso-adjusted estimators enhance, or, at least, do not deteriorate the precision compared with that associated with the unadjusted estimator.

Card 5 · To adjust or not to adjust? Estimating the average treatment effect in randomized experiments with missing covariates

Complete randomization allows for consistent estimation of the average treatment effect based on the difference in means of the outcomes without strong modeling assumptions on the outcome-generating process. Appropriate use of the pretreatment covariates can further improve the estimation efficiency…

Read full passage excerpt

Complete randomization allows for consistent estimation of the average treatment effect based on the difference in means of the outcomes without strong modeling assumptions on the outcome-generating process. Appropriate use of the pretreatment covariates can further improve the estimation efficiency. However, missingness in covariates is common in experiments and raises an important question: should we adjust for covariates subject to missingness, and if so, how? The unadjusted difference in means is always unbiased. The complete-covariate analysis adjusts for all completely observed covariates and improves the efficiency of the difference in means if at least one completely observed covariate is predictive of the outcome. Then what is the additional gain of adjusting for covariates subject to missingness? A key insight is that the missingness indicators act as fully observed pretreatment covariates as long as missingness is not affected by the treatment, and can thus be used in covari …

Model-reported card use

The model did not report that any card materially affected this completion.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

designs arise when treatment is assigned at the level of groups, such as villages, schools, or classrooms, rather than to individuals.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

This case does not have both with-card continuation text and a model-reported use record. It remains among all 280 tests but does not enter the reported-use comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.