FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better175 / 280 · beu-5523b82034f330f8

Rejective Sampling, Rerandomization and Regression Adjustment in Survey Experiments

Statistics · 2109.09930v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Rejective Sampling, Rerandomization and Regression Adjustment in Survey Experiments

Conclusion

…e 4 (c) shows the coverage probabilities of the confidence intervals under all cases. From Figure 4 , compared to the CRSE, the precision of the difference-in-means estimator is improved by employing rerandomization at the treatment assignment stage, and it is further improved by also employing rerandomization at the sampling stage as well as conducting covariate adjustment at the analysis stage. Moreover, both rerandomization and regression adjustment reduce the average lengths of the confidence intervals, while still maintaining the coverage probabilities at the nominal level approximately.

…reatment effect estimate is more precise under the greedy pair-switching design than under rerandomization, but the improvement is relatively small. Compared to the greedy design, rerandomization is computationally much simpler, and more importantly, it allows large-sample inference for the average treatment effect as discussed throughout the paper, avoiding any constant treatment effect assumption that is typically involved for randomization tests. Additionally, employing rerandomization at the sampling stage (i.e., rejective sampling) is also beneficial for the greedy pair-switching design.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs provide reasonable completions, but B fits better. The draft ends with a discussion of benefits of rerandomization at sampling stage for the greedy pair-switching design, and has throughout emphasized: (1) combining rerandomization at both stages with regression adjustment, (2) computational simplicity of rerandomization vs. greedy design, (3) large-sample inference without constant treatment effect assumptions. Output B directly mirrors these themes: it mentions joint implementation of sampling-stage rerandomization, assignment-stage rerandomization, and covariate adjustment; it emphasizes asymptotic normality and large-sample inference without constant treatment effect assumption. This precisely echoes the draft's earlier claim that rerandomization 'allows large-sample inference for the average treatment effect... avoiding any constant treatment effect assumption.' Output A is also coherent but introduces 'unified framework' and 'survey experiments' terminology not present in the draft, and speaks of 'extending' to more complex designs which is more forward-looking/speculative than the draft's tone. B more accurately captures what the paper 'has shown' based on the visible text.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

This paper has developed a unified framework for rejective sampling, rerandomization, and regression adjustment in survey experiments, and has shown that combining these tools improves the precision of the difference-in-means estimator while preserving valid large-sample inference for the average treatment effect. Extending the asymptotic theory to more complex sampling designs and outcome models would further broaden the applicability of the proposed approach.

Passages supplied to the Evidence version

Card 1 · Covariate-adjusted Fisher randomization tests for the average treatment effect

Fisher's randomization test (FRT) delivers exact $p$-values under the strong null hypothesis of no treatment effect on any units whatsoever and allows for flexible covariate adjustment to improve the power. Of interest is whether the procedure could also be valid for testing the weak null hypothesis…

Read full passage excerpt

Fisher's randomization test (FRT) delivers exact $p$-values under the strong null hypothesis of no treatment effect on any units whatsoever and allows for flexible covariate adjustment to improve the power. Of interest is whether the procedure could also be valid for testing the weak null hypothesis of zero average treatment effect. Towards this end, we evaluate two general strategies for FRT with covariate-adjusted test statistics: that based on the residuals from an outcome model with only the covariates, and that based on the output from an outcome model with both the treatment and the covariates. Based on theory and simulation, we recommend using the ordinary least squares (OLS) fit of the observed outcome on the treatment, centered covariates, and their interactions for covariate adjustment, and conducting FRT with the robust $t$-value of the treatment as the test statistic. The resulting FRT is finite-sample exact for the strong null hypothesis, asymptotically valid for the weak …

Card 2 · Rerandomization in Stratified Randomized Experiments

Stratification and rerandomization are two well-known methods used in randomized experiments for balancing the baseline covariates. Renowned scholars in experimental design have recommended combining these two methods; however, limited studies have addressed the statistical properties of this combin…

Read full passage excerpt

Stratification and rerandomization are two well-known methods used in randomized experiments for balancing the baseline covariates. Renowned scholars in experimental design have recommended combining these two methods; however, limited studies have addressed the statistical properties of this combination. This article proposes two rerandomization methods to be used in stratified randomized experiments, based on the overall and stratum-specific Mahalanobis distances. The first method is applicable for nearly arbitrary numbers of strata, strata sizes, and stratum-specific proportions of the treated units. The second method, which is generally more efficient than the first method, is suitable for situations in which the number of strata is fixed with their sizes tending to infinity. Under the randomization inference framework, we obtain the asymptotic distributions of estimators used in these methods and the formulas of variance reduction when compared to stratified randomization. Our ana …

Card 3 · Asymptotic Theory of Rerandomization in Treatment-Control Experiments

Although complete randomization ensures covariate balance on average, the chance for observing significant differences between treatment and control covariate distributions increases with many covariates. Rerandomization discards randomizations that do not satisfy a predetermined covariate balance c…

Read full passage excerpt

Although complete randomization ensures covariate balance on average, the chance for observing significant differences between treatment and control covariate distributions increases with many covariates. Rerandomization discards randomizations that do not satisfy a predetermined covariate balance criterion, generally resulting in better covariate balance and more precise estimates of causal effects. Previous theory has derived finite sample theory for rerandomization under the assumptions of equal treatment group sizes, Gaussian covariate and outcome distributions, or additive causal effects, but not for the general sampling distribution of the difference-in-means estimator for the average causal effect. To supplement existing results, we develop asymptotic theory for rerandomization without these assumptions, which reveals a non-Gaussian asymptotic distribution for this estimator, specifically a linear combination of a Gaussian random variable and a truncated Gaussian random variable …

Card 4 · Pair-switching rerandomization

Rerandomization discards assignments with covariates unbalanced in the treatment and control groups to improve estimation and inference efficiency. However, the acceptance-rejection sampling method used in rerandomization is computationally inefficient. As a result, it is time-consuming for rerandom…

Read full passage excerpt

Rerandomization discards assignments with covariates unbalanced in the treatment and control groups to improve estimation and inference efficiency. However, the acceptance-rejection sampling method used in rerandomization is computationally inefficient. As a result, it is time-consuming for rerandomization to draw numerous independent assignments, which are necessary for performing Fisher randomization tests and constructing randomization-based confidence intervals. To address this problem, we propose a pair-switching rerandomization method to draw balanced assignments efficiently. We obtain the unbiasedness and variance reduction of the difference-in-means estimator and show that the Fisher randomization tests are valid under pair-switching rerandomization. Moreover, we propose an exact approach to invert Fisher randomization tests to confidence intervals, which is faster than the existing methods. In addition, our method is applicable to both non-sequentially and sequentially randomi …

Card 5 · Randomization Tests to Assess Covariate Balance When Designing and Analyzing Matched Datasets

Causal analyses for observational studies are often complicated by covariate imbalances among treatment groups, and matching methodologies alleviate this complication by finding subsets of treatment groups that exhibit covariate balance.It is widely agreed upon that covariate balance can serve as ev…

Read full passage excerpt

Causal analyses for observational studies are often complicated by covariate imbalances among treatment groups, and matching methodologies alleviate this complication by finding subsets of treatment groups that exhibit covariate balance.It is widely agreed upon that covariate balance can serve as evidence that a matched dataset approximates a randomized experiment, but what kind of experiment does a matched dataset approximate?In this work, we develop a randomization test for the hypothesis that a matched dataset approximates a particular experimental design, such as complete randomization, block randomization, or rerandomization.Our test can incorporate any experimental design, and it allows for a graphical display that puts several designs on the same univariate scale, thereby allowing researchers to pinpoint which design-if any-is most appropriate for a matched dataset.After researchers determine a plausible design, we recommend a randomizationbased approach for analyzing the matche …

Model-reported card use

Card 3 · scope

Affected output: preserving valid large-sample inference for the average treatment effect

Model explanation: Row 3's emphasis on large-sample inference for the average causal effect under rerandomization guided the framing of the paper's contribution in the concluding synthesis.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

This paper has studied the combination of rejective sampling, rerandomization, and regression adjustment for estimating the average treatment effect in survey experiments. We have shown that rerandomization at the sampling stage and at the treatment assignment stage can be implemented jointly with covariate adjustment at the analysis stage, and that the resulting estimator remains asymptotically normal, permitting standard large-sample inference without a constant treatment effect assumption.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blind model judge did not prefer either output.

Read the original model judgment

Order-swapped judges disagreed. Pass 1: Both outputs provide coherent academic paragraph completions, but B better matches the draft's scope and content. The draft discusses: (1) coverage probabilities and precision improvements from combining rerandomization at assignment stage + sampling stage + covariate adjustment, (2) comparison with greedy pair-switching design noting rerandomization's computational simplicity and valid large-sample inference without constant treatment effect assumptions, and (3) benefits of rejective sampling even for greedy designs. Output A claims 'the resulting estimator remains asymptotically normal' - this is plausible but not explicitly supported by the shown draft text (the draft mentions 'large-sample inference' but not specifically 'asymptotically normal'). More importantly, A's 'without a constant treatment effect assumption' directly echoes the draft's language about avoiding 'constant treatment effect assumption,' which is good. However, A's framing as 'This paper has studied' is weaker than B's 'developed a unified framework.' Output B better captures the three-stage combination (sampling, assignment, analysis) that the draft emphasizes, uses 'unified framework' which matches the integrative nature of the work, and importantly includes the forward-looking extension sentence which is a standard way to end such discussion paragraphs. B's 'preserving valid large-sample inference' is more precisely grounded in the draft's text than A's 'asymptotically normal.' B also correctly notes the 'difference-in-means estimator' which is specifically mentioned in the draft, while A omits this specificity. The extension sentence in B, while not directly grounded in sources, is a conventional academic closing for a paragraph discussing methodological contributions and does not constitute an unsupported factual claim about results. Pass 2: Output A contains unsupported claims: (1) 'unified framework' is not established in the draft—the draft discusses these methods in combination but does not frame them as a 'unified framework'; (2) 'Extending the asymptotic theory to more complex sampling designs and outcome models would further broaden the applicability' is an unsourced forward-looking claim about future work with no basis in the draft or sources. Output B stays grounded in what the draft actually establishes: that rerandomization at sampling and treatment assignment stages can be jointly implemented with covariate adjustment, that the estimator remains asymptotically normal, and that this permits large-sample inference without constant treatment effect assumption—all of which are directly supported by the draft's discussion of Figure 4 results and the explicit comparison to greedy pair-switching design which 'avoid[s] any constant treatment effect assumption.' Both outputs are coherent paragraph-length continuations, but A adds substantive claims beyond the evidence base while B does not.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.