FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better171 / 280 · beu-709a12a6edcb44e0

Causal inference methods for combining randomized trials and observational studies: a review

Statistics · 2011.08047v4

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Causal inference methods for combining randomized trials and observational studies: a review

1 Introduction

…n criteria which can lead to a trial sample that differs markedly from the population potentially eligible for the treatment. Therefore, the findings from RCTs can lack generalizability to a target population of interest. This concern is related to the aim of external validity , central in medical research ( Concato et al.,, 2000 ; Rothwell,, 2005 ; Green and Glasgow,, 2006 ; Frieden,, 2017 ) policy research ( Martel Garcia and Wantchekon,, 2010 ; Deaton and Cartwright,, 2018 ; Deaton et al.,, 2019 ; Jeong and Namkoong,, 2022 ) , and other fields such as advertising ( Gordon et al.,, 2019 ) .

…er assumptions such as unconfoundedness it is possible to estimate a causal treatment effect from observational data. In practice, methods such as matching, inverse propensity weighting (IPW), or augmented IPW (AIPW) are used ( Imbens and Rubin,, 2015 ) . Even when a confounder is unobserved, solutions exist at the price of additional assumptions, for example the front-door criterion ( Pearl,, 1993 ) , instrumental variables ( Angrist et al.,, 1996 ; Hernán and Robins,, 2006 ; Imbens,, 2014 ) , and sensitivity analysis ( Cornfield et al.,, 1959 ; Rosenbaum and Rubin,, 1983 ; Imbens,, 2003 ) .

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

The draft ends with a paragraph discussing methods for causal inference from observational data when confounders are unobserved, listing instrumental variables, front-door criterion, and sensitivity analysis. The task asks for one coherent academic paragraph to continue. Output A abstains with 'complete', arguing the draft already makes a complete move. However, the draft does not make a complete move: it introduces solutions for unobserved confounders but stops abruptly after listing them without any concluding synthesis, transition, or forward-looking statement. The paragraph feels unfinished—it lists methods parenthetically without explaining their relationships, limitations, or how they connect to the broader argument about generalizability vs. observational methods. Output B provides a substantive continuation that elaborates on instrumental variables specifically, noting both their promise (handling unobserved confounding) and a key practical limitation (potential invalidity of instruments). This adds value by developing one of the listed methods with nuance that fits the paper's evident concern with practical challenges in causal inference. The content about invalid instruments is well-supported by the sources (specifically the 'Two Robust Tools for Inference about Causal Effects with Invalid Instruments' source). B's text could be inserted with minor editing (it doesn't explicitly connect back to the list format, but flows naturally as elaboration). A's abstention is premature because the paragraph genuinely needs a concluding or developing move to feel complete.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

Instrumental variable methods, for instance, can identify causal effects when unobserved confounding is present, provided that a valid instrument affects the outcome only through the treatment and is unrelated to unmeasured confounders, although in practice some putative instruments are likely to be invalid.

Passages supplied to the Evidence version

Card 1 · Bounds on the conditional and average treatment effect with unobserved confounding factors

For observational studies, we study the sensitivity of causal inference when treatment assignments may depend on unobserved confounders. We develop a loss minimization approach for estimating bounds on the conditional average treatment effect (CATE) when unobserved confounders have a bounded effect…

Read full passage excerpt

For observational studies, we study the sensitivity of causal inference when treatment assignments may depend on unobserved confounders. We develop a loss minimization approach for estimating bounds on the conditional average treatment effect (CATE) when unobserved confounders have a bounded effect on the odds ratio of treatment selection. Our approach is scalable and allows flexible use of model classes in estimation, including nonparametric and black-box machine learning methods. Based on these bounds for the CATE, we propose a sensitivity analysis for the average treatment effect (ATE). Our semi-parametric estimator extends/bounds the augmented inverse propensity weighted (AIPW) estimator for the ATE under bounded unobserved confounding. By constructing a Neyman orthogonal score, our estimator of the bound for the ATE is a regular root-$n$ estimator so long as the nuisance parameters are estimated at the $o_p(n^{-1/4})$ rate. We complement our methodology with optimality results sho …

Card 2 · Estimation of causal effects using instrumental variables with nonignorable missing covariates: Application to effect of type of delivery NICU on premature infants

Understanding how effective high-level NICUs (neonatal intensive care units that have the capacity for sustained mechanical assisted ventilation and high volume) are compared to low-level NICUs is important and valuable for both individual mothers and for public policy decisions. The goal of this pa…

Read full passage excerpt

Understanding how effective high-level NICUs (neonatal intensive care units that have the capacity for sustained mechanical assisted ventilation and high volume) are compared to low-level NICUs is important and valuable for both individual mothers and for public policy decisions. The goal of this paper is to estimate the effect on mortality of premature babies being delivered in a high-level NICU vs. a low-level NICU through an observational study where there are unmeasured confounders as well as nonignorable missing covariates. We consider the use of excess travel time as an instrumental variable (IV) to control for unmeasured confounders. In order for an IV to be valid, we must condition on confounders of the IV—outcome relationship, for example, month prenatal care started must be conditioned on for excess travel time to be a valid IV. However, sometimes month prenatal care started is missing, and the missingness may be nonignorable because it is related to the not fully measured mo …

Card 3 · Stronger instruments via integer programming in an observational study of late preterm birth outcomes

In an optimal nonbipartite match, a single population is divided into matched pairs to minimize a total distance within matched pairs. Nonbipartite matching has been used to strengthen instrumental variables in observational studies of treatment effects, essentially by forming pairs that are similar…

Read full passage excerpt

In an optimal nonbipartite match, a single population is divided into matched pairs to minimize a total distance within matched pairs. Nonbipartite matching has been used to strengthen instrumental variables in observational studies of treatment effects, essentially by forming pairs that are similar in terms of covariates but very different in the strength of encouragement to accept the treatment. Optimal nonbipartite matching is typically done using network optimization techniques that can be quick, running in polynomial time, but these techniques limit the tools available for matching. Instead, we use integer programming techniques, thereby obtaining a wealth of new tools not previously available for nonbipartite matching, including fine and near-fine balance for several nominal variables, forced near balance on means and optimal subsetting. We illustrate the methods in our on-going study of outcomes of late-preterm births in California, that is, births of 34 to 36 weeks of gestation …

Card 4 · Causal Inference with a Graphical Hierarchy of Interventions

Our results motivate the study of estimation theory for the edge g-formula, since we show it arises both in mediation analysis, and in settings where treatment assignment has unobserved causes, such as models associated with Pearl’s front-door criterion.

Card 5 · Two Robust Tools for Inference about Causal Effects with Invalid Instruments

Instrumental variables have been widely used to estimate the causal effect of a treatment on an outcome. Existing confidence intervals for causal effects based on instrumental variables assume that all of the putative instrumental variables are valid; a valid instrumental variable is a variable that…

Read full passage excerpt

Instrumental variables have been widely used to estimate the causal effect of a treatment on an outcome. Existing confidence intervals for causal effects based on instrumental variables assume that all of the putative instrumental variables are valid; a valid instrumental variable is a variable that affects the outcome only by affecting the treatment and is not related to unmeasured confounders. However, in practice, some of the putative instrumental variables are likely to be invalid. This paper presents two tools to conduct valid inference and tests in the presence of invalid instruments. First, we propose a simple and general approach to construct confidence intervals based on taking unions of well-known confidence intervals. Second, we propose a novel test for the null causal effect based on a collider bias. Our two proposals, especially when fused together, outperform traditional instrumental variable confidence intervals when invalid instruments are present, and can also be used …

Model-reported card use

Card 5 · fact

Affected output: provided that a valid instrument affects the outcome only through the treatment and is unrelated to unmeasured confounders

Model explanation: The row's definition of instrument validity supplied the specific condition attached to the IV claim rather than a generic mention of instrumental variables.

Card 5 · scope

Affected output: although in practice some putative instruments are likely to be invalid

Model explanation: The row's caveat bounded the IV claim by noting that the validity condition is often violated in practice.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

No continuation text was produced.

Model action：complete

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The with-card arm produced text and reported card use, but the no-card arm produced no text. This counts as reported use, without entering the two-text preference comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.