FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better172 / 280 · beu-8b7b29c267c1ffda

Causal inference methods for combining randomized trials and observational studies: a review

Statistics · 2011.08047v4

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Causal inference methods for combining randomized trials and observational studies: a review

8 Conclusion

…analyses with the medical expert committee are necessary to re-define a target population of interest on which generalization is possible and medically relevant. As it is generally the case, beyond methodological and theoretical guarantees, a major step to be taken before applying a set of methods is to clearly state the causal question and estimand(s) and the associated identifiability requirements. This task is even more complex when combining data sets. A primary and fundamental concern is whether outcome, treatment, and covariates are comparable in the two studies ( Lodi et al.,, 2019 ) .

…nces of adding covariates that are shifted between the two populations while not being treatment effect modifiers, resulting in variance inflation ( Colnet et al., 2022b, ) . In its current state, the field probably lacks work on covariate selection and its impact on bias and variance. Some recent works propose the use of causal graphs to select optimal adjustment sets that allow the reduction of the variance of the final estimation ( Rotnitzky and Smucler,, 2019 ; Witte et al.,, 2020 ; Guo and Perković,, 2020 ) , but such methods have not yet been developed for generalization or data fusion.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Output A is invalid because it provides contradictory signals: empty text with action 'invalid'. The action 'invalid' is reserved for outputs with material contradictions, invented results, or serious topic drift—none of which apply to an empty output. This makes Output A unusable. Output B appropriately uses 'complete' with no text. Evaluating the draft: the paragraph ends by noting that causal graph methods for optimal adjustment sets (variance reduction) 'have not yet been developed for generalization or data fusion.' This is a natural concluding observation for this section. The draft has already (1) established the complexity of stating causal questions when combining datasets, (2) raised comparability concerns with citation, (3) discussed variance inflation from shifted covariates, and (4) noted the gap in extending optimal adjustment set methods to generalization/data fusion. Adding another paragraph would risk becoming redundant or speculative without a clear next argumentative move. The 'complete' action is therefore appropriate and usable here.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

No continuation text was produced.

Model action：invalid

Passages supplied to the Evidence version

Card 1 · Graphical Criteria for Efficient Total Effect Estimation Via Adjustment in Causal Linear Models

Abstract Covariate adjustment is a commonly used method for total causal effect estimation. In recent years, graphical criteria have been developed to identify all valid adjustment sets, that is, all covariate sets that can be used for this purpose. Different valid adjustment sets typically provide…

Read full passage excerpt

Abstract Covariate adjustment is a commonly used method for total causal effect estimation. In recent years, graphical criteria have been developed to identify all valid adjustment sets, that is, all covariate sets that can be used for this purpose. Different valid adjustment sets typically provide total causal effect estimates of varying accuracies. Restricting ourselves to causal linear models, we introduce a graphical criterion to compare the asymptotic variances provided by certain valid adjustment sets. We employ this result to develop two further graphical tools. First, we introduce a simple variance decreasing pruning procedure for any given valid adjustment set. Second, we give a graphical characterization of a valid adjustment set that provides the optimal asymptotic variance among all valid adjustment sets. Our results depend only on the graphical structure and not on the specific error variances or edge coefficients of the underlying causal linear model. They can be applied …

Card 2 · Efficient adjustment sets for population average treatment effect estimation in non-parametric causal graphical models

The method of covariate adjustment is often used for estimation of population average causal treatment effects in observational studies. Graphical rules for determining all valid covariate adjustment sets from an assumed causal graphical model are well known. Restricting attention to causal linear m…

Read full passage excerpt

The method of covariate adjustment is often used for estimation of population average causal treatment effects in observational studies. Graphical rules for determining all valid covariate adjustment sets from an assumed causal graphical model are well known. Restricting attention to causal linear models, a recent article ( Henckel et al.,, 2019 ) derived two novel graphical criteria: one to compare the asymptotic variance of linear regression treatment effect estimators that control for certain distinct adjustment sets and another to identify the optimal adjustment set that yields the adjusted least squares treatment effect estimator with the smallest asymptotic variance among consistent adjusted least squares estimators.

Card 3 · Efficient adjustment sets in causal graphical models with hidden variables

Summary We study the selection of adjustment sets for estimating the interventional mean under a point exposure dynamic treatment regime, that is, a treatment rule that depends on the subject’s covariates. We assume a nonparametric causal graphical model with, possibly, hidden variables and at least…

Read full passage excerpt

Summary We study the selection of adjustment sets for estimating the interventional mean under a point exposure dynamic treatment regime, that is, a treatment rule that depends on the subject’s covariates. We assume a nonparametric causal graphical model with, possibly, hidden variables and at least one adjustment set comprised of observable variables. We provide the definition of a valid adjustment set for a point exposure dynamic treatment regime, which generalizes the existing definition for a static intervention. We show that there exists an adjustment set, referred to as optimal minimal, that yields the nonparametric estimator of the interventional mean with the smallest asymptotic variance among those that are based on observable minimal adjustment sets. An observable minimal adjustment set is a valid adjustment set such that all its variables are observable and the removal of any of its variables destroys its validity. We provide similar optimality results for the class of obser …

Card 4 · Towards unique and unbiased causal effect estimation from data with hidden variables

Causal effect estimation from observational data is a crucial but challenging task. Currently, only a limited number of data-driven causal effect estimation methods are available. These methods either provide only a bound estimation of the causal effect of a treatment on the outcome, or generate a u…

Read full passage excerpt

Causal effect estimation from observational data is a crucial but challenging task. Currently, only a limited number of data-driven causal effect estimation methods are available. These methods either provide only a bound estimation of the causal effect of a treatment on the outcome, or generate a unique estimation of the causal effect, but making strong assumptions on data and having low efficiency. In this paper, we identify a practical problem setting and propose an approach to achieving unique and unbiased estimation of causal effects from data with hidden variables. For the approach, we have developed the theorems to support the discovery of the proper covariate sets for confounding adjustment (adjustment sets). Based on the theorems, two algorithms are proposed for finding the proper adjustment sets from data with hidden variables to obtain unbiased and unique causal effect estimation. Experiments with synthetic datasets generated using five benchmark Bayesian networks and four r …

Model-reported card use

Card 2 · fact

Affected output: graphical criteria for comparing the asymptotic variance of estimators based on distinct valid adjustment sets and for identifying the adjustment set that yields the smallest asymptotic variance among consistent estimators

Model explanation: This row supplied the specific two-part content of the graphical criteria (variance comparison and optimal set identification) that the generated sentence reports as prior work.

Card 3 · fact

Affected output: an optimal minimal adjustment set that minimizes the asymptotic variance among estimators based on observable minimal adjustment sets

Model explanation: This row supplied the concrete optimality result for the hidden-variable setting, including the term 'optimal minimal adjustment set' and the scope 'among observable minimal adjustment sets', which the generated sentence attributes to Rotnitzky and Smucler (2019).

Card 1 · terminology

Affected output: comparing the asymptotic variance of estimators based on distinct valid adjustment sets

Model explanation: This row provided the field-native phrasing 'compare the asymptotic variances provided by valid adjustment sets', which shaped the terminology of the first clause.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

No continuation text was produced.

Model action：complete

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The with-card arm returned a card-use record but no continuation text. It is excluded from reported use with text and remains in all 280 first attempts.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.