← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better173 / 280 · beu-88a7a4d6069f4a9c

Tutorial: Introduction to computational causal inference using reproducible Stata, R and Python code

Statistics · 2012.09920v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Tutorial: Introduction to computational causal inference using reproducible Stata, R and Python code

8 Conclusion

Overall, all the methods introduced here include the estimation of the G-formula (non-parametrically or parametrically), which is a generalization of standardization, and the inverse probability of treatment weighting (IPTW). [ 4 ] However, there are other estimators based on matching strategies that we did not discuss in our tutorial. ( 18 )

…different estimators we introduced in the tutorial. Overall, all the methods showed a consistent result for the ATE. In a well-balanced data set, all of the methods used to address confounding consistently provide the same result ( 3 ). The RHC data (demonstrated in this paper) is used to teach causal inference methods because of its extremely well balanced distribution of confounders across levels of the treatment (RHC). However, in most observational studies, data is not usually well balanced and there are potentially near violations of the positivity assumption that must always be checked.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

The draft ends with a critical methodological point: that most observational studies have poorly balanced data with potential positivity violations, unlike the well-balanced RHC example. This creates a clear argumentative need for a continuation that addresses what to do in these more common, problematic settings. Output A directly addresses this by discussing estimator choice consequences and the need for overlap/positivity diagnostics—precisely the logical next move given the draft's warning about real-world data challenges. Output B, while not invalid, pivots to a generic 'this is just a starting point' conclusion and mentions reproducible code, which is not referenced in the visible draft and constitutes a broader framing move that ignores the specific methodological tension the author just raised. The draft does not mention reproducible code, making B's reference to it somewhat unmoored. A maintains the argumentative thread about estimator performance under practical constraints; B changes the subject.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

In such settings, the choice among estimators becomes consequential, and analysts should report diagnostics for overlap and positivity alongside the effect estimate.

Passages supplied to the Evidence version

Card 1 · Multiply Robust Causal Inference with Double Negative Control Adjustment for Categorical Unmeasured Confounding

Unmeasured confounding is a threat to causal inference in observational studies. In recent years, use of negative controls to mitigate unmeasured confounding has gained increasing recognition and popularity. Negative controls have a longstanding tradition in laboratory sciences and epidemiology to r…

Read full passage excerpt

Unmeasured confounding is a threat to causal inference in observational studies. In recent years, use of negative controls to mitigate unmeasured confounding has gained increasing recognition and popularity. Negative controls have a longstanding tradition in laboratory sciences and epidemiology to rule out non-causal explanations, although they have been used primarily for bias detection. Recently, Miao et al. (2018) have described sufficient conditions under which a pair of negative control exposure and outcome variables can be used to nonparametrically identify the average treatment effect (ATE) from observational data subject to uncontrolled confounding. In this paper, we establish nonparametric identification of the ATE under weaker conditions in the case of categorical unmeasured confounding and negative control variables. We also provide a general semiparametric framework for obtaining inferences about the ATE while leveraging information about a possibly large number of measured …

Card 2 · Bias and High-Dimensional Adjustment in Observational Studies of Peer Effects

Peer effects, in which an individual’s behavior is affected by peers’ behavior, are posited by multiple theories in the social sciences. Randomized field experiments that identify peer effects, however, are often expensive or infeasible, so many studies of peer effects use observational data, which…

Read full passage excerpt

Peer effects, in which an individual’s behavior is affected by peers’ behavior, are posited by multiple theories in the social sciences. Randomized field experiments that identify peer effects, however, are often expensive or infeasible, so many studies of peer effects use observational data, which is expected to suffer from confounding. Here we show, in the context of information and media diffusion, that high-dimensional adjustment of a nonexperimental control group (660 million observations) using propensity score models produces estimates of peer effects statistically indistinguishable from those using a large randomized experiment (215 million observations). Compared with the experiment, naive observational estimators overstate peer effects by over 300% and commonly available variables (e.g., demographics) offer little bias reduction. Adjusting for a measure of prior behaviors closely related to the focal behavior reduces this bias by 91%, while models adjusting for over 3700 past …

Card 3 · Causal Inference through a Witness Protection Program

One of the most fundamental problems in causal inference is the estimation of a causal effect when variables are confounded. This is difficult in an observational study, because one has no direct evidence that all confounders have been adjusted for. We introduce a novel approach for estimating causa…

Read full passage excerpt

One of the most fundamental problems in causal inference is the estimation of a causal effect when variables are confounded. This is difficult in an observational study, because one has no direct evidence that all confounders have been adjusted for. We introduce a novel approach for estimating causal effects that exploits observational conditional independencies to suggest "weak" paths in a unknown causal graph. The widely used faithfulness condition of Spirtes et al. is relaxed to allow for varying degrees of "path cancellations" that imply conditional independencies but do not rule out the existence of confounding causal paths. The outcome is a posterior distribution over bounds on the average causal effect via a linear programming approach and Bayesian inference. We claim this approach should be used in regular practice along with other default tools in observational studies.

Card 5 · A Gaussian Process Framework for Overlap and Causal Effect Estimation with High-Dimensional Covariates

Abstract A powerful tool for the analysis of nonrandomized observational studies has been the potential outcomes model. Utilization of this framework allows analysts to estimate average treatment effects. This article considers the situation in which high-dimensional covariates are present and revis…

Read full passage excerpt

Abstract A powerful tool for the analysis of nonrandomized observational studies has been the potential outcomes model. Utilization of this framework allows analysts to estimate average treatment effects. This article considers the situation in which high-dimensional covariates are present and revisits the standard assumptions made in causal inference. We show that by employing a flexible Gaussian process framework, the assumption of strict overlap leads to very restrictive assumptions about the distribution of covariates, results for which can be characterized using classical results from Gaussian random measures as well as reproducing kernel Hilbert space theory. In addition, we propose a strategy for data-adaptive causal effect estimation that does not rely on the strict overlap assumption. These findings reveal under a focused framework the stringency that accompanies the use of the treatment positivity assumption in high-dimensional settings.

Model-reported card use

Card 4 · direction

Affected output: analysts should report diagnostics for overlap and positivity alongside the effect estimate

Model explanation: This row's emphasis on the stringency of the positivity assumption in high-dimensional settings guided the choice to close the conclusion by directing attention to overlap and positivity diagnostics rather than to estimator performance.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

Beyond the estimators covered here, the broader family of causal methods continues to expand, and readers should treat this tutorial as a starting point rather than an exhaustive survey. The reproducible code provided for each method is intended to lower the barrier to applying these approaches in practice, and we encourage readers to adapt it to their own data and research questions.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blind model judge did not prefer either output.

Read the original model judgment

Order-swapped judges disagreed. Pass 1: Both outputs are usable continuations, but B better satisfies the task requirements. The draft discusses specific estimators (G-formula, IPTW, matching) and their behavior on well-balanced vs. unbalanced data, ending with a warning about positivity violations. Output B provides a natural, coherent paragraph that: (1) acknowledges the tutorial's scope limitation ('starting point rather than an exhaustive survey'), which connects to the prior sentence about matching strategies not discussed; (2) transitions to practical next steps with reproducible code, which fits the tutorial context. Output A, while topically relevant to positivity, is too short (one sentence) and reads more like a trailing clause than a complete paragraph. The task asks for 'one coherent academic paragraph,' and B delivers substantive paragraph-length content that bridges the technical discussion to practical implementation. A's brevity and somewhat abrupt diagnostic recommendation doesn't flow as naturally from the draft's closing emphasis on the RHC data's teaching purpose and the general warning about observational studies. Pass 2: Both outputs provide coherent continuations, but B is more useful and precisely targeted. The draft explicitly discusses that 'in most observational studies, data is not usually well balanced and there are potentially near violations of the positivity assumption that must always be checked.' Output B directly follows this thread by discussing the consequentiality of estimator choice in such settings and the need to report overlap/positivity diagnostics—this is grounded in the draft's actual content and concerns. Output A, while not unsupported, introduces new topics (reproducible code, exhaustive survey, starting point) that are not mentioned in the draft and represent a broader, more generic conclusion that doesn't continue the specific line of reasoning about balance, positivity, and estimator behavior in observational studies. The draft's paragraph builds from consistent results in well-balanced data to the more realistic, problematic case of poorly balanced data; B continues this logical arc while A pivots away to meta-commentary about the tutorial itself.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.