← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better34 / 280 · beu-f9504a2b075c6ab3

Incremental causal effects: an introduction and review

Statistics · 2110.10532v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Incremental causal effects: an introduction and review

7 Discussion

… In our time-varying setup, this corresponds to assuming that, given the past data, subjects’ values of future data are independent of whether or not they are censored. This assumption allows for identification and estimation of the estimand ψ ⁡ ( δ ) \psi(\delta) . The estimator is very similar to what we outlined in Section 4 , but includes an inverse weighting term for the probability of censoring. This is common in time-varying causal analyses with missing data (e.g., Robins et al. (2000) ), and takes the same form as inverse propensity weighting to account for missing potential outcomes.

…tes a few of its own unique problems. First, as we highlighted in Remarks 1 and 11 , we can allow δ \delta to vary with covariates and time. Second, we might hope to characterize the stochastic treatment intervention distribution in more detail than just the parameter δ \delta . For example, we may be interested in estimating the average number of treatments, the first treatment, or the modal number of treatments under the incremental propensity score intervention. In this case, we would need to develop an estimator for the distribution of treatment attendance over time for various δ \delta .

This note serves as an introduction to and a review of Kennedy (2019) , which describes a class of causal effects based on incremental

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Both outputs accurately continue the sentence about Kennedy (2019) and incremental propensity score interventions. Output A completes with a concise definition: 'propensity score interventions that shift the odds of treatment by a user-specified factor δ.' This is accurate and fits well. Output B provides the same core definition but adds crucial scholarly context: that identification is possible 'without requiring the propensity score to be bounded away from zero and one.' This additional information is highly relevant because (1) it reflects a key contribution of Kennedy (2019) explicitly noted in the sources—the relaxation of positivity restrictions, which distinguishes this work from standard approaches, and (2) it connects to the broader methodological significance highlighted in the sources about weaker overlap assumptions. The draft is an introductory/review sentence, so providing this distinguishing feature helps establish why the paper matters. Output B's extra clause is substantive, not redundant, and integrates seamlessly without requiring repair. While Output A is usable and correct, Output B better serves author utility by capturing the paper's key innovation in the same breath.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

propensity score interventions. The incremental propensity score intervention shifts the odds of treatment by a user-specified factor δ, and the resulting estimand can be identified and estimated without requiring the propensity score to be bounded away from zero and one.

Passages supplied to the Evidence version

Card 1 · Approximate Residual Balancing: Debiased Inference of Average Treatment Effects in High Dimensions

Summary There are many settings where researchers are interested in estimating average treatment effects and are willing to rely on the unconfoundedness assumption, which requires that the treatment assignment be as good as random conditional on pretreatment variables. The unconfoundedness assumptio…

Read full passage excerpt

Summary There are many settings where researchers are interested in estimating average treatment effects and are willing to rely on the unconfoundedness assumption, which requires that the treatment assignment be as good as random conditional on pretreatment variables. The unconfoundedness assumption is often more plausible if a large number of pretreatment variables are included in the analysis, but this can worsen the performance of standard approaches to treatment effect estimation. We develop a method for debiasing penalized regression adjustments to allow sparse regression methods like the lasso to be used for √n-consistent inference of average treatment effects in high dimensional linear models. Given linearity, we do not need to assume that the treatment propensities are estimable, or that the average treatment effect is a sparse contrast of the outcome model parameters. Rather, in addition to standard assumptions used to make lasso regression on the outcome model consistent und …

Card 2 · Nonparametric causal effects based on incremental propensity score interventions

In this section we first describe a new class of stochastic dynamic intervention, which we call incremental propensity score interventions, and give some motivation and examples. We go on to show that these interventions are nonparametrically identified without requiring any positivity restrictions…

Read full passage excerpt

In this section we first describe a new class of stochastic dynamic intervention, which we call incremental propensity score interventions, and give some motivation and examples. We go on to show that these interventions are nonparametrically identified without requiring any positivity restrictions on the propensity scores (e.g., the propensity scores do not need to be bounded away from zero and one). Then we describe the efficiency theory for estimating mean outcomes under these interventions, based on a new result for general stochastic interventions that depend on the observational treatment distribution.

Card 3 · Debiased Inverse Propensity Score Weighting for Estimation of Average Treatment Effects with High-Dimensional Confounders

We consider estimation of average treatment effects given observational data with high-dimensional pretreatment variables. Existing methods for this problem typically assume some form of sparsity for the regression functions. In this work, we introduce a debiased inverse propensity score weighting (…

Read full passage excerpt

We consider estimation of average treatment effects given observational data with high-dimensional pretreatment variables. Existing methods for this problem typically assume some form of sparsity for the regression functions. In this work, we introduce a debiased inverse propensity score weighting (DIPW) scheme for average treatment effect estimation that delivers $\sqrt{n}$-consistent estimates when the propensity score follows a sparse logistic regression model; the outcome regression functions are permitted to be arbitrarily complex. We further demonstrate how confidence intervals centred on our estimates may be constructed. Our theoretical results quantify the price to pay for permitting the regression functions to be unestimable, which shows up as an inflation of the variance of the estimator compared to the semiparametric efficient variance by a constant factor, under mild conditions. We also show that when outcome regressions can be estimated faster than a slow $1/\sqrt{ \log n} …

Card 4 · Assessing the causal effects of a stochastic intervention in time series data: Are heat alerts effective in preventing deaths and hospitalizations?

The methodological development of this paper is motivated by the need to address the following scientific question: does the issuance of heat alerts prevent adverse health effects? Our goal is to address this question within a causal inference framework in the context of time series data. A key chal…

Read full passage excerpt

The methodological development of this paper is motivated by the need to address the following scientific question: does the issuance of heat alerts prevent adverse health effects? Our goal is to address this question within a causal inference framework in the context of time series data. A key challenge is that causal inference methods require the overlap assumption to hold: each unit (i.e., a day) must have a positive probability of receiving the treatment (i.e., issuing a heat alert on that day). In our motivating example, the overlap assumption is often violated: the probability of issuing a heat alert on a cooler day is zero. To overcome this challenge, we propose a stochastic intervention for time series data which is implemented via an incremental time-varying propensity score (ItvPS). The ItvPS intervention is executed by multiplying the probability of issuing a heat alert on day $t$ -- conditional on past information up to day $t$ -- by an odds ratio $δ_t$. First, we introduce …

Card 5 · Nonparametric causal effects based on incremental propensity score interventions

In this section we first describe a new class of stochastic dynamic intervention, which we call incremental propensity score interventions, and give some motivation and examples. We go on to show that these interventions are nonparametrically identified without requiring any positivity restrictions…

Read full passage excerpt

In this section we first describe a new class of stochastic dynamic intervention, which we call incremental propensity score interventions, and give some motivation and examples. We go on to show that these interventions are nonparametrically identified without requiring any positivity restrictions on the propensity scores (e.g., the propensity scores do not need to be bounded away from zero and one). Then we describe the efficiency theory for estimating mean outcomes under these interventions, based on a new result for general stochastic interventions that depend on the observational treatment distribution.

Model-reported card use

Card 2 · terminology

Affected output: incremental propensity score intervention

Model explanation: The row supplies the field-native term for the class of interventions the sentence names.

Card 2 · fact

Affected output: can be identified and estimated without requiring the propensity score to be bounded away from zero and one

Model explanation: The row establishes the identification property that the sentence asserts about the incremental propensity score intervention.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

propensity score interventions that shift the odds of treatment by a user-specified factor δ.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Both outputs provide grounded continuations about incremental propensity score interventions. Output A completes with a concise description of the intervention mechanism (shifting odds by factor δ). Output B adds more substance: it also mentions identification/estimation without positivity restrictions (bounded away from 0 and 1), which is directly supported by the Kennedy (2019) source snippets that state 'these interventions are nonparametrically identified without requiring any positivity restrictions on the propensity scores.' Output B provides a more complete, useful sentence that captures both the mechanism and a key theoretical advantage, making it more informative for a review introduction. Both are factually grounded, but B better satisfies 'one useful sentence' by delivering more substantive content from the source.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.