← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better85 / 280 · beu-fe6f5426982279c4

A Review on the Optimal Fingerprinting Approach in Climate Change Studies

Climate and environmental science · 2205.10508v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Review on the Optimal Fingerprinting Approach in Climate Change Studies

1 Introduction

The optimal fingering printing approach proposed in a series of papers starting from Allen and Tett (1999) (herein AT99) followed by Allen and Stott (2003) and Stott et al (2003) ), has been the corner stone in the detection and assessment of climate change due to human activities in the last 20 years, and has been the adopted methods by numerous Intergovernmental Panel on Climate Change (IPCC) reports on the climate change. Recently, McKitrick (2021) (M21) offered a very critical assessment on Allen and Tett (1999) , suggesting there were serious problems with the approach.

The criticism of McKitrick (2021) is well summarized in its abstract ”AT99 stated the GM Theorem incorrectly, omitting a critical condition altogether, their GLS method cannot satisfy the GM conditions, and their variance estimator is inconsistent by construction. Additionally, they did not formally state the null hypothesis of the RCT (residual consistency test) nor identify which of the GM conditions it tests, nor did they prove its distribution and critical values, rendering it uninformative as a specification test.”

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Output A provides a coherent, usable continuation that directly addresses the critique introduced in the draft. It appropriately scopes the significance of McKitrick's criticism by noting the implications for IPCC attribution confidence without overreaching or inventing specific results. The paragraph flows naturally from the quoted abstract summary, explaining that the critique targets statistical foundations (GM Theorem conditions, GLS optimality) and connects this to real-world consequences for climate attribution confidence. Output B is marked 'invalid' with empty text, making it unusable by definition regardless of any reasoning behind the invalidation. The draft clearly needs a next move—it introduces McKitrick's critique but has not yet explained its significance or scope—so an abstention would be inappropriate, and the 'invalid' action provides no usable content.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

No continuation text was produced.

Model action:invalid

Passages supplied to the Evidence version

Card 1 · Data-driven rate-optimal specification testing in regression models

We propose new data-driven smooth tests for a parametric regression function. The smoothing parameter is selected through a new criterion that favors a large smoothing parameter under the null hypothesis. The resulting test is adaptive rate-optimal and consistent against Pitman local alternatives ap…

Read full passage excerpt

We propose new data-driven smooth tests for a parametric regression function. The smoothing parameter is selected through a new criterion that favors a large smoothing parameter under the null hypothesis. The resulting test is adaptive rate-optimal and consistent against Pitman local alternatives approaching the parametric model at a rate arbitrarily close to $1/\sqrt{n}$. Asymptotic critical values come from the standard normal distribution and the bootstrap can be used in small samples. A general formalization allows one to consider a large class of linear smoothing methods, which can be tailored for detection of additive alternatives.

Card 2 · VALID HETEROSKEDASTICITY ROBUST TESTING

Tests based on heteroskedasticity robust standard errors are an important technique in econometric practice. Choosing the right critical value, however, is not simple at all: conventional critical values based on asymptotics often lead to severe size distortions, and so do existing adjustments inclu…

Read full passage excerpt

Tests based on heteroskedasticity robust standard errors are an important technique in econometric practice. Choosing the right critical value, however, is not simple at all: conventional critical values based on asymptotics often lead to severe size distortions, and so do existing adjustments including the bootstrap. To avoid these issues, we suggest to use smallest size-controlling critical values, the generic existence of which we prove in this article for the commonly used test statistics. Furthermore, sufficient and often also necessary conditions for their existence are given that are easy to check. Granted their existence, these critical values are the canonical choice: larger critical values result in unnecessary power loss, whereas smaller critical values lead to overrejections under the null hypothesis, make spurious discoveries more likely, and thus are invalid. We suggest algorithms to numerically determine the proposed critical values and provide implementations in accompa …

Card 3 · Corrected portmanteau tests for VAR models with time-varying variance

The problem of test of fit for Vector AutoRegressive (VAR) processes with unconditionally heteroscedastic errors is studied. The volatility structure is deterministic but time-varying and allows for changes that are commonly observed in economic or financial multivariate series. Our analysis is base…

Read full passage excerpt

The problem of test of fit for Vector AutoRegressive (VAR) processes with unconditionally heteroscedastic errors is studied. The volatility structure is deterministic but time-varying and allows for changes that are commonly observed in economic or financial multivariate series. Our analysis is based on the residual autocovariances and autocorrelations obtained from Ordinary Least Squares (OLS), Generalized Least Squares (GLS)and Adaptive Least Squares (ALS) estimation of the autoregressive parameters. The ALS approach is the GLS approach adapted to the unknown time-varying volatility that is then estimated by kernel smoothing. The properties of the three types of residual autocovariances and autocorrelations are derived. In particular it is shown that the ALS and GLS residual autocorrelations are asymptotically equivalent. It is also found that the asymptotic distribution of the OLS residual autocorrelations can be quite different from the standard chi-square asymptotic distribution o …

Card 4 · A Consistent Heteroskedasticity‐Robust LM‐Type Specification Test for Semiparametric Models

ABSTRACT This article develops a heteroskedasticity‐robust Lagrange Multiplier‐type specification test for semiparametric regression models. The test is able to detect a wide class of deviations from the null hypothesis. The test statistic is based on the estimates from the restricted semiparametric…

Read full passage excerpt

ABSTRACT This article develops a heteroskedasticity‐robust Lagrange Multiplier‐type specification test for semiparametric regression models. The test is able to detect a wide class of deviations from the null hypothesis. The test statistic is based on the estimates from the restricted semiparametric model, can be computed in a regression‐based way, and is asymptotically standard normal under the null. I apply the test to Environmental Engel Curves and find that while the relationship between income and emissions may be quadratic, its shape changes with other variables in the model. Monte Carlo studies suggest that my test controls size well, has good power, and is fairly robust to the tuning parameter choice.

Card 5 · Extremal Dependence-Based Specification Testing of Time Series

We propose a specification test for conditional location--scale models based on extremal dependence properties of the standardized residuals. We do so comparing the left-over serial extremal dependence -- as measured by the pre-asymptotic tail copula -- with that arising under serial independence at…

Read full passage excerpt

We propose a specification test for conditional location--scale models based on extremal dependence properties of the standardized residuals. We do so comparing the left-over serial extremal dependence -- as measured by the pre-asymptotic tail copula -- with that arising under serial independence at different lags. Our main theoretical results show that the proposed Portmanteau-type test statistics have nuisance parameter-free asymptotic limits. The test statistics are easy to compute, as they only depend on the standardized residuals, and critical values are likewise easily obtained from the limiting distributions. This contrasts with extant tests (based, e.g., on autocorrelations of squared residuals), where test statistics depend on the parameter estimator of the model and critical values may need to be bootstrapped. We show that our test performs well in simulations. An empirical application to S&P 500 constituents illustrates that our test can uncover violations of residual serial …

Model-reported card use

Card 2 · fact

Affected output: conventional asymptotic critical values often produce severe size distortions, and existing corrections including the bootstrap do not fully resolve the problem

Model explanation: This row supplied the specific econometric fact that conventional asymptotic critical values and bootstrap adjustments both suffer size distortions, which grounds the opening sentence's claim about specification testing.

Card 1 · terminology

Affected output: Specification testing

Model explanation: This row supplied the field-native term 'specification testing' used to frame the paragraph's opening move.

Card 5 · direction

Affected output: This literature provides a natural reference point for evaluating the residual consistency test

Model explanation: This row's framing of specification tests based on standardized residuals guided the choice to connect the econometric specification-testing literature to the residual consistency test under discussion.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

The critique centers on the statistical foundations of the optimal fingerprinting method, particularly the conditions under which generalized least squares estimators retain their optimality properties. These concerns, if valid, would have implications for the confidence attributed to attribution results reported in successive IPCC assessments.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The with-card arm returned a card-use record but no continuation text. It is excluded from reported use with text and remains in all 280 first attempts.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.