FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better99 / 280 · beu-dffec37e9902bd85

A Review on the Optimal Fingerprinting Approach in Climate Change Studies

Climate and environmental science · 2205.10508v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Review on the Optimal Fingerprinting Approach in Climate Change Studies

4 Conclusion

…ol{\beta}} losing the BLUE property all together, but only make it a restricted BLUE as indicated in Remark 9 , provided 𝐂 ~ N = 𝐂 N \tilde{\mathbf{C}}_{N}=\mathbf{\mathbf{C}}_{N} . In the case of 𝐂 ~ N ≠ 𝐂 N \tilde{\mathbf{C}}_{N}\neq\mathbf{\mathbf{C}}_{N} , 𝜷 ~ \tilde{\boldsymbol{\beta}} would still be a consistent and unbiased estimator under standard conditions of the linear regression if the independence between ”the two worlds” is maintained by the simulations of the climate models. Our remarks made in 2.1 - 2.2 should address most of “The five assumptions” raised by McKitrick (2021) .

…T) proposed by AT99 is a valid test for checking on the agreement between the covariances ( 𝐂 ~ N \tilde{\mathbf{C}}_{N} vs 𝐂 N \mathbf{\mathbf{C}}_{N} ) of the two worlds as shown in 2.3 , which means that a proper control of the probability of the type I error (rejecting when the underlying covariance matrix of the null simulation does match 𝐂 N \mathbf{\mathbf{C}}_{N} of the physical world) can be realized. As mentioned in Remarks 11 and 12 , its control on the probability of the type II error (not rejecting when the covariances of the two worlds are actually different) need further study.

Moreover, we outline the connection between the ”optimal fingerprinting” approach and the Feasible Generalized Least Square, which we hope would connect the field of climate change studies

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs complete the sentence appropriately by connecting climate change studies to broader statistical literature. Output B is preferable because it specifies 'with estimated covariance matrices,' which is more precise and accurate. The 'optimal fingerprinting' approach in climate science specifically involves generalized least squares with an estimated covariance structure (accounting for natural variability), not just generic linear regression and hypothesis testing. Output B's narrower scope better matches the technical content of the Feasible Generalized Least Squares method, where the covariance matrix is estimated rather than known. Output A's broader phrasing ('hypothesis testing') introduces a slightly tangential element that isn't directly implied by the FGLS connection.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

to the broader statistical literature on linear regression and hypothesis testing.

Passages supplied to the Evidence version

Card 1 · Violation of the Sphericity Assumption and Its Effect on Type-I Error Rates in Repeated Measures ANOVA and Multi-Level Linear Models (MLM)

We investigated the effects of violations of the sphericity assumption on Type I error rates for different methodical approaches of repeated measures analysis using a simulation approach. In contrast to previous simulation studies on this topic, up to nine measurement occasions were considered. Effe…

Read full passage excerpt

We investigated the effects of violations of the sphericity assumption on Type I error rates for different methodical approaches of repeated measures analysis using a simulation approach. In contrast to previous simulation studies on this topic, up to nine measurement occasions were considered. Effects of the level of inter-correlations between measurement occasions on Type I error rates were considered for the first time. Two populations with non-violation of the sphericity assumption, one with uncorrelated measurement occasions and one with moderately correlated measurement occasions, were generated. One population with violation of the sphericity assumption combines uncorrelated with highly correlated measurement occasions. A second population with violation of the sphericity assumption combines moderately correlated and highly correlated measurement occasions. From these four populations without any between-group effect or within-subject effect 5,000 random samples were drawn. Fina …

Card 2 · Sharp minimax tests for large covariance matrices and adaptation

We consider the detection problem of correlations in a $p$-dimensional Gaussian vector, when we observe $n$ independent, identically distributed random vectors, for $n$ and $p$ large. We assume that the covariance matrix varies in some ellipsoid with parameter $\alpha >1/2$ and total energy bounded…

Read full passage excerpt

We consider the detection problem of correlations in a $p$-dimensional Gaussian vector, when we observe $n$ independent, identically distributed random vectors, for $n$ and $p$ large. We assume that the covariance matrix varies in some ellipsoid with parameter $\alpha >1/2$ and total energy bounded by $L>0$. We propose a test procedure based on a U-statistic of order 2 which is weighted in an optimal way. The weights are the solution of an optimization problem, they are constant on each diagonal and non-null only for the $T$ first diagonals, where $T=o(p)$. We show that this test statistic is asymptotically Gaussian distributed under the null hypothesis and also under the alternative hypothesis for matrices close to the detection boundary. We prove upper bounds for the total error probability of our test procedure, for $\alpha>1/2$ and under the assumption $T=o(p)$ which implies that $n=o(p^{2\alpha})$. We illustrate via a numerical study the behavior of our test procedure. Moreover, w …

Card 3 · Sharp minimax tests for large Toeplitz covariance matrices with repeated observations

Note that, most of the literature on testing the null hypothesis ( 1 ), either focus on finding the asymptotic behavior of the test statistic under the null hypothesis, or control in addition the type II error probability for one fixed unknown matrix under the alternative, whereas our main interest…

Read full passage excerpt

Note that, most of the literature on testing the null hypothesis ( 1 ), either focus on finding the asymptotic behavior of the test statistic under the null hypothesis, or control in addition the type II error probability for one fixed unknown matrix under the alternative, whereas our main interest is to quantify the worst type II error probabilities, i.e. uniformly over a large set of possible covariance matrices.

Card 4 · Performance of Statistical Tests for Single-Source Detection Using Random Matrix Theory

This paper introduces a unified framework for the detection of a single source with a sensor array in the context where the noise variance and the channel between the source and the sensors are unknown at the receiver. The Generalized Maximum Likelihood Test is studied and yields the analysis of the…

Read full passage excerpt

This paper introduces a unified framework for the detection of a single source with a sensor array in the context where the noise variance and the channel between the source and the sensors are unknown at the receiver. The Generalized Maximum Likelihood Test is studied and yields the analysis of the ratio between the maximum eigenvalue of the sampled covariance matrix and its normalized trace. Using recent results from random matrix theory, a practical way to evaluate the threshold and thep-value of the test is provided in the asymptotic regime where the numberKof sensors and the numberNof observations per sensor are large but have the same order of magnitude. The theoretical performance of the test is then analyzed in terms of Receiver Operating Characteristic (ROC) curve. It is, in particular, proved that both Type I and Type II error probabilities converge to zero exponentially as the dimensions increase at the same rate, and closed-form expressions are provided for the error expone …

Card 5 · Sharp minimax tests for large Toeplitz covariance matrices with repeated observations

Note that, most of the literature on testing the null hypothesis ( 1 ), either focus on finding the asymptotic behavior of the test statistic under the null hypothesis, or control in addition the type II error probability for one fixed unknown matrix under the alternative, whereas our main interest…

Read full passage excerpt

Note that, most of the literature on testing the null hypothesis ( 1 ), either focus on finding the asymptotic behavior of the test statistic under the null hypothesis, or control in addition the type II error probability for one fixed unknown matrix under the alternative, whereas our main interest is to quantify the worst type II error probabilities, i.e. uniformly over a large set of possible covariance matrices.

Model-reported card use

The model did not report that any card materially affected this completion.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

to the broader statistical literature on linear regression with estimated covariance matrices.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

This case does not have both with-card continuation text and a model-reported use record. It remains among all 280 tests but does not enter the reported-use comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.