FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better81 / 280 · beu-e388652256443e63

Reduction Methods in Climate Dynamics -- A Brief Review

Climate and environmental science · 2209.12240v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Reduction Methods in Climate Dynamics -- A Brief Review

1. Introduction

…a high resolution on very powerful computers but typically only short runs can be achieved at high resolution. For some questions that involve longer timescales, one can consider a hierarchy of models [ 145 ] where high resolution ESM/GCMs can be approximated using lower resolution ESMs and simpler Earth system models of intermediate complexity (EMICs). These models usually contain several simplifications and a larger grid size compared to the ESMs/GCMs and they therefore need less computing time. However, EMICs are still mostly too complex for analytical study without further simplification.

…e formally linked through a rigorous reduction procedure. For example, one of the key problems in climate science is the determination of Equilibrium Climate Sensitivity (ECS) [ 26 ] . This is defined to be the equilibrium increase in global mean surface temperature that results from a doubling of CO 2 in the atmosphere and as such is the key scientific link between anthropogenic emissions and the climate [ 89 , 61 ] . A rigorous justification of ECS would need to span the model hierarchy as it involves reduction of ESM runs to an equation that just considers global radiation balance [ 68 ] .

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs provide coherent continuations, but they serve different argumentative purposes. The draft ends by stating that 'A rigorous justification of ECS would need to span the model hierarchy as it involves reduction of ESM runs to an equation that just considers global radiation balance.' Output A shifts to a methodological concern about statistical discrepancy models affecting ECS inference—this is factually grounded in the sources but introduces a somewhat tangential point about inference uncertainty rather than completing the reduction argument. Output B directly addresses the conceptual necessity of the reduction procedure itself, explaining why isolating global radiation balance matters for understanding ECS feedbacks. This fits more tightly with the draft's closing emphasis on 'reduction of ESM runs to an equation that just considers global radiation balance.' Output B completes the argumentative move about model hierarchy and reduction that the draft initiated, whereas Output A pivots to a related but distinct concern about statistical methodology. The draft's focus on 'rigorous justification' spanning the hierarchy is better served by B's conceptual unpacking of why the reduction is necessary than by A's shift to discrepancy model dependence.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

A central challenge in this context is that the inferred value of ECS can depend on the statistical model used to represent structural discrepancies between the reduced energy balance model and the full ESM output.

Passages supplied to the Evidence version

Card 1 · Dependence of Inferred Climate Sensitivity on the Discrepancy Model

We consider the effect of different temporal error structures on the inference of equilibrium climate sensitivity 1 1 1 ECS is defined as the realized equilibrium surface warming—globally-averaged surface air temperature—for a doubling of CO 2 (ECS), in the context of an energy balance model (EBM) t…

Read full passage excerpt

We consider the effect of different temporal error structures on the inference of equilibrium climate sensitivity 1 1 1 ECS is defined as the realized equilibrium surface warming—globally-averaged surface air temperature—for a doubling of CO 2 (ECS), in the context of an energy balance model (EBM) that is commonly employed in analyzing earth system models (ESM) and observations. We consider error structures ranging from uncorrelated (IID normal) to AR(1) to Gaussian correlation (Gaussian Process GP) to analyze the abrupt 4xCO 2 CMIP5 experiment in twenty-one different ESMs. For seven of the ESMs, the posterior distribution of ECS is seen to depend rather weakly on the discrepancy model used suggesting that the discrepancies were largely uncorrelated.

Card 2 · Climate response and sensitivity: time scales and late tipping points

Climate response metrics are used to quantify the Earth’s climate response to anthropogenic changes of atmospheric CO 2 . Equilibrium climate sensitivity (ECS) is one such metric that measures the equilibrium response to CO 2 doubling. However, both in their estimation and their usage, such metrics…

Read full passage excerpt

Climate response metrics are used to quantify the Earth’s climate response to anthropogenic changes of atmospheric CO 2 . Equilibrium climate sensitivity (ECS) is one such metric that measures the equilibrium response to CO 2 doubling. However, both in their estimation and their usage, such metrics make assumptions on the linearity of climate response, although it is known that, especially for larger forcing levels, response can be nonlinear. Such nonlinear responses may become visible immediately in response to a larger perturbation, or may only become apparent after a long transient period. In this paper, we illustrate some potential problems and caveats when estimating ECS from transient simulations. We highlight ways that very slow time scales may lead to poor estimation of ECS even if there is seemingly good fit to linear response over moderate time scales. Moreover, such slow processes might lead to late abrupt responses (late tipping points) associated with a system’s nonlineari …

Card 3 · Extreme Sensitivity and Climate Tipping Points

Abstract A climate state close to a tipping point will have a degenerate linear response to perturbations, which can be associated with extreme values of the equilibrium climate sensitivity (ECS). In this paper we contrast linearized (‘instantaneous’) with fully nonlinear geometric (‘two-point’) not…

Read full passage excerpt

Abstract A climate state close to a tipping point will have a degenerate linear response to perturbations, which can be associated with extreme values of the equilibrium climate sensitivity (ECS). In this paper we contrast linearized (‘instantaneous’) with fully nonlinear geometric (‘two-point’) notions of ECS, in both presence and absence of tipping points. For a stochastic energy balance model of the global mean surface temperature with two stable regimes, we confirm that tipping events cause the appearance of extremes in both notions of ECS. Moreover, multiple regimes with different mean sensitivities are visible in the two-point ECS. We confirm some of our findings in a physics-based multi-box model of the climate system.

Card 4 · Beyond Forcing Scenarios: Predicting Climate Change through Response Operators in a Coupled General Circulation Model

Abstract Global Climate Models are key tools for predicting the future response of the climate system to a variety of natural and anthropogenic forcings. Here we show how to use statistical mechanics to construct operators able to flexibly predict climate change. We perform our study using a fully c…

Read full passage excerpt

Abstract Global Climate Models are key tools for predicting the future response of the climate system to a variety of natural and anthropogenic forcings. Here we show how to use statistical mechanics to construct operators able to flexibly predict climate change. We perform our study using a fully coupled model - MPI-ESM v.1.2 - and for the first time we prove the effectiveness of response theory in predicting future climate response to CO2 increase on a vast range of temporal scales, from inter-annual to centennial, and for very diverse climatic variables. We investigate within a unified perspective the transient climate response and the equilibrium climate sensitivity, and assess the role of fast and slow processes. The prediction of the ocean heat uptake highlights the very slow relaxation to a newly established steady state. The change in the Atlantic Meridional Overturning Circulation (AMOC) and of the Antarctic Circumpolar Current (ACC) is accurately predicted. The AMOC strength …

Card 5 · Dependence of Inferred Climate Sensitivity on the Discrepancy Model

We consider the effect of different temporal error structures on the inference of equilibrium climate sensitivity 1 1 1 ECS is defined as the realized equilibrium surface warming—globally-averaged surface air temperature—for a doubling of CO 2 (ECS), in the context of an energy balance model (EBM) t…

Read full passage excerpt

We consider the effect of different temporal error structures on the inference of equilibrium climate sensitivity 1 1 1 ECS is defined as the realized equilibrium surface warming—globally-averaged surface air temperature—for a doubling of CO 2 (ECS), in the context of an energy balance model (EBM) that is commonly employed in analyzing earth system models (ESM) and observations. We consider error structures ranging from uncorrelated (IID normal) to AR(1) to Gaussian correlation (Gaussian Process GP) to analyze the abrupt 4xCO 2 CMIP5 experiment in twenty-one different ESMs. For seven of the ESMs, the posterior distribution of ECS is seen to depend rather weakly on the discrepancy model used suggesting that the discrepancies were largely uncorrelated.

Model-reported card use

Card 1 · fact

Affected output: the inferred value of ECS can depend on the statistical model used to represent structural discrepancies between the reduced energy balance model and the full ESM output

Model explanation: This row supplies the concrete fact that ECS inference from an energy balance model depends on the discrepancy model, which is the substantive content of the new paragraph.

Card 2 · terminology

Affected output: structural discrepancies

Model explanation: This row's framing of assumptions underlying ECS estimation informed the use of 'structural discrepancies' to describe the gap between reduced and full models.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

The reduction from an ESM to a global radiation balance equation is not merely a practical convenience but a conceptual necessity: it isolates the feedbacks that determine ECS from the internal variability and spatial detail that obscure them.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output A is grounded in the sources: the first source explicitly discusses how 'the posterior distribution of ECS is seen to depend' on 'the discrepancy model used' when inferring ECS from ESMs via energy balance models, which directly supports the claim about inferred ECS depending on the statistical model for discrepancies. This continues the paragraph's theme of rigorous reduction procedures and ECS justification. Output B makes an unsupported claim that the reduction 'isolates the feedbacks that determine ECS from the internal variability and spatial detail that obscure them' — this presents a purpose/benefit ('conceptual necessity,' 'isolates feedbacks,' 'obscure them') that is not stated in any source. The sources discuss ECS inference, discrepancy models, and timescales, but none state that reduction isolates feedbacks from internal variability or that spatial detail 'obscures' feedbacks. This is an invented explanatory mechanism.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.