FlowingFlowing
← Back to all development cases
Development case · blind model judgment pending independent human review; not a formal Benchmark conclusion.
AutocompleteEvidence version preferred95 / 162 · 5ab26034a3b6d744

Long-term modulation of solar cycles

Astrophysics · 2302.14845v1

FLOWING EVIDENCE BENCHMARK

How do we tell whether Evidence helps?

At the same writing position, with the same model and task, how does supplying retrieved paper passages change the first output? We compare matched versions and retain ties, unusable outputs and incomplete reviews.

SAME DRAFT · EVIDENCE ON OR OFF

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED INPUTS

SHARED BY BOTH

Manuscript context, model, task and prompt

A · With Evidence

Retrieved paper passages supplied

B · Without Evidence

No retrieved passages supplied

LLM

Same model and version

A → first output

B → first output

Blind judge agent

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Order check: X / Y → Y / X

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The judge sees the same draft and both first outputs without knowing which received Evidence.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are autocomplete positions stratified?

Before seeing generation outcomes, we check whether a retrieved passage contains a specific proposition that directly supports the next writing move. Those positions appear in the left opportunity group; the rest are ordinary positions on the right. We select a balanced sample from admitted papers in each field. The 50/50 split is experimental, not a measure of how often either type occurs in writing.

AUTOCOMPLETE · 110

Biology, statistics and astrophysics

55 positions on each side; statistics uses the ten-paper rerun.

AUTOCOMPLETE · 52

Psychology and climate science

26 positions on each side; psychology includes nine papers and climate science four.

How are the table percentages calculated?

Across five fields there are 81 source-opportunity positions. The original blind review preferred Evidence in 51; a task check moved one empty Evidence continuation to both unusable, leaving 50 in public counts. In the ordinary group, another pair of empty outputs moved from no winner to both unusable. Original verdicts remain visible on case pages.

50Evidence version preferred
÷
81All positions in this group
=
62%Evidence preference in this group

Source contribution is a separate review: 16 of 21 Evidence wins entered into source review directly used retrieved papers; another 29 wins await review.

These are development-stage model judgments pending independent human review. They are not formal Benchmark conclusions and do not, on their own, establish causality.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Long-term modulation of solar cycles

6 Implications for the dynamo theory > 6.1 Introduction to the solar dynamo

The solar magnetic cycle is maintained by a dynamo process, operating in the solar convection zone (SCZ). Thus, it is natural to expect that the variations in the solar cycle are caused by some mechanisms in the solar dynamo. Here we identify the causes of the variations in the solar cycle and demonstrate them by presenting results from some illustrative models. Let us first summarise the mechanism of the solar dynamo.

…so-called Babcock–Leighton process is clearly identified in the observed magnetic field data on the solar surface ( Mordvinov et al, 2022 , e.g.,) . The observed correlation between the polar field (or its proxy) at the solar minima and the amplitude of the next cycle ( Wang and Sheeley, 2009 ; Kitchatinov and Olemskoy, 2011 ; Muñoz-Jaramillo et al, 2013 ; Priyal et al, 2014 ) and the flux budgets of the observed and the generated poloidal and toroidal fields ( Cameron and Schüssler, 2015 ) suggest that the Babcock–Leighton process is possibly the main source of the poloidal field in the Sun.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionJudgment: Evidence version preferred

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

In Babcock–Leighton dynamo models, the poloidal flux generated at the surface must be transported back to the base of the convection zone, and a poleward meridional circulation at the solar surface provides the required return path.

Passages supplied to the Evidence version

A mean-field Babcock-Leighton solar dynamo model with long-term variability ↗

At the same time, the magnetic field of the following spot will also decay and connect with the polar magnetic field, which has opposite polarity. This process will at some point annihilate the flux in the polar region, causing a poloidal polarity reversal. In this case the poloidal and toroidal fie…

Read full passage excerpt

At the same time, the magnetic field of the following spot will also decay and connect with the polar magnetic field, which has opposite polarity. This process will at some point annihilate the flux in the polar region, causing a poloidal polarity reversal. In this case the poloidal and toroidal fields regeneration processes are spatially separated (recall Figure 1 h to Figure 1 j); a mechanism transporting the new polar surface magnetic flux generated by the Babcock-Leighton mechanism to the bottom of the convection zone is necessary. Facing this requirement, initially a meridional circulation flow was assumed in Babcock-Leighton dynamo modeling. As a matter of fact, a poleward meridional flow is indeed observed at the solar surface ( Duvall Jr,, 1979 ) .

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

In the Babcock–Leighton framework, the toroidal field generated by differential rotation near the base of the convection zone rises to the surface as tilted bipolar active regions, and the decay and dispersal of these regions supply the poloidal field that seeds the next cycle.

Judgment and task check

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

In this rationale, Output A = No Evidence; Output B = Evidence.

Output B is directly grounded in the supplied source, which explicitly states that 'a mechanism transporting the new polar surface magnetic flux generated by the Babcock-Leighton mechanism to the bottom of the convection zone is necessary' and that 'initially a meridional circulation flow was assumed in Babcock-Leighton dynamo modeling' with 'a poleward meridional flow is indeed observed at the solar surface.' Output A, while plausible dynamo physics, introduces multiple unsupported claims: (1) that toroidal field is 'generated by differential rotation near the base of the convection zone' — not mentioned in the source; (2) that it 'rises to the surface as tilted bipolar active regions' — not mentioned; (3) that 'decay and dispersal of these regions supply the poloidal field' — the source mentions decay and connection with polar field causing reversal, but not this specific causal chain as described. The draft's preceding paragraph already establishes the Babcock-Leighton process as identified in surface data and correlated with polar fields; Output B naturally continues by explaining the transport mechanism problem and solution that the source details, making it the more grounded completion.

Source review confirmed that this continuation directly used specific, verifiable information from the retrieved paper.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.