FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better78 / 280 · beu-b032369aa66f6bdf

A Robust Bayesian Meta-Analysis for Estimating the Hubble Constant via Time Delay Cosmography

Psychology and cognitive neuroscience · 2308.13018v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Robust Bayesian Meta-Analysis for Estimating the Hubble Constant via Time Delay Cosmography

1 Introduction

… have improved data quality and inferential accuracy of existing methods, for example, Riess et al. (2019) , Riess et al. (2021) , and Riess et al. (2022) . Various methods have also been developed under different physical processes to better understand the systematic error and test the standard cosmological model.

…. The faint image at the center is the intervening lensing galaxy, and the two bright images located to the south and north from the center are the two lensed images of the same quasar. We call it a strong gravitational lensing effect ( Schneider, Wambsganss and Kochanek, 2006 ; Treu, 2010 ) . The travel times of light photons for each lensed image can be different depending on the paths they take because the length of each path may differ and photons may pass through different gravitational potential of the intervening galaxy. We call such differences between their travel times time delays .

For each pair of lensed images (one unique pair for a doubly-lensed system, and three unique pairs for a quadruply-lensed system), time delay cosmography models the additional travel distance of the longer route by a physical equation that interweaves cosmological parameters and measurable quantities. Two of the measurable quantities are time delay and Fermat potential difference, and these can be separately estimated from two different types of data. Time delay estimates are obtained from several time series data of brightness of multiply-lensed quasar images ( Tewes et al., 2013 ; Eulaers et al., 2013 ; Liao et al., 2015 ; Tak et al., 2017 ; Courbin et al., 2018 ; Bonvin

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs complete the citation list and introduce Fermat potential differences as the second measurable quantity, fitting the draft's structure. However, Output A is superior in fit and scope. The draft already establishes that time delays come from 'time series data of brightness' and names specific papers for that quantity. Output A continues this pattern cleanly: it finishes the time-delay citations with two more recent papers, then states Fermat potential differences 'are inferred from high-resolution imaging of the lens and its environment.' This is appropriately scoped—general enough to be accurate without overcommitting to specifics. Output B adds parenthetical citations (Suyu et al., 2010; Wong et al., 2020) for Fermat potential differences. While plausible, these citations are not verified by the provided sources, and more importantly, they risk being slightly mismatched: Wong et al., 2020 is not in the source snippets, and Suyu et al., 2010 is absent too. The draft's pattern for time delays was to cite papers without specifying what each paper did; Output B breaks this parallelism by adding a new citation format and specific papers that aren't source-verified. Output A's 'lens and its environment' also better matches the source snippets' emphasis on lensing galaxies, clusters, and line-of-sight structure (e.g., SBS 1520+530 sources discuss 'galaxy cluster' and 'groups or clusters of galaxies along the line of sight'). Output B's 'lens system' is slightly narrower. Both are usable, but Output A requires less repair and maintains tighter consistency with the draft's established style and scope.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

et al., 2019; Millon et al., 2020), whereas Fermat potential differences are inferred from high-resolution imaging of the lens and its environment.

Passages supplied to the Evidence version

Card 1 · Bayesian estimates of astronomical time delays between gravitationally lensed stochastic light curves

The gravitational field of a galaxy can act as a lens and deflect the light emitted by a more distant object such as a quasar. Strong gravitational lensing causes multiple images of the same quasar to appear in the sky. Since the light in each gravitationally lensed image traverses a different path…

Read full passage excerpt

The gravitational field of a galaxy can act as a lens and deflect the light emitted by a more distant object such as a quasar. Strong gravitational lensing causes multiple images of the same quasar to appear in the sky. Since the light in each gravitationally lensed image traverses a different path length from the quasar to the Earth, fluctuations in the source brightness are observed in the several images at different times. The time delay between these fluctuations can be used to constrain cosmological parameters and can be inferred from the time series of brightness data or light curves of each image. To estimate the time delay, we construct a model based on a state-space representation for irregularly observed time series generated by a latent continuous-time Ornstein–Uhlenbeck process. We account for microlensing, an additional source of independent long-term extrinsic variability, via a polynomial regression. Our Bayesian strategy adopts a Metropolis–Hastings within Gibbs sampler …

Card 2 · iPTF16geu: A multiply imaged, gravitationally lensed type Ia supernova

= 0.409. This phenomenon was identified because the light from the stellar explosion was magnified more than 50 times by the curvature of space around matter in an intervening galaxy. We used high-spatial-resolution observations to resolve four images of the lensed supernova, approximately 0.3 arc s…

Read full passage excerpt

= 0.409. This phenomenon was identified because the light from the stellar explosion was magnified more than 50 times by the curvature of space around matter in an intervening galaxy. We used high-spatial-resolution observations to resolve four images of the lensed supernova, approximately 0.3 arc seconds from the center of the foreground galaxy. The observations probe a physical scale of ~1 kiloparsec, smaller than is typical in other studies of extragalactic gravitational lensing. The large magnification and symmetric image configuration imply close alignment between the lines of sight to the supernova and to the lens. The relative magnifications of the four images provide evidence for substructures in the lensing galaxy.

Card 3 · The Rewards of Patience: An 822 Day Time Delay in the Gravitational Lens SDSS J1004+4112

We present 107 new epochs of optical monitoring data for the four brightest images of the gravitational lens SDSS J1004+4112 observed between 2006 October and 2007 June. Combining this data with the previously obtained light curves, we determine the time delays between images A, B, and C. We confirm…

Read full passage excerpt

We present 107 new epochs of optical monitoring data for the four brightest images of the gravitational lens SDSS J1004+4112 observed between 2006 October and 2007 June. Combining this data with the previously obtained light curves, we determine the time delays between images A, B, and C. We confirm our previous measurement, that B leads A by Δ τ BA = 40.6 ± 1.8 days, and find that image C leads image A by Δ τ CA = 821.6 ± 2.1 days. The lower limit on the remaining delay is that image D lags image A by Δ τ AD > 1250 days. Based on the microlensing of images A and B, we estimate that the accretion disk size at a rest wavelength of 2300 Å is 10 14.8 ± 0.3 cm for a disk inclination of , which is consistent with the microlensing disk size-black hole mass correlation function given our estimate of the black hole mass from the Mg II line width of log M BH / M ☉ = 8.4 ± 0.2. The long delays allow us to fill in the seasonal gaps and assemble a continuous, densely sampled light curve spanning 5 …

Card 4 · The lensing system towards the doubly imaged quasar SBS 1520+530

The gravitational potential responsible for the lensing effect in SBS 1520+530 is studied over length scales from a few arc-seconds to a few arc-minutes. For this purpose, we use sharply deconvolved Hubble Space Telescope images in the optical and near-IR, in combination with ground based optical da…

Read full passage excerpt

The gravitational potential responsible for the lensing effect in SBS 1520+530 is studied over length scales from a few arc-seconds to a few arc-minutes. For this purpose, we use sharply deconvolved Hubble Space Telescope images in the optical and near-IR, in combination with ground based optical data obtained over a wider field-of-view. In particular, we have carried out a multi-color analysis in order to identify groups or clusters of galaxies along the line of sight. Photometric redshifts are measured for 139 galaxies unveiling significant excesses of galaxies 1.0´ NW and 1.7´ SW of the main lensing galaxy. The photometric redshift inferred both for the main lensing galaxy and for the galaxy concentrations is . This is in rough agreement with the measured spectroscopic redshift of the main lensing galaxy, (Burud et al. [CITE]), suggesting that it is part of a larger group or cluster. We investigate the impact of including the galaxy cluster, first on the modelling of the lensing sys …

Card 5 · Time delay and lens redshift for the doubly imaged BAL quasar SBS 1520+530

We present optical R-band light curves of the gravitationally lensed quasar SBS 1520+530 derived from data obtained at the Nordic Optical Telescope. A time delay of days (1σ) is determined from the light curves. In addition, spectra of SBS 1520+530 obtained at the Keck Observatory are spatially deco…

Read full passage excerpt

We present optical R-band light curves of the gravitationally lensed quasar SBS 1520+530 derived from data obtained at the Nordic Optical Telescope. A time delay of days (1σ) is determined from the light curves. In addition, spectra of SBS 1520+530 obtained at the Keck Observatory are spatially deconvolved in order to extract the spectrum of the faint lensing galaxy, free of any contamination by the light from the bright quasar images. This spectrum indicates a lens redshift , in agreement with one of the absorption systems found in the quasar spectra. The best mass model of the system includes a second nearby galaxy and a cluster of galaxies in addition to the main lensing galaxy. Adopting this model and an , cosmology, our time-delay measurement yields a Hubble constant of (1σ error).

Model-reported card use

The model did not report that any card materially affected this completion.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

et al., 2019 ; Millon et al., 2020 ) , while Fermat potential differences are inferred from high-resolution imaging of the lens system ( Suyu et al., 2010 ; Wong et al., 2020 ) .

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

This case does not have both with-card continuation text and a model-reported use record. It remains among all 280 tests but does not enter the reported-use comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.