FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better49 / 280 · beu-59ebf8f7756fb9f8

The DAWES review: Gravitational-wave burst astrophysics

Astrophysics · 2410.12105v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

The DAWES review: Gravitational-wave burst astrophysics

12 conclusions

…magnetic or neutrino counterpart. The observed waveform could, in principle, be used to place limits on the characteristic mass and size of the progenitor system, albeit contingent on the likely unknown distance ( Bécsy et al., 2020 ) . Such constraints would be based on there being a maximum compactness and a maximum luminosity for any potential gravitational-wave source. If the luminosity distance to the source is known from an electromagnetic counterpart, we could also place constraints on the gravitational-wave energy, and set lower limits on the mass and size of the astrophysical source.

…l relativity, and the first direct evidence that binary neutron stars are the progenitors of short GRBs. However, this is just the beginning. The next generation of observatories will hear every binary black hole merger in the observable Universe, and almost all of the binary neutron star mergers well back before the peak of star formation. However, there are many other potential sources of gravitational waves that have yet to be discovered. A detection of a burst of gravitational waves from a different type of source is one of the most promising potential next gravitational-wave discoveries.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

The draft ends with a statement about binary neutron star mergers being confirmed as progenitors of short GRBs, followed by a transition: 'However, this is just the beginning.' It then discusses future observatories and ends with 'A detection of a burst of gravitational waves from a different type of source is one of the most promising potential next gravitational-wave discoveries.' This is a complete argumentative move: it sets up future prospects and ends on an open note about other source types. A continuation could work, but abstention is also reasonable. Output A provides a brief completion that accurately summarizes the GW170817/GRB 170817A association (well-supported by the sources and the draft's own text) and notes that extending this to other source classes remains open. This fits the draft's pivot toward future discoveries and other source types. It doesn't invent anything and connects naturally to the 'different type of source' mentioned at the end. Output B is unusable due to a material unsupported claim: it references 'The DAWES review' as if it were a known, cited work in this manuscript. There is no such review mentioned anywhere in the draft or sources. This appears to be a hallucinated source/review name. The rest of the paragraph is generic and could apply to many papers, but the specific invocation of 'The DAWES review' makes it invalid as it falsely attributes content to a non-existent manuscript element.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

The association of GW170817 with GRB 170817A confirmed binary neutron star mergers as progenitors of short gamma-ray bursts, establishing the first direct link between a gravitational-wave event and its electromagnetic counterpart. Extending this association to other source classes, however, remains an open observational challenge.

Passages supplied to the Evidence version

Card 1 · Off-axis short GRBs from structured jets as counterparts to GW events

Binary neutron star mergers are considered to be the most favorable sources that produce electromagnetic (EM) signals associated with gravitational waves (GWs). These mergers are the likely progenitors of short duration gamma-ray bursts (GRBs). The brief gamma-ray emission (the “prompt” GRB emission…

Read full passage excerpt

Binary neutron star mergers are considered to be the most favorable sources that produce electromagnetic (EM) signals associated with gravitational waves (GWs). These mergers are the likely progenitors of short duration gamma-ray bursts (GRBs). The brief gamma-ray emission (the “prompt” GRB emission) is produced by ultra-relativistic jets, as a result, this emission is strongly beamed over a small solid angle along the jet. It is estimated to be a decade or more before a short GRB jet within the LIGO volume points along our line of sight.

Card 2 · The rate and luminosity function of Short GRBs

We compare the luminosity function and rate inferred from the BATSE short hard bursts (SHBs) peak flux distribution with the redshift and luminosity distributions of SHBs observed by Swift/HETE II. The Swift/HETE II SHB sample is incompatible with SHB population that follows the star formation rate.…

Read full passage excerpt

We compare the luminosity function and rate inferred from the BATSE short hard bursts (SHBs) peak flux distribution with the redshift and luminosity distributions of SHBs observed by Swift/HETE II. The Swift/HETE II SHB sample is incompatible with SHB population that follows the star formation rate. However, it is compatible with a distribution of delay times after the SFR. This would be the case if SHBs are associated with binary neutron star mergers. The implied SHB rates that we find range from ∼ 8 to ∼ 30h703Gpc−3yr−1. This rate is a much higher than what was previously estimated and it is comparable to the rate of neutron star mergers estimated from statistics of binary pulsars. If GRBs are produced in mergers the implied rate practically guarantees detection by LIGO II and possibly even by LIGO I, if we are lucky. Our analysis, which is based on observed short hard burst is limited to bursts with luminosities above 1049erg/sec. Weaker bursts may exist but if so they are hardly de …

Card 3 · The rate, luminosity function and time delay of non-Collapsar short GRBs

We estimate the rate and the luminosity function of short (hard) Gamma-Ray Bursts (sGRBs) that are non-Collapsars, using the peak fluxes and redshifts of BATSE, Swift and Fermi GRBs. Following Bromberg et al., we select a sub-sample of Swift bursts which are most likely non-Collapsars. We find that…

Read full passage excerpt

We estimate the rate and the luminosity function of short (hard) Gamma-Ray Bursts (sGRBs) that are non-Collapsars, using the peak fluxes and redshifts of BATSE, Swift and Fermi GRBs. Following Bromberg et al., we select a sub-sample of Swift bursts which are most likely non-Collapsars. We find that these sGRBs are delayed relative to the global star formation rate (SFR) with a typical delay time of a 3–4 Gyr (depending on the SFR model). However, if two or three sGRB at high redshifts have been missed because of selection effects, a distribution of delay times of ∝ 1/t would be also compatible. The current event rate of these non-Collapsar sGRBs with Liso > 5 × 1049 erg s−1 is |$4.1_{-1.9}^{+2.3}\,\mathrm{Gpc}^{-3}\,\mathrm{yr}^{-1}$|⁠. The rate was significantly larger around z ∼ 1 and it declines since that time. The luminosity function we find is a broken power law with a break at |$2.0_{-0.4}^{+1.4} \times 10^{52} \,\mathrm{erg\,s}^{-1}$| and power-law indices |$0.95_{-0.1 2}^{+0.1 …

Card 4 · The luminosity and redshift distributions of short-duration GRBs

Using the BATSE peak flux distribution we rederive the short GRBs luminosity function and compare it with the observed redshift distribution of long bursts. We show that both distributions are compatible with the assumption that short as well as long bursts follow the star formation rate. In this ca…

Read full passage excerpt

Using the BATSE peak flux distribution we rederive the short GRBs luminosity function and compare it with the observed redshift distribution of long bursts. We show that both distributions are compatible with the assumption that short as well as long bursts follow the star formation rate. In this case the difference between the two observed distributions can be interpreted as arising mostly from differences in the detector's sensitivity to long and short bursts, while the local rate of short bursts is Gpc-3 yr-1. We also consider the possibility that short GRBs may be associated with binary neutron star mergers and estimate the effect of the merging time delay on the luminosity function and redshift distribution. We find that in this case the local rate of short GRBs is ~ Gpc-3 yr-1. Assuming that all binary merging systems lead to short GRBs, we find a typical jet opening angle of .

Card 5 · Gravitational Waves and Gamma-Rays from a Binary Neutron Star Merger: GW170817 and GRB 170817A

Abstract On 2017 August 17, the gravitational-wave event GW170817 was observed by the Advanced LIGO and Virgo detectors, and the gamma-ray burst (GRB) GRB 170817A was observed independently by the Fermi Gamma-ray Burst Monitor, and the Anti-Coincidence Shield for the Spectrometer for the Internation…

Read full passage excerpt

Abstract On 2017 August 17, the gravitational-wave event GW170817 was observed by the Advanced LIGO and Virgo detectors, and the gamma-ray burst (GRB) GRB 170817A was observed independently by the Fermi Gamma-ray Burst Monitor, and the Anti-Coincidence Shield for the Spectrometer for the International Gamma-Ray Astrophysics Laboratory . The probability of the near-simultaneous temporal and spatial observation of GRB 170817A and GW170817 occurring by chance is 5.0 × 10 − 8 . We therefore confirm binary neutron star mergers as a progenitor of short GRBs. The association of GW170817 and GRB 170817A provides new insight into fundamental physics and the origin of short GRBs. We use the observed time delay of ( + 1.74 ± 0.05 ) s between GRB 170817A and GW170817 to: (i) constrain the difference between the speed of gravity and the speed of light to be between − 3 × 10 − 15 and + 7 × 10 − 16 times the speed of light, (ii) place new bounds on the violation of Lorentz invariance, (iii) present a …

Model-reported card use

Card 5 · fact

Affected output: confirmed binary neutron star mergers as progenitors of short gamma-ray bursts

Model explanation: This row supplied the concrete confirmed result that the GW170817/GRB 170817A association established binary neutron star mergers as short GRB progenitors, which the draft's conclusion had not yet stated.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

The DAWES review has surveyed the observational and theoretical landscape of gravitational-wave burst astrophysics, from the established compact binary mergers to the unmodeled and poorly modeled signals that current searches may still miss. Realizing the full discovery potential of the next generation of observatories will require not only improved sensitivity but also waveform-agnostic detection methods capable of identifying sources whose gravitational-wave emission is not yet predicted.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output A is grounded in the draft and sources. It directly continues the manuscript's discussion by referencing GW170817/GRB 170817A (mentioned in the draft's preceding paragraph and detailed in the sources) and notes that extending this association to other source classes remains challenging—which aligns with the draft's closing statement about other potential gravitational-wave sources yet to be discovered. The claim about 'first direct link' is supported by source 5 ('we therefore confirm binary neutron star mergers as a progenitor of short GRBs') and the draft's own text ('the first direct evidence that binary neutron stars are the progenitors of short GRBs'). Output B introduces 'The DAWES review'—an entity completely absent from all supplied sources and the draft. This is a fabricated source/publication. It also makes unsupported claims about 'waveform-agnostic detection methods' and sources 'whose gravitational-wave emission is not yet predicted' that go beyond anything in the provided materials. The draft mentions 'different type of source' and 'burst of gravitational waves,' but does not discuss unmodeled signals, waveform-agnostic methods, or any review called DAWES. These are substantive inventions that mislead about the literature base.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.