FlowingFlowing

BENCHMARK

Does adding literature cards improve autocomplete?

The evaluation covers 280 academic writing positions across seven fields, with 40 positions per field. Every position and literature card was fixed before generation; at each position we generated one version with cards and one without, retaining the first output from each.

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

AUTOCOMPLETE · THE SAME 280 FIRST ATTEMPTS

How often were cards used, and did writing improve overall?

Seven fields · 40 positions each

OVERALL PREFERENCE · ALL 280

49%

AI judge preferred cards in 138/280

138With cards better26About the same107Without cards better5Neither usable4Inconclusive

CARD ADOPTION FUNNEL

The purple bars below show the share of all 280 attempts

All frozen positions280/280 · 100%

Text plus model-reported card use

50%140/280

Both wrote text; comparable pairs

107/280· 38%

That is 76% of the 140 adopted cases; in the other 33 (24%), the no-card arm was blank.

Use is reported by the generator, not an evidence-opportunity label.

WHEN CARDS WERE USED AND BOTH WROTE

With-card version preferred

65%70/107
About the same7% · 7/107
Without cards better28% · 30/107

COMPARISON · NOT COUNTED AS VALID ADOPTION

39%55/140 with-card wins in the overall review

130 had no use report and 10 reported use without valid text; also 25 ties, 54 no-card wins, 3 neither usable, and 3 inconclusive. This group does not mean evidence was unsuitable.

The 65% and 39% on the right use different denominators and review rubrics and cannot be directly subtracted. Under the same overall review, cards were preferred in 83 of 140 adopted cases (59%) and 55 of 140 not counted as adopted (39%).

Why include positions where evidence may have little opportunity to help?

All 280 positions and cards were fixed before generation; at each position only card availability changed. Keeping every first output avoids selecting favorable cases after seeing results. The 49% describes this cohort, not the win share where evidence clearly had an opportunity to help. Limited opportunity also does not guarantee zero effect.

To split this total into clear versus limited evidence opportunity, each position needs an independent label based only on the draft and cards, followed by group sizes and win shares. This cohort lacks complete, consistent labels, so two reliable rates cannot be filled in; the 50% adoption rate cannot substitute for that split.

218 pairs had text on both sides; 48 had at least one deliberate stop; 14 contained an invalid output; 4 judgments were inconclusive. The AI judge assessed whether a stop fit the draft, without independent human review.

See detailed results for all seven fields →

Current autocomplete evaluation

How often was card use reported, and how did written pairs compare?

The evaluation covers forty writing positions in each of seven fields. Card use is self-reported by the generator; the preferences below cover only tests where both sides produced text.

Quantitative biology

13 papers · 40 completions · 183 cards

All 40: 18 with-card wins, 4 ties, 18 no-card wins, 0 neither usable, 0 inconclusive

40 tests

Model-reported card use

55%· 22/40

22 of 40 produced text and reported card use

When both versions wrote text, which was better?

With cards better

71%· 12/17

About the same

0%· 0/17

Without cards better

29%· 5/17

Of 22 reported uses, 17 had text on both sides: 12 with-card wins, 0 ties, 5 no-card wins; the no-card arm had no text in 5.

Statistics

24 papers · 40 completions · 195 cards

All 40: 21 with-card wins, 4 ties, 13 no-card wins, 0 neither usable, 2 inconclusive

40 tests

Model-reported card use

57%· 23/40

23 of 40 produced text and reported card use

The previous 25/40 included two with-card outputs without text; corrected here.

When both versions wrote text, which was better?

With cards better

69%· 11/16

About the same

13%· 2/16

Without cards better

19%· 3/16

Of 23 reported uses, 16 had text on both sides: 11 with-card wins, 2 ties, 3 no-card wins; the no-card arm had no text in 7.

Astrophysics

20 papers · 40 completions · 200 cards

All 40: 24 with-card wins, 2 ties, 13 no-card wins, 1 neither usable, 0 inconclusive

40 tests

Model-reported card use

57%· 23/40

23 of 40 produced text and reported card use

When both versions wrote text, which was better?

With cards better

77%· 10/13

About the same

0%· 0/13

Without cards better

23%· 3/13

Of 23 reported uses, 13 had text on both sides: 10 with-card wins, 0 ties, 3 no-card wins; the no-card arm had no text in 10.

Psychology and cognitive neuroscience

26 papers · 40 completions · 186 cards

All 40: 20 with-card wins, 4 ties, 16 no-card wins, 0 neither usable, 0 inconclusive

40 tests

Model-reported card use

63%· 25/40

25 of 40 produced text and reported card use

When both versions wrote text, which was better?

With cards better

47%· 9/19

About the same

0%· 0/19

Without cards better

53%· 10/19

Of 25 reported uses, 19 had text on both sides: 9 with-card wins, 0 ties, 10 no-card wins; the no-card arm had no text in 6.

Climate and environmental science

20 papers · 40 completions · 191 cards

All 40: 13 with-card wins, 3 ties, 24 no-card wins, 0 neither usable, 0 inconclusive

40 tests

Model-reported card use

35%· 14/40

14 of 40 produced text and reported card use

When both versions wrote text, which was better?

With cards better

56%· 5/9

About the same

0%· 0/9

Without cards better

44%· 4/9

Of 14 reported uses, 9 had text on both sides: 5 with-card wins, 0 ties, 4 no-card wins; the no-card arm had no text in 5.

Mathematics

10 papers · 40 completions · 200 cards

All 40: 22 with-card wins, 5 ties, 11 no-card wins, 0 neither usable, 2 inconclusive

40 tests

Model-reported card use

45%· 18/40

18 of 40 produced text and reported card use

When both versions wrote text, which was better?

With cards better

78%· 14/18

About the same

6%· 1/18

Without cards better

17%· 3/18

Of 18 reported uses, 18 had text on both sides: 14 with-card wins, 1 ties, 3 no-card wins; the no-card arm had no text in 0.

Computer science (LLM / 3D vision)

18 papers · 40 completions · 200 cards

All 40: 20 with-card wins, 4 ties, 12 no-card wins, 4 neither usable, 0 inconclusive

40 tests

Model-reported card use

38%· 15/40

15 of 40 produced text and reported card use

When both versions wrote text, which was better?

With cards better

60%· 9/15

About the same

27%· 4/15

Without cards better

13%· 2/15

Of 15 reported uses, 15 had text on both sides: 9 with-card wins, 4 ties, 2 no-card wins; the no-card arm had no text in 0.

What to know before reading the results

Positions fixed in advance: Forty positions were selected in each of seven fields; positions and cards were fixed before any outputs were generated.

Matched comparison: The writing position, model and task stay the same; only whether literature cards are supplied changes.

No repeated retries: Each side keeps its first output and is not regenerated based on the result.

Who judges quality: An AI judge compares anonymized outputs; this is not an independent human review.

Who reports card use: Card use is self-reported by the generation model without case-by-case human review.

Results are inspectable: All 280 cases show the draft, literature cards and first outcomes from both sides; judged cases also show the review rationale.

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

CASES FROM THIS EVALUATION

Explore all 280 autocomplete tests

Select any number above to filter to its cases. Each case shows the manuscript, cards and both first outcomes; judged cases also show the review rationale.

Preferences come from an anonymized AI judge without independent human review; four unresolved disagreements are listed as inconclusive.

Overall
Subset
Field

Showing 280 / 280 tests · page 1 / 24

AutocompleteOverall: Without cards betterbeu-8f58833fb268bb31

Deciphering genomic codes using advanced NLP techniques: a scoping review

Quantitative biology · 2411.16084v1

… identify critical regulatory elements such as transcription-factor binding sites, enhancer-promoter interactions, chromatin accessibility, and gene expression patterns [ 7 , 8 ] .

Open full evidence →AutocompleteOverall: Without cards betterbeu-018cb8c48d7476c7

Generative AI for Controllable Protein Sequence Design: A Survey

Quantitative biology · 2402.10516v1

… fic structural and functional constraints. To this end, researchers have utilized conditional generative models and optimization algorithms to realize such tailored protein design.

Open full evidence →AutocompleteOverall: With cards betterbeu-b9e0f2f3fea4a333

A Review of Link Prediction Applications in Network Biology

Quantitative biology · 2312.01275v1

… topology, incorporating knowledge from diverse sources like protein-protein interaction databases, gene expression profiles, or disease association data ( breit2020openbiolink ) .

Open full evidence →AutocompleteOverall: Without cards betterbeu-40adbed355c2a3b5

A Review of Link Prediction Applications in Network Biology

Quantitative biology · 2312.01275v1

… ) . This temporal understanding is particularly crucial for tracking disease progression, understanding gene expression changes, or following the evolution of biological pathways.

Open full evidence →AutocompleteOverall: Without cards betterbeu-265f9f8e2509097b

A Review of BioTree Construction in the Context of Information Fusion: Priors, Methods, Applications and Trends

Quantitative biology · 2410.04815v2

… time ordering into a unified framework, constructing lineage trees that represent the developmental trajectories of cells based on statistical modeling of gene expression patterns.

Open full evidence →AutocompleteOverall: With cards betterbeu-89bf16c0a707dfcf

A Review of BioTree Construction in the Context of Information Fusion: Priors, Methods, Applications and Trends

Quantitative biology · 2410.04815v2

… ical systems. While deep learning approaches have begun to leverage data-driven strategies for fusion, classical methods remain inadequate in addressing this integration challenge.

Open full evidence →AutocompleteOverall: With cards betterbeu-405da15edfd07669

A Unified View of Deep Learning for Reaction and Retrosynthesis Prediction: Current Status and Future Challenges

Quantitative biology · 2306.15890v1

… xplores SSL strategies for retrosynthesis but it fails to achieve a strong performance. Therefore, exploring more powerful SSL strategies for both problems is a feasible direction.

Open full evidence →AutocompleteOverall: With cards betterbeu-9ffd365d52d8311f

Transformers in Protein: A Survey

Quantitative biology · 2505.20098v2

… foundational principles, we examined leading models such as AlphaFold, ESM, ProtTrans, and RoseTTAFold, highlighting their design innovations and contributions to protein science.

Open full evidence →AutocompleteOverall: Without cards betterbeu-587d3034bbbe3477

Generalized Modeling: A survey and guide

Quantitative biology · 2112.00360v2

… ynamical stability and bifurcations of steady states, prediction of responses to different types of perturbations, and identification of important parameters and parameter regions.

Open full evidence →AutocompleteOverall: With cards betterbeu-e41bb369f3385f36

From Words to Molecules: A Survey of Large Language Models in Chemistry

Quantitative biology · 2402.01439v1

… arkable abilities in various domains. These models excel at understanding and generating complex text, making them crucial for tasks that demand deep textual analysis and creation.

Open full evidence →AutocompleteOverall: Without cards betterbeu-77d4bcceaac1b33e

Large Language Models in Bioinformatics: A Survey

Quantitative biology · 2503.04490v3

… that combine deep learning with symbolic AI Colelough and Regli 2025 or constraint-based modeling Bystrova et al. 2024 may enable causality-aware predictions in biological systems.

Open full evidence →AutocompleteOverall: Without cards betterbeu-5ded0d3c9dde7209

Generalized Modeling: A survey and guide

Quantitative biology · 2112.00360v2

… ittle effort. In the following we explain the basic idea of GM in Sec. 2. After this philosophical introduction, we provide a comprehensive review of previous works that used GM in

Open full evidence →
1 / 24

← Back to home