FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better151 / 280 · beu-fbd10587698c74f8

Transformers in Protein: A Survey

Quantitative biology · 2505.20098v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Transformers in Protein: A Survey

I Introduction

Protein informatics, a cornerstone of bioinformatics, is dedicated to deciphering protein structures, functions, and interactions. The exponential growth of biological data—particularly in proteomics—has intensified the demand for computational methods capable of efficiently and accurately interpreting complex protein datasets. While traditional approaches have laid a foundational understanding, their limitations in handling the scale and intricacy of modern protein data have spurred the adoption of more sophisticated models.

… models have emerged as a transformative solution across disciplines [ 1 , 2 , 3 , 4 , 5 , 6 ] , owing to their ability to process variable-length sequences and model long-range dependencies via self-attention mechanisms. These attributes are especially valuable in protein informatics, where sequence-structure-function relationships often hinge on distal interactions and hierarchical patterns [ 7 , 8 ] . Notably, transformer-based architectures (e.g., BERT, GPT) have achieved breakthroughs in protein structure prediction [ 9 ] , interaction analysis [ 10 ] , and functional annotation [ 11 ] .

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs provide coherent continuations, but they serve different argumentative functions. The draft establishes that transformers have achieved breakthroughs in specific protein informatics tasks (structure prediction, interaction analysis, functional annotation) with citations [9-11]. The natural next move is to articulate the gap or motivation for the present survey. Output B directly addresses this by identifying a critical gap: despite task-specific advances, the field lacks 'a unified account' and 'coherent taxonomy' of how transformers are adapted to protein-specific challenges. This fits the typical survey paper structure where, after establishing background and prior breakthroughs, the authors state what is missing and how their work fills it. The language ('This survey addresses that gap') explicitly signals the manuscript's contribution and scope. Output A pivots to self-supervised pretraining and foundation models, which is plausible but less fitting. The draft already mentions 'transformer-based architectures (e.g., BERT, GPT)'—the foundation model paradigm is already implicit. Reiterating this as 'increasingly leverage large-scale unlabeled... data' adds incremental description rather than advancing the argument. The phrase 'mirrors the broader trajectory of foundation models in biomedicine' drifts toward general biomedicine rather than staying tightly focused on protein informatics. It reads more like a middle-of-introduction elaboration than the logical next move after listing specific breakthroughs. Output B's scope is more appropriate: it transitions from 'here are advances' to 'here is what's missing and what we do about it'—the standard survey paper move. It requires no substantial repair to insert, and its taxonomy-framing aligns with the sources' emphasis on comprehensive organization (e.g., source 2's 'comprehensive survey' structure). Output A is usable but less argumentatively productive at this specific position.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

Beyond these task-specific advances, transformer-based protein models increasingly leverage large-scale unlabeled sequence and structure data through self-supervised pretraining, yielding representations that transfer across diverse downstream tasks. This paradigm shift mirrors the broader trajectory of foundation models in biomedicine, where a single pretrained backbone can be adapted to numerous applications.

Passages supplied to the Evidence version

Card 1 · A Comprehensive Survey of Foundation Models in Medicine

Foundation models (FMs) are large-scale deep learning models trained on massive datasets, often using self-supervised learning techniques. These models serve as a versatile base for a wide range of downstream tasks, including those in medicine and healthcare. FMs have demonstrated remarkable success…

Read full passage excerpt

Foundation models (FMs) are large-scale deep learning models trained on massive datasets, often using self-supervised learning techniques. These models serve as a versatile base for a wide range of downstream tasks, including those in medicine and healthcare. FMs have demonstrated remarkable success across multiple healthcare domains. However, existing surveys in this field do not comprehensively cover all areas where FMs have made significant strides. In this survey, we present a comprehensive review of FMs in medicine, focusing on their evolution, learning strategies, flagship models, applications, and associated challenges. We examine how prominent FMs, such as the BERT and GPT families, are transforming various aspects of healthcare, including clinical large language models, medical image analysis, and omics research. Additionally, we provide a detailed taxonomy of FM-enabled healthcare applications, spanning clinical natural language processing, medical computer vision, graph lear …

Card 2 · Advances of Deep Learning in Protein Science: A Comprehensive Survey

Protein representation learning plays a crucial role in understanding the structure and function of proteins, which are essential biomolecules involved in various biological processes. In recent years, deep learning has emerged as a powerful tool for protein modeling due to its ability to learn comp…

Read full passage excerpt

Protein representation learning plays a crucial role in understanding the structure and function of proteins, which are essential biomolecules involved in various biological processes. In recent years, deep learning has emerged as a powerful tool for protein modeling due to its ability to learn complex patterns and representations from large-scale protein data. This comprehensive survey aims to provide an overview of the recent advances in deep learning techniques applied to protein science. The survey begins by introducing the developments of deep learning based protein models and emphasizes the importance of protein representation learning in drug discovery, protein engineering, and function annotation. It then delves into the fundamentals of deep learning, including convolutional neural networks, recurrent neural networks, attention models, and graph neural networks in modeling protein sequences, structures, and functions, and explores how these techniques can be used to extract mea …

Card 3 · GoBERT: Gene Ontology Graph Informed BERT for Universal Gene Function Prediction

Exploring the functions of genes and gene products is crucial to a wide range of fields, including medical research, evolutionary biology, and environmental science. However, discovering new functions largely relies on expensive and exhaustive wet lab experiments. Existing methods of automatic funct…

Read full passage excerpt

Exploring the functions of genes and gene products is crucial to a wide range of fields, including medical research, evolutionary biology, and environmental science. However, discovering new functions largely relies on expensive and exhaustive wet lab experiments. Existing methods of automatic function annotation or prediction mainly focus on protein function prediction with sequence, 3D-structures or protein family information. In this study, we propose to tackle the gene function prediction problem by exploring Gene Ontology graph and annotation with BERT (GoBERT) to decipher the underlying relationships among gene functions. Our proposed novel function prediction task utilizes existing functions as inputs and generalizes the function prediction to gene and gene products. Specifically, two pre-train tasks are designed to jointly train GoBERT to capture both explicit and implicit relations of functions. Neighborhood prediction is a self-supervised multi-label classification task that …

Model-reported card use

Card 2 · fact

Affected output: transformer-based protein models increasingly leverage large-scale unlabeled sequence and structure data through self-supervised pretraining, yielding representations that transfer across diverse downstream tasks

Model explanation: This row supplied the factual premise that deep learning protein models learn representations from large-scale protein data, which I used to ground the claim about self-supervised pretraining on large-scale data.

Card 1 · terminology

Affected output: foundation models in biomedicine, where a single pretrained backbone can be adapted to numerous applications

Model explanation: This row supplied the field-specific terminology of foundation models and the framing of a versatile pretrained base adapted to many downstream tasks, which shaped how I expressed the paradigm shift.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

Despite these advances, the application of transformers to protein informatics remains fragmented across tasks, architectures, and data modalities, and a unified account of how these models are adapted to protein-specific challenges is still lacking. This survey addresses that gap by systematically reviewing transformer-based approaches across protein structure prediction, function annotation, and interaction modeling, and by organizing the field's methodological choices into a coherent taxonomy.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the no-Evidence version. Its original rationale appears below.

Read the original model judgment

Output A contains unsupported claims. It states that transformer-based protein models 'increasingly leverage large-scale unlabeled sequence and structure data through self-supervised pretraining, yielding representations that transfer across diverse downstream tasks' and that 'This paradigm shift mirrors the broader trajectory of foundation models in biomedicine, where a single pretrained backbone can be adapted to numerous applications.' While the sources discuss foundation models and self-supervised learning in general terms (Source 1 mentions FMs use self-supervised learning and BERT/GPT transform healthcare; Source 2 mentions deep learning learns from large-scale protein data), none of the sources specifically state that transformer-based protein models use self-supervised pretraining on unlabeled sequence and structure data, or that this yields transferable representations, or that a single pretrained backbone can be adapted to numerous protein applications. The claim about 'unlabeled sequence and structure data' and 'single pretrained backbone' adapted to 'numerous applications' in protein informatics is an extrapolation not directly supported by the sources. The draft itself mentions transformer breakthroughs but does not mention pretraining, unlabeled data, or transferable representations. Output B is grounded in the draft and sources. It notes that transformer applications are 'fragmented across tasks, architectures, and data modalities' — which aligns with Source 2's survey structure covering multiple techniques (CNNs, RNNs, attention, GNNs) and applications (structure prediction, interaction prediction, function prediction). The claim that 'a unified account... is still lacking' is supported by Source 2's stated purpose to provide 'a comprehensive survey' and Source 1's note that 'existing surveys in this field do not comprehensively cover all areas.' Output B's statement that 'This survey addresses that gap' is consistent with the draft's survey framing and Source 2's explicit survey purpose. The taxonomy mention is supported by Source 1's 'detailed taxonomy of FM-enabled healthcare applications.' Output B preserves the draft's meaning and scope without adding unsupported factual claims.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.