FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better8 / 280 · beu-9ffd365d52d8311f

Transformers in Protein: A Survey

Quantitative biology · 2505.20098v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Transformers in Protein: A Survey

VI CONCLUSION

The complex nature of protein systems necessitates cross-disciplinary collaboration among computational biologists, structural biologists, machine learning experts, and domain scientists [ 125 ] [ 40 ] . Such collaboration will accelerate the translation of Transformer models into practical workflows for drug discovery, synthetic biology, and personalized medicine. Furthermore, co-development of models and wet-lab validation pipelines will ensure that computational predictions have real-world biological relevance.

…become a cornerstone in protein informatics, offering robust mechanisms for learning long-range dependencies and capturing complex biological patterns. This survey has presented a systematic review of Transformer-based architectures and their applications across critical domains, including protein structure prediction, function annotation, protein-protein interactions, and computational drug discovery. Starting from foundational principles, we examined leading models such as AlphaFold, ESM, ProtTrans, and RoseTTAFold, highlighting their design innovations and contributions to protein science.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Both outputs provide coherent continuations that transition from the survey summary to future challenges. However, Output B is superior in fit and scope. The draft concludes a systematic review of Transformer architectures (AlphaFold, ESM, ProtTrans, RoseTTAFold) and their applications. Output B directly addresses the core scientific challenge—sequence-structure-function integration—that is the logical next frontier given the models surveyed, and it mirrors language from Source 1 about 'joint sequence-structure-function co-design frameworks' without being unsupported. Output A raises valid but more generic concerns (data scarcity, computational cost, interpretability) that, while plausible, are less specifically tied to the Transformer-based protein informatics focus of the preceding text. Output B's emphasis on co-design frameworks follows naturally from the 'co-development of models and wet-lab validation pipelines' mentioned in the prior paragraph and the survey's architecture-focused scope. Output A's concerns about 'attention-based representations' being hard to interpret is somewhat at odds with Source 5's discussion of AlphaFold's interpretable confidence scores. Output B requires no repair; Output A is usable but slightly less precisely scoped to the manuscript's trajectory.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

Despite these advances, challenges remain in modeling the sequence-structure-function relationship and in ensuring robust generalization beyond the regions of protein space spanned by the training data. Future progress will likely depend on joint sequence-structure-function co-design frameworks that model the fitness landscape more effectively than models treating these modalities independently.

Passages supplied to the Evidence version

Card 1 · A Model-Centric Review of Deep Learning for Protein Design

Deep learning has transformed protein design, enabling accurate structure prediction, sequence optimization, and de novo protein generation. Advances in single-chain protein structure prediction via AlphaFold2, RoseTTAFold, ESMFold, and others have achieved near-experimental accuracy, inspiring succ…

Read full passage excerpt

Deep learning has transformed protein design, enabling accurate structure prediction, sequence optimization, and de novo protein generation. Advances in single-chain protein structure prediction via AlphaFold2, RoseTTAFold, ESMFold, and others have achieved near-experimental accuracy, inspiring successive work extended to biomolecular complexes via AlphaFold Multimer, RoseTTAFold All-Atom, AlphaFold 3, Chai-1, Boltz-1 and others. Generative models such as ProtGPT2, ProteinMPNN, and RFdiffusion have enabled sequence and backbone design beyond natural evolution-based limitations. More recently, joint sequence-structure co-design models, including ESM3, have integrated both modalities into a unified framework, resulting in improved designability. Despite these advances, challenges still exist pertaining to modeling sequence-structure-function relationships and ensuring robust generalization beyond the regions of protein space spanned by the training data. Future advances will likely focus …

Card 2 · Protein Folding Neural Networks Are Not Robust

Deep neural networks such as AlphaFold and RoseTTAFold predict remarkably accurate structures of proteins compared to other algorithmic approaches. It is known that biologically small perturbations in the protein sequence do not lead to drastic changes in the protein structure. In this paper, we dem…

Read full passage excerpt

Deep neural networks such as AlphaFold and RoseTTAFold predict remarkably accurate structures of proteins compared to other algorithmic approaches. It is known that biologically small perturbations in the protein sequence do not lead to drastic changes in the protein structure. In this paper, we demonstrate that RoseTTAFold does not exhibit such a robustness despite its high accuracy, and biologically small perturbations for some input sequences result in radically different predicted protein structures. This raises the challenge of detecting when these predicted protein structures cannot be trusted. We define the robustness measure for the predicted structure of a protein sequence to be the inverse of the root-mean-square distance (RMSD) in the predicted structure and the structure of its adversarially perturbed sequence. We use adversarial attack methods to create adversarial protein sequences, and show that the RMSD in the predicted protein structure ranges from 0.119Å to 34.162Å wh …

Card 3 · Exploring Post-Training Quantization of Protein Language Models

Recent advancements in unsupervised protein language models (ProteinLMs), like ESM-1b and ESM-2, have shown promise in different protein prediction tasks. However, these models face challenges due to their high computational demands, significant memory needs, and latency, restricting their usage on…

Read full passage excerpt

Recent advancements in unsupervised protein language models (ProteinLMs), like ESM-1b and ESM-2, have shown promise in different protein prediction tasks. However, these models face challenges due to their high computational demands, significant memory needs, and latency, restricting their usage on devices with limited resources. To tackle this, we explore post-training quantization (PTQ) for ProteinLMs, focusing on ESMFold, a simplified version of AlphaFold based on ESM-2 ProteinLM. Our study is the first attempt to quantize all weights and activations of ProteinLMs. We observed that the typical uniform quantization method performs poorly on ESMFold, causing a significant drop in TM-Score when using 8-bit quantization. We conducted extensive quantization experiments, uncovering unique challenges associated with ESMFold, particularly highly asymmetric activation ranges before Layer Normalization, making representation difficult using low-bit fixed-point formats. To address these challe …

Card 4 · Leveraging Multi-modal Representations to Predict Protein Melting Temperatures

Accurately predicting protein melting temperature changes (Delta Tm) is fundamental for assessing protein stability and guiding protein engineering. Leveraging multi-modal protein representations has shown great promise in capturing the complex relationships among protein sequences, structures, and…

Read full passage excerpt

Accurately predicting protein melting temperature changes (Delta Tm) is fundamental for assessing protein stability and guiding protein engineering. Leveraging multi-modal protein representations has shown great promise in capturing the complex relationships among protein sequences, structures, and functions. In this study, we develop models based on powerful protein language models, including ESM-2, ESM-3 and AlphaFold, using various feature extraction methods to enhance prediction accuracy. By utilizing the ESM-3 model, we achieve a new state-of-the-art performance on the s571 test dataset, obtaining a Pearson correlation coefficient (PCC) of 0.50. Furthermore, we conduct a fair evaluation to compare the performance of different protein language models in the Delta Tm prediction task. Our results demonstrate that integrating multi-modal protein representations could advance the prediction of protein melting temperatures.

Card 5 · Recent advances in interpretable machine learning using structure-based protein representations

Recent advancements in machine learning (ML) are transforming the field of structural biology. For example, AlphaFold, a groundbreaking neural network for protein structure prediction, has been widely adopted by researchers. The availability of easy-to-use interfaces and interpretable outcomes from…

Read full passage excerpt

Recent advancements in machine learning (ML) are transforming the field of structural biology. For example, AlphaFold, a groundbreaking neural network for protein structure prediction, has been widely adopted by researchers. The availability of easy-to-use interfaces and interpretable outcomes from the neural network architecture, such as the confidence scores used to color the predicted structures, have made AlphaFold accessible even to non-ML experts. In this paper, we present various methods for representing protein 3D structures from low- to high-resolution, and show how interpretable ML methods can support tasks such as predicting protein structures, protein function, and protein-protein interactions. This survey also emphasizes the significance of interpreting and visualizing ML-based inference for structure-based protein representations that enhance interpretability and knowledge discovery. Developing such interpretable approaches promises to further accelerate fields including …

Model-reported card use

Card 1 · direction

Affected output: Despite these advances, challenges remain in modeling the sequence-structure-function relationship and in ensuring robust generalization beyond the regions of protein space spanned by the training data.

Model explanation: This row supplied the specific remaining challenges to name in the conclusion's outlook, steering the paragraph toward sequence-structure-function modeling and generalization rather than a generic future-work statement.

Card 1 · direction

Affected output: Future progress will likely depend on joint sequence-structure-function co-design frameworks that model the fitness landscape more effectively than models treating these modalities independently.

Model explanation: This row provided the forward-looking direction—joint sequence-structure-function co-design and fitness-landscape modeling—that shapes the second sentence of the conclusion.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

Despite these advances, several challenges remain before Transformer-based protein models can be routinely deployed in practice. Data scarcity in protein function annotation, the computational cost of training and inference at proteome scale, and the limited interpretability of attention-based representations continue to constrain their broader adoption.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output A is grounded in the first source, which explicitly discusses 'challenges still exist pertaining to modeling sequence-structure-function relationships and ensuring robust generalization beyond the regions of protein space spanned by the training data' and states that 'Future advances will likely focus on joint sequence-structure-function co-design frameworks that are able to model the fitness landscape more effectively than models that treat these modalities independently.' The language closely tracks the source while fitting the survey's concluding tone. Output B introduces three specific challenges—'Data scarcity in protein function annotation,' 'computational cost of training and inference at proteome scale,' and 'limited interpretability of attention-based representations'—none of which are supported by the supplied sources. While source 3 mentions computational demands of ProteinLMs and source 5 mentions interpretability, neither supports these as general challenges for 'Transformer-based protein models' in the way B frames them, and 'data scarcity in protein function annotation' is entirely unsupported. The draft itself does not mention these challenges. B's claims about what 'continues to constrain their broader adoption' constitute unsupported material factual claims.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.