FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better156 / 280 · beu-1fbfa982fdb55b45

From Words to Molecules: A Survey of Large Language Models in Chemistry

Quantitative biology · 2402.01439v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

From Words to Molecules: A Survey of Large Language Models in Chemistry

1 Introduction

Humans understand and describe their environment using natural language, which reflects the complexity of human thought. The emergence of Large Language Models (LLMs) marks a significant advancement in artificial intelligence, showcasing remarkable abilities in various domains. These models excel at understanding and generating complex text, making them crucial for tasks that demand deep textual analysis and creation.

…similar to how syntax operates, while molecules are formed within specific physical constraints, echoing the principles of grammar. This parallel suggests the potential for encoding chemical information into LLMs in a manner comparable to natural language. Despite the conceptual parallels, the languages of chemistry and human communication differ substantially in their semantics. Consequently, incorporating chemical knowledge into LLMs presents a complex challenge, with numerous approaches being explored to leverage LLMs in the field of chemistry, making it a subject of considerable interest.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Both outputs provide usable continuations, but B is superior in fit and author utility. The draft establishes: (1) chemistry and natural language have structural parallels (syntax/grammar), (2) they differ in semantics, creating challenges for incorporating chemical knowledge into LLMs, and (3) numerous approaches are being explored. Output A provides a generic transition ('Building on this analogy...') that merely restates the already-established point that molecular structures are being treated as language. It adds no concrete development and reads as filler. Output B instead advances the argument by immediately exemplifying the 'numerous approaches' mentioned in the draft's closing clause. It introduces SELFormer with specific, accurate details from the source (transformer-based, SELFIES notation, pre-training scale, contrast with SMILES) that directly illustrate how chemical language models address the semantic challenge. This specificity makes B more useful to an academic author who needs to ground their claims. The SELFIES/SMILES comparison in B also elegantly echoes the draft's syntax/grammar vs. semantics distinction—SELFIES's guaranteed validity addresses a 'syntax' problem, while property prediction tasks engage with chemical 'semantics.' B requires no repair to insert; A would leave the reader expecting examples that don't arrive.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

A growing body of work has begun to treat string-based chemical notations as a language in their own right, applying natural language processing architectures to molecular representation learning. SELFormer, for instance, is a transformer-based chemical language model pre-trained on two million drug-like compounds using SELFIES, a notation chosen for its validity and robustness relative to SMILES, and fine-tuned for molecular property prediction tasks.

Passages supplied to the Evidence version

Card 1 · SELFormer: Molecular Representation Learning via SELFIES Language Models

Automated computational analysis of the vast chemical space is critical for numerous fields of research such as drug discovery and material science. Representation learning techniques have recently been employed with the primary objective of generating compact and informative numerical expressions o…

Read full passage excerpt

Automated computational analysis of the vast chemical space is critical for numerous fields of research such as drug discovery and material science. Representation learning techniques have recently been employed with the primary objective of generating compact and informative numerical expressions of complex data. One approach to efficiently learn molecular representations is processing string-based notations of chemicals via natural language processing (NLP) algorithms. Majority of the methods proposed so far utilize SMILES notations for this purpose; however, SMILES is associated with numerous problems related to validity and robustness, which may prevent the model from effectively uncovering the knowledge hidden in the data. In this study, we propose SELFormer, a transformer architecture-based chemical language model that utilizes a 100% valid, compact and expressive notation, SELFIES, as input, in order to learn flexible and high-quality molecular representations. SELFormer is pre- …

Card 2 · polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics

Polymers are a vital part of everyday life. Their chemical universe is so large that it presents unprecedented opportunities as well as significant challenges to identify suitable application-specific candidates. We present a complete end-to-end machine-driven polymer informatics pipeline that can s…

Read full passage excerpt

Polymers are a vital part of everyday life. Their chemical universe is so large that it presents unprecedented opportunities as well as significant challenges to identify suitable application-specific candidates. We present a complete end-to-end machine-driven polymer informatics pipeline that can search this space for suitable candidates at unprecedented speed and accuracy. This pipeline includes a polymer chemical fingerprinting capability called polyBERT (inspired by Natural Language Processing concepts), and a multitask learning approach that maps the polyBERT fingerprints to a host of properties. polyBERT is a chemical linguist that treats the chemical structure of polymers as a chemical language. The present approach outstrips the best presently available concepts for polymer property prediction based on handcrafted fingerprint schemes in speed by two orders of magnitude while preserving accuracy, thus making it a strong candidate for deployment in scalable architectures includin …

Card 3 · Interactive Molecular Discovery with Natural Language

Natural language is expected to be a key medium for various human-machine interactions in the era of large language models. When it comes to the biochemistry field, a series of tasks around molecules (e.g., property prediction, molecule mining, etc.) are of great significance while having a high tec…

Read full passage excerpt

Natural language is expected to be a key medium for various human-machine interactions in the era of large language models. When it comes to the biochemistry field, a series of tasks around molecules (e.g., property prediction, molecule mining, etc.) are of great significance while having a high technical threshold. Bridging the molecule expressions in natural language and chemical language can not only hugely improve the interpretability and reduce the operation difficulty of these tasks, but also fuse the chemical knowledge scattered in complementary materials for a deeper comprehension of molecules. Based on these benefits, we propose the conversational molecular design, a novel task adopting natural language for describing and editing target molecules. To better accomplish this task, we design ChatMol, a knowledgeable and versatile generative pre-trained model, enhanced by injecting experimental property information, molecular spatial knowledge, and the associations between natural …

Card 4 · Beyond Chemical Language: A Multimodal Approach to Enhance Molecular Property Prediction

We present a novel multimodal language model approach for predicting molecular properties by combining chemical language representation with physicochemical features. Our approach, MULTIMODAL-MOLFORMER, utilizes a causal multistage feature selection method that identifies physicochemical features ba…

Read full passage excerpt

We present a novel multimodal language model approach for predicting molecular properties by combining chemical language representation with physicochemical features. Our approach, MULTIMODAL-MOLFORMER, utilizes a causal multistage feature selection method that identifies physicochemical features based on their direct causal effect on a specific target property. These causal features are then integrated with the vector space generated by molecular embeddings from MOLFORMER. In particular, we employ Mordred descriptors as physicochemical features and identify the Markov blanket of the target property, which theoretically contains the most relevant features for accurate prediction. Our results demonstrate a superior performance of our proposed approach compared to existing state-of-the-art algorithms, including the chemical language-based MOLFORMER and graph neural networks, in predicting complex tasks such as biodegradability and PFAS toxicity estimation. Moreover, we demonstrate the ef …

Model-reported card use

Card 1 · fact

Affected output: SELFormer, for instance, is a transformer-based chemical language model pre-trained on two million drug-like compounds using SELFIES, a notation chosen for its validity and robustness relative to SMILES, and fine-tuned for molecular property prediction tasks.

Model explanation: This row supplied the concrete prior-work example (SELFormer, SELFIES, two million drug-like compounds, property prediction) that the paragraph uses to open the survey's discussion of chemical language models.

Card 2 · terminology

Affected output: treat string-based chemical notations as a language in their own right, applying natural language processing architectures to molecular representation learning

Model explanation: This row's framing of chemical structure as a chemical language processed with NLP concepts guided the terminology used to describe the field's approach to string-based notations.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

Building on this analogy, recent work has begun to treat molecular structures as a language that LLMs can process, motivating a growing body of methods that apply these models across chemical tasks.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Both outputs provide coherent continuations, but B is substantially more useful for a researcher. Output A is vague and generic—it states that 'recent work has begun to treat molecular structures as a language' without grounding this in any of the supplied sources. It adds no specific methods, models, or evidence. Output B, by contrast, directly incorporates source-grounded content: it names SELFormer (from the first source), specifies it is transformer-based, notes the SELFIES notation with its advantages over SMILES, and mentions pre-training on two million drug-like compounds and fine-tuning for property prediction. This gives the paragraph concrete substance that matches the draft's theme of encoding chemical information into LLMs. B also better satisfies the task request for 'one coherent academic paragraph' with meaningful length and detail, whereas A is underdeveloped and would require significant expansion to be publication-ready.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.