FlowingFlowing
← Back to all development cases
Development case · blind model judgment pending independent human review; not a formal Benchmark conclusion.
AutocompleteTie6 / 162 · 7dce670e0a924e05

Metagenomic Analysis using Phylogenetic Placement -- A Review of the First Decade

Quantitative biology · 2202.03534v2

FLOWING EVIDENCE BENCHMARK

How do we tell whether Evidence helps?

At the same writing position, with the same model and task, how does supplying retrieved paper passages change the first output? We compare matched versions and retain ties, unusable outputs and incomplete reviews.

SAME DRAFT · EVIDENCE ON OR OFF

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED INPUTS

SHARED BY BOTH

Manuscript context, model, task and prompt

A · With Evidence

Retrieved paper passages supplied

B · Without Evidence

No retrieved passages supplied

LLM

Same model and version

A → first output

B → first output

Blind judge agent

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Order check: X / Y → Y / X

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The judge sees the same draft and both first outputs without knowing which received Evidence.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are autocomplete positions stratified?

Before seeing generation outcomes, we check whether a retrieved passage contains a specific proposition that directly supports the next writing move. Those positions appear in the left opportunity group; the rest are ordinary positions on the right. We select a balanced sample from admitted papers in each field. The 50/50 split is experimental, not a measure of how often either type occurs in writing.

AUTOCOMPLETE · 110

Biology, statistics and astrophysics

55 positions on each side; statistics uses the ten-paper rerun.

AUTOCOMPLETE · 52

Psychology and climate science

26 positions on each side; psychology includes nine papers and climate science four.

How are the table percentages calculated?

Across five fields, 51 of 81 left-column positions preferred the Evidence version. Ties, pairs where both versions were unusable and incomplete reviews remain in the denominator.

51Evidence version preferred
÷
81All positions in this group
=
63%Evidence preference in this group

Source contribution is a separate review: 16 of 22 Evidence wins entered into source review directly used retrieved papers; another 29 wins await review.

These are development-stage model judgments pending independent human review. They are not formal Benchmark conclusions and do not, on their own, establish causality.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Metagenomic Analysis using Phylogenetic Placement -- A Review of the First Decade

1 Introduction

…utionary relationships between these query sequences and the reference species/sequences, and thus go beyond simple species identification. Phylogenetic placement has found applications in a variety of situations, such as data cleaning and retention ( Mahe2017 ) , inference of new clades ( Dunthorn2014 ; Bass2018a ) , estimation of ecological profiles ( Keck2018 ) , identification of low-coverage genomes of viral strains ( Muhlemann2020a ) , phylogenetic analysis of viruses such as SARS-CoV-2 ( Morel2020sars ; Turakhia2021 ) , and in clinical studies of microbial diseases ( Srinivasan2012 ) .

When analyzing the resulting data, there are two complementary interpretations of phylogenetic placement: (1) as a set of individual sequences, placed with respect to the reference phylogeny, e. g., for taxonomic assignment, phylo-geographic tracing, or even possible clinical relevance; (2) as a combined distribution of sequences on the tree, characterizing the sampled environment at a given point in time or space to examine the composition of a species community as a whole, for instance as a means of sample ordination and visualization, and association with environmental variables.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionJudgment: Tie

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Evidence

Same model and position · retrieved passages supplied · first output

These two interpretations are not mutually exclusive, and many placement-based studies move between them, using individual placements to validate or interpret the aggregate distribution and vice versa. The choice of interpretation also shapes downstream analysis, since per-sequence placements support taxonomic and phylogenetic questions while the combined distribution supports community-level comparisons across samples.

Passages supplied to the Evidence version

Minimizing the number of optimizations for efficient community dynamic flux balance analysis

Here, we show that a basis for the space of internal fluxes can be chosen for each microbe in a community and this basis can be used to simulate forward by solving a relatively inexpensive system of linear equations at most time steps, instead of the full optimization problem. Using our method, we c…

Read full passage excerpt

Here, we show that a basis for the space of internal fluxes can be chosen for each microbe in a community and this basis can be used to simulate forward by solving a relatively inexpensive system of linear equations at most time steps, instead of the full optimization problem. Using our method, we can use this solution as long as the resulting metabolic activity remains within the optimization problem’s constraints (i.e. the solution remains feasible). As the solution becomes infeasible, it first becomes a feasible but degenerate solution to the optimization problem, and we can solve a different but related optimization problem to choose an appropriate basis to continue forward simulation. We show using an eight species community that this is an efficient and robust method for computing dynamic flux balance analysis simulations, and so is capable of simulating communities of organisms.

No Evidence

Same model and position · no retrieved passages · first output

These two interpretations are not mutually exclusive, and many placement-based studies move between them, using individual placements to validate or interpret the aggregate distribution and vice versa. The choice of interpretation also shapes downstream analysis, since per-sequence placements support taxonomic and phylogenetic questions while the combined distribution supports community-level comparisons across samples.

Blind model judgment

The blind model judge did not prefer either output.

Read the original model judgment

Both outputs are identical. They provide a coherent continuation that logically extends the draft's discussion of two complementary interpretations of phylogenetic placement. The paragraph correctly notes that the interpretations are not mutually exclusive, describes how studies move between them, and explains how the choice shapes downstream analysis—connecting per-sequence placements to taxonomic/phylogenetic questions and combined distributions to community-level comparisons. This is a natural, grounded extension of the draft's framework without introducing unsupported claims, invented citations, or manuscript-specific methods/results. The source provided is irrelevant to this topic (it concerns flux balance analysis), but neither output attempts to use it, which is appropriate. No citation markers are generated. Both outputs are equally usable.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.