FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteAbout the same97 / 280 · beu-d6fe76b321cc1f89

A Machine Learning Tutorial for Operational Meteorology, Part II: Neural Networks and Deep Learning

Climate and environmental science · 2211.00147v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Machine Learning Tutorial for Operational Meteorology, Part II: Neural Networks and Deep Learning

3 Neural network application and discussion > 3.4 Classification

… rally good skill (AUC > > 0.9, CSI > > 0.7 red; dash-dot line Fig. 10 a; red square Fig. 10 ), the result is worse than all other methods discussed so far (Fig. 10 b).

The last neural network trained for task (1) is a CNN. To be explicit, recall that the CNN uses the raw images as inputs and convolves them to extract features. The result of training a CNN on the sub-SEVIR data provides one of the best performing machine learning methods (black line Fig. 10 a; red star Fig. 10 ), matching the skill of the gradient boosting trees and the ANN trained on the engineered features. Note that the CNN only marginally outperforms the other methods on the performance diagram and is likely not a significant difference.

It might be surprising to see that the ANNs do not substantially outperform the tree based methods on this task despite the added complexity of neural networks and their training. This is a common pitfall for machine learning users. In fact, there is growing evidence that the tree based methods can often outperform neural networks and deep learning on tabular data ( Shwartz-Ziv and Armon, 2022 , i.e., data contained in a spreadsheet;) . A distinction is made between tabular and non-tabular datasets here because spatial details can contain substantial information for the machine learning task and isn’t always easily quantified into a tabular dataset. For example, consider assessing a storm’s tornadic potential. While using composite radar reflectivity as a feature could be useful (e.g., strong radar value means a strong storm), there is likely more information contained

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: About the same

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

This rationale combines reviews with different display orders.

Order-swapped judges disagreed. Pass 1: Both outputs complete the sentence well, continuing the argument about spatial information in radar data and connecting to CNNs. Output A is more concise and fits better with the draft's measured, academic tone. The draft has been careful not to overstate claims (e.g., 'could be useful,' 'likely more information'), and Output A matches this restraint with 'what convolutional neural networks are designed to exploit'—a factual, well-established claim. Output B adds 'more effectively than tabular methods,' which slightly overreaches by introducing a comparative performance claim that isn't directly supported by the preceding text and could require additional qualification. The draft's point is that spatial information exists and isn't easily tabularized, not that CNNs necessarily outperform tabular methods on this specific task. Output A's narrower claim about what CNNs are 'designed to exploit' is more precisely aligned with the argument structure. Pass 2: Both outputs complete the sentence effectively and make the same core point: spatial structure in radar data contains more information than single values, making CNNs more suitable than tabular methods. Output A is slightly preferable because (1) it uses 'any single value' which more precisely contrasts with the draft's 'composite radar reflectivity as a feature' and 'strong radar value means a strong storm' phrasing, whereas B's 'single point value' introduces a slightly different term ('point value' vs. 'composite reflectivity'); (2) A's 'operate directly on the image' more explicitly explains the mechanism by which CNNs exploit spatial information, which better serves the draft's educational purpose of distinguishing tabular vs. non-tabular approaches; (3) A's 'more effectively than tabular methods' ties back more cleanly to the draft's earlier contrast between tree-based/ANN methods and the tabular/non-tabular distinction being established. B's ending 'are designed to exploit' is somewhat more passive and doesn't reinforce the comparative framework as strongly.

Evidence

Same model and position · retrieved passages supplied · first output

in the spatial structure of the radar echo than in a single point value, and this spatial information is what convolutional neural networks are designed to exploit.

Passages supplied to the Evidence version

Card 1 · Classification of diffusion modes in single-particle tracking data: Feature-based versus deep-learning approach

Single-particle trajectories measured in microscopy experiments contain important information about dynamic processes occurring in a range of materials including living cells and tissues. However, extracting that information is not a trivial task due to the stochastic nature of the particles' moveme…

Read full passage excerpt

Single-particle trajectories measured in microscopy experiments contain important information about dynamic processes occurring in a range of materials including living cells and tissues. However, extracting that information is not a trivial task due to the stochastic nature of the particles' movement and the sampling noise. In this paper, we adopt a deep-learning method known as a convolutional neural network (CNN) to classify modes of diffusion from given trajectories. We compare this fully automated approach working with raw data to classical machine learning techniques that require data preprocessing and extraction of human-engineered features from the trajectories to feed classifiers like random forest or gradient boosting. All methods are tested using simulated trajectories for which the underlying physical model is known. From the results it follows that CNN is usually slightly better than the feature-based methods, but at the cost of much longer processing times. Moreover, ther …

Card 2 · Ensemble machine learning model trained on a new synthesized dataset generalizes well for stress prediction using wearable devices

INTRODUCTION: Advances in wearable sensor technology have enabled the collection of biomarkers that may correlate with levels of elevated stress. While significant research has been done in this domain, specifically in using machine learning to detect elevated levels of stress, the challenge of prod…

Read full passage excerpt

INTRODUCTION: Advances in wearable sensor technology have enabled the collection of biomarkers that may correlate with levels of elevated stress. While significant research has been done in this domain, specifically in using machine learning to detect elevated levels of stress, the challenge of producing a machine learning model capable of generalizing well for use on new, unseen data remain. Acute stress response has both subjective, psychological and objectively measurable, biological components that can be expressed differently from person to person, further complicating the development of a generic stress measurement model. Another challenge is the lack of large, publicly available datasets labeled for stress response that can be used to develop robust machine learning models. In this paper, we first investigate the generalization ability of models built on datasets containing a small number of subjects, recorded in single study protocols. Next, we propose and evaluate methods comb …

Card 3 · Application of machine learning in the determination of impact parameter in the Sn132+Sn124 system

Background: $^{132}\mathrm{Sn}+^{124}\mathrm{Sn}$ collisions at a beam energy of 270 $\mathrm{MeV}/\mathrm{nucleon}$ were performed at the Radioactive Isotope Beam Factory (RIBF) in RIKEN to investigate the nuclear equation of state. Reconstructing the impact parameter is one of the important tasks…

Read full passage excerpt

Background: $^{132}\mathrm{Sn}+^{124}\mathrm{Sn}$ collisions at a beam energy of 270 $\mathrm{MeV}/\mathrm{nucleon}$ were performed at the Radioactive Isotope Beam Factory (RIBF) in RIKEN to investigate the nuclear equation of state. Reconstructing the impact parameter is one of the important tasks in the experiment as it relates to many observables.Purpose: In this work, we employ three commonly used algorithms in machine learning, the artificial neural network (ANN), the convolutional neural network (CNN), and the light gradient boosting machine (LightGBM), to determine the impact parameter by analyzing either the charged particle spectra or several features simulated with events from the ultrarelativistic quantum molecular dynamics (UrQMD) model.Method: To closely imitate experimental data and investigate the generalizability of the trained machine learning algorithms, incompressibility of nuclear equation of state and the in-medium nucleon-nucleon cross sections are varied in the U …

Card 4 · Boost then Convolve: Gradient Boosting Meets Graph Neural Networks

Graph neural networks (GNNs) are powerful models that have been successful in various graph representation learning tasks. Whereas gradient boosted decision trees (GBDT) often outperform other machine learning methods when faced with heterogeneous tabular data. But what approach should be used for g…

Read full passage excerpt

Graph neural networks (GNNs) are powerful models that have been successful in various graph representation learning tasks. Whereas gradient boosted decision trees (GBDT) often outperform other machine learning methods when faced with heterogeneous tabular data. But what approach should be used for graphs with tabular node features? Previous GNN models have mostly focused on networks with homogeneous sparse features and, as we show, are suboptimal in the heterogeneous setting. In this work, we propose a novel architecture that trains GBDT and GNN jointly to get the best of both worlds: the GBDT model deals with heterogeneous features, while GNN accounts for the graph structure. Our model benefits from end-to-end optimization by allowing new trees to fit the gradient updates of GNN. With an extensive experimental comparison to the leading GBDT and GNN models, we demonstrate a significant increase in performance on a variety of graphs with tabular features. The code is available: https:// …

Card 5 · Machine learning based texture analysis of patella from X-rays for detecting patellofemoral osteoarthritis

OBJECTIVE: To assess the ability of texture features for detecting radiographic patellofemoral osteoarthritis (PFOA) from knee lateral view radiographs. DESIGN: We used lateral view knee radiographs from The Multicenter Osteoarthritis Study (MOST) public use datasets (n = 5507 knees). Patellar regio…

Read full passage excerpt

OBJECTIVE: To assess the ability of texture features for detecting radiographic patellofemoral osteoarthritis (PFOA) from knee lateral view radiographs. DESIGN: We used lateral view knee radiographs from The Multicenter Osteoarthritis Study (MOST) public use datasets (n = 5507 knees). Patellar region-of-interest (ROI) was automatically detected using landmark detection tool (BoneFinder), and subsequently, these anatomical landmarks were used to extract three different texture ROIs. Hand-crafted features, based on Local Binary Patterns (LBP), were then extracted to describe the patellar texture. First, a machine learning model (Gradient Boosting Machine) was trained to detect radiographic PFOA from the LBP features. Furthermore, we used end-to-end trained deep convolutional neural networks (CNNs) directly on the texture patches for detecting the PFOA. The proposed classification models were eventually compared with more conventional reference models that use clinical assessments and par …

Model-reported card use

The model did not report that any card materially affected this completion.

No Evidence

Same model and position · no retrieved passages · first output

in the spatial structure of the radar echo than in any single value, which is why convolutional neural networks that operate directly on the image can exploit this information more effectively than tabular methods.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

This case does not have both with-card continuation text and a model-reported use record. It remains among all 280 tests but does not enter the reported-use comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.