FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better235 / 280 · beu-4502571e273ace82

A Machine Learning Tutorial for Operational Meteorology, Part II: Neural Networks and Deep Learning

Climate and environmental science · 2211.00147v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

A Machine Learning Tutorial for Operational Meteorology, Part II: Neural Networks and Deep Learning

3 Neural network application and discussion > 3.5 Regression

…that are occurring in a SEVIR image. Like the previous section, the goal is to compare the neural network methods presented in this paper to the traditional machine learning methods of Part 1. To make the comparison more concise, we only show the best performing regression model trained on the sub-SEVIR dataset, which was the gradient boosted trees. Recall that for this regression task, only data examples that had more than one flash in them were used as the training data. The performance of the gradient boosted tree on the validation dataset is shown in Fig. 12 a and the red bar in Fig. 13 .

…ined and their performance is characterized in the same way as regression in Part 1. The first neural network trained is the ANN using the engineered features as inputs. Akin to the results of the classification task, this ANN achieves similar performance to the gradient boosted trees. The ANN has a high density of points that follow the diagonal in Fig. 12 b and has a mean absolute error, root mean square error and R 2 R^{2} values very close to the gradient boosted tree (blue bar Fig. 13 ). That being said, the bias of the ANN is larger than the bias of the gradient boosted tree (Fig. 13 ).

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

The draft is in the middle of a results section comparing neural network methods to traditional machine learning for a regression task on SEVIR data. The paragraph before the cursor has just discussed the ANN with engineered features and its comparison to gradient boosted trees. The manuscript clearly needs continuation: the author mentions 'The first neural network trained is the ANN' and discusses 'the bias of the ANN,' implying more neural network architectures will follow (likely CNN or other architectures mentioned earlier in the paper). Output A provides a logical bridge that transitions to comparing remaining neural network architectures, maintaining the paper's structure and flow. Output B abstains with 'complete,' but the task clearly needs a next move—there are more neural networks to discuss, and ending here would leave the section structurally incomplete with an unresolved 'first neural network' implying sequels. While B's abstention is not invalid per se, it is less usable because continuing is necessary. A's text is appropriately scoped, fits the argument, and can be inserted without repair.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

The remaining neural network architectures are then trained on the same regression task and evaluated with the same metrics. Because the ANN and gradient boosted tree perform comparably, the comparison among the neural network methods themselves becomes the more informative result for this task.

Passages supplied to the Evidence version

Card 1 · Guitar Effects Recognition and Parameter Estimation with Convolutional Neural Networks

Despite the popularity of guitar effects, there is very little existing research on classification and parameter estimation of specific plugins or effect units from guitar recordings. In this paper, convolutional neural networks were used for classification and parameter estimation for 13 overdrive,…

Read full passage excerpt

Despite the popularity of guitar effects, there is very little existing research on classification and parameter estimation of specific plugins or effect units from guitar recordings. In this paper, convolutional neural networks were used for classification and parameter estimation for 13 overdrive, distortion and fuzz guitar effects. A novel dataset of processed electric guitar samples was assembled, with four sub-datasets consisting of monophonic or polyphonic samples and discrete or continuous settings values, for a total of about 250 hours of processed samples. Results were compared for networks trained and tested on the same or on a different sub-dataset. We found that discrete datasets could lead to equally high performance as continuous ones, whilst being easier to design, analyse and modify. Classification accuracy was above 80\%, with confusion matrices reflecting similarities in the effects timbre and circuits design. With parameter values between 0.0 and 1.0, the mean absolu …

Card 2 · Time Series Analysis of Blockchain-Based Cryptocurrency Price Changes

In this paper we apply neural networks and Artificial Intelligence (AI) to historical records of high-risk cryptocurrency coins to train a prediction model that guesses their price. This paper's code contains Jupyter notebooks, one of which outputs a timeseries graph of any cryptocurrency price once…

Read full passage excerpt

In this paper we apply neural networks and Artificial Intelligence (AI) to historical records of high-risk cryptocurrency coins to train a prediction model that guesses their price. This paper's code contains Jupyter notebooks, one of which outputs a timeseries graph of any cryptocurrency price once a CSV file of the historical data is inputted into the program. Another Jupyter notebook trains an LSTM, or a long short-term memory model, to predict a cryptocurrency's closing price. The LSTM is fed the close price, which is the price that the currency has at the end of the day, so it can learn from those values. The notebook creates two sets: a training set and a test set to assess the accuracy of the results. The data is then normalized using manual min-max scaling so that the model does not experience any bias; this also enhances the performance of the model. Then, the model is trained using three layers -- an LSTM, dropout, and dense layer-minimizing the loss through 50 epochs of trai …

Card 3 · Data-driven advice for interpreting local and global model predictions in bioinformatics problems

However, Lundberg et al. [2020] pointed out a potential bias of CFCs which depends on the distance from the root of a tree. The by now immensely popular alternative, SHapley Additive exPlanation (SHAP) values appear to mitigate this bias but are computationally much more expensive. Here we contribut…

Read full passage excerpt

However, Lundberg et al. [2020] pointed out a potential bias of CFCs which depends on the distance from the root of a tree. The by now immensely popular alternative, SHapley Additive exPlanation (SHAP) values appear to mitigate this bias but are computationally much more expensive. Here we contribute a thorough comparison of the explanations computed by both methods on a set of 165 publicly available classification problems in order to provide data-driven algorithm recommendations to current researchers. For random forests, we find extremely high similarities and correlations of both local and global SHAP values and CFC scores leading to very similar rankings and interpretations.

Card 4 · Evaluating and predicting the Efficiency Index for Stereotactic Radiosurgery Plans using RapidMiner GO(JAVA) Based Artificial Intelligence Algorithms

Evaluation the prediction of Efficiency index by DVH parameter for SRS treatment plans using Supervised Machine learning and the performance of predictive model algorithms of RapidMiner GO in the parameter prediction are investigated. Dose volume histogram (DVH) based Efficiency index was calculated…

Read full passage excerpt

Evaluation the prediction of Efficiency index by DVH parameter for SRS treatment plans using Supervised Machine learning and the performance of predictive model algorithms of RapidMiner GO in the parameter prediction are investigated. Dose volume histogram (DVH) based Efficiency index was calculated for 100 clinical SRS plans generated by Leksell Gamma plan, and the results were compared to predicted values produced by machine learning toolbox of RapidMiner Go, algorithms are namely, Generalized linear model (GLR), Decision Tree Model, Support Vector Machine (SVM), Gradient Boosted Trees (GBT), Random Forest (RF) and Deep learning Model (DL). Root mean square error (RMSE), Average absolute error, Absolute relative error, squared correlation and model building time were determined to evaluate the performance of each algorithm. The GLR algorithm model had square correlation of 0.974 with the smallest RMSE of 0.01, relatively high prediction speed, and fast model building time with 2.812 …

Model-reported card use

The model did not report that any card materially affected this completion.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

No continuation text was produced.

Model action：complete

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

This case does not have both with-card continuation text and a model-reported use record. It remains among all 280 tests but does not enter the reported-use comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.