← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better83 / 280 · beu-31b4bddd89140356

70 years of machine learning in geoscience in review

Climate and environmental science · 2006.13311v3

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

70 years of machine learning in geoscience in review

2 Contemporary Machine Learning in Geoscience > 2.9 The State of ML on Geoscience

Granulometry on SEM images is an example of an application in category I, where previously sediments were hand-measured in images [ Dramsch et al., 2018 ] . Applying large deformation diffeomorphic mapping of seismic data was computationally infeasible for matching 4D seismic data, however, made feasible by applying a U-net architecture to the problem of category II [ Dramsch et al., 2019 ] . The problem of earthquake magnitude prediction falls into category III due to the complexity of the system but was nevertheless approached with neural networks [ Panakkat and Adeli, 2007 ] .

…ed access. These papers include varying degrees of depth in application and model validation. One of the primary influences for the current increase in publications are new fields such as automatic seismic interpretation, as well as, publications soliciting and encouraging machine learning publications. Computer vision models were relatively straight forward to transfer to seismic interpretation tasks, with papers in this sub-sub-field ranging from single 2D line salt identification models with limited validation to 3D multi-facies interpretation with validation on a separate geographic area.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Output B is invalid by its own declaration (action: "invalid" with empty text), making it unusable regardless of content. Output A provides a coherent continuation that synthesizes the preceding discussion about categories I-III with the subsequent text about validation depth. The paragraph bridges the categorical framework to the specific issue of validation unevenness in seismic ML papers, creating logical flow. The mention of 'single 2D line salt identification models with limited validation to 3D multi-facies interpretation validated on a separate geographic area' directly echoes language already present in the visible draft, ensuring seamless insertion. The point about comparison difficulties is a natural academic observation that follows from describing heterogeneous validation practices. The output appropriately scopes itself to one paragraph and does not overreach with unsupported claims.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

No continuation text was produced.

Model action:invalid

Passages supplied to the Evidence version

Card 1 · A Machine Learning Benchmark for Facies Classification

Unfortunately, and unlike other areas of research such as computer vision, there is a lack of large publicly-available annotated datasets for seismic interpretation that can be used to train and benchmark machine learning models. To address this problem, some researchers resort to annotating their o…

Read full passage excerpt

Unfortunately, and unlike other areas of research such as computer vision, there is a lack of large publicly-available annotated datasets for seismic interpretation that can be used to train and benchmark machine learning models. To address this problem, some researchers resort to annotating their own training and testing dataset. For example, in the Netherlands F3 block, Zhao, (2018) annotated 40 inlines, Di et al., (2018) annotated 12 inlines, while Rutherford Ildstad and Bormann, (2017) only annotated a single inline for their model. The limited number of annotated sections in understandable given that the annotation process is time-consuming, requires subject matter expertise, and can be quite subjective.

Card 2 · Penobscot Dataset: Fostering Machine Learning Development for Seismic Interpretation

We have seen in the past years the flourishing of machine and deep learning algorithms in several applications such as image classification and segmentation, object detection and recognition, among many others. This was only possible, in part, because datasets like ImageNet -- with +14 million label…

Read full passage excerpt

We have seen in the past years the flourishing of machine and deep learning algorithms in several applications such as image classification and segmentation, object detection and recognition, among many others. This was only possible, in part, because datasets like ImageNet -- with +14 million labeled images -- were created and made publicly available, providing researches with a common ground to compare their advances and extend the state-of-the-art. Although we have seen an increasing interest in machine learning in geosciences as well, we will only be able to achieve a significant impact in our community if we collaborate to build such a common basis. This is even more difficult when it comes to the Oil&Gas industry, in which confidentiality and commercial interests often hinder the sharing of datasets with others. In this letter, we present the Penobscot interpretation dataset, our contribution to the development of machine learning in geosciences, more specifically in seismic inte …

Card 3 · Learning to label seismic structures with deconvolution networks and weak labels

Recently, there has been increasing interest in using deep learning techniques for various seismic interpretation tasks. However, unlike shallow machine learning models, deep learning models are often far more complex and can have hundreds of millions of free parameters. This not only means that lar…

Read full passage excerpt

Recently, there has been increasing interest in using deep learning techniques for various seismic interpretation tasks. However, unlike shallow machine learning models, deep learning models are often far more complex and can have hundreds of millions of free parameters. This not only means that large amounts of computational resources are needed to train these models, but more critically, they require vast amounts of labeled training data as well. In this work, we show how automatically-generated weak labels can be effectively used to overcome this problem and train powerful deep learning models for labeling seismic structures in large seismic volumes. To achieve this, we automatically generate thousands of weak labels and use them to train a deconvolutional network for labeling fault, salt dome, and chaotic regions within the Netherlands F3 block. Furthermore, we show how modifying the loss function to take into account the weak training labels helps reduce false positives in the lab …

Card 4 · Parallel Computation of PDFs on Big Spatial Data Using Spark

We consider big spatial data, which is typically produced in scientific areas such as geological or seismic interpretation. The spatial data can be produced by observation (e.g. using sensors or soil instrument) or numerical simulation programs and correspond to points that represent a 3D soil cube…

Read full passage excerpt

We consider big spatial data, which is typically produced in scientific areas such as geological or seismic interpretation. The spatial data can be produced by observation (e.g. using sensors or soil instrument) or numerical simulation programs and correspond to points that represent a 3D soil cube area. However, errors in signal processing and modeling create some uncertainty, and thus a lack of accuracy in identifying geological or seismic phenomenons. Such uncertainty must be carefully analyzed. To analyze uncertainty, the main solution is to compute a Probability Density Function (PDF) of each point in the spatial cube area. However, computing PDFs on big spatial data can be very time consuming (from several hours to even months on a parallel computer). In this paper, we propose a new solution to efficiently compute such PDFs in parallel using Spark, with three methods: data grouping, machine learning prediction and sampling. We evaluate our solution by extensive experiments on dif …

Card 5 · A Machine Learning Benchmark for Facies Classification

Unfortunately, and unlike other areas of research such as computer vision, there is a lack of large publicly-available annotated datasets for seismic interpretation that can be used to train and benchmark machine learning models. To address this problem, some researchers resort to annotating their o…

Read full passage excerpt

Unfortunately, and unlike other areas of research such as computer vision, there is a lack of large publicly-available annotated datasets for seismic interpretation that can be used to train and benchmark machine learning models. To address this problem, some researchers resort to annotating their own training and testing dataset. For example, in the Netherlands F3 block, Zhao, (2018) annotated 40 inlines, Di et al., (2018) annotated 12 inlines, while Rutherford Ildstad and Bormann, (2017) only annotated a single inline for their model. The limited number of annotated sections in understandable given that the annotation process is time-consuming, requires subject matter expertise, and can be quite subjective.

Model-reported card use

Card 1 · fact

Affected output: The scarcity of large, publicly available annotated datasets for seismic interpretation has been a persistent obstacle to training and benchmarking machine learning models in this domain, leading many researchers to annotate their own training and testing data.

Model explanation: This row supplied the core factual claim that annotated seismic datasets are scarce and that researchers therefore annotate their own data, which the generated sentence states.

Card 1 · fact

Affected output: For instance, in the Netherlands F3 block, Zhao (2018) annotated 40 inlines, Di et al. (2018) annotated 12 inlines, and Rutherford Ildstad and Bormann (2017) annotated only a single inline for their models.

Model explanation: This row supplied the concrete annotated inline counts and the specific studies, which the generated sentence reproduces as examples.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

Across these categories, the depth of model validation varies considerably, from single 2D line salt identification models with limited validation to 3D multi-facies interpretation validated on a separate geographic area. This unevenness complicates direct comparison of reported performance between studies.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The with-card arm returned a card-use record but no continuation text. It is excluded from reported use with text and remains in all 280 first attempts.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.