FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better236 / 280 · beu-03a573652589d24d

Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey

Climate and environmental science · 2312.03014v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Foundation Models for Weather and Climate Data Understanding: A Comprehensive Survey

IX Challenges, Outlook, and Opportunities > IX-D Generalizability of Models

…causal conclusions from climate models using XAI tools refers to the limited causality these tools can provide. Physics-guided AI, also known as knowledge-guided or physics-informed AI, is one avenue researchers are exploring to impose physical realism and mitigate the effects of false correlations on predictive algorithms [ 248 , 294 , 244 , 246 ] . However, research in this area is still nascent. Thus, while strides have been made in enhancing the interpretability of AI models, substantial challenges remain, highlighting the need for continued research and development in this critical area.

…her and climate analysis, foundation models may exhibit suboptimal performance when predicting Non-IID data beyond the training dataset. A notable example of this is the use of foundation models for the prediction of extreme events outside their trained distribution. These biased and anomalous data often induce significant performance degradation in the model. This is especially the case as the warming climate alters the Earth’s spatiotemporal distribution. The existing relationships that currently describe the predictive variables and extreme climate events may no longer apply in the future.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Both outputs provide coherent continuations, but Output A better serves the manuscript's argument. The draft establishes two problems: (1) foundation models fail on Non-IID/extreme event prediction due to distributional shift from climate change, and (2) existing relationships may no longer apply. Output A directly extends this by identifying spatial and temporal heterogeneity of observations as a further constraint on generalizability, maintaining focus on the core problem of model failure under changing conditions. It specifically addresses 'data-rich regions' transferring poorly to 'sparsely observed areas' and regimes with 'few analogues'—concretely elaborating the distributional shift problem without introducing new solution concepts. Output B pivots to 'transfer learning and domain adaptation' as 'strategies,' which shifts the paragraph toward solutions rather than continuing the diagnosis of problems. This creates a mild argumentative discontinuity: the draft ends with 'relationships may no longer apply,' establishing unresolved challenges, and Output B's turn to solutions feels premature. Additionally, Output B's claim that effectiveness 'remains unevenly characterized' is more abstract and less directly grounded in the specific climate-AI context established. Output A maintains tighter thematic coherence with the visible draft's emphasis on spatiotemporal distribution changes and their consequences for model reliability.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

Transfer learning and domain adaptation offer complementary strategies for extending pretrained weather and climate foundation models to unseen regions, variables, or time periods, yet their effectiveness under distribution shift remains unevenly characterized across tasks.

Passages supplied to the Evidence version

Card 1 · Analysis of Speech Separation Performance Degradation on Emotional Speech Mixtures

Despite recent strides made in Speech Separation, most models are trained on datasets with neutral emotions. Emotional speech has been known to degrade performance of models in a variety of speech tasks, which reduces the effectiveness of these models when deployed in real-world scenarios. In this p…

Read full passage excerpt

Despite recent strides made in Speech Separation, most models are trained on datasets with neutral emotions. Emotional speech has been known to degrade performance of models in a variety of speech tasks, which reduces the effectiveness of these models when deployed in real-world scenarios. In this paper we perform analysis to differentiate the performance degradation arising from the emotions in speech from the impact of out-of-domain inference. This is measured using a carefully designed test dataset, Emo2Mix, consisting of balanced data across all emotional combinations. We show that even models with strong out-of-domain performance such as Sepformer can still suffer significant degradation of up to 5.1 dB SI-SDRi on mixtures with strong emotions. This demonstrates the importance of accounting for emotions in real-world speech separation applications.

Card 2 · An Encryption Method of ConvMixer Models without Performance Degradation

In this paper, we propose an encryption method for ConvMixer models with a secret key. Encryption methods for DNN models have been studied to achieve adversarial defense, model protection and privacy-preserving image classification. However, the use of conventional encryption methods degrades the pe…

Read full passage excerpt

In this paper, we propose an encryption method for ConvMixer models with a secret key. Encryption methods for DNN models have been studied to achieve adversarial defense, model protection and privacy-preserving image classification. However, the use of conventional encryption methods degrades the performance of models compared with that of plain models. Accordingly, we propose a novel method for encrypting ConvMixer models. The method is carried out on the basis of an embedding architecture that ConvMixer has, and models encrypted with the method can have the same performance as models trained with plain images only when using test images encrypted with a secret key. In addition, the proposed method does not require any specially prepared data for model training or network modification. In an experiment, the effectiveness of the proposed method is evaluated in terms of classification accuracy and model protection in an image classification task on the CIFAR10 dataset.

Card 3 · PUMA: Performance Unchanged Model Augmentation for Training Data Removal

Preserving the performance of a trained model while removing unique characteristics of marked training data points is challenging. Recent research usually suggests retraining a model from scratch with remaining training data or refining the model by reverting the model optimization on the marked dat…

Read full passage excerpt

Preserving the performance of a trained model while removing unique characteristics of marked training data points is challenging. Recent research usually suggests retraining a model from scratch with remaining training data or refining the model by reverting the model optimization on the marked data points. Unfortunately, aside from their computational inefficiency, those approaches inevitably hurt the resulting model's generalization ability since they remove not only unique characteristics but also discard shared (and possibly contributive) information. To address the performance degradation problem, this paper presents a novel approach called Performance Unchanged Model Augmentation~(PUMA). The proposed PUMA framework explicitly models the influence of each training data point on the model's generalization ability with respect to various performance criteria. It then complements the negative impact of removing marked data by reweighting the remaining data optimally. To demonstrate …

Card 4 · Record-breaking temperatures reveal a warming climate

We present a mathematical analysis of records drawn from independent random variables with a drifting mean. To leading order the change in the record rate is proportional to the ratio of the drift velocity to the standard deviation of the underlying distribution. We apply the theory to time series o…

Read full passage excerpt

We present a mathematical analysis of records drawn from independent random variables with a drifting mean. To leading order the change in the record rate is proportional to the ratio of the drift velocity to the standard deviation of the underlying distribution. We apply the theory to time series of daily temperatures for given calendar days, obtained from historical climate recordings of European and American weather stations as well as re-analysis data. We conclude that the change in the mean temperature has increased the rate of record-breaking events in a moderate but significant way: for the European station data covering the time period 1976–2005, we find that about 5 of the 17 high temperature records observed on average in 2005 can be attributed to the warming climate.

Model-reported card use

The model did not report that any card materially affected this completion.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

Beyond distributional shift, the generalizability of weather and climate foundation models is further constrained by the spatial and temporal heterogeneity of the observational record. Models pretrained on data-rich regions or well-sampled historical periods may transfer poorly to sparsely observed areas and to regimes for which the training corpus provides few analogues.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

This case does not have both with-card continuation text and a model-reported use record. It remains among all 280 tests but does not enter the reported-use comparison.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.