FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better63 / 280 · beu-e5d887c8e52a8217

Generalizable machine learning for stress monitoring from wearable devices: A systematic literature review

Psychology and cognitive neuroscience · 2209.15137v3

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Generalizable machine learning for stress monitoring from wearable devices: A systematic literature review

4 Discussion > 4.3 Lack of Statistical Power

…noted specific methods for establishing thresholds within biomarkers to utilise as indicators of periods of high stress (low, moderate, high). Any potential thresholds established by the machine learning algorithms, (specifically tree-based methods) during training were not examined in detail to determine any potential time-varying dynamics between the biomarkers. Ghiasi et al. [ 83 ] proposed combining HRV and EDA correlates as a single index, rather than treating each as separate indicators of ANS changes. They reported good results when validating this metric on two experimental protocols.

… of encountering a Type II error by up to 20% [ 84 ] . Statistical power has three parts: effect size (a statistical measure), sample size (number of observations or participants) and significance (typically 0.05 [ 73 ] ). Power analysis assists researchers in determining the smallest sample size suitable to detect the effect of a given experiment at a desired level of significance, as collecting larger samples are likely costlier and much harder. The use of machine learning in behavioral science experiments does not automatically negate the need for sufficient statistical power [ 75 , 36 ] .

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

The draft discusses biomarkers, machine learning thresholds, HRV/EDA indices, then transitions to statistical power—specifically Type II errors, effect size, sample size, and significance. The paragraph ends by stating ML doesn't negate need for sufficient statistical power. The task is to continue with one coherent academic paragraph. Output A provides a generic statement about low-powered studies detecting fewer true effects, inflated effect sizes, and replication failure. While accurate and related to the topic, it merely restates what the preceding paragraph already established (Type II errors, power components) and adds nothing new to the manuscript's specific argument about ML in stress/biomarker research. It's redundant continuation. Output B productively extends the argument by identifying a concrete methodological problem in the reviewed studies: small sample sizes relative to feature dimensionality in wearable signal classification. This directly connects the general statistical power discussion to the manuscript's actual subject (biomarker/ML stress studies), raises the specific issue of overfitting vs. reproducible stress response, and notes limitations on generalizability and cross-study comparison. This advances the critical review in a way that fits the draft's trajectory. Output B is more useful because it applies the general power principle to the specific research context, making a novel critical point rather than repeating established claims.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

In low-powered studies, true effects are less likely to be detected, while any effects that are detected tend to be inflated and less likely to replicate.

Passages supplied to the Evidence version

Card 1 · Power contours: optimising sample size and precision in experimental psychology and human neuroscience

Low powered studies are less able to detect a true effect (and so make Type II errors), and are therefore more likely to report spurious effects (Type I errors) compared with high-powered studies. Furthermore, any real effects that are detected are likely to have inflated effect sizes ( Ioannidis200…

Read full passage excerpt

Low powered studies are less able to detect a true effect (and so make Type II errors), and are therefore more likely to report spurious effects (Type I errors) compared with high-powered studies. Furthermore, any real effects that are detected are likely to have inflated effect sizes ( Ioannidis2008 ; Colquhoun2014 ) . These problems are common across many scientific disciplines, and estimates of power across studies in the neurosciences ( Button2013 ) yield power values in the range 8%-30%, far below the desired level of >=80%. The prevalence of low-powered studies has filled some areas of the literature with effects that fail to replicate and may well be spurious ( Ioannidis2005 ; OSC2015 ) . Most discussion of increasing statistical power has focussed on recruiting larger sample sizes, because for a given effect size, power is a function of sample size (see Figure 1 d).

Card 2 · Statistical power for cluster analysis

BACKGROUND: Cluster algorithms are gaining in popularity in biomedical research due to their compelling ability to identify discrete subgroups in data, and their increasing accessibility in mainstream software. While guidelines exist for algorithm selection and outcome evaluation, there are no firml…

Read full passage excerpt

BACKGROUND: Cluster algorithms are gaining in popularity in biomedical research due to their compelling ability to identify discrete subgroups in data, and their increasing accessibility in mainstream software. While guidelines exist for algorithm selection and outcome evaluation, there are no firmly established ways of computing a priori statistical power for cluster analysis. Here, we estimated power and classification accuracy for common analysis pipelines through simulation. We systematically varied subgroup size, number, separation (effect size), and covariance structure. We then subjected generated datasets to dimensionality reduction approaches (none, multi-dimensional scaling, or uniform manifold approximation and projection) and cluster algorithms (k-means, agglomerative hierarchical clustering with Ward or average linkage and Euclidean or cosine distance, HDBSCAN). Finally, we directly compared the statistical power of discrete (k-means), "fuzzy" (c-means), and finite mixture …

Card 3 · Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances

Abstract This work presents a statistically principled method for estimating the required number of instances in the experimental comparison of multiple algorithms on a given problem class of interest. This approach generalises earlier results by allowing researchers to design experiments based on t…

Read full passage excerpt

Abstract This work presents a statistically principled method for estimating the required number of instances in the experimental comparison of multiple algorithms on a given problem class of interest. This approach generalises earlier results by allowing researchers to design experiments based on the desired best, worst, mean or median-case statistical power to detect differences between algorithms larger than a certain threshold. Holm’s step-down procedure is used to maintain the overall significance level controlled at desired levels, without resulting in overly conservative experiments. This paper also presents an approach for sampling each algorithm on each instance, based on optimal sample size ratios that minimise the total required number of runs subject to a desired accuracy in the estimation of paired differences. A case study investigating the effect of 21 variants of a custom-tailored Simulated Annealing for a class of scheduling problems is used to illustrate the applicati …

Card 4 · Detecting the skewness of data from the sample size and the five-number summary

For clinical studies with continuous outcomes, when the data are potentially skewed, researchers may choose to report the whole or part of the five-number summary (the sample median, the first and third quartiles, and the minimum and maximum values), rather than the sample mean and standard deviatio…

Read full passage excerpt

For clinical studies with continuous outcomes, when the data are potentially skewed, researchers may choose to report the whole or part of the five-number summary (the sample median, the first and third quartiles, and the minimum and maximum values), rather than the sample mean and standard deviation. For the studies with skewed data, if we include them in the classical meta-analysis for normal data, it may yield misleading or even wrong conclusions. In this paper, we develop a flow chart and three new tests for detecting the skewness of data from the sample size and the five-number summary. Simulation studies demonstrate that our new tests are able to control the type I error rates, and meanwhile provide good statistical power. A real data example is also analyzed to demonstrate the usefulness of the skewness tests in meta-analysis and evidence-based practice.

Card 5 · Power contours: optimising sample size and precision in experimental psychology and human neuroscience

Low powered studies are less able to detect a true effect (and so make Type II errors), and are therefore more likely to report spurious effects (Type I errors) compared with high-powered studies. Furthermore, any real effects that are detected are likely to have inflated effect sizes ( Ioannidis200…

Read full passage excerpt

Low powered studies are less able to detect a true effect (and so make Type II errors), and are therefore more likely to report spurious effects (Type I errors) compared with high-powered studies. Furthermore, any real effects that are detected are likely to have inflated effect sizes ( Ioannidis2008 ; Colquhoun2014 ) . These problems are common across many scientific disciplines, and estimates of power across studies in the neurosciences ( Button2013 ) yield power values in the range 8%-30%, far below the desired level of >=80%. The prevalence of low-powered studies has filled some areas of the literature with effects that fail to replicate and may well be spurious ( Ioannidis2005 ; OSC2015 ) . Most discussion of increasing statistical power has focussed on recruiting larger sample sizes, because for a given effect size, power is a function of sample size (see Figure 1 d).

Model-reported card use

Card 1 · fact

Affected output: true effects are less likely to be detected, while any effects that are detected tend to be inflated and less likely to replicate

Model explanation: Supplied the concrete consequences of low power used in the new paragraph, replacing a generic statement about power being important.

Card 5 · expression

Affected output: In low-powered studies, true effects are less likely to be detected, while any effects that are detected tend to be inflated

Model explanation: Provided the contrastive observation-then-consequence sentence shape used to open the paragraph.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

In the reviewed studies, sample sizes were frequently small relative to the number of features extracted from wearable signals, which increases the risk that reported classification performance reflects overfitting rather than a reproducible stress response. This imbalance between feature dimensionality and the number of participants limits the generalizability of the reported models and complicates comparison across studies.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output A is a safe, grounded continuation that accurately summarizes the first source about low-powered studies and Type II errors, directly connecting to the draft's discussion of statistical power and the 20% Type II error mention. However, it is quite short and somewhat redundant with what the draft already states. Output B attempts a more substantive continuation relevant to the manuscript's machine learning focus, but introduces unsupported claims: 'sample sizes were frequently small relative to the number of features extracted from wearable signals' and 'increases the risk that reported classification performance reflects overfitting' and 'This imbalance between feature dimensionality and the number of participants' — none of these claims about feature dimensionality, overfitting, or specific sample sizes in 'reviewed studies' are supported by any supplied source or the draft text. The sources discuss statistical power, cluster analysis, algorithm comparison, and skewness detection, but none mention wearable signals, feature extraction, dimensionality relative to participants, or overfitting in classification models. Despite B being more topically ambitious and paragraph-like, it contains material unsupported claims. Since unsupportedClaim=true requires usable=false, but B is not 'unusable' in the sense of being empty or malformed — it's a coherent paragraph — I need to check: the instruction says 'Any output with unsupportedClaim=true must have usable=false.' So B must be marked usable=false. Re-evaluating: A is usable, grounded, but minimal. B has unsupported claims and must be usable=false. However, A is very short — arguably not a full paragraph. But it is a coherent continuation. Given the constraint that unsupportedClaim=true → usable=false, B is not usable. A is usable but minimal. The preference should be A since B is unusable. Wait — let me re-read: A says 'In low-powered studies, true effects are less likely to be detected, while any effects that are detected tend to be inflated and less likely to replicate.' This is supported by the first source. But is it a 'paragraph'? It's one sentence. The task asks for 'one coherent academic paragraph.' A single sentence may not satisfy 'paragraph' requirements. But it's not empty. B is multiple sentences, more paragraph-like, but unsupported. Given the strict rule, B must be usable=false. Between a minimal but usable A and an unusable B, preference goes to A. Actually, re-checking: is A really supported? The draft mentions 'Type II error by up to 20%' and sources discuss low power. A restates this. But does it add anything? It's a continuation. It's safe. Let me verify B's claims again: 'sample sizes were frequently small relative to the number of features extracted from wearable signals' — no source mentions wearable signals or feature counts. 'overfitting rather than a reproducible stress response' — no source mentions overfitting or stress response reproducibility. 'This imbalance between feature dimensionality and the number of participants' — no source mentions feature dimensionality. These are all unsupported. So B: unsupportedClaim=true, usable=false. A: no unsupported claim, usable=true. Preference: A.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.