FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better214 / 280 · beu-2de05c68db34e2de

Addressing Outcome Reporting Bias in Meta-analysis: A Selection Model Perspective

Psychology and cognitive neuroscience · 2408.05747v2

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Addressing Outcome Reporting Bias in Meta-analysis: A Selection Model Perspective

4 Discussion

Overall, the parameter variations tested here ( γ = 0 \gamma=0 to 3 3 , β = 0 \beta=0 to 3 3 ) do not drastically alter the results in Figure 2 but reveal how selection function shape and strictness can impact the magnitude of adjustment. Nevertheless, since the true mechanism driving outcome reporting bias is unknown in practice, we strongly recommend using a sensitivity analysis approach with multiple plausible selection functions and parameter combinations. This enhances the robustness of conclusions and helps quantify the potential impact of ORB on meta-analytic estimates.

…model framework, a common method in publication bias (PB) literature. This methodology allows one to incorporate contributions from unreported study outcomes based on different assumed missing data mechanisms, specified via selection functions. Our proposed selection functions expand on existing methods, including those from previous works like Copas et al., [12] , by being more flexible in the missing data mechanisms assumed, utilizing information from all identified unreported study outcomes, and jointly estimating both treatment effect and heterogeneity variance via a random effects model.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Both outputs provide usable continuations, but A is substantially more valuable at this position. The draft ends with a strong recommendation for sensitivity analysis, and Output A directly explains *why* this recommendation is methodologically necessary—namely, that the unidentifiability of the true outcome reporting mechanism makes sensitivity analysis essential rather than optional. This creates a coherent logical arc: the preceding paragraph notes the unknown mechanism, recommends sensitivity analysis, and A explains the epistemic basis for that recommendation. The fit is tight and the scope is appropriate. Output B is factually accurate and cites relevant related work (Copas-Heckman-type analyses for GLMMs, which appears in the sources). However, it introduces a new topical direction—generalized linear mixed models for rare-event meta-analysis—that is not clearly connected to the current manuscript's focus. The draft discusses selection functions in a random effects model with parameters γ and β, but does not mention GLMMs, rare events, or the specific Copas-Heckman extension described in the source. This creates a scope mismatch: B reads like a literature review addition that drifts toward a different methodological domain rather than completing the current argument. The paragraph would need substantial repair to connect back to the manuscript's actual framework. A wins on fit, appropriate scope, and argumentative coherence.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

The selection function framework adopted here is closely related to Copas-Heckman-type sensitivity analyses, which have recently been extended to generalized linear mixed models for rare-event meta-analysis.

Passages supplied to the Evidence version

Card 1 · Copas-Heckman-type sensitivity analysis for publication bias in rare-event meta-analysis under generalized linear mixed models

In systematic reviews and meta-analyses, publication bias (PB) is one of the serious concerns and mainly induced by selective publication of academic literatures. Although many methods have been proposed to deal with PB, almost all the methods are based on the normal-normal (NN) random-effects model…

Read full passage excerpt

In systematic reviews and meta-analyses, publication bias (PB) is one of the serious concerns and mainly induced by selective publication of academic literatures. Although many methods have been proposed to deal with PB, almost all the methods are based on the normal-normal (NN) random-effects model assuming that data are normally distributed in both the within-study and the between-study levels. For rare-event meta-analysis where data contain rare occurrences of events, the standard NN random-effects model may perform poorly. Instead, some generalized linear mixed models (GLMMs) which employ the exact distribution for the number of events in within-study level provide alternatives and have been widely used in practice. However, limited methods can be applied to deal with PB in the GLMMs. To address this limitation, we propose a framework of sensitivity analysis for evaluating the impact of PB in various GLMMs. The proposed framework is developed based on the famous Copas-Heckman-type …

Card 2 · Generalized random forests

We propose generalized random forests, a method for nonparametric statistical estimation based on random forests (Breiman [Mach. Learn. 45 (2001) 5–32]) that can be used to fit any quantity of interest identified as the solution to a set of local moment equations. Following the literature on local m…

Read full passage excerpt

We propose generalized random forests, a method for nonparametric statistical estimation based on random forests (Breiman [Mach. Learn. 45 (2001) 5–32]) that can be used to fit any quantity of interest identified as the solution to a set of local moment equations. Following the literature on local maximum likelihood estimation, our method considers a weighted set of nearby training examples; however, instead of using classical kernel weighting functions that are prone to a strong curse of dimensionality, we use an adaptive weighting function derived from a forest designed to express heterogeneity in the specified quantity of interest. We propose a flexible, computationally efficient algorithm for growing generalized random forests, develop a large sample theory for our method showing that our estimates are consistent and asymptotically Gaussian and provide an estimator for their asymptotic variance that enables valid confidence intervals. We use our approach to develop new methods for …

Card 3 · Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies

BACKGROUND: Random-effects meta-analysis is commonly performed by first deriving an estimate of the between-study variation, the heterogeneity, and subsequently using this as the basis for combining results, i.e., for estimating the effect, the figure of primary interest. The heterogeneity variance…

Read full passage excerpt

BACKGROUND: Random-effects meta-analysis is commonly performed by first deriving an estimate of the between-study variation, the heterogeneity, and subsequently using this as the basis for combining results, i.e., for estimating the effect, the figure of primary interest. The heterogeneity variance estimate however is commonly associated with substantial uncertainty, especially in contexts where there are only few studies available, such as in small populations and rare diseases. METHODS: Confidence intervals and tests for the effect may be constructed via a simple normal approximation, or via a Student-t distribution, using the Hartung-Knapp-Sidik-Jonkman (HKSJ) approach, which additionally uses a refined estimator of variance of the effect estimator. The modified Knapp-Hartung method (mKH) applies an ad hoc correction and has been proposed to prevent counterintuitive effects and to yield more conservative inference. We performed a simulation study to investigate the behaviour of the …

Card 4 · Incorporating genuine prior information about between-study heterogeneity in random effects pairwise and network meta-analyses

Background: Pairwise and network meta-analyses using fixed effect and random effects models are commonly applied to synthesise evidence from randomised controlled trials. The models differ in their assumptions and the interpretation of the results. The model choice depends on the objective of the an…

Read full passage excerpt

Background: Pairwise and network meta-analyses using fixed effect and random effects models are commonly applied to synthesise evidence from randomised controlled trials. The models differ in their assumptions and the interpretation of the results. The model choice depends on the objective of the analysis and knowledge of the included studies. Fixed effect models are often used because there are too few studies with which to estimate the between-study standard deviation from the data alone. Objectives: The aim is to propose a framework for eliciting an informative prior distribution for the between-study standard deviation in a Bayesian random effects meta-analysis model to genuinely represent heterogeneity when data are sparse. Methods: We developed an elicitation method using external information such as empirical evidence and experts' beliefs on the 'range' of treatment effects in order to infer the prior distribution for the between-study standard deviation. We also developed the m …

Card 5 · A Random Effects Model-based Method of Moments Estimation of Causal Effect in Mendelian Randomization Studies

Recent advances in genotyping technology have delivered a wealth of genetic data, which is rapidly advancing our understanding of the underlying genetic architecture of complex diseases. Mendelian Randomization (MR) leverages such genetic data to estimate the causal effect of an exposure factor on a…

Read full passage excerpt

Recent advances in genotyping technology have delivered a wealth of genetic data, which is rapidly advancing our understanding of the underlying genetic architecture of complex diseases. Mendelian Randomization (MR) leverages such genetic data to estimate the causal effect of an exposure factor on an outcome from observational studies. In this paper, we utilize genetic correlations to summarize information on a large set of genetic variants associated with the exposure factor. Our proposed approach is a generalization of the MR-inverse variance weighting (IVW) approach where we can accommodate many weak and pleiotropic effects. Our approach quantifies the variation explained by all valid instrumental variables (IVs) instead of estimating the individual effects and thus could accommodate weak IVs. This is particularly useful for performing MR estimation in small studies, or minority populations where the selection of valid IVs is unreliable and thus has a large influence on the MR estim …

Model-reported card use

Card 1 · fact

Affected output: Copas-Heckman-type sensitivity analyses, which have recently been extended to generalized linear mixed models for rare-event meta-analysis

Model explanation: This row supplies the specific fact that Copas-Heckman-type sensitivity analysis has been extended to GLMMs for rare-event meta-analysis, which the draft alone did not provide.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

A key limitation of the selection model framework is that the true outcome reporting mechanism is unidentifiable from the observed data alone, so the resulting adjustment depends on the assumed selection function. This dependence is why the sensitivity analysis recommended above is not merely a robustness check but a necessary component of the method, since different plausible selection functions can yield different adjusted estimates even when the underlying data are held fixed.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the no-Evidence version. Its original rationale appears below.

Read the original model judgment

Both outputs provide coherent continuations, but A better satisfies the task requirements. Output A directly addresses the gap between the two draft paragraphs: it explains why sensitivity analysis is necessary (the unidentifiable mechanism) and connects back to the recommendation in the first paragraph, creating a logical bridge. It also stays within the scope of what the sources and draft support—the draft explicitly states 'the true mechanism driving outcome reporting bias is unknown' and recommends sensitivity analysis. Output B introduces a connection to Copas-Heckman-type analyses for GLMMs and rare-event meta-analysis, which, while mentioned in source [1], is not directly relevant to the draft's content. The draft discusses selection functions expanding on Copas et al. [12] in general terms, but does not mention GLMMs or rare-event meta-analysis. Output B's mention of these specific extensions introduces content that is topically related but not supported by the draft's actual scope. Output A provides a more self-contained, coherent paragraph that advances the manuscript's argument without importing unsupported specific extensions.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.