FlowingFlowing
← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better161 / 280 · beu-b58a5bcdc06bc95a

Data Integration by combining big data and survey sample data for finite population inference

Statistics · 2003.12156v3

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Data Integration by combining big data and survey sample data for finite population inference

1 Introduction

… Kaplan et al. (2014) , Keiding and Louis (2016) , Lohr and Raghunathan (2017) , Sax et al. (2003) , and Tam and Kim (2018) ). Specific discussion of these biases can be found in Baeza-Yates (2018) for web data; Brodie et al. (2018) on data from smart phones and wearable devices; and Olteanu et al. (2019) for social media data. The weighting methods considered in Valliant and Dever (2011) and Elliott and Valliant (2017) are based on a missing-at-random assumption (MAR) of Rubin (1976) . The MAR assumption is a strong assumption and there is no way to verify this assumption from the data only.

…ata integration. Bethlehem (2016) discussed practical issues in sample matching for solving the selection bias in the non-probability sample. While matching-based imputation is promising and potentially useful in practice, it is still based on the missing-at-random assumption. Chen et al. (2020) also considered a weighting adjustment method based on parametric model assumptions on the selection mechanism for the non-probability sample, but the MAR assumption is still required. Rao (2020) provided comprehensive reviews of statistical methods of data integration for finite population inference.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Both outputs provide coherent continuations, but they serve different argumentative purposes. Output A offers a broad methodological critique about untestable assumptions and the need for auxiliary information—this is accurate but somewhat generic and could apply to many contexts. It doesn't advance the specific thread about data integration methods that the draft has been building. Output B, by contrast, introduces doubly robust estimation as a specific recent development that directly addresses the MAR limitation discussed in the preceding text. The draft explicitly critiques MAR assumptions in Valliant and Dever (2011), Elliott and Valliant (2017), Bethlehem (2016), and Chen et al. (2020). Output B's content about doubly robust estimation—remaining consistent if either model is correct—is precisely the kind of methodological advance that responds to this critique. The sources confirm this is an active research direction (second source describes this exact two-step approach). Output B maintains the paper's trajectory: from identifying the MAR problem in existing methods to introducing a solution. Output A's pivot to 'auxiliary information or external data sources' is vaguer and less clearly connected to the specific methods just reviewed. Output B is more targeted and advances the argument with a concrete methodological response.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

More recently, doubly robust estimation has been proposed to combine a non-probability sample with a probability sample that provides high-dimensional covariate information, where variable selection and finite population mean estimation are carried out in two steps and the resulting estimator remains root n consistent if either the sampling probability or the outcome model is correctly specified.

Passages supplied to the Evidence version

Card 2 · Sampling Techniques for Big Data Analysis

Summary In analysing big data for finite population inference, it is critical to adjust for the selection bias in the big data. In this paper, we propose two methods of reducing the selection bias associated with the big data sample. The first method uses a version of inverse sampling by incorporati…

Read full passage excerpt

Summary In analysing big data for finite population inference, it is critical to adjust for the selection bias in the big data. In this paper, we propose two methods of reducing the selection bias associated with the big data sample. The first method uses a version of inverse sampling by incorporating auxiliary information from external sources, and the second one borrows the idea of data integration by combining the big data sample with an independent probability sample. Two simulation studies show that the proposed methods are unbiased and have better coverage rates than their alternatives. In addition, the proposed methods are easy to implement in practice.

Card 3 · Doubly Robust Inference when Combining Probability and Non-Probability Samples with High Dimensional Data

Summary We consider integrating a non-probability sample with a probability sample which provides high dimensional representative covariate information of the target population. We propose a two-step approach for variable selection and finite population inference. In the first step, we use penalized…

Read full passage excerpt

Summary We consider integrating a non-probability sample with a probability sample which provides high dimensional representative covariate information of the target population. We propose a two-step approach for variable selection and finite population inference. In the first step, we use penalized estimating equations with folded concave penalties to select important variables and show selection consistency for general samples. In the second step, we focus on a doubly robust estimator of the finite population mean and re-estimate the nuisance model parameters by minimizing the asymptotic squared bias of the doubly robust estimator. This estimating strategy mitigates the possible first-step selection error and renders the doubly robust estimator root n consistent if either the sampling probability or the outcome model is correctly specified.

Card 4 · On valid descriptive inference from non-probability sample

We examine the conditions under which descriptive inference can be based directly on the observed distribution in a non-probability sample, under both the super-population and quasi-randomisation modelling approaches. Review of existing estimation methods reveals that the traditional formulation of…

Read full passage excerpt

We examine the conditions under which descriptive inference can be based directly on the observed distribution in a non-probability sample, under both the super-population and quasi-randomisation modelling approaches. Review of existing estimation methods reveals that the traditional formulation of these conditions may be inadequate due to potential issues of under-coverage or heterogeneous mean beyond the assumed model. We formulate unifying conditions that are applicable to both type of modelling approaches. The difficulties of empirically validating the required conditions are discussed, as well as valid inference approaches using supplementary probability sampling. The key message is that probability sampling may still be necessary in some situations, in order to ensure the validity of descriptive inference, but it can be much less resource-demanding provided the presence of a big non-probability sample.

Card 5 · Statistical matching of non-Gaussian data

The statistical matching problem is a data integration problem with structured missing data. The general form involves the analysis of multiple datasets that only have a strict subset of variables jointly observed across all datasets. The simplest version involves two datasets, labelled A and B, wit…

Read full passage excerpt

The statistical matching problem is a data integration problem with structured missing data. The general form involves the analysis of multiple datasets that only have a strict subset of variables jointly observed across all datasets. The simplest version involves two datasets, labelled A and B, with three variables of interest $X, Y$ and $Z$. Variables $X$ and $Y$ are observed in dataset A and variables $X$ and $Z$ are observed in dataset $B$. Statistical inference is complicated by the absence of joint $(Y, Z)$ observations. Parametric modelling can be challenging due to identifiability issues and the difficulty of parameter estimation. We develop computationally feasible procedures for the statistical matching of non-Gaussian data using suitable data augmentation schemes and identifiability constraints. Nearest-neighbour imputation is a common alternative technique due to its ease of use and generality. Nearest-neighbour matching is based on a conditional independence assumption tha …

Model-reported card use

Card 2 · fact

Affected output: doubly robust estimation has been proposed to combine a non-probability sample with a probability sample that provides high-dimensional covariate information

Model explanation: This row supplied the concrete prior-work fact that a doubly robust approach integrates a non-probability sample with a probability sample carrying high-dimensional covariate information, which the paragraph's review of data-integration methods had not yet covered.

Card 2 · fact

Affected output: the resulting estimator remains root n consistent if either the sampling probability or the outcome model is correctly specified

Model explanation: This row supplied the specific double-robustness guarantee and its condition, allowing the sentence to state the method's property rather than a generic description.

Card 1 · direction

Affected output: doubly robust estimation has been proposed to combine a non-probability sample with a probability sample

Model explanation: This row established that combining a big data sample with an independent probability sample is a recognized data-integration strategy, guiding the choice to add this class of methods to the literature review.

Card 3 · scope

Affected output: More recently, doubly robust estimation has been proposed

Model explanation: This row framed the role of supplementary probability sampling in non-probability-sample inference, keeping the added sentence scoped to methods that combine the two sample types rather than to non-probability samples alone.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

These approaches share a common reliance on untestable assumptions about the selection mechanism, which motivates methods that incorporate auxiliary information or external data sources to reduce sensitivity to such assumptions.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output A is a grounded continuation that accurately reflects the second source (Doubly Robust Inference when Combining Probability and Non-Probability Samples with High Dimensional Data), describing the two-step approach with penalized estimating equations, variable selection, and doubly robust estimation with root n consistency if either model is correct. This matches the source's description of 'a two-step approach for variable selection and finite population inference' and 'the doubly robust estimator root n consistent if either the sampling probability or the outcome model is correctly specified.' Output B makes an unsupported claim: it states that these approaches share 'a common reliance on untestable assumptions about the selection mechanism,' which while plausible as a general statistical concept, is not supported by any of the supplied sources. The sources discuss MAR assumptions, conditional independence assumptions, and selection consistency, but none frame the commonality as 'untestable assumptions about the selection mechanism' as a motivating factor for methods incorporating auxiliary information. The phrase 'which motivates methods that incorporate auxiliary information or external data sources to reduce sensitivity to such assumptions' introduces an unsupported causal/motivational claim not found in the sources. The first source mentions using 'auxiliary information from external sources,' but does not present this as motivated by reducing sensitivity to untestable assumptions about selection mechanisms. Output B's claim is a field-knowledge extrapolation, not closed-book supported.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.