← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better278 / 280 · beu-7668cf2c0f008535

Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation

Computer science · 2604.15270v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Enhancing Large Language Models with Retrieval Augmented Generation for Software Testing and Inspection Automation

body

Studies show that models can effectively interpret inspection comments and generate appropriate revisions [34]. In parallel, automated comment generation has gained traction, where Machine Learning and NLP methods are leveraged to analyze code and produce defect-focused annotations or fix suggestions [ 35, 36]. These approaches improve review efficiency, but remain limited in their ability to express uncertainty or identify gaps in understanding [37]. In another study, Sun et al. [38] provided the first systematic investigation of the adoption and usage of AI-based code review actions on GitHub, categorizing available tools by their review

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Output A is superior because it directly continues the specific categorization framework established by Sun et al. [38] in the source material. The draft ends mid-sentence with 'categorizing available tools by their review,' and Output A completes this with 'granularity, distinguishing PR-level, file-level, and hunk-level review actions' — which precisely matches the source's description of categorizing 'by their review granularity (i.e., PR, file, or hunk).' Output A also adds the substantive contribution about 'analyzing the factors that influence whether AI-generated comments lead to code changes,' which is the third bullet point in the source. This continuation is source-grounded, specific, and maintains the academic register. Output B is vague and generic ('capabilities and analyzing how developers integrate them into their workflows') — while not incorrect, it misses the specific granularity taxonomy that the source emphasizes and that the draft was clearly building toward. Output B could apply to almost any technology adoption study, whereas Output A captures the distinctive contribution of this particular paper.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

granularity, distinguishing PR-level, file-level, and hunk-level review actions, and analyzing the factors that influence whether AI-generated comments lead to code changes.

Passages supplied to the Evidence version

Card 1 · Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions

Example comments from a PR-level review action (Integral-Healthcare/robin-ai-reviewer), a file-level review action (anc95/ChatGPT-CodeReview), and a hunk-Level review action (coderabbitai/ai-pr-reviewer). machine learning techniques to model the factors that influence the effectiveness of comments.…

Read full passage excerpt

Example comments from a PR-level review action (Integral-Healthcare/robin-ai-reviewer), a file-level review action (anc95/ChatGPT-CodeReview), and a hunk-Level review action (coderabbitai/ai-pr-reviewer). machine learning techniques to model the factors that influence the effectiveness of comments. Our contributions are as follows. • We provide the first systematic study of the adoption and usage of AI-based code review actions on GitHub, categorising tools by their review granularity (i.e., PR, file, or hunk) and behaviour (e.g., action triggers). • We introduce an LLM-assisted framework for assess- ing whether code review comments are actionable and whether they have been addressed, achieving high accu- racy compared to human annotations. • We identify and explain thefactors that influence whether AI-generated comments lead to code changes, offering design implications for code review tools

Card 2 · An Embodied Generalist Agent in 3D World

Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images ye…

Read full passage excerpt

Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e.g., 3D grounding, embodied reasoning and acting. We argue these limitations significantly hinder current models from performing real-world tasks and approaching general intelligence. To this end, we introduce LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. LEO is trained with a unified task interface, model architecture, and objective in two stages: (i) 3D vision-language (VL) alignment and (ii) 3D vision-language-action (VLA) instruction tuning. We collect large-scale datasets comprising d …

Card 3 · Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation

Embodied scene understanding requires not only comprehending visual-spatial information that has been observed but also determining where to explore next in the 3D physical world. Existing 3D Vision-Language (3D-VL) models primarily focus on grounding objects in static observations from 3D reconstru…

Read full passage excerpt

Embodied scene understanding requires not only comprehending visual-spatial information that has been observed but also determining where to explore next in the 3D physical world. Existing 3D Vision-Language (3D-VL) models primarily focus on grounding objects in static observations from 3D reconstruction, such as meshes and point clouds, but lack the ability to actively perceive and explore their environment. To address this limitation, we introduce \underline{\textbf{M}}ove \underline{\textbf{t}}o \underline{\textbf{U}}nderstand (\textbf{\model}), a unified framework that integrates active perception with \underline{\textbf{3D}} vision-language learning, enabling embodied agents to effectively explore and understand their environment. This is achieved by three key innovations: 1) Online query-based representation learning, enabling direct spatial memory construction from RGB-D frames, eliminating the need for explicit 3D reconstruction. 2) A unified objective for grounding and explori …

Card 4 · Scale, Don't Fine-tune: Guiding Multimodal LLMs for Efficient Visual Place Recognition at Test-Time

Visual Place Recognition (VPR) has evolved from handcrafted descriptors to deep learning approaches, yet significant challenges remain. Current approaches, including Vision Foundation Models (VFMs) and Multimodal Large Language Models (MLLMs), enhance semantic understanding but suffer from high comp…

Read full passage excerpt

Visual Place Recognition (VPR) has evolved from handcrafted descriptors to deep learning approaches, yet significant challenges remain. Current approaches, including Vision Foundation Models (VFMs) and Multimodal Large Language Models (MLLMs), enhance semantic understanding but suffer from high computational overhead and limited cross-domain transferability when fine-tuned. To address these limitations, we propose a novel zero-shot framework employing Test-Time Scaling (TTS) that leverages MLLMs' vision-language alignment capabilities through Guidance-based methods for direct similarity scoring. Our approach eliminates two-stage processing by employing structured prompts that generate length-controllable JSON outputs. The TTS framework with Uncertainty-Aware Self-Consistency (UASC) enables real-time adaptation without additional training costs, achieving superior generalization across diverse environments. Experimental results demonstrate significant improvements in cross-domain VPR pe …

Card 5 · Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and a…

Read full passage excerpt

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>9 …

Model-reported card use

Card 1 · fact

Affected output: granularity, distinguishing PR-level, file-level, and hunk-level review actions

Model explanation: The row supplies the specific granularity categories (PR, file, hunk) that the draft's unfinished clause 'categorizing available tools by their review' requires to complete the sentence.

Card 1 · fact

Affected output: analyzing the factors that influence whether AI-generated comments lead to code changes

Model explanation: The row provides the concrete analytical focus of Sun et al.'s study—factors influencing whether AI-generated comments lead to code changes—which completes the description of the systematic investigation.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

capabilities and analyzing how developers integrate them into their workflows.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output A directly continues from the draft's mention of 'categorizing available tools by their review' with specific, source-grounded details from Sun et al. [38]. The source explicitly states they 'categorising tools by their review granularity (i.e., PR, file, or hunk)' and 'identify and explain the factors that influence whether AI-generated comments lead to code changes.' Output A accurately captures both the granularity levels (PR-level, file-level, hunk-level) and the analytical focus on factors influencing code changes. Output B, while plausible and not unsupported, invents 'capabilities and analyzing how developers integrate them into their workflows' — the source does not mention 'capabilities' as a categorization dimension or 'developer integration into workflows' as a focus. The source emphasizes granularity categorization and factors leading to code changes, not workflow integration. Output A is more precisely grounded in the available evidence and completes the sentence with the actual substance of the study.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.