← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better271 / 280 · beu-f1de2c5b1888babf

From Human to Machine Refactoring: Assessing GPT-4's Impact on Python Class Quality and Readability

Computer science · 2601.13139v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

From Human to Machine Refactoring: Assessing GPT-4's Impact on Python Class Quality and Readability

body

Prior studies investigate how LLMs can generate, recommend, or validate refactorings, often combining them with static analysis, traditional refactoring tools, or employing hybrid prompting tech- niques. In this section, we first review related work on automated code refactoring supported by LLMs, then discuss broader efforts applying LLMs to software engineering tasks that inform our ap- proach and evaluation design. 2.1 Automated Code Refactoring Recent studies highlighted the growing role of LLM in automated software refactoring. Several works integrated LLMs with tradi- tional program analysis to improve the precision and safety of automated transformations. EM-Assist [32, 33] augments the clas- sic Extract Method refactoring technique by combining iterative LLM suggestions

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

Both outputs complete the sentence about EM-Assist appropriately, but B is preferable. The draft establishes that EM-Assist 'augments the classic Extract Method refactoring technique by combining iterative LLM suggestions' and needs a continuation explaining what it combines them with. Output B better matches the source description: the source states EM-Assist 'generates refactoring suggestions using LLMs, then validates and enhances them with static analysis in the IDE.' B captures both validation ('validate') and refinement ('refine') aspects, plus the IDE context ('before they are applied in the IDE'). Output A's 'identify and validate extractable code fragments' is accurate but slightly narrower—focusing only on identification/validation rather than the full pipeline of validation and refinement that the source describes. B also flows better from 'iterative LLM suggestions' by showing the complete workflow (validate → refine → apply). Both are usable and source-grounded, but B is more comprehensive and precise.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

with static analysis to validate and refine candidate refactorings before they are applied in the IDE.

Passages supplied to the Evidence version

Card 1 · EM-Assist: Safe Automated ExtractMethod Refactoring with LLMs

It bridges the gap between suggesting and applying Extract Method refactoring, and shrinks the gap between refactoring suggestions and developer practices. EM-Assist generates refactoring suggestions using LLMs, then validates and enhances them with static analysis in the IDE

Card 2 · Emergent Abilities in Large Language Models: A Survey

Large Language Models (LLMs) are leading a new technological revolution as one of the most promising research streams toward artificial general intelligence. The scaling of these models, accomplished by increasing the number of parameters and the magnitude of the training datasets, has been linked t…

Read full passage excerpt

Large Language Models (LLMs) are leading a new technological revolution as one of the most promising research streams toward artificial general intelligence. The scaling of these models, accomplished by increasing the number of parameters and the magnitude of the training datasets, has been linked to various so-called emergent abilities that were previously unobserved. These emergent abilities, ranging from advanced reasoning and in-context learning to coding and problem-solving, have sparked an intense scientific debate: Are they truly emergent, or do they simply depend on external factors, such as training dynamics, the type of problems, or the chosen metric? What underlying mechanism causes them? Despite their transformative potential, emergent abilities remain poorly understood, leading to misconceptions about their definition, nature, predictability, and implications. In this work, we shed light on emergent abilities by conducting a comprehensive review of the phenomenon, addressi …

Card 3 · VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI

Recent advancements in Multi-modal Large Language Models (MLLMs) have opened new avenues for applications in Embodied AI. Building on previous work, EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating egocentric video understanding capabilities. To bridge the gap between MLL…

Read full passage excerpt

Recent advancements in Multi-modal Large Language Models (MLLMs) have opened new avenues for applications in Embodied AI. Building on previous work, EgoThink, we introduce VidEgoThink, a comprehensive benchmark for evaluating egocentric video understanding capabilities. To bridge the gap between MLLMs and low-level control in Embodied AI, we design four key interrelated tasks: video question-answering, hierarchy planning, visual grounding and reward modeling. To minimize manual annotation costs, we develop an automatic data generation pipeline based on the Ego4D dataset, leveraging the prior knowledge and multimodal capabilities of GPT-4o. Three human annotators then filter the generated data to ensure diversity and quality, resulting in the VidEgoThink benchmark. We conduct extensive experiments with three types of models: API-based MLLMs, open-source image-based MLLMs, and open-source video-based MLLMs. Experimental results indicate that all MLLMs, including GPT-4o, perform poorly ac …

Card 4 · RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering

Remote Sensing Image Captioning (RSIC) presents unique challenges and plays a critical role in applications. Traditional RSIC methods often struggle to produce rich and diverse descriptions. Recently, with advancements in VLMs, efforts have emerged to integrate these models into the remote sensing d…

Read full passage excerpt

Remote Sensing Image Captioning (RSIC) presents unique challenges and plays a critical role in applications. Traditional RSIC methods often struggle to produce rich and diverse descriptions. Recently, with advancements in VLMs, efforts have emerged to integrate these models into the remote sensing domain and to introduce descriptive datasets specifically designed to enhance VLM training. This paper proposes RS-MoE, a first Mixture of Expert based VLM specifically customized for remote sensing domain. Unlike traditional MoE models, the core of RS-MoE is the MoE Block, which incorporates a novel Instruction Router and multiple lightweight Large Language Models (LLMs) as expert models. The Instruction Router is designed to generate specific prompts tailored for each corresponding LLM, guiding them to focus on distinct aspects of the RSIC task. This design not only allows each expert LLM to concentrate on a specific subset of the task, thereby enhancing the specificity and accuracy of the …

Card 5 · Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning re…

Read full passage excerpt

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, address the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that in …

Model-reported card use

Card 1 · fact

Affected output: with static analysis to validate and refine candidate refactorings before they are applied in the IDE.

Model explanation: The row supplies the specific mechanism (static analysis validation and enhancement in the IDE) that completes the unfinished clause about EM-Assist's approach.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

with static analysis to identify and validate extractable code fragments.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Both outputs complete the sentence about EM-Assist by describing how it combines LLM suggestions with static analysis. Output B is preferable because it more accurately reflects the source: the snippet states EM-Assist 'generates refactoring suggestions using LLMs, then validates and enhances them with static analysis in the IDE.' B captures both 'validates' and the refinement/enhancement aspect ('refine'), plus correctly notes the IDE context ('before they are applied in the IDE' aligns with 'in the IDE'). Output A says 'identify and validate extractable code fragments,' which introduces 'identify' and 'extractable code fragments'—the source says static analysis is used to 'validate and enhance' suggestions, not to identify fragments. B's 'validate and refine' is closer to 'validates and enhances,' and 'candidate refactorings' is more general and accurate than A's 'extractable code fragments' which narrows the scope. Both are usable and source-grounded, but B preserves more of the source's meaning with less editorial invention.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.