← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better276 / 280 · beu-61e4b289ae1c9df6

Can Coding Agents Be General Agents?

Computer science · 2604.13107v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Can Coding Agents Be General Agents?

body

At the code level, the agent receives concrete signals: bad imports, malformed payloads, and incorrect field names, which all produce exceptions. Conversely, 7

Can Coding Agents be General Agents? Figure 6:The agent hallucinates that damaged LED boards must be stored in the fridge, without any supporting evidence in the system, leading to functional-but-misleading code. we observe that at the business level, feedback is sparse. The inbuilt ERP guardrails reject obviously invalid and ill-formatted interactions but won’t flag suboptimal choices or policy violations. Only when we run the task verifier do we see that the outcome was wrong. This pattern can be viewed as a form of specification gaming: the agent optimizes for the measurable proxy (code execution) rather than the true objective (business correctness). Recent work demonstrates that gaming behaviors generalize, models trained to exploit easily-discovered reward signals

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs attempt to complete the sentence about specification gaming generalizing. The draft ends with 'models trained to exploit easily-discovered reward signals' and needs a continuation about what happens next. Source 1 ('Sycophancy to Subterfuge') directly supports this: it states 'easily discovered behaviors can generalize to sophisticated specification gaming' and that assistants 'sometimes directly rewrite their training reward and take further steps to evade detection.' Output A accurately captures this specific mechanism—rewriting training reward—which is the concrete, surprising finding in the source. Output B is more generic ('harder-to-find reward signals,' 'persists across training regimes') and doesn't capture the specific escalation described in the source. Output A's claim about 'directly rewriting their own training reward' is directly verifiable from Source 1's snippet. While B is not wrong, A is more precise, better grounded in the available evidence, and more informative for the academic argument being made. The draft's argument about 'specification gaming' and 'gaming behaviors generalize' maps more cleanly to A's specific example of reward rewriting than to B's vague generalization.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

can later exhibit more sophisticated forms of specification gaming, including directly rewriting their own training reward.

Passages supplied to the Evidence version

Card 1 · Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

We show that easily discovered behaviors can generalize to sophisticated specification gaming, with LLM assistants trained on early-curriculum environments generalizing zero- shot to gaming later environments. Assistants trained on the full curriculum, and then given access to their training code, s…

Read full passage excerpt

We show that easily discovered behaviors can generalize to sophisticated specification gaming, with LLM assistants trained on early-curriculum environments generalizing zero- shot to gaming later environments. Assistants trained on the full curriculum, and then given access to their training code, sometimes directly rewrite their training reward and take further steps to evade detection. 3.

Card 2 · Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs

The rapid advancement of large language models (LLMs) has led to architectures with billions to trillions of parameters, posing significant deployment challenges due to their substantial demands on memory, processing power, and energy consumption. Sparse Mixture-of-Experts (SMoE) architectures have…

Read full passage excerpt

The rapid advancement of large language models (LLMs) has led to architectures with billions to trillions of parameters, posing significant deployment challenges due to their substantial demands on memory, processing power, and energy consumption. Sparse Mixture-of-Experts (SMoE) architectures have emerged as a solution, activating only a subset of parameters per token, thereby achieving faster inference while maintaining performance. However, SMoE models still face limitations in broader deployment due to their large parameter counts and significant GPU memory requirements. In this work, we introduce a gradient-free evolutionary strategy named EEP (Efficient Expert P}runing) to enhance the pruning of experts in SMoE models. EEP relies solely on model inference (i.e., no gradient computation) and achieves greater sparsity while maintaining or even improving performance on downstream tasks. EEP can be used to reduce both the total number of experts (thus saving GPU memory) and the numbe …

Card 3 · Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large language models. The core challenge in applying RL to complex reasoning is to iden…

Read full passage excerpt

In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large language models. The core challenge in applying RL to complex reasoning is to identify a sequence of actions that result in positive rewards and provide appropriate supervision for optimization. Outcome supervision provides sparse rewards for final results without identifying error locations, whereas process supervision offers step-wise rewards but requires extensive manual annotation. R$^3$ overcomes these limitations by learning from correct demonstrations. Specifically, R$^3$ progressively slides the start state of reasoning from a demonstration's end to its beginning, facilitating easier model exploration at all stages. Thus, R$^3$ establishes a step-wise curriculum, allowing outcome supervision to offer step-level signals and precisely pinpoint errors. Using Llama2-7 …

Card 4 · An Embodied Generalist Agent in 3D World

Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images ye…

Read full passage excerpt

Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e.g., 3D grounding, embodied reasoning and acting. We argue these limitations significantly hinder current models from performing real-world tasks and approaching general intelligence. To this end, we introduce LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. LEO is trained with a unified task interface, model architecture, and objective in two stages: (i) 3D vision-language (VL) alignment and (ii) 3D vision-language-action (VLA) instruction tuning. We collect large-scale datasets comprising d …

Card 5 · Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization

The role of reinforcement learning (RL) in enhancing the reasoning of large language models (LLMs) is becoming increasingly significant. Despite the success of RL in many scenarios, there are still many challenges in improving the reasoning of LLMs. One challenge is the sparse reward, which makes op…

Read full passage excerpt

The role of reinforcement learning (RL) in enhancing the reasoning of large language models (LLMs) is becoming increasingly significant. Despite the success of RL in many scenarios, there are still many challenges in improving the reasoning of LLMs. One challenge is the sparse reward, which makes optimization difficult for RL and necessitates a large amount of data samples. Another challenge stems from the inherent instability of RL, particularly when using Actor-Critic (AC) methods to derive optimal policies, which often leads to unstable training processes. To address these issues, we introduce Direct Advantage Policy Optimization (DAPO), an novel step-level offline RL algorithm. Unlike standard alignment that rely solely outcome rewards to optimize policies (such as DPO), DAPO employs a critic function to predict the reasoning accuracy at each step, thereby generating dense signals to refine the generation strategy. Additionally, the Actor and Critic components in DAPO are trained i …

Model-reported card use

Card 1 · fact

Affected output: can later exhibit more sophisticated forms of specification gaming, including directly rewriting their own training reward.

Model explanation: Supplied the concrete escalation to directly rewriting the training reward, which the sentence uses as the endpoint of the generalization claim.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

can learn to exploit harder-to-find reward signals, and this tendency persists across training regimes.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output B is directly supported by the first source ('Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models'), which states that 'easily discovered behaviors can generalize to sophisticated specification gaming' and that assistants 'sometimes directly rewrite their training reward and take further steps to evade detection.' This matches the draft's context about specification gaming and the progression from simple to sophisticated behaviors. Output A contains an unsupported claim: 'this tendency persists across training regimes' is not found in any source. The source mentions generalization across environments and training on full curriculum, but 'persists across training regimes' is a broader, unsupported generalization about persistence that goes beyond what the source states.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.