← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better275 / 280 · beu-d7d637ec5da5db85

It's Not About Whom You Train: An Analysis of Corporate Education in Software Engineering

Computer science · 2604.06580v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

It's Not About Whom You Train: An Analysis of Corporate Education in Software Engineering

body

However, despite these investments, many organizations lack mechanisms to assess the effectiveness of their programs [20], and determining the return on investment in technology training remains an open problem [5]. This article is part of a multiphase research effort on corporate education in SE. In the first phase, a Systematic Mapping of the Literature identified significant gaps in research on training in SE [6]. In the second phase, a quantitative analysis with 282 Brazilian professionals identified the central factors of

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = without cards · B = with cards

The draft ends mid-sentence describing what the second phase of research identified: 'the central factors of...' This clearly needs completion. Output B directly completes this with the three factors found in the source (cognitive engagement, variety of activities, and instructor performance), which matches the cited study with 282 Brazilian professionals. Output A drifts to broader scope ('characteristics of training programs,' 'adoption and outcomes') that goes beyond what the source study actually identified and introduces concepts (adoption) not mentioned in the source. Output B is more accurate, precisely scoped to the actual findings, and fits the grammatical structure 'central factors of [X]' where X should be the outcome variable being predicted. Both are usable, but B is tighter and more source-faithful.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

perceived training quality and effectiveness, including cognitive engagement, variety of activities, and instructor performance.

Passages supplied to the Evidence version

Card 1 · Corporate Training in Brazilian Software Engineering: A Quantitative Study of Professional Perceptions

However, there is a knowledge gap regarding the factors that drive quality and effectiveness of such training from the professionals’ perspective, and no validated instrument exists for assessing these factors in the software engineering (SE) domain.Objective:This study aims to quantitatively analyz…

Read full passage excerpt

However, there is a knowledge gap regarding the factors that drive quality and effectiveness of such training from the professionals’ perspective, and no validated instrument exists for assessing these factors in the software engineering (SE) domain.Objective:This study aims to quantitatively analyze which factors influence SE professionals’ per- ceptions of corporate training quality and effectiveness.Method: A quantitative survey was conducted with 282 Brazilian SE profes- sionals. A structured questionnaire was developed and polychoric correlation was adopted for data analysis.Results:Three tightly correlated factors (cognitive engagement, variety of activities, and instructor performance) emerged as the strongest predictors of per- ceived training quality and effectiveness.

Card 2 · Behavior and Representation in Open-Weight Large Language Models for Combinatorial Optimization: From Feature Extraction to Algorithm Selection

Recent advances in Large Language Models (LLMs) open new perspectives for automation in optimization, yet little is known about whether their internal representations capture problem structure or algorithmic behavior. We investigate whether representations learned by frozen, open-weight LLMs for com…

Read full passage excerpt

Recent advances in Large Language Models (LLMs) open new perspectives for automation in optimization, yet little is known about whether their internal representations capture problem structure or algorithmic behavior. We investigate whether representations learned by frozen, open-weight LLMs for combinatorial optimization instances can support downstream decision tasks. The goal is not to replace exact feature extractors or to propose a new algorithm, but to assess whether such representations are reusable for feature recovery and algorithm selection. Our methodology combines direct querying, which tests explicit feature extraction, with probing analyses of whether this information is implicitly encoded in the hidden layers. The probing framework is further extended to a per-instance algorithm selection task. Experiments span four benchmark problems, three instance representations, and five open-weight models from 3B to 120B parameters across the Llama instruct and GPT reasoning famili …

Card 3 · Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoning

Recently, Multimodal Large Language Models (MLLMs) have achieved significant success across multiple disciplines due to their exceptional instruction-following capabilities and extensive world knowledge. However, whether these MLLMs possess human-like compositional reasoning abilities remains an ope…

Read full passage excerpt

Recently, Multimodal Large Language Models (MLLMs) have achieved significant success across multiple disciplines due to their exceptional instruction-following capabilities and extensive world knowledge. However, whether these MLLMs possess human-like compositional reasoning abilities remains an open problem. To unveil their reasoning behaviors, we first curate a \textbf{M}ultimodal \textbf{A}ssumptive \textbf{R}ea\textbf{s}oning Benchmark (MARS-Bench) in this paper. Interestingly, we find that most prevalent MLLMs can be easily fooled by the introduction of a presupposition into the question, whereas such presuppositions appear naive to human reasoning. Besides, we also propose a simple yet effective method, Active Deduction (AD), a novel reinforcement learning paradigm to encourage the model to actively perform composite deduction before reaching a final decision. Equipped with the proposed AD method, a MLLM demonstrates significant improvements in assumptive reasoning abilities with …

Card 4 · Emergent Abilities in Large Language Models: A Survey

Large Language Models (LLMs) are leading a new technological revolution as one of the most promising research streams toward artificial general intelligence. The scaling of these models, accomplished by increasing the number of parameters and the magnitude of the training datasets, has been linked t…

Read full passage excerpt

Large Language Models (LLMs) are leading a new technological revolution as one of the most promising research streams toward artificial general intelligence. The scaling of these models, accomplished by increasing the number of parameters and the magnitude of the training datasets, has been linked to various so-called emergent abilities that were previously unobserved. These emergent abilities, ranging from advanced reasoning and in-context learning to coding and problem-solving, have sparked an intense scientific debate: Are they truly emergent, or do they simply depend on external factors, such as training dynamics, the type of problems, or the chosen metric? What underlying mechanism causes them? Despite their transformative potential, emergent abilities remain poorly understood, leading to misconceptions about their definition, nature, predictability, and implications. In this work, we shed light on emergent abilities by conducting a comprehensive review of the phenomenon, addressi …

Card 5 · Composition of Experts: A Modular Compound AI System Leveraging Large Language Models

Large Language Models (LLMs) have achieved remarkable advancements, but their monolithic nature presents challenges in terms of scalability, cost, and customization. This paper introduces the Composition of Experts (CoE), a modular compound AI system leveraging multiple expert LLMs. CoE leverages a…

Read full passage excerpt

Large Language Models (LLMs) have achieved remarkable advancements, but their monolithic nature presents challenges in terms of scalability, cost, and customization. This paper introduces the Composition of Experts (CoE), a modular compound AI system leveraging multiple expert LLMs. CoE leverages a router to dynamically select the most appropriate expert for a given input, enabling efficient utilization of resources and improved performance. We formulate the general problem of training a CoE and discuss inherent complexities associated with it. We propose a two-step routing approach to address these complexities that first uses a router to classify the input into distinct categories followed by a category-to-expert mapping to obtain desired experts. CoE offers a flexible and cost-effective solution to build compound AI systems. Our empirical evaluation demonstrates the effectiveness of CoE in achieving superior performance with reduced computational overhead. Given that CoE comprises o …

Model-reported card use

Card 1 · fact

Affected output: perceived training quality and effectiveness, including cognitive engagement, variety of activities, and instructor performance.

Model explanation: Supplied the specific factors and the outcome variable that complete the draft's unfinished clause about the second phase's quantitative analysis.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

corporate education in software engineering, including the characteristics of training programs and the factors that influence their adoption and outcomes.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output B directly continues the sentence with specific findings from Source [6] (the Corporate Training in Brazilian Software Engineering paper), naming the three factors that emerged from the quantitative analysis with 282 Brazilian professionals: 'cognitive engagement, variety of activities, and instructor performance.' This matches the source snippet exactly and honestly reports the row's content. Output A, while topically related, adds unsupported claims: it mentions 'characteristics of training programs' and 'factors that influence their adoption and outcomes'—neither 'adoption' nor 'outcomes' are discussed in the source; the source focuses on 'perceived training quality and effectiveness,' not adoption or outcomes. The phrase 'including the characteristics of training programs' is also vague and not directly grounded in the source's specific three-factor finding. Output B precisely completes what the draft started (the second phase identified the central factors of...) with the exact factors from the cited study, making it more source-grounded and requiring less editing to be accurate.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.