← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWith cards better274 / 280 · beu-615de9a11fdea5f9

It's Not About Whom You Train: An Analysis of Corporate Education in Software Engineering

Computer science · 2604.06580v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

It's Not About Whom You Train: An Analysis of Corporate Education in Software Engineering

body

For example, professionals at the start of their careers may be evalu- ating predominantlyonboardingexperiences, whose introductory nature tends to produce high perceptions of gain. In this scenario, the observed pattern would reflect, at least partially, differences in the type of training received by each band. Although the descrip- tive pattern is consistent across the four significant items and the post-hoctests identify specific pairs, future studies should include typological controls to isolate the effect of experience from the effect of training type. 5.4 Structural Conditions and Content Gaps Regional differences reflect access conditions (protected time, percep- tion of competitiveness), not instructional quality.The two items significant by state (Q7 and Q10) refer to structural conditions, not pedagogical quality dimensions. This converges with the analy- sis of improvement suggestions [24], which identified “Time and structural conditions”

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: With cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

The draft ends mid-sentence with 'Time and structural conditions' and needs continuation. Both outputs attempt to complete this, but with critical differences in accuracy to the source. Output A correctly identifies 'Time and structural conditions' as 'a recurring theme' (frequency 28, not the highest), and frames it as 'access-related constraints shape professionals' perceptions independently of instructional content' — this accurately reflects the source data where T3 (Time and structural conditions) ranks third in frequency, and aligns with the paper's argument that regional differences reflect structural access conditions, not pedagogical quality. Output B claims it is 'the most frequent category of requested change' — this is factually wrong per the source snippet, which shows T1 (Practical applicability, n=87) and T2 (Pedagogical quality, n=70) both exceed T3 (n=28). The source explicitly states 'The predominant demand is concentrated in T1.' Output B's 'most frequent' claim is a material contradiction of the source data, making it invalid. Output A's more modest 'recurring theme' characterization is accurate and fits the argument that structural conditions matter but are not the primary concern.

Evidence

Same model and position · retrieved passages supplied · first output

Preferred by blind review

as a recurring theme, indicating that access-related constraints shape professionals' perceptions independently of instructional content.

Passages supplied to the Evidence version

Card 1 · The professional's opinion: Suggestions for improving the corporate education training process in Software Engineering

The final frequency exceeding the number of participants (174) reflects the multifaceted nature of the suggestions. Tabela 2. Frequency of identified themes Code Theme Frequency T1 Practical applicability and alignment with needs 87 T2 Pedagogical quality, didactics, and organization 70 T3 Time and…

Read full passage excerpt

The final frequency exceeding the number of participants (174) reflects the multifaceted nature of the suggestions. Tabela 2. Frequency of identified themes Code Theme Frequency T1 Practical applicability and alignment with needs 87 T2 Pedagogical quality, didactics, and organization 70 T3 Time and structural conditions for learning 28 T4 Incentives, recognition, and institutional sup- port 27 T5 Interaction, mentoring, and social exchange 16 The predominant demand is concentrated in T1 (n= 87), followed by T2 (n= 70).

Card 2 · Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Task

Background. Large Language Models (LLMs) are increasingly being applied to Software Engineering (SE) tasks, achieving high accuracy across problems such as clone detection, vulnerability prediction, and code summarization. However, their high computational demands and energy consumption raise sustai…

Read full passage excerpt

Background. Large Language Models (LLMs) are increasingly being applied to Software Engineering (SE) tasks, achieving high accuracy across problems such as clone detection, vulnerability prediction, and code summarization. However, their high computational demands and energy consumption raise sustainability concerns and hinder their use on consumer hardware and resource-constrained platforms. A common way to report the computational cost of an LLM in the literature and industry is to use the number of Floating Point Operations (FLOPs) required to perform a pass over the network. Aims. This paper investigates the implications of energy-aware knowledge distillation for SE, aiming to improve model efficiency while maintaining performance and to determine whether FLOPs is a reliable energy-aware metric. Method. We conduct a controlled experiment using Morph, a Many-Objective Optimization-based distillation methodology, to empirically examine whether FLOPs accurately reflect energy consumpt …

Card 3 · Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and a…

Read full passage excerpt

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>9 …

Card 4 · Corporate Training in Brazilian Software Engineering: A Quantitative Study of Professional Perceptions

However, since no reliability or factor analysis was conducted, the instrument remains exploratory and requires future psychometric validation, including reliability analysis (Cronbach’s alpha or McDonald’s omega), exploratory and confirmatory factor analysis, and cross-cultural replications. A comp…

Read full passage excerpt

However, since no reliability or factor analysis was conducted, the instrument remains exploratory and requires future psychometric validation, including reliability analysis (Cronbach’s alpha or McDonald’s omega), exploratory and confirmatory factor analysis, and cross-cultural replications. A companion study, currently under review, examines whether these patterns hold across different professional roles, experience levels, and organizational contexts by applying significance tests across sociodemographic variables on the same dataset.

Card 5 · Emergent Abilities in Large Language Models: A Survey

Large Language Models (LLMs) are leading a new technological revolution as one of the most promising research streams toward artificial general intelligence. The scaling of these models, accomplished by increasing the number of parameters and the magnitude of the training datasets, has been linked t…

Read full passage excerpt

Large Language Models (LLMs) are leading a new technological revolution as one of the most promising research streams toward artificial general intelligence. The scaling of these models, accomplished by increasing the number of parameters and the magnitude of the training datasets, has been linked to various so-called emergent abilities that were previously unobserved. These emergent abilities, ranging from advanced reasoning and in-context learning to coding and problem-solving, have sparked an intense scientific debate: Are they truly emergent, or do they simply depend on external factors, such as training dynamics, the type of problems, or the chosen metric? What underlying mechanism causes them? Despite their transformative potential, emergent abilities remain poorly understood, leading to misconceptions about their definition, nature, predictability, and implications. In this work, we shed light on emergent abilities by conducting a comprehensive review of the phenomenon, addressi …

Model-reported card use

Card 1 · fact

Affected output: as a recurring theme, indicating that access-related constraints shape professionals' perceptions independently of instructional content.

Model explanation: The row supplies the specific theme label and its recurrence in the suggestion analysis, which the sentence uses to characterize the structural conditions finding.

No Evidence

Same model and position · no retrieved passages · first output

Not preferred in this comparison

as the most frequent category of requested change, indicating that practitioners themselves prioritize organizational conditions over instructional content.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output A correctly identifies 'Time and structural conditions' as 'a recurring theme' from the source [24], which shows T3 'Time and structural conditions for learning' had frequency 28 (not the most frequent). The source does NOT say it was 'the most frequent category of requested change' — T1 'Practical applicability and alignment with needs' at 87 was the most frequent, followed by T2 at 70. Output B's claim that structural conditions were 'the most frequent category' directly contradicts the source data. Additionally, Output B's claim that 'practitioners themselves prioritize organizational conditions over instructional content' is unsupported: the source shows practitioners prioritized practical applicability (T1) and pedagogical quality (T2) far more than structural conditions (T3). Output A's 'recurring theme' is accurate (28 responses, a notable theme), and its claim that 'access-related constraints shape professionals' perceptions independently of instructional content' aligns with the draft's argument that regional differences reflect 'access conditions... not instructional quality.'

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.