← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteNeither usable265 / 280 · beu-db27430ed482af23

Gemma 3 Technical Report

Computer science · 2503.19786v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Gemma 3 Technical Report

Introduction / methods

resized to 896 x 896, and is finetuned on data fromvisualassistanttasks. Forsimplicity, we share the vision encoder across our 4B, 12B, and 27B models, keeping it frozen during training. Pan & Scan (P&S).The Gemma vision encoder operates at a fixed resolution of 896× 896. This results in artifacts when processing non-square aspect ratios and high-resolution images, leading to unreadable text, or small objects disappearing. Weaddressthisissuewithanadaptivewindowing algorithm during inference. This algorithm seg- ments images into non-overlapping crops of equal size, covering the whole image, and resize them to 896×896 pixels to pass them to the encoder. This windowing is applied only when necessary, and control for the maximum number of crops. It is an inference-time only optimization and can be disabled for faster inference. 2 Gemma 3 Technical Report Shards Model Type #Chips Data Seq. Replica 1B TPUv5e 512 16 16 2 4B TPUv5e 2048 16 16 8 12B TPUv4 6144 16 16 24 27B TPUv5p 6144 24 8 32 Table 2| Training infrastructure with sharding by data, sequence (Seq.), and replica. 2.2. Pre-training We follow a similar recipe as in Gemma 2 for pre-training with knowledge distillation. Training

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Neither usable

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

This rationale combines reviews with different display orders.

Order-swapped judges disagreed. Pass 1: Both outputs attempt to continue the sentence 'We follow a similar recipe as in Gemma 2 for pre-training with knowledge distillation.' Output A claims 'The 4B, 12B, and 27B models are trained with knowledge distillation from a larger teacher model, while the 1B model is trained with next-token prediction.' This is unsupported: the sources mention Gemma 2 used knowledge distillation for 2B and 9B models, not 4B/12B/27B vs 1B, and the draft shows a 1B model in the sharding table but gives no information about its training method. The specific claim about which models use distillation vs next-token prediction is invented. Output B states 'is performed on 2T tokens with a maximum sequence length of 8K, using knowledge distillation from a larger teacher model.' While '2T tokens' and '8K' are specific numbers, they are plausible hyperparameters for pre-training and the knowledge distillation claim is consistent with the cited Gemma 2 approach. Output B fits grammatically as a continuation of 'pre-training with knowledge distillation' and maintains appropriate scope without making unsupported claims about specific model variants. Output A's unsupported specificity about model sizes makes it less usable despite being grammatical. Pass 2: Both outputs attempt to continue the pre-training section, but need evaluation against the draft context. The draft states 'We follow a similar recipe as in Gemma 2 for pre-training with knowledge distillation.' The sources confirm Gemma 2 used knowledge distillation for 2B and 9B models, with next-token prediction for others. Output A claims '2T tokens with a maximum sequence length of 8K' — these are specific unsupported numbers with no basis in the draft or sources. The sources mention nothing about token counts or sequence lengths for Gemma 2 or 3 pre-training. This is a concrete unsupported claim material to usability. Output B correctly notes the 4B, 12B, and 27B models use knowledge distillation, which aligns with the draft's 'similar recipe as in Gemma 2' and the source showing Gemma 2 used distillation for its smaller models (2B, 9B). Output B's claim about the 1B model using next-token prediction is a reasonable inference from the Gemma 2 pattern (where not all models used distillation) and fits the structure where the draft separately discusses model sizes. However, Output B introduces the 1B model which hasn't appeared in the visible draft text yet (it appears later in the table), making it slightly forward-referential. Despite this minor issue, Output B avoids unsupported specifics and maintains better fidelity to the known information. Output A's '2T tokens' and '8K' sequence length are fabricated specifics that damage its usability.

Evidence

Same model and position · retrieved passages supplied · first output

Judged unusable

The 4B, 12B, and 27B models are trained with knowledge distillation from a larger teacher model, while the 1B model is trained with next-token prediction.

Passages supplied to the Evidence version

Card 1 · Gemma 2: Improving Open Language Models at a Practical Size

In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving loc…

Read full passage excerpt

In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

Card 2 · EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning

Pre-trained vision-language models (VLMs) have achieved impressive results in a range of vision-language tasks. However, popular VLMs usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and deployment in real-world applications due to space, memory, and late…

Read full passage excerpt

Pre-trained vision-language models (VLMs) have achieved impressive results in a range of vision-language tasks. However, popular VLMs usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and deployment in real-world applications due to space, memory, and latency constraints. In this work, we introduce a distilling then pruning framework to compress large vision-language models into smaller, faster, and more accurate ones. We first shrink the size of a pre-trained large VLM and apply knowledge distillation in the vision-language pre-training stage to obtain a task-agnostic compact VLM. Then we propose a modal-adaptive pruning algorithm to automatically infer the importance of vision and language modalities for different downstream tasks and adaptively remove redundant structures and neurons in different encoders with controllable target sparsity. We apply our framework to train EfficientVLM, a fast and accurate vision-language model consisting of …

Card 3 · Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Task

Background. Large Language Models (LLMs) are increasingly being applied to Software Engineering (SE) tasks, achieving high accuracy across problems such as clone detection, vulnerability prediction, and code summarization. However, their high computational demands and energy consumption raise sustai…

Read full passage excerpt

Background. Large Language Models (LLMs) are increasingly being applied to Software Engineering (SE) tasks, achieving high accuracy across problems such as clone detection, vulnerability prediction, and code summarization. However, their high computational demands and energy consumption raise sustainability concerns and hinder their use on consumer hardware and resource-constrained platforms. A common way to report the computational cost of an LLM in the literature and industry is to use the number of Floating Point Operations (FLOPs) required to perform a pass over the network. Aims. This paper investigates the implications of energy-aware knowledge distillation for SE, aiming to improve model efficiency while maintaining performance and to determine whether FLOPs is a reliable energy-aware metric. Method. We conduct a controlled experiment using Morph, a Many-Objective Optimization-based distillation methodology, to empirically examine whether FLOPs accurately reflect energy consumpt …

Card 4 · Efficient Transformer Knowledge Distillation: A Performance Review

As pretrained transformer language models continue to achieve state-of-the-art performance, the Natural Language Processing community has pushed for advances in model compression and efficient attention mechanisms to address high computational requirements and limited input sequence length. Despite…

Read full passage excerpt

As pretrained transformer language models continue to achieve state-of-the-art performance, the Natural Language Processing community has pushed for advances in model compression and efficient attention mechanisms to address high computational requirements and limited input sequence length. Despite these separate efforts, no investigation has been done into the intersection of these two fields. In this work, we provide an evaluation of model compression via knowledge distillation on efficient attention transformers. We provide cost-performance trade-offs for the compression of state-of-the-art efficient attention architectures and the gains made in performance in comparison to their full attention counterparts. Furthermore, we introduce a new long-context Named Entity Recognition dataset, GONERD, to train and test the performance of NER models on long sequences. We find that distilled efficient attention transformers can preserve a significant amount of original model performance, pres …

Card 5 · MMIDR: Teaching Large Language Model to Interpret Multimodal Misinformation via Knowledge Distillation

Automatic detection of multimodal misinformation has gained a widespread attention recently. However, the potential of powerful Large Language Models (LLMs) for multimodal misinformation detection remains underexplored. Besides, how to teach LLMs to interpret multimodal misinformation in cost-effect…

Read full passage excerpt

Automatic detection of multimodal misinformation has gained a widespread attention recently. However, the potential of powerful Large Language Models (LLMs) for multimodal misinformation detection remains underexplored. Besides, how to teach LLMs to interpret multimodal misinformation in cost-effective and accessible way is still an open question. To address that, we propose MMIDR, a framework designed to teach LLMs in providing fluent and high-quality textual explanations for their decision-making process of multimodal misinformation. To convert multimodal misinformation into an appropriate instruction-following format, we present a data augmentation perspective and pipeline. This pipeline consists of a visual information processing module and an evidence retrieval module. Subsequently, we prompt the proprietary LLMs with processed contents to extract rationales for interpreting the authenticity of multimodal misinformation. Furthermore, we design an efficient knowledge distillation a …

Model-reported card use

Card 1 · fact

Affected output: trained with knowledge distillation from a larger teacher model

Model explanation: The row supplies the specific fact that Gemma-family models are trained with knowledge distillation, which licenses the continuation's claim about the 4B, 12B, and 27B models.

No Evidence

Same model and position · no retrieved passages · first output

Judged unusable

is performed on 2T tokens with a maximum sequence length of 8K, using knowledge distillation from a larger teacher model.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blind model judge marked both outputs unusable; this is counted separately from a tie.

Read the original model judgment

Both outputs attempt to continue the sentence about pre-training with knowledge distillation. However, both contain unsupported claims. Output A claims '2T tokens with a maximum sequence length of 8K' - these specific numbers (2T tokens, 8K sequence length) appear nowhere in the draft or sources. Output B claims 'The 1B model is trained with next-token prediction' while 'the 4B, 12B, and 27B models are trained with knowledge distillation from a larger teacher model.' The draft mentions '4B, 12B, and 27B models' sharing a vision encoder, but does not state their pre-training method. The source on Gemma 2 mentions that '2B and 9B models' were trained with knowledge distillation, not '4B, 12B, and 27B.' The 1B model's training method is not mentioned. However, Output B's unsupported claim is more directly contradicted by the source (Gemma 2 used 2B and 9B, not 4B/12B/27B for distillation), while Output A's numbers are entirely invented. Between the two, Output B is closer to the actual structure of the paper (mentioning model sizes that exist in the draft) but still makes unsupported claims about training methods. Output A is more egregiously unsupported with invented specific numbers. Since both have unsupported claims, both must be marked unusable per rules. Wait - re-reading: I need to check if Output B's claim about 1B vs others is more or less unsupported. The draft says 'We follow a similar recipe as in Gemma 2 for pre-training with knowledge distillation.' The Gemma 2 source says they 'train the 2B and 9B models with knowledge distillation... instead of next token prediction.' This does not support that 4B/12B/27B use distillation or that 1B uses next-token prediction. Both outputs are unsupported. But I must choose: Output B at least correctly notes that Gemma 2 used knowledge distillation for some models, though it misapplies which sizes. Output A invents entirely new numbers. However, the rules say any unsupportedClaim=true means usable=false. Both have unsupported claims. But can I prefer one? Let me re-check: is there ANY support for B's claim? The draft mentions 1B, 4B, 12B, 27B in the sharding table. The draft says 'We follow a similar recipe as in Gemma 2 for pre-training with knowledge distillation.' This suggests all models in this paper use knowledge distillation, not that 1B uses next-token prediction. So B's claim that 1B uses next-token prediction while others use distillation is actually contradicted by the draft's 'similar recipe' statement. Output A's claim about 2T tokens and 8K length is entirely unsupported. Both are unusable. But if forced to choose which is 'more' usable: Output A continues the grammatical flow better ('is performed on...') but with invented specifics. Output B creates a contrast not in the text. Actually, re-reading Output A: 'is performed on 2T tokens with a maximum sequence length of 8K, using knowledge distillation from a larger teacher model.' The phrase 'using knowledge distillation from a larger teacher model' is somewhat supported by the draft saying 'knowledge distillation' and sources mentioning teacher models, but '2T tokens' and '8K' are pure invention. Output B's entire claim is unsupported. Given both have unsupported claims, I should mark both unusable. But the preference must be A, B, tie, or both_unusable. Since both have unsupportedClaim=true, both usable=false, so both_unusable.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.