← Back to all development cases
This case was judged by an anonymized AI reviewer and has not been independently reviewed by people; inspect the draft, cards, and both outputs to assess it.
AutocompleteWithout cards better264 / 280 · beu-3fbda54ea7ba521d

Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding

Computer science · 2608.02980v1

FLOWING EVIDENCE BENCHMARK

How do we compare writing with and without cards?

At the same writing position, the same model produces one first result with cards and one without. All 280 pairs enter the overall anonymized review, which assesses whether a decision not to continue fits the draft.

Overall: 138 with-card wins, 26 ties, 107 no-card wins, 5 pairs where neither was usable, and 4 inconclusive judgments. Both sides wrote text in 218 pairs; 48 had a deliberate stop and 14 included an invalid output.

SAME DRAFT · WITH OR WITHOUT CARDS

01 · FIXED WRITING POSITION

MANUSCRIPT

Same writing position ▌

Matched autocomplete at the same draft position

02 · TWO MATCHED VERSIONS

SHARED BY BOTH

Draft, writing task, and generation model

A · With cards

Relevant literature cards supplied

B · Without cards

No literature cards supplied

Generation model

Same model and version

A → first output

B → first output

Anonymized AI judge

First outputs are anonymized as X and Y

Continuation review: accuracy, fit to the writing task and usability
Returns: X preferred / tie / Y preferred / both unusable

Review again with the answer order swapped

HOW DOES BLIND JUDGING WORK?

① Anonymize both outputs
The AI judge sees the same draft and both first outputs without knowing which received cards.

② Compare and swap order
The judge applies task-specific criteria in X/Y and then Y/X order.

③ Review disagreements
A third pass resolves disagreements. Incomplete reviews remain in the denominator.

How are positions and cards selected?

We selected 40 writing positions in each of seven fields, for 280 comparisons. The position and relevant literature cards were set before each comparison, so both versions began from the same draft.

The main result retains every frozen position, including places where cards might not add a useful next move. This avoids selecting cases based on their outputs, but 49% describes the whole cohort, not a win rate where evidence clearly had an opportunity to help. This cohort did not record that opportunity label for each position before generation, so those two groups cannot yet be separated reliably.

CARD USE · 140/280

Reported card use in continuations with text

The model reports whether a card affected its continuation. In 140 of 280 tests it produced continuation text and reported card use. The previous count of 150 included 10 cases without continuation text and has been corrected.

COMPARISON · 70 / 7 / 30

Which written version was better?

Of the 107 reported-use cases where both versions produced text, the blinded AI judge preferred the with-card version 70 times, found 7 about the same, and preferred the without-card version 30 times.

How are the two table percentages calculated?

ADOPTION COVERAGE

140reported use with text
÷
280all positions
=
50%

WITH-CARD VERSION PREFERRED

70with cards better
÷
107comparable tests
=
65%

The percentages use different groups: 50% counts continuations with text reporting card use among all 280 tests; 65% covers the 107 such cases with text on both sides. The generating model reports card use, while an anonymized AI judge compares writing quality.

Card use is self-reported by the generator. The anonymized AI judge sees the draft and source snippets, so source-aware preference remains possible; there was no independent human review. The overall result describes these 280 frozen positions, not a win probability for any writing task.

Manuscript writing position

Text excerpt · not a PDF page

Research manuscript · excerpt

Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding

4.1 Evaluation on 3D Referential Grounding

igate catastrophic for- getting of Qwen’s original capabilities, we co-train on 2D datasets: referential grounding (RefCOCO, RefCOCO+, RefCOCOg [27]), instance segmentation (COCO [34]), captioning and QA (LLaV A-Instruct-150k [35]), and in- struction fine-tuning (Alpaca [44]). 4.1. Evaluation on 3D Referential Grounding Datasets.We evaluate on the validation sets of three ScanNet-based [14] 3D referential grounding benchmarks: SR3D, NR3D [1], and ScanRefer [10]. While SR3D com- prises 88k synthetic utterances, NR3D (41k) and ScanRefer (51k) feature complex, human-annotated queries. Follow- ing recent work [3, 7, 24, 25], we operate directly on noisy, raw sensor RGB-D point clouds rather than clean, post- processed meshes. Although this setup introduces sensor- mesh misalignments that can degrade performance [24, 25], it better reflects practical embodied learning scenarios. Evaluation Metrics.We report standard Top-1 accuracy, where a prediction is correct if the highest-confidence pre- dicted bounding box achieves an Intersection over Union (IoU) with the ground-truth box above a threshold (0.25, 0.5). Since our model predicts segmentation masks, we convert masks to bounding boxes by thresholding at their extreme corners. Baselines.Following prior work [57, 58], we compare Qwen-3D against state-of-the-art expert (non-LMM) and LLM-based approaches. Expert baselines include two- stage [59, 60] and single-stage [3, 25] methods.

↓ Next paragraph was generated here

The highlighted text is near the writing position. This is a layout of the evaluation input, not a PDF screenshot.

A matched comparison at the same writing positionOverall AI review: Without cards better

Both arms use the same model, task and manuscript context. Only the retrieved passages differ. The first recorded output from each arm is shown below.

Read overall AI review rationale

A = with cards · B = without cards

Both outputs appropriately continue the baseline comparison section, but Output A has a material unsupported claim. The draft mentions 'LLM-based approaches' as a category to compare against, and Output A names specific models (LEO and Robin3D) as 'general-purpose 3D LLMs' evaluated 'under the same raw point cloud protocol.' However, the sources do not establish that LEO or Robin3D are evaluated on the same 'raw sensor RGB-D point clouds' protocol described in this draft. The LEO source mentions '3D vision-language alignment' and '3D vision-language-action instruction tuning' but does not specify raw point cloud evaluation. The Robin3D source mentions 'five widely-used 3D multimodal learning benchmarks' but does not confirm raw point cloud protocol. More critically, the draft's context is about Qwen-3D's specific evaluation on SR3D, NR3D, and ScanRefer with raw RGB-D point clouds—there is no evidence that LEO and Robin3D are 'typically evaluated under the same raw point cloud protocol' on these specific benchmarks. This is an invented specificity about the manuscript's actual comparison setup. Output B, while more generic, avoids unsupported specifics and accurately describes a common technical approach (adapting 2D VL models to 3D via projection layers or 3D-aware prompting) that fits the 'LLM-based approaches' category without making false claims about which specific models are compared or their protocols. However, Output A's unsupported claim is not so severe as to make it unusable—it could potentially be repaired—so both remain usable, with B preferred for accuracy.

Evidence

Same model and position · retrieved passages supplied · first output

Not preferred in this comparison

LLM-based baselines include general-purpose 3D LLMs such as LEO and Robin3D, which are typically evaluated under the same raw point cloud protocol.

Passages supplied to the Evidence version

Card 1 · An Embodied Generalist Agent in 3D World

Our proposed agent, referred to as LEO, is trained with shared LLM-based model architectures, objectives, and weights in two stages: (i) 3D vision-language alignment and (ii) 3D vision-language-action instruction tuning.

Card 2 · Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation

Embodied scene understanding requires not only comprehending visual-spatial information that has been observed but also determining where to explore next in the 3D physical world. Existing 3D Vision-Language (3D-VL) models primarily focus on grounding objects in static observations from 3D reconstru…

Read full passage excerpt

Embodied scene understanding requires not only comprehending visual-spatial information that has been observed but also determining where to explore next in the 3D physical world. Existing 3D Vision-Language (3D-VL) models primarily focus on grounding objects in static observations from 3D reconstruction, such as meshes and point clouds, but lack the ability to actively perceive and explore their environment. To address this limitation, we introduce \underline{\textbf{M}}ove \underline{\textbf{t}}o \underline{\textbf{U}}nderstand (\textbf{\model}), a unified framework that integrates active perception with \underline{\textbf{3D}} vision-language learning, enabling embodied agents to effectively explore and understand their environment. This is achieved by three key innovations: 1) Online query-based representation learning, enabling direct spatial memory construction from RGB-D frames, eliminating the need for explicit 3D reconstruction. 2) A unified objective for grounding and explori …

Card 3 · Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding

In recent years, vision language pre-training frameworks have made significant progress in natural language processing and computer vision, achieving remarkable performance improvement on various downstream tasks. However, when extended to point cloud data, existing works mainly focus on building ta…

Read full passage excerpt

In recent years, vision language pre-training frameworks have made significant progress in natural language processing and computer vision, achieving remarkable performance improvement on various downstream tasks. However, when extended to point cloud data, existing works mainly focus on building task-specific models, and fail to extract universal 3D vision-language embedding that generalize well. We carefully investigate three common tasks in semantic 3D scene understanding, and derive key insights into the development of a pre-training model. Motivated by these observations, we propose a vision-language pre-training framework 3DVLP (3D vision-language pre-training with object contrastive learning), which transfers flexibly on 3D vision-language downstream tasks. 3DVLP takes visual grounding as the proxy task and introduces Object-level IoU-guided Detection (OID) loss to obtain high-quality proposals in the scene. Moreover, we design Object-level Cross-Contrastive alignment (OCC) task …

Card 4 · LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness

Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D scene understanding capabilities has been hindered by the la…

Read full passage excerpt

Recent advancements in Large Multimodal Models (LMMs) have greatly enhanced their proficiency in 2D visual understanding tasks, enabling them to effectively process and understand images and videos. However, the development of LMMs with 3D scene understanding capabilities has been hindered by the lack of large-scale 3D vision-language datasets and powerful 3D encoders. In this paper, we introduce a simple yet effective framework called LLaVA-3D. Leveraging the strong 2D visual understanding priors from LLaVA, our LLaVA-3D efficiently adapts LLaVA for 3D scene understanding without compromising 2D understanding capabilities. To achieve this, we utilize the 3D position embeddings to enhance the 2D CLIP Patches with 3D spatial context information and construct 3D patches. By integrating the 3D position embeddings into 2D LMMs and employing joint 2D and 3D vision-language instruction tuning, we establish a unified architecture for both 2D visual understanding and 3D scene understanding. In …

Card 5 · Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning

Recent advancements in 3D Large Language Models (3DLLMs) have highlighted their potential in building general-purpose agents in the 3D real world, yet challenges remain due to the lack of high-quality robust instruction-following data, leading to limited discriminative power and generalization of 3D…

Read full passage excerpt

Recent advancements in 3D Large Language Models (3DLLMs) have highlighted their potential in building general-purpose agents in the 3D real world, yet challenges remain due to the lack of high-quality robust instruction-following data, leading to limited discriminative power and generalization of 3DLLMs. In this paper, we introduce Robin3D, a powerful 3DLLM trained on large-scale instruction-following data generated by our novel data engine, Robust Instruction Generation (RIG) engine. RIG generates two key instruction data: 1) the Adversarial Instruction-following data, which features mixed negative and positive samples to enhance the model's discriminative understanding. 2) the Diverse Instruction-following data, which contains various instruction styles to enhance model's generalization. As a result, we construct 1 million instruction-following data, consisting of 344K Adversarial samples, 508K Diverse samples, and 165K benchmark training set samples. To better handle these complex i …

Model-reported card use

Card 1 · terminology

Affected output: general-purpose 3D LLMs

Model explanation: This row supplies the term 'general-purpose' and the LLM-based framing for 3D agents, shaping how the baseline category is named.

Card 5 · fact

Affected output: LLM-based baselines include general-purpose 3D LLMs such as LEO and Robin3D

Model explanation: This row establishes Robin3D as a 3D LLM evaluated on 3D multimodal benchmarks, licensing its inclusion as an LLM-based baseline.

No Evidence

Same model and position · no retrieved passages · first output

Preferred by blind review

LLM-based baselines include methods that adapt 2D vision-language models to 3D grounding through learned projection layers or 3D-aware prompting.

Earlier card-use comparison

The earlier diagnostic compared only cases with reported card use and text on both sides; its scope differs from the overall AI review above.

The blinded model review preferred the Evidence version. Its original rationale appears below.

Read the original model judgment

Output A is grounded in the sources and draft. The draft explicitly mentions comparing against 'LLM-based approaches' following prior work [57, 58], and the sources describe LEO as an 'Embodied Generalist Agent in 3D World' and Robin3D as a '3D Large Language Model' — both are general-purpose 3D LLMs. The 'raw point cloud protocol' is directly supported by the draft's statement that they 'operate directly on noisy, raw sensor RGB-D point clouds.' Output B invents 'learned projection layers or 3D-aware prompting' as the mechanism for LLM-based baselines. While LLaVA-3D (in sources) uses '3D position embeddings' and 'joint 2D and 3D vision-language instruction tuning,' this is not described as 'learned projection layers or 3D-aware prompting,' and the draft does not specify these mechanisms. The general claim about how LLM-based methods adapt 2D VLM to 3D is an unsupported extrapolation not directly stated in either draft or sources.

The passages are input material; their presence does not mean an output used them or that they support every claim. Draft and source passages are excerpted; no simulated scores or PDF appear here.