Vision Tower Bench

Choosing a vision tower for a text-only LLM

We probed sixteen frozen vision towers across layers. Most towers lose geometry by their final layer, projectors mostly recover that loss rather than add to it, and few-label recognition separates towers that full labels cannot.

Finding 1Geometry fades by the final layerSee the evidence ↓Finding 2Projectors repair the final layer, not the best oneSee the evidence ↓Finding 3Few labels separate towers that full labels cannotSee the evidence ↓

PaperRepo 🤗Towers

Version

Why this exists

Adding vision to a base LLM means choosing a tower, layer, and projector. Full-model benchmarks do not isolate those choices, so we probe frozen features before multimodal training.

What the capability axes mean

Recognition

Can a probe identify the object, with plenty of labels and with almost none?

Geometry

Can it recover depth and surface orientation from a single image?

Cross-view

Can it match points across views or object instances?

Occlusion

How much recognition survives when half the image is hidden?

Capability Profiles

Each column measures a separate capability. Close values may be unresolved.

Tower	Few-label1% top-1 ↑	Full-label100% top-1 ↑	Depthd1 ↑	Normalsdeg ↓	Cross-viewNAVI@2cm ↑	Occlusion50% kept ↑
DINOv3	0.530	0.915	0.768	19.08	0.541	40%
DINOv2	0.377	0.908	0.702	18.85	0.539	52%
SigLIP2	0.656	0.917	0.670	23.90	0.401	49%
Muse Glimmer	0.703	0.918	0.659	24.30	0.214	52%
GLM 5.3 Flash	0.683	0.919	0.643	24.35	0.382	45%
Nemotron 3.5 Super	0.480	0.902	0.690	23.06	0.502	36%
Nemotron 3 Omni	0.544	0.905	0.641	24.39	0.419	46%
Nemotron 3.5 Bolt	0.484	0.895	—	—	—	—
Qwen3.8	0.463	0.878	0.674	24.24	0.378	30%
Kimi K2.6	0.388	0.888	0.660	25.51	0.365	28%
DeepSeek V4.1	0.369	0.877	0.654	25.27	0.392	35%
Gemma 4	0.329	0.876	0.555	27.25	0.304	30%
Kimi K3	0.228	0.852	0.655	27.03	0.333	20%
MiMo V2.6 Pro	0.240	0.825	0.648	25.53	0.363	—
MiMo V2.6 Flash	0.239	0.830	0.653	25.55	0.367	—
MiniMax M3	0.052	0.324	0.543	31.48	0.235	18%

Recognition standardizes channels before PCA. GLM 5.3 Flash, Muse Glimmer, SigLIP2 and DINOv3 sit within 0.004 at full labels; treat them as tied.
Depth and normals use the best matched tower layer. NAVI uses the final tower layer.
Nemotron 3.5 Bolt ships without its tower stage, so recognition reads its merged stage.
One outlier channel dominates MiMo V2.6's final tower layer, which leaves its occlusion readout at chance.
↑ higher is better, ↓ lower is better.

Method, in short

Towers stay frozen. We tap eight depths plus merged and projected stages. Features cache once; only the readouts train. Recognition uses ImageNet-100, geometry uses DIODE and KITTI, and correspondence uses NAVI, ScanNet and SPair.

Heads are reduced to 512 dimensions with train-split PCA, and recognition standardizes channels first. Each cell selects a learning rate on validation and runs three seeds.

Figure 1

How the bench taps a model

Results

Finding 1

Geometry fades by the final layer

Geometry peaks before the final layer in 56 of 60 arms, but DINO and RADIO towers barely fade. DINOv3 and Nemotron 3.5 Super lose no depth at their final layer, and DINOv2, Nemotron 3 Omni and GLM 5.3 Flash lose under 0.03 d1. Qwen3.8, Muse Glimmer, Kimi K3 and DeepSeek V4.1 lose more than 0.10.

Figure 2

Geometry peaks early

Stars mark each tower's depth peak in the matched arm.

The original six towers’ matched sweeps used unseeded PCA.

Finding 2

Projectors repair the final layer, not the best one

Against the final tower layer, the projector wins 30 of 48 DIODE comparisons. Against the best intermediate layer, it is ahead on depth only for GLM 5.3 Flash and Gemma 4. Most of the projector's gain recovers what the final layer lost.

Figure 3

The best tower layer beats the projector

The best layer is chosen on test scores, so the gap is descriptive.

Projector	DIODE				KITTI	NAVI	ScanNet	SPair
Projector	Depth (M)	Depth (R)	Normals (M)	Normals (R)	KITTI	NAVI	ScanNet	SPair
Muse Glimmer	+.026	+.093	+0.79°	+2.89°	−.004	+.106	+.032	+.133
Kimi K2.6	+.017	+.020	+2.20°	+3.11°	−.005	+.057	+.008	+.115
Kimi K3	+.078	+.027	+2.50°	+2.83°	+.001	+.059	+.021	−.014
Qwen3.8	+.058	+.160	+4.31°	+4.44°	−.002	+.019	−.019	+.022
Gemma 4	+.101	+.056	+2.72°	+2.31°	+.026	+.042	−.002	+.012
GLM 5.3 Flash	+.038	−.072	+0.97°	+0.10°	−.002	+.037	−.003	+.015
DeepSeek V4.1	+.080	−.026	+3.33°	−0.09°	−.009	+.012	−.033	−.023
Nemotron 3 Omni	−.009	−.050	+0.37°	−0.47°	−.011	+.002	−.021	−.011
MiniMax M3	−.094	−.116	−3.15°	+0.50°	−.098	−.009	−.025	−.013
Nemotron 3.5 Super	−.057	−.151	−1.55°	−1.60°	−.016	−.056	−.067	−.131
MiMo V2.6 Pro	+.097	−.364	+6.73°	−0.29°	+.018	+.008	−.017	+.027
MiMo V2.6 Flash	+.173	−.063	+8.90°	+0.48°	+.033	+.029	−.010	+.019

Figure 4

Projector advantage over the final tower layer

—Positive favors the Projector.
—Normals report error removed.
—M = capacity-matched to 512 dimensions.
—R = raw token width.

Projector better, interval excludes zero
Tower better, interval excludes zero
Unresolved

Finding 3

Few labels separate towers that full labels cannot

With full labels, 12 of 16 towers sit between 0.876 and 0.919. With 1% labels they spread from 0.05 to 0.70, led by Muse Glimmer, GLM 5.3 Flash and SigLIP2. Standardizing channels before PCA lifted Muse Glimmer from 0.43 to 0.70 and MiMo V2.6 from chance to 0.24.

Figure 5

Recognition rankings change with label budget

Channels are standardized before PCA.

How to choose

Pick the tower first, then the layer, then the projector. With no dominant requirement, start with SigLIP2. It is top three on recognition at both label budgets, with no weak axis.

For spatial work, tap an intermediate layer and use Figure 2 to place it. The final layer is safe for DINOv3, DINOv2 and the Nemotron towers.

If the priority is	Start with
Depth or ScanNet correspondence	DINOv3. Its final layer is its depth peak.
Normals or NAVI correspondence	DINOv2, DINOv3.
Spatial work inside an existing VLM	Nemotron 3.5 Super.
Recognition with few labels	Muse Glimmer, GLM 5.3 Flash, SigLIP2. Standardize channels first.
Recognition with abundant labels	GLM 5.3 Flash, Muse Glimmer, SigLIP2, DINOv3.
Robustness to occlusion	DINOv2, Muse Glimmer, SigLIP2.

What v3 does not establish

—These are frozen-representation results, not evaluations of a working VLM.
—Best layers are picked on test scores, so Figures 2 and 3 are descriptive.
—MiMo V2.6's final tower layer is dominated by one channel, which distorts its final-layer cells and projector comparisons.
—Nemotron 3.5 Bolt ships without a tower stage, and Nemotron 3.5 Super's tower cells precede its final LayerNorm.
—Near-ties, attention pooling, and the small geometry test sets remain unresolved.

Representation explorer

What the features look like

Qualitative PCA of frozen patch tokens. Visible structure is not a benchmark score.

source · 448²

DINOv3 · tower · 28 x 28

DINOv3 traces the subject smoothly

The eagle reads as one region with a soft gradient. First on depth, NAVI and ScanNet, and its final layer is its depth peak.

NAVI recall@2cm0.541 · rank 1 of 15

recognition top-10.915 · rank 4 of 16

1% labels top-10.530 · rank 5 of 16

Changelog

Oct 2026

Vision Tower Bench v3 adds five new Towers.

Oct 2026

Vision Towers for MiMo V2.6 Pro and MiMo V2.6 Flash published on Hugging Face.

Oct 2026

Vision Towers for Nemotron 3.5 Super and Nemotron 3.5 Bolt published on Hugging Face.

Oct 2026

Vision Tower Bench v2.1 reruns low-label recognition with channels standardized before PCA.

Sept 2026

Vision Towers for Gemma 4, MiniMax M3 and GLM 5.3 Flash published on Hugging Face.

Sept 2026

Vision Towers for Deepseek V4.1 Flash and Nemotron 3 Nano Omni published on Hugging Face.

Sept 2026

Vision Tower Bench v2 adds five Towers and expands the correspondence and robustness results.

Sept 2026

Vision Tower Bench v1 released, covering six Towers on geometry, semantic and correspondence.

Sept 2026

Vision Towers for Kimi K2.6, Kimi K3, Muse Glimmer and Qwen3.8 published on Hugging Face.