🔒 Unlocking Few-Shot Capabilities 🔓
in LVLMs via
Prompt Conditioning and Head Selection

ECCV 2026
Adhemar de Senneville1, Xavi Bou2, Rafael Grompone1, Jérémy Anger1, Gabriele Facciolo1,3
1ENS Paris-Saclay, Centre Borelli 2AMIAD, Pole Recherche 3Institut Universitaire de France
arXiv Code Poster

LVLMs are worse than CLIP at Few-shot and Zero-shot classification unless...

1. You use features from the right attention heads. 2. You use the right prompt.

Select a prompt to display the attention map of the head best suited to the task.

Selected attention head Layer 23 · Head 6
Bird-specialized attention on the first bird and aircraft example
Bird-specialized attention on the second bird and aircraft example
A Unified Framework

HEC: Head Ensemble Classifiers

HEC (Head Ensemble Classifiers)

CLIP-based vs HEC (Ours). CLIP-based methods encode class names and support set images independently to construct a zero-shot and a few-shot classifier respectively. Our method keeps the same two-classifier structure. However, both distributions go through a shared LLM decoder which can be conditioned by a text prompt to include guidance on the domain or the classes of the support set. The few-shot classifier (HEC-V) and zero-shot classifier (HEC-T) build on separate sparse sets of vision-heads and text-heads, respectively. Similarly to CLIP-based methods, the two classifiers can be combined into a single classifier (HEC-VT).

A Unified Training-free Framework
HEC covers few-shot and zero-shot classification in one framework, without fine-tuning.
HEC-V

Selecting Vision-Heads with GDA

HEC-V vision-head selection with Gaussian Discriminant Analysis

Vision-heads are better than the model’s output at few-shot classification. HEC-V evaluates the prompt-conditioned features of each attention head using Gaussian Discriminant Analysis (GDA) on the support set. It then selects a sparse set of the 20 most discriminative vision-heads and combines their predictions into the few-shot classifier.

Prompt-conditioned Domain Adaptation
HEC-V extracts prompt-conditioned domain-specific few-shot features from LVLMs, which was previously impossible with vision-only or CLIP backbones.
HEC-T

Selecting Text-Heads Only Once

Text-heads are better than the model’s output at zero-shot classification. HEC-T selects a sparse set of the 20 most discriminative text-heads and combines their predictions into the zero-shot classifier.

Text-Head Transferability
By selecting the top 20 text-heads only once on ImageNet, HEC-T improves the zero-shot performance of Qwen2-VL by 10.1 points.
HEC-VT

Combining Vision and Text Heads

GDA
Previous CLIP SOTA
81.5%
SAVs
Previous LVLM SOTA
80.0%
HEC-VT
Ours
83.0%

Higher is better. Average 4-shot accuracy across 12 datasets.

Bridging the Gap
HEC-VT combines HEC-V and HEC-T classifiers, bridging the gap between LVLM-based and CLIP-based few-shot classification.

Contact

For questions, contact adhemar.de_senneville@ens-paris-saclay.fr

BibTeX

@article{de2026unlocking,
  title={Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection},
  author={de Senneville, Adhemar and Bou, Xavier and Anger, J{\'e}r{\'e}my and Grompone, Rafael and Facciolo, Gabriele},
  journal={arXiv preprint arXiv:2603.24181},
  year={2026}
}

Acknowledgements. This work was partially funded by AID-DGA (l’Agence de l’Innovation de Défense à la Direction Générale de l’Armement, Ministère des Armées), and was also partly funded by the ANR-DFG project BOFOR ANR-24-CE92-0048. This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD011016525 made by GENCI. We thank Pierrick Bournez for his insightful feedback while reviewing the paper. Finally, we thank Constantin Godard for his support during the NPC internship. This website's template was inspired by the INSID3 project page.