FactEmbed

Abstract

Composed Image Retrieval (CIR) aims to retrieve a target image given a reference image and a modification text. Although recent approaches leveraging Multimodal Large Language Models (MLLMs) have shown significant progress, we argue that the MLLM’s compositional understanding remains underutilized. This stems from directly mapping composed queries into a retrieval embedding via contrastive supervision alone, without explicitly resolving the intended visual modifications. In this work, we introduce Factorized MLLM Embeddings (FactEmbed), which decouples compositional query comprehension from building discriminative embeddings. Rather than directly projecting the composed query into a single embedding without internal reasoning, FactEmbed employs a set of factorized latents that simulate the intended visual modifications via an intermediate target imagination objective. This process encourages the latents to ground into target semantics before forming discriminative representations for retrieval. Subsequently, to propagate these factorized representations throughout the model for retrieval, we optimize a factor-level contrastive loss while distilling the decomposed representations into a deeper attention layer of the MLLM. Finally, for efficient inference, we retain the [EOS] token to establish a global-level embedding on top of these factorized latents, while we exploit the mean-pooled factorized latents to rerank a short candidate list at negligible cost, as both are produced by the same forward pass. Extensive evaluations demonstrate that FactEmbed outperforms prior works across CIR benchmarks.

Method

FactEmbed appends K = 8 learnable factorized latents before the [EOS] token on both the query side (reference image + modification text) and the target side (target image). Four components shape them.

Target imagination

A lightweight decoder on an intermediate layer reconstructs the target’s DINOv2 patch features from the latents, on top of a coarse 2×2 layout prior, so the latents must encode the fine-grained factors.

Factor-level InfoNCE

Query and target latent sets are matched with optimal transport, and the matching score is trained contrastively alongside the instance-level InfoNCE on [EOS].

Attention distillation

The decoder’s per-latent alpha masks are distilled into a deeper attention layer of the MLLM, propagating the spatial decomposition into the backbone.

Efficient rerank

At inference, one forward pass gives the global [EOS] embedding for retrieval and the mean-pooled latents for reranking the top-100 shortlist.

Reference image, target image, DINOv2 PCA of the target, PCA of the features imagined from the query, and the argmax of the decoder alpha masks, for a dress query and a shoe query. — **Target imagination.** For each composed query: target DINOv2 features (DINOv2 PCA), features imagined by the decoder from the query (Recon PCA) on a joint PCA basis, and, per spatial position, the latent with the largest decoder alpha weight (Alpha Mask Argmax; each color is a different latent). On top of the coarse layout prior, the latents recover the fine details.

Alpha-mask argmax maps for three garments under no layout prior, a 1x1 layout prior and the default 2x2 layout prior. — **Impact of the layout prior.** Without a prior (top) the latents collapse into coarse global semantics. With the default 2×2 prior (bottom) they isolate semantically editable parts.

Why does it help?

MLLM-based CIR usually compresses the composed query into a single [EOS] embedding and trains it with a contrastive loss. That mapping gives the model no signal about which parts of the reference the text modifies and which it preserves, even though CIR edits are mostly localized. FactEmbed inserts an intermediate step that resolves the edit first.

Imagination forces factorization

The 2×2 layout prior already explains the target’s coarse layout, so the latents can only lower the reconstruction loss by encoding the residual fine-grained factors. Squeezing them through a few bottlenecks makes each latent cover a coherent part, with no dense annotations.

Factors are aligned for retrieval

Reconstruction alone can decompose query and target faithfully yet place them in incompatible spaces. The factor-level InfoNCE aligns the query factors with the target factors directly.

The backbone inherits the decomposition

Distilling the decoder’s alpha masks into a deeper attention layer moves the spatial decomposition out of the auxiliary head and into the layers that produce the retrieval embedding.

Fine detail is recovered at ranking time

Aggregating many factors into one [EOS] vector dilutes fine-grained evidence. Reranking the top-100 with the mean-pooled latents restores it, from the same forward pass and one extra vector per image.

Results

Refined-CIRR and Refined-FashionIQ

Method	Refined-CIRR				Refined-FashionIQ (Avg.)
Method	R@1	R@5	R_s@1	Avg.	R@10	R@50
VLM2Vec-v2 2B	59.93	86.79	81.47	70.70	46.18	67.27
MM-Embed 8B	60.64	87.73	81.54	71.09	60.30	79.97
TME	65.05	90.53	86.86	75.96	68.74	85.53
Qwen3-VL-Emb 8B	67.03	90.45	84.60	75.82	68.61	84.25
CSMCIR^*	67.16	90.96	86.15	76.66	69.55	85.83
Qwen2.5-VL (Base)	64.64	89.51	84.65	74.65	66.82	83.55
ReCALL	67.41	90.99	87.12	77.27	69.48	85.70
FactEmbed (Ours)	68.99	91.29	87.45	78.22	71.13	86.41

Selected rows from the paper’s main table; bold and underline mark the best and second-best among the rows shown. Avg. on Refined-CIRR is (R@1 + R_s@1)/2; on Refined-FashionIQ it is the mean over dress, shirt and top&tee. ^* uses additional gallery image captions.

CIRR test

Method	Recall@N				Recall_subset@N			Avg.
Method	R@1	R@5	R@10	R@50	R@1	R@2	R@3	Avg.
CIR-LVLM	53.64	83.76	90.60	97.93	79.12	92.33	96.67	81.44
QuRe	52.22	82.53	90.31	98.17	78.51	91.28	96.48	80.52
CCIN	53.41	84.05	91.17	98.00	–	–	–	–
TME	53.42	82.99	90.24	98.15	81.04	92.58	96.94	82.01
Air-Know	51.97	82.25	90.42	98.25	79.21	91.94	96.87	80.73
ConeSep	53.06	83.45	90.60	98.34	81.23	92.62	97.24	82.34
Qwen3-VL-Emb 8B^‡	51.95	80.92	88.58	97.49	75.23	90.29	95.66	78.08
CSMCIR^*	53.88	85.06	91.69	98.58	80.10	92.24	96.92	82.58
Qwen2.5-VL (Base)^‡	51.98	81.90	90.27	98.19	77.69	91.85	96.92	79.80
ReCALL^‡	53.76	83.76	91.54	98.34	81.11	93.35	97.52	82.44
FactEmbed (Ours)	56.60	84.87	91.78	98.41	81.86	93.78	97.61	83.37

Avg. is (R@5 + R_subset@1)/2.

FashionIQ validation

Method	Dress		Shirt		Top&Tee		Avg.
Method	R@10	R@50	R@10	R@50	R@10	R@50	R@10	R@50
CIR-LVLM	50.42	73.60	58.59	75.86	59.61	78.99	56.21	76.14
CCIN	49.38	72.58	55.93	74.14	57.93	77.56	54.41	74.76
TME	49.73	71.69	56.43	74.44	59.31	78.94	55.15	75.02
QuRe	46.80	69.81	53.53	72.87	57.47	77.77	52.60	73.48
Air-Know	51.03	73.08	56.76	75.05	59.47	79.02	55.75	75.72
ConeSep	50.96	73.02	56.98	75.22	58.80	79.40	55.58	75.88
Qwen3-VL-Emb 8B^‡	47.84	69.46	52.36	71.34	55.58	74.40	51.93	71.73
CSMCIR^*	52.45	74.81	57.70	75.76	61.14	80.98	57.10	77.18
Qwen2.5-VL (Base)^‡	45.81	70.65	55.50	73.45	57.62	76.95	52.98	73.68
ReCALL^‡	48.93	72.63	54.76	73.60	57.98	78.74	53.89	74.99
FactEmbed (Ours)	53.25	74.67	58.34	76.30	61.55	79.65	57.71	76.87

Bold and underline mark the best and second-best results. ^‡ reproduced with the official implementation. ^* uses additional image captions for gallery images.

Interactive demo

Browse validation queries (reference image + modification text) and their top-10 retrievals; the ground-truth target is outlined in green. Captions default to the refined queries. The model weights are on Hugging Face, and the demo also runs standalone.

FashionIQ

[EOS] retrieval versus Efficient rerank, Qwen2.5-VL-7B backbone.

CIRR

Efficient rerank with three FactEmbed backbones: Qwen2.5-VL-7B, Qwen3-VL-8B and Qwen3-VL-Embedding-8B.

BibTeX

@article{moon2026factembed,
  title   = {Factorized MLLM Embeddings for Composed Image Retrieval},
  author  = {Moon, WonJun and Kim, Seongchan and Kim, Dahun and Kim, Seungryong},
  journal = {arXiv preprint arXiv:2611.05193},
  year    = {2026}
}