Multi3IR: A Benchmark for Multi-perspective
Multi-domain Multi-modal Information Retrieval

Seokwon Song, Sohyeon Kim, Gunhee Kim
Seoul National University
EMNLP 2026
Paper Code 🤗 Dataset

Abstract

Information retrieval increasingly targets open-ended queries that admit diverse perspectives. Existing IR benchmarks focus on closed-ended queries, and even open-ended ones mostly have supporting documents from a single subject domain and a single modality. We introduce Multi³IR, a benchmark that evaluates how well retrievers cover the multifaceted perspectives of open-ended queries across diverse domains and modalities. It comprises 104.9K Stack Exchange queries, each annotated with perspective descriptions that capture the query's implicit viewpoints, over a 1.01M-document corpus of web text and web images with judged gold documents per perspective. We also propose SPIN (Steering Perspectives by Injecting Noise), a parameter- and label-efficient method that learns noise vectors to steer a frozen retriever's embeddings toward diverse yet meaningful semantic directions. Existing multimodal retrievers suffer from single-perspective bias; SPIN substantially improves perspective coverage on Multi³IR and generalizes to unseen open-ended IR benchmarks.

Motivation: Can Retrievers Find Every Perspective?

Ask “Why are so few foods blue?” and there is no single answer. Biology explains why blue pigments are rare, psychology explains how people perceive blue food, and chemistry explains why blueberries turn green when baked. The evidence is spread across domains, in both text and images. Yet current retrievers compress the query into one embedding and return documents for only the dominant perspective. A useful retriever must find evidence for every perspective.

The query 'Why are so few foods blue?' branches into three perspectives, each linked to supporting text and image documents from biology, psychology, social science and chemistry.
An open-ended query has several implicit perspectives, each supported by documents from different domains and modalities.

Multi³IR Framework

Benchmark Construction

Data construction pipeline: Step 1 post selection from Stack Exchange, Step 2 perspective extraction and verification, Step 3 document retrieval and verification.
A Stack Exchange post goes in; perspectives with exclusively supporting text and image documents come out.
1Post Selection. Open-ended questions and their community answers are collected from 77 Stack Exchange sites in 5 categories.
2Perspective Extraction & Verification. GPT-5-mini turns the answers into perspective descriptions. Unfaithful perspectives (Bespoke-MiniCheck-7B) and redundant ones (cosine > 0.8) are removed.
3Document Retrieval & Verification. Text documents come from C4 (BM25 + dense rerank) and image documents from Google Image Search. Qwen3-VL-30B-A3B keeps only documents that support their own perspective and no sibling perspective.

SPIN: Steering Perspectives by Injecting Noise

Algorithm 1 SPIN Optimization

Require: frozen encoder \(M\) with \(L\) layers, training set \(\mathcal{X}\), learnable noise vectors \(\mathcal{E}=\{\boldsymbol{\varepsilon}_1,\ldots,\boldsymbol{\varepsilon}_m\}\), injection layer \(l\), sigmoid temperature \(\tau\), bias \(b\)

  1. for batch \(\mathcal{B}=\{(q_n,\mathcal{P}_n)\}_{n=1}^{N}\subset\mathcal{X}\) do
  2. for \(n=1,\ldots,N\) do
  3. \(\mathbf{h}_l^n\leftarrow M_{0\rightarrow l}(q_n)\) ▷ representation of \(q_n\) at layer \(l\)
  4. \(\mathbf{S}_n\leftarrow\{M_{l\rightarrow L}(\mathbf{h}_l^n+\boldsymbol{\varepsilon})\mid\boldsymbol{\varepsilon}\in\mathcal{E}\}\) ▷ steered embeddings
  5. \(\mathbf{T}_n\leftarrow\{M(p)\mid p\in\mathcal{P}_n\}\) ▷ target embeddings
  6. end for
  7. \(p_{\mathbf{s}}(\mathbf{t})\leftarrow\sigma\big((\cos(\mathbf{s},\mathbf{t})-b)/\tau\big)\) ▷ prob. that \(\mathbf{s}\) covers \(\mathbf{t}\)
  8. \(\mathcal{L}_{\text{pos}}\leftarrow-\frac{1}{N}\sum_{n}\frac{1}{|\mathbf{T}_n|}\sum_{\mathbf{t}\in\mathbf{T}_n}\log\big(1-\prod_{\mathbf{s}\in\mathbf{S}_n}(1-p_{\mathbf{s}}(\mathbf{t}))\big)\)
  9. \(\mathcal{L}_{\text{neg}}\leftarrow-\frac{1}{N}\sum_{n}\frac{1}{|\mathbf{T}_{\neq n}|}\sum_{\mathbf{t}\in\mathbf{T}_{\neq n}}\sum_{\mathbf{s}\in\mathbf{S}_n}\log\big(1-p_{\mathbf{s}}(\mathbf{t})\big)\) ▷ in-batch negatives
  10. Minimize \(\mathcal{L}=\mathcal{L}_{\text{pos}}+\mathcal{L}_{\text{neg}}\)
  11. end for
Training illustration
SPIN training: the query passes through frozen layers up to the middle layer, the hidden state is perturbed by learnable noises, and the frozen remaining layers produce steered embeddings matched against target embeddings of the perspective descriptions.
Training: learnable noises injected at the middle layer branch one query into several steered embeddings, each matched against the perspective descriptions.
Inference illustration
SPIN inference: the learned noises are frozen and each steered embedding retrieves its own ranked list.
Inference: the learned noises are frozen; each steered embedding retrieves its own ranked list, and the lists are merged by round robin.
1Split a Frozen Retriever. The encoder is split at the middle layer (0.5L), where the hidden states still hold several interpretations of the query.
2Inject Learnable Noise. m = 5 noise vectors are added to the hidden states, and the remaining frozen layers produce m steered embeddings. Only the noise table is trained (20.5K parameters).
3Noisy-OR Coverage Loss. Every perspective description must be covered by at least one steered embedding, and no negative may be covered by any.
4Round-Robin Retrieval. At inference the m rankings are merged by round robin, so the final list covers several perspectives instead of repeating the dominant one.

Multi³IR Samples

Perspective Coverage Evaluation

Preliminary: Single-Perspective Bias

Hit@20 by perspective rank for Qwen3-VL, GME-Qwen2 and MM-Embed. With the original query, Hit@20 drops from about 0.6 to near zero by the third perspective; with oracle perspective queries it stays high for all four.
Hit@20 per perspective, ranked. (a) Retrieval with the original query. (b) Retrieval with the gold perspective descriptions, merged by round robin.

Do retrievers cover all perspectives? No. With the original query, only the dominant perspective is covered; Hit@20 falls close to zero by the third perspective for all three retrievers.

Why do they miss the others? Querying with the gold perspective descriptions recovers them, so the document embeddings are already well separated.

The bottleneck is on the query side. One query embedding fails to capture the multiple meanings of an open-ended query.

Main Results

Scroll horizontally to view all columns.

Perspective coverage (%) on the Multi³IR-Bench test split (1.0K queries, 4.8K perspectives, 1.01M candidates). HC@k counts a perspective as covered when one of its gold documents is in the top-k; SC@k uses a GPT-5-mini support judge (κ = 0.84 against humans). m is the number of query embeddings; FT-Doc / FT-Persp mark document- or perspective-level supervision. The best non-oracle result per retriever is shown in bold; Oracle uses the gold perspective descriptions as queries.
Retriever Method m Supervision Hard Coverage Soft Coverage
FT-DocFT-Persp HC@10HC@100 SC@10SC@20

Key Findings

Current retrievers have single-perspective bias. With the original query, multimodal retrievers cover only the dominant perspective; Hit@20 falls close to zero by the third perspective.

The bottleneck is the query embedding. Querying with the gold perspective descriptions recovers the missed documents, so the document embeddings are already well separated.

SPIN improves coverage with few parameters and cheap labels. With 20.5K trainable parameters and perspective-level supervision only, SPIN outperforms LLM-Expansion and ARE on all three retrievers, and the middle injection layer matters most.

BibTeX

@article{song2026multi3ir,
  title={MULTI3IR: A Benchmark for Multi-perspective Multi-domain Multi-modal Information Retrieval},
  author={Song, Seokwon and Kim, Sohyeon and Kim, Gunhee},
  journal={arXiv preprint arXiv:2608.30949},
  year={2026}
}