Think Before You Link

Abstract

Multimodal entity linking grounds entity mentions in text and images to knowledge-base entries. These systems degrade on rare entities, but prior work measures rarity primarily through popularity-based metrics such as pageviews. We broaden this view using knowledge-graph structural metrics that capture how well an entity is documented and connected. These metrics identify many rare entities that popularity metrics miss. Across the resulting rare-entity slices, state-of-the-art accuracy drops by 15.4–39.9%, showing that different rarity definitions expose different failure modes. To address these failures, we introduce a simple, training-free framework in which a reasoning-capable vision-language model iteratively searches and reasons over Wikipedia, gathering evidence dynamically. Controlled experiments show that reasoning and retrieval are complementary. Reasoning alone does not significantly improve accuracy on rare entities. Retrieval without reasoning improves rare-entity accuracy but can hurt overall accuracy. Their combination performs best. On MERLIN, a multilingual multimodal entity linking benchmark over five languages (Hindi, Indonesian, Japanese, Tamil, Vietnamese), our best system improves over the state of the art by 6.9% overall and by up to 23.3% on rare-entity slices. We release MERLIN-Rare, rare-entity test slices for targeted evaluation, with our framework.

Rarity Beyond Popularity

Prior work usually defines rare entities with popularity signals such as Wikipedia pageviews or incoming links. Popularity is only one dimension of rarity. An entity can receive attention but still have little structured or cross-lingual information. Limited documentation reduces the available textual evidence. Sparse knowledge-graph structure provides fewer relations, and low cross-lingual coverage makes it harder to connect a source-language mention to an English knowledge-base entry.

We study 15 metrics. Nine Wikipedia metrics measure attention and documentation through pageviews, backlinks, article size, revisions, editors, categories, external links, references, and images. Six Wikidata metrics measure structure and cross-lingual coverage through incoming links, outgoing links, language editions, statements, qualifiers, and entity age. An entity is rare on a metric when its value falls in the bottom 5% of the test-set distribution.

Different metrics identify different tails. Their bottom-5% sets overlap by only 37% on average, and some pairs overlap by only 10%. This means that popularity-only evaluation leaves many poorly documented or structurally sparse entities untested.

Heatmap of pairwise overlap among the rare-entity sets defined by 15 rarity metrics. — Pairwise Jaccard overlap between bottom-5% rare-entity sets. The mean pairwise overlap is 37%.

On the current state-of-the-art Cultural Pangea model, accuracy drops by 15.4–39.9% across these slices. The drop is 37.7% for pageviews and 37.0% for Wikidata statement count, even though those metrics identify largely different entities. The effect is also not created by the 5% cutoff. Accuracy declines smoothly toward the sparse end, and the main results remain stable with 1%, 5%, and 10% thresholds.

Bar chart showing Cultural Pangea accuracy losses from 15.4 to 39.9 percent across rarity metrics. — Cultural Pangea accuracy change on every bottom-5% rarity slice relative to the full test set.

Line charts showing accuracy decreasing toward the sparse end for language editions and statement count. — Accuracy decreases toward the sparse end of both structural metrics. Our system degrades less steeply than Cultural Pangea.

Framework

Our framework is training-free. A vision-language model reads the image, text, and mention, searches Wikipedia, incorporates the returned snippets, and decides whether to search again. Unlike methods that rerank a fixed candidate set, each search can depend on the evidence returned by earlier searches.

Returned evidence updates the model’s reasoning state, so the next query can retrieve different content. In this example, visual evidence changes a generic search into one that identifies the ship.

We force the first search in every retrieval configuration because the 2B models often answered without calling the tool, even when instructed to search. After that first call, the model decides whether and how to continue. It can make up to 20 retrieval calls. A second pass reads the complete reasoning trace and returns only the final Wikipedia title.

Reasoning and Retrieval

Controlled experiments with matched Qwen3-VL variants separate the effects of reasoning and retrieval. Reasoning alone does not significantly improve accuracy on rare entities. Retrieval improves rare-entity accuracy, but it can hurt overall accuracy when the model does not deliberate over the returned evidence. Their combination performs best.

Qwen3-VL 8B accuracy on the full MERLIN test set.
Model	No retrieval	Embedding retrieval
Instruct	83.5	82.9
Thinking	84.2	87.9

The matched Qwen variants establish this controlled interaction. GLM-4.6V-Flash provides a narrower check. With its thinking mode, retrieval improves all 15 rare-entity slices, but its non-thinking mode cannot reliably sustain the search loop. The GLM result supports the benefit of reasoning with retrieval on rare entities, but it is not a complete replication of the four-way Qwen comparison.

Results

On MERLIN, our best system reaches 87.9% average accuracy across Hindi, Indonesian, Japanese, Tamil, and Vietnamese. This is +6.9% over Cultural Pangea. The advantage grows on rare entities, reaching +23.3%. Fourteen of the 15 rare-slice gains are larger than the full-test-set gain.

Summary of gains across five languages, rare-entity slices, and a representative language-edition slice. — Our best system improves over Cultural Pangea on all five languages, with larger gains on the rare-entity slices.

Sorted bar chart showing gains over Cultural Pangea from 5.5 to 23.3 percent across 15 rarity metrics. — The gain reaches +23.3% for qualifiers, +22.1% for statements, and +21.7% for Wikidata outgoing links.

Horizontal bar chart showing that our system degrades less than Cultural Pangea on 14 of 15 rarity dimensions. — Our system degrades less than Cultural Pangea on 14 of 15 rarity dimensions.

Bar chart comparing a 4B reasoning model with retrieval against an 8B instruction model. — 4B-Think with embedding retrieval matches 8B-Instruct on the full set and leads by about 5–7% on structural rare slices.

Search Behavior

Embedding retrieval finds the target early, and the second search is often the most useful refinement. Later calls have sharply diminishing returns. Thinking models usually stop after one or two searches and spend more time reasoning between calls. Instruct models continue searching, even as their queries become less diverse and more repetitive.

Charts showing target retrieval by search number and cumulative retrieval success. — The second search is the most productive refinement step. Additional searches contribute less.

Charts showing instruction-model queries becoming less diverse and more repetitive over repeated searches. — Instruct-model query quality declines over long search chains, with verbatim repetition reaching 34%.

MERLIN-Rare

MERLIN-Rare is the union of the 15 rare-entity slices used in the paper. It contains 1,105 entity mentions, 790 unique images, five languages, and 105 prediction sets. Each example includes its rarity metrics, percentiles, slice flags, and the corresponding model predictions and traces.

The dataset is available on Hugging Face. The repository guide describes the files and evaluation scripts.

from datasets import load_dataset

dataset = load_dataset(
    "neulab/merlin-rare",
    "hi",
    split="test",
)

structurally_sparse = dataset.filter(
    lambda row: row["rare_language_editions"]
)

Citation

@article{pengpun2026think,
  title  = {Think Before You Link: Rarity, Reasoning, and
            Retrieval in Multilingual Entity Linking},
  author = {Pengpun, Parinthapat and Khanuja, Simran and
            Neubig, Graham},
  year   = {2026}
}

Acknowledgments

We thank Ibrahim AlRayes for his help with this project, and Jean de Dieu Nyandwi and Zaid Sheikh for sharing resources that supported this work. We also thank the members of NeuLab for their helpful feedback.

This work was supported in part by a research grant from the Defence Science and Technology Agency (DSTA), Singapore.

Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking

Iteration 1

Iteration 2

Iteration 3