Reranking
Reranking is a second scoring pass that reorders an already-retrieved set of candidates using a slower, more accurate model than the one that produced the set.
It exists because the two things a retrieval system wants — searching a large collection quickly, and judging relevance accurately — are served by different mechanisms.
Fast search relies on precomputation. Each chunk’s embedding is computed once, in advance, without any knowledge of the query it will eventually be matched against. That is what makes searching millions of vectors feasible, and it is also the limitation: the chunk’s representation was fixed before anyone knew what would be asked of it.
A reranker gives up the precomputation. It takes the query and one candidate together, processes them jointly, and scores the pair. This is far more accurate — the model can attend to how the specific query relates to the specific text — and far more expensive, because nothing can be computed ahead of time and every candidate requires its own model pass.
The resolution is a two-stage pipeline: retrieve broadly and cheaply, then rescore a small candidate set precisely. This shape is old, predating embeddings, and is standard in classical information retrieval.
In practice
The typical arrangement: retrieve 50 to 100 candidates from the vector index, rerank them, pass the top 5 to 10 to the language model.
The model doing the reranking is usually a cross-encoder — an architecture that takes query and document as a single joined input and outputs a relevance score. Contrast with the bi-encoder used for the first stage, which encodes query and document separately and compares the results. The names describe exactly the difference that matters: joint processing versus independent processing.
Other mechanisms also carry the label:
- LLM-based reranking, where a general-purpose language model is prompted to score or order candidates. Flexible, slower, more expensive per candidate.
- Fusion methods such as reciprocal rank fusion, which merge rankings from several retrievers (typically dense and keyword) by position rather than score. Often called reranking, though no model is involved and nothing is rescored.
- Rule-based reordering on metadata — boosting recent documents, demoting deprecated ones. Also often called reranking.
Cost scales with candidate count. Reranking 100 candidates is roughly ten times the work of reranking 10, so the candidate count is the main tuning knob and typically the dominant term in a reranker’s latency contribution.
Commonly confused with
Retrieval. Retrieval selects a candidate set from the whole collection. Reranking only reorders a set it has been given; it can never surface something retrieval missed. This is the most consequential distinction in the entry: if the correct chunk was not in the candidate set, no reranker can recover it, and the failure is a retrieval failure regardless of what the reranker does.
Filtering. Filtering removes candidates by a condition — date, permission, source. Reranking reorders without removing, though it is common to truncate afterwards, which produces the same observable effect.
Hybrid search. Running two retrieval methods (usually dense and keyword) and combining their results. It happens at the first stage and produces a candidate set; reranking is a second stage applied to one. The two frequently appear together and the merge step of hybrid search is sometimes itself described as reranking.
Ranking. The general term for ordering results by relevance. Reranking specifically means doing it again, to an already-ordered set.
Usage notes
Whether reranking helps is corpus-dependent and measurable. Its benefit comes from correcting first-stage ordering errors, so it helps most where the first stage is noisy — long chunks, ambiguous queries, near-duplicate documents — and least where the first stage already places the right chunk first. The general claim “reranking improves retrieval” is too coarse to act on; the measurement is whether precision at the k actually passed to the model improves on a given eval set.
“Reranker” usually means a hosted or open cross-encoder model, but the word is applied loosely to any second-pass reordering, including pure heuristics. Worth asking which is meant.
Latency is the usual reason it is dropped. A reranker sits on the critical path between retrieval and generation and cannot be overlapped with either. Reducing candidate count is the standard mitigation.
Scores are not comparable across rerankers, and generally not calibrated. A score of 0.8 from one model means nothing in particular relative to 0.8 from another, and thresholding on absolute values requires calibrating against a specific model and corpus.