Vector Index
A vector index is a data structure that stores embeddings and can find those nearest to a given query vector without comparing it against every stored vector individually.
The problem it solves is scale. Finding the closest vectors by brute force means computing a similarity score against every item in the collection — exact, simple, and linear in collection size. At a few thousand vectors that is fine. At tens of millions it is not, particularly under a latency budget.
A vector index trades exactness for speed. It organises vectors so that a search can skip most of the collection, examining only the regions likely to contain near neighbours. The result is approximate: the returned neighbours are usually the true nearest ones, but not guaranteed to be. How often the approximation is wrong is tunable, and trades against speed and memory.
The word describes three different things in ordinary usage, and separating them prevents most confusion: the algorithm (HNSW, IVF), the built structure holding a particular collection, and the product resource a managed service bills for.
In practice
The two algorithm families encountered most often:
Graph-based (HNSW and relatives). Vectors become nodes in a layered graph, each connected to some of its near neighbours. A search enters at the top layer, walks greedily toward the query, and descends. Generally fast with good accuracy; memory-hungry, since the graph edges are stored alongside the vectors.
Cluster-based (IVF and relatives). Vectors are partitioned into cells around centroids. A search identifies the few cells nearest the query and scans only those. Lower memory; accuracy depends on how many cells are probed, which is a per-query setting.
Both are commonly combined with quantisation — compressing vectors to fewer bits per dimension — to cut memory at some accuracy cost.
Three parameters recur across implementations regardless of algorithm:
- one controlling build quality (how thoroughly the structure is constructed — slower build, better search)
- one controlling search effort (how much of the structure to examine per query — slower search, better recall)
- one controlling memory via quantisation or dimensionality
The distinctive operational property is that indexes are expensive to build and cheap to query. Adding vectors incrementally is supported by most implementations but degrades structure quality over time, so periodic rebuilds are normal maintenance. Deletion is frequently implemented as a tombstone rather than a true removal, with space reclaimed only on rebuild.
Commonly confused with
Vector database. An index is a data structure. A vector database is a system built around one, adding persistence, metadata filtering, access control, replication, backups, and an API. A vector index can be a library running inside your process with no server at all.
Embedding. The embedding is the individual vector. The index is the structure that stores many of them and searches across them. See embedding.
Database index (the SQL sense). A B-tree index answers exact lookups and range queries, and returns exactly the matching rows. A vector index answers nearest-neighbour queries approximately. The shared word conceals a fundamental difference: one is about equality and ordering, the other about distance and similarity.
Retriever. The component that takes a query and returns relevant chunks. It usually uses a vector index, and may also use keyword search, filters, and reranking. The retriever is the interface; the index is one of its parts.
Approximate nearest neighbour (ANN). The problem — find items close to a query point, allowing some error. A vector index is an implementation of an ANN solution. “ANN index” and “vector index” are used interchangeably in practice.
Usage notes
“Index” is used as a verb, a structure, and a billable unit, often in one sentence: “index your documents into an index, and you’ll be billed per index.” When precision matters, name the layer.
“Flat index” means no index at all. Most libraries offer a brute-force mode under that name — exact, linear-time search. It is a reasonable choice for small collections, and worth knowing that it exists, because “we’re using a flat index” and “we’re not using an index” describe the same configuration.
Recall in this context is not the same measurement as recall in evaluation. Index recall measures how often the approximate search returns the true nearest neighbours. Retrieval recall measures how often the returned chunks contain the answer. An index can have near-perfect recall while the retrieval is useless, because the true nearest vectors were the wrong chunks.
Metadata filtering interacts with the algorithm. Filtering before search shrinks the candidate set but may break the index’s structural assumptions; filtering after search may leave too few results. Implementations differ substantially here, and the behaviour is often under-documented.
Managed services frequently do not disclose the algorithm, presenting a tuning surface instead. That is a product decision, not an absence of an index underneath.