Context Window
A context window is the maximum amount of text, measured in tokens, that a language model can process in a single request — a hard architectural limit covering the input and, in most accounting, the generated output as well.
Everything the model considers when producing a response must fit inside it: the system instructions, the conversation history, any retrieved chunks, the user’s question, and the answer being written. The model has no memory outside this window. Anything not present in it does not exist for that request.
The limit is a property of the model, not of the service or the request. It is fixed at training time by architectural and training choices, and cannot be raised by configuration.
Sizes have grown by several orders of magnitude since large language models became widely available, from a few thousand tokens to, in some models, over a million. That growth is what made “put the whole corpus in the prompt” a plausible alternative to retrieval for small corpora.
In practice
In a retrieval pipeline, the context window is the budget that prompt assembly spends. A request typically contains:
| Component | Share |
|---|---|
| System instructions | small, fixed |
| Conversation history | grows per turn |
| Retrieved chunks | the largest and most variable part |
| User query | small |
| Room reserved for the answer | must be left free |
The last row is the one most often forgotten. If the window is filled to capacity with input, there is no room to generate, and the request either fails or returns a truncated response.
Assembly is therefore a packing problem: fit as many useful chunks as possible while reserving output space and keeping history. When it doesn’t fit, something is dropped — the oldest turns, the lowest-ranked chunks, or the tail of a chunk cut mid-sentence. Which of these happens is a design decision, and in many systems it is an accident.
Two properties matter beyond the raw limit.
Cost scales with what you put in it, not with the window’s size. Input tokens are billed per request, so a long prompt is a recurring cost, and using more of an available window is a decision with a price.
Usage across a long window is uneven. Material placed in the middle of a very long context tends to be used less reliably than material at the beginning or end — an effect observed widely enough to have a common name, lost in the middle. A model advertising a large window can accept that much input; it does not follow that it attends to all of it equally.
Commonly confused with
Context length / max tokens / window size. Synonyms in general use. Note that some APIs expose a max_tokens parameter that limits only the output, which is a different quantity from the total window despite the similar name — a frequent source of confusion when a request fails.
Token limit. Ambiguous. It may mean the context window, the output cap, or a rate limit on tokens per minute imposed by the provider. The three are unrelated constraints and are all abbreviated the same way in error messages.
Memory. A conversational system’s memory is an application-level construct — storing, summarising, and reinjecting prior turns. The context window is the fixed capacity that memory strategies exist to work around. A system with sophisticated memory still has the same window.
Chunk size. A chunk is a retrieval unit; the context window holds several of them plus everything else. Chunk size is a corpus decision, window size is a model property. See chunk.
Attention span. Not a technical term. Used informally to describe the uneven-usage effect above, and best avoided in precise writing.
Usage notes
Advertised size and effective size differ. A model may accept a very large input while performing measurably worse on tasks requiring information from deep within it. Published window sizes describe capacity, not uniform quality across that capacity, and treating the two as equivalent is a common planning error.
Token counts are model-specific. The same text produces different token counts under different tokenisers, and estimates in words or characters are approximations. A rough working figure for English prose is around 750 tokens per page, but this varies with content — code, tables, and non-English text tokenise less efficiently, sometimes substantially.
Whether output counts against the window varies by model and provider. Most architectures share one budget between input and output; some services document separate input and output maxima. Check the specific model rather than assuming.
Prompt caching changes the economics without changing the limit. A stable prefix can often be cached and billed at a reduced rate on subsequent requests, but it still occupies the same space in the window.