Context Window

Definition. The maximum number of tokens a model can attend to in a single request, comprising the system prompt, the full message history, any retrieved documents, and the generated output. It is a hard architectural limit set at training time, not a soft preference. Current production models range from roughly 8,000 tokens on small efficient models to over a million on long-context variants.

How it works. Manage the window as a budget with explicit allocations. Implement a context assembler that reserves space for output first, then the system prompt, then the most recent conversation turns, then retrieved evidence, and drops or summarises the oldest material when the total exceeds the limit. For long-running conversations, replace raw history with a rolling summary plus the last N turns.

Example. A document assistant keeps a 200,000-token window. Users paste a 150,000-token annual report and ask follow-up questions. After six turns of answers averaging 400 tokens each, plus 2,000 tokens of retrieved excerpts per turn, the request exceeds the window and the API returns a context-length error. The team adds a summarisation step that compresses the report into a 12,000-token structured outline plus a retrieval index, keeping every request under 30,000 tokens and cutting cost by roughly 80 percent.

Use case. Context window sizing decides architecture. Small windows force retrieval and chunking; large windows allow whole-document reasoning but invite cost and recall problems. It is the primary constraint in code assistants, contract analysis, long-form research agents and multi-turn customer support.

Warnings. A large window is not a substitute for retrieval. Stuffing 500,000 tokens into a prompt is slow, expensive, and measurably less accurate than retrieving the right 8,000. Also note that providers may bill cached and uncached input differently, so reusing a stable prefix is a real cost lever.

Limitations. Effective attention degrades with length, a phenomenon often called lost in the middle. Positional encodings extrapolate imperfectly beyond training lengths. And because the window is shared with output, a request that fills it leaves no room to answer, producing truncated or empty responses.

← Browse all glossary terms