LLM Context Windows Explained: Tokens, Sizes, and How to Use Them Well

Ask a language model a question and it doesn't reach into an infinite memory — it reads a fixed-size window of text, thinks, and writes back. That window is the single most important constraint to understand when building anything real on top of an LLM. Get it right and your assistant stays accurate and affordable; ignore it and you get truncated inputs, forgotten instructions, and surprising bills. Here's what a context window is, how big they've gotten, and how to use one well.

What a context window is

A context window is the maximum amount of text a model can consider in a single request, measured in tokens. The critical thing to understand is that everything shares that one budget:

• Your prompt — the instructions and the question.
• Any documents or data you paste or retrieve into the request.
• The conversation history — every earlier turn you send back for continuity.
• The model's response — the answer it generates counts against the same limit.

That last point trips people up. If a model has a 128K-token window and your input already fills 126K of it, there's only room left for a short reply. The window is a shared desk, not a bottomless inbox — prompt, docs, history, and answer all have to fit on it at once. When the total would exceed the limit, something gets cut, usually the oldest history or the tail of a long document.

Tokens vs words

Models don't count in words — they count in tokens. A token is a chunk of text, often a word-piece, and a useful rule of thumb is that one token is roughly ¾ of a word. Common words are a single token; longer or unusual words split into several.

So the working conversions are:

• 1,000 tokens ≈ 750 words ≈ about 1.5 pages of normal prose.
• 8K tokens ≈ 6,000 words — a long article or a short chapter.
• 32K tokens ≈ 24,000 words — a lengthy report.
• 128K tokens ≈ 96,000 words — a short novel.
• 200K tokens ≈ 150,000 words — a full-length book.
• 1M tokens ≈ 750,000 words — a small shelf of books.

Because tokenization varies by language and content — code, numbers, and non-English text often use more tokens per word — treat these as estimates, not guarantees. When it matters, count the tokens in your actual prompt rather than eyeballing the word count.

How big windows have gotten

Context windows have grown remarkably fast. Early production models worked with around 4K tokens — a few pages. That grew to 8K, then 32K, and the jump continued to 128K and 200K. By 2026, frontier models converged on windows around 1M tokens, and some push higher still.

That trajectory changed what's practical. A 4K window meant carefully summarizing everything before you sent it. A million-token window means you can hand a model an entire codebase, a stack of contracts, or a long support history and ask questions across all of it in one shot. The capability is real — but bigger windows come with tradeoffs worth understanding before you fill them.

Why bigger isn't always better

A large window is a ceiling, not a target. Two things push back against stuffing it full:

• Recall degrades in the middle. Models are reliably good at using information near the start and end of a long input, but accuracy for facts buried in the middle tends to drop — the well-documented "lost in the middle" effect. A fact technically inside the window isn't guaranteed to be used well.
• Every input token is billed. You pay for tokens going in, not just coming out. Filling a giant window on every request — especially by resending long histories turn after turn — gets expensive quickly. Before you design around a huge prompt, it's worth checking what it costs at your expected volume.

Bigger windows buy you headroom and flexibility. They don't buy you a free pass to be careless with what you put in them.

How to use a context window well

A few habits keep your workflows accurate and cheap:

• Leave comfortable headroom. Don't design prompts that ride the edge of the limit. Reserve clear room for the response, and keep a buffer so a slightly longer document doesn't silently truncate.
• Put the important things at the edges. Given the middle-recall dip, place your key instructions and the most relevant material near the beginning or end of the prompt, not buried in a wall of text.
• Trim history deliberately. Don't resend the entire conversation every turn out of habit. Summarize older turns or keep a rolling window of the most recent, relevant exchanges.
• Prefer retrieval over stuffing. Instead of pasting your whole corpus into every request, use retrieval (RAG) to pull in only the passages that matter for the current question. It's more accurate — the model sees signal, not noise — and far cheaper, because you're not paying to re-read everything each time.

That last point is the big one. The instinct with a large window is to dump everything in and let the model sort it out. The better pattern is almost always to send less, but the right less: retrieve the relevant chunks, keep the prompt focused, and let the window's size be insurance rather than a strategy.

The takeaway

A context window is simply the model's working memory for one request — shared across your prompt, your documents, the history, and the answer, and measured in tokens rather than words. Windows have grown from a few pages to hundreds of thousands of words. But the winning move isn't to fill every token; it's to keep prompts focused, put key information where the model reads it best, and lean on retrieval so you're paying for signal instead of noise.

On LLMGraph the retrieval, chunking, and prompt assembly are handled for you — you point a workflow at your content and it sends the model the right context, not all of it. That's the difference between a window that works for you and one you're constantly fighting.