You can usually name the moment it happens. Forty minutes into a session that started well, the model reintroduces an approach you rejected at the beginning. A formatting rule from your first message quietly stops being applied. The prose still reads fine, which is what makes it dangerous.
The instinct is to blame the model for getting lazy. The real cause is a hard architectural limit called the context window, plus a second, quieter problem that shows up long before you reach that limit.
A context window is the maximum amount of text a language model can hold in view at once, measured in tokens. It covers everything the model reads and everything it writes in a single turn. Anything outside it does not exist as far as the model is concerned.
That definition explains one of two failure modes. Both matter, and the one almost nobody writes about is the one you hit first.
What a context window actually is
Tokens are the unit of measurement. In English, one token runs to roughly three-quarters of a word, so 1,000 tokens is about 750 words. A 200,000-token window holds somewhere near 150,000 words, or about 500 printed pages. A one-million-token window holds around 750,000 words.
Those numbers sound generous. They rarely feel that way in practice, for a reason most people never learn: models are stateless between turns.
Every time you press send, the system resends the entire visible conversation and the model reprocesses it from the beginning. Nothing persists on its own. What feels like continuity is a transcript being re-read, every single time. This is why “please remember what I told you” is not a request a model can act on. There is no store to remember from. There is a window, and the window is a budget.
What is actually inside the window
Most people picture the window holding their chat. It holds considerably more:
• The system prompt, meaning the product instructions you never see
• Tool definitions, and the raw output of every tool the assistant has called
• Uploaded files, attachments, and any retrieved documents
• The complete conversation history so far
• Your current message
• Reserved room for the reply itself
In coding and agent tools, the middle items dominate. A single file read or one web fetch can consume more of the budget than an hour of typing. That is why a session can feel crowded when you have barely written anything.
Input and output share the same budget
The window is not divided between what you send and what comes back. They compete for the same space. Send 190,000 tokens into a 200,000-token window and roughly 10,000 remain for the answer. This is the mechanism behind replies that stop mid-thought or arrive suspiciously thin after a large paste.
Anthropic's developer documentation exposes this accounting directly. During tool use, the API passes the model a running note in the form “Token usage: 35000/200000; 165000 remaining”. The model gets a fuel gauge. You usually do not.
The two ways long sessions break
Here is the distinction most explanations skip, and the one that changes how you work.
Failure one: overflow, the hard wall
Overflow is what people mean by “hitting the context limit”. The conversation grows past the window, and the oldest material is cut, dropped, or replaced with a summary. The break is clean and total. The model does not have partial access to a truncated message. It has none.
Overflow is easy to reason about. It is also not usually what is degrading your session.
Failure two: context rot, the slow slide
Long before the window fills, output quality starts sliding. The term for this is context rot, formalised by research from Chroma that tested 18 frontier models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 variants. The finding was uniform. Every model tested grew less reliable as input length increased, even on simple retrieval tasks.
The important detail is that rot arrives well before capacity does. A model with a 200,000-token window can show meaningful degradation at 50,000. The decline is gradual rather than a cliff edge, which is precisely why it goes unnoticed.
Telling the two apart
Overflow removes information. Rot degrades the use of information that is still sitting right there.
If the model has no idea what you said at message three, that is overflow. If it can quote message three back to you and still keeps breaking the rule inside it, that is rot.
Why bigger context windows do not fix it
The reasonable next thought is to switch to a model with a larger window. The benchmark data argues otherwise.
Advertised size against effective size
NVIDIA's RULER benchmark tested 17 long-context models across 13 tasks at input lengths from 4,000 to 128,000 tokens. Instead of testing simple retrieval, RULER added multi-hop tracing and aggregation work. The results separate the number on the box from the number that holds up.
Almost every model scored near-perfect on the plain needle-in-a-haystack test at its claimed length, then came apart on harder tasks at those same lengths. RULER's authors set a quality threshold and reported the longest input each model could handle while still clearing it. They called that the effective context length.
One entry deserves attention. A Llama 3 variant marketed with a one-million-token window cleared the threshold only to 16,000 tokens, a gap of roughly 98% between the advertised figure and the working one.
The pattern repeats elsewhere. Independent analysis comparing RULER against Fiction.liveBench and Databricks' long-context retrieval tests converges on a usable band of roughly 16,000 to 64,000 tokens for most models, far below advertised capacity.
None of this makes large windows useless. Epoch AI's tracking shows the longest context windows growing by roughly 30 times per year since mid-2023, with effective use improving even faster on the benchmarks they measure. Gemini 1.5 Pro cleared RULER's threshold beyond 128,000 tokens where most rivals failed at 32,000. Capacity and competence are both climbing. They are simply different numbers, and only one of them appears in the marketing.
The lost-in-the-middle problem
Position inside the window matters as much as total length. Nelson Liu and colleagues at Stanford documented a U-shaped accuracy curve: models perform best when the relevant information sits at the very start or the very end of the input, and worst when it sits somewhere in the middle.
The size of the sag is striking. GPT-3.5-Turbo's multi-document question answering dropped by more than 20% depending purely on where the answer document was placed. At its worst position the model scored below its own closed-book result of 56.1%, meaning it did better answering from memory with no documents than with the correct document buried in the middle.
The practical consequence lands directly on how you write prompts. Your carefully composed opening instructions occupy one of the two strong positions, right up until the conversation grows around them. Then they slide into the weak middle and stay there.
Why the architecture makes this inevitable
Transformers work by letting every token attend to every other token. For n tokens that produces n² pairwise relationships, so the relational load grows quadratically while the model's ability to weight those relationships stays fixed.
Anthropic frames the consequence as an attention budget. Context behaves as a finite resource with diminishing marginal returns, and every token you add spends part of that budget. Degradation is gentler in some models than others, but it shows up in all of them. Anthropic's engineering write-up on effective context engineering sets out the reasoning in full.
This is the part worth internalising. Context rot is not a defect waiting on a patch. It falls out of how the architecture works.
What your AI app does when the window fills
Between the research and your daily experience sits a product layer, and it behaves differently from tool to tool.
Three mechanisms with three different symptoms
• Truncation drops the oldest content outright. Cheap to implement, and the information is simply gone.
• Compaction summarises the conversation so far and restarts from that summary. Continuity survives, specificity does not.
• Persistent memory writes selected facts outside the window so they survive across sessions. This is a separate system built on top of the model rather than a property of it.
Compaction causes the most confusion, because it feels like the model changed personality mid-conversation. Anthropic's support documentation states plainly that Claude summarises earlier messages as a conversation approaches the limit, and that Claude Sonnet 5 compacts automatically at 500,000 tokens. In Claude Code the summariser is tuned to hold onto decisions and unresolved bugs while discarding redundant tool output.
Nothing tells you which details were judged redundant. That is the whole problem in one sentence.
The visibility problem
Ask most people using ChatGPT or Claude in a browser how full their context window is and they cannot answer, because the interface never says. The chat scrolls on forever, which implies the window does too.
Developer tools have started closing the gap. Claude Code exposes a /context command that reports current usage, and /compact for compressing on demand. Recent VS Code builds added a context window indicator for Copilot sessions. Consumer chat surfaces have largely not followed.
Until they do, you are flying without an instrument. Which makes the next part the practical core of this article.
Diagnosing what actually went wrong
Six symptoms cover almost every long-session failure. Match what you are seeing to the cause before you change anything.
| What you observe | Likely cause | What to do about it |
|---|---|---|
| A constraint from early on is gone, and the model cannot recall it at all | Overflow or compaction | Restate the constraint in your newest message rather than pointing back |
| The model can quote the constraint but keeps ignoring it | Context rot or position bias | Move it to the end and make it the entire message, with nothing else competing |
| An approach you already rejected comes back | The rejection was dropped during compaction | Keep a running “already ruled out” list and repaste it when you notice drift |
| An attached file is ignored or misquoted | The file was pushed out, or was never re-read after compaction | Re-attach it, or better, paste only the excerpt that matters |
| Answers get shorter and shallower with no change in the questions | Input has squeezed the space reserved for output | Trim pasted material, or start a fresh session |
| Tone or formatting drifts without you noticing when | Style rules are buried in the weak middle of the window | Re-anchor format rules in the same message as the request |
The thirty-second test
Before rewriting anything, ask the model to list every constraint it is currently working under. What comes back tells you which failure mode you are in. A short or wrong list means overflow. A complete list followed by continued rule-breaking means rot. Either way you now know whether to restate or restart, and you spent half a minute finding out.
How to keep long sessions working
The fixes are habits rather than settings. They cost seconds each and they compound.
Re-anchor instead of repeating
The most common mistake is scrolling up and writing “as I said earlier, keep it under 400 words”. That sentence points at a weak position in the window and asks the model to go fetch. Restating the constraint inside your current message puts it in the strongest position available, at the recency end of the U-curve.
The weak version asks the model to remember. The strong version simply hands it the rule again, in the same breath as the request. It reads redundant. It works.
Write a handoff brief before you need one
When a session nears the end of its useful life, the way to keep the work is a short structured brief you paste into a fresh thread:
• Objective: what we are producing, in one line
• Locked decisions: what has been settled and is not up for revisiting
• Constraints: the rules that must hold
• Ruled out: approaches already rejected, with the reason attached
• Open question: the single thing being worked on right now
Two minutes of writing beats forty minutes arguing with a model that has lost the plot. Keep these briefs in a text file and they stop being disposable, becoming reusable project state you carry between tools.
For builders: the same idea at system scale
Anthropic's guidance for agents reduces to finding the smallest set of high-signal tokens that produce the outcome you want. In practice that means clearing raw tool results once they have been processed, keeping long-lived state in files rather than in message history, retrieving just in time instead of preloading everything, and handing focused sub-tasks to sub-agents that start with clean windows.
Capacity is the wrong metric to optimise. Signal-to-noise ratio inside the window determines output quality, and that ratio is something you control.
What to expect as windows keep growing
The frontier is moving in the right direction. Context windows have been growing at roughly 30 times per year, and by Epoch AI's measure the input length at which leading models still reach 80% accuracy has been improving faster than raw capacity. Gemini 1.5 Pro's RULER result showed that graceful long-context behaviour is achievable, and current flagship models hold comprehension considerably further out than their predecessors managed.
What will not change is the shape of the problem. Quadratic scaling does not stop being quadratic. As long as attention is finite and every added token spends part of it, the ratio of useful signal to accumulated noise will govern output quality more than the size of the container does.
Which points at the thing most people have backwards. The skill worth building is not learning to fill a bigger window. It is learning to recognise when a session has stopped being worth continuing, and to leave while the work is still good.
Comments
Join the discussion and share your perspective.