You’re three hours into a research session with an AI assistant. You’ve fed it your product brief, customer interview notes, and a draft outline. You ask it to refine the intro based on everything you’ve discussed. It responds with something generic—like it never saw any of your earlier messages.
You didn’t do anything wrong. The assistant just ran out of memory.
Context windows have hard limits
Every AI model has a maximum context window—the total number of tokens (roughly words and punctuation) it can hold in a single conversation. For Claude, that’s 200,000 tokens on the Sonnet and Opus models. For GPT-4o, it’s 128,000. Once you exceed that limit, the model starts dropping the oldest messages to make room for new ones.
A token isn’t exactly a word. Short words like “it” or “to” are single tokens. Longer words like “conversation” might be two or three. Punctuation, formatting, and code add up faster than you expect. A 1,000-word document is usually around 1,300 tokens. A 10-message back-and-forth with code snippets and formatting can easily hit 5,000.
If you’re working in a long session—drafting email sequences, debugging workflows, refining a landing page—you can hit the limit without realizing it. The assistant doesn’t warn you. It just starts forgetting.
What gets dropped first
Most models use a sliding window. When the conversation exceeds the token limit, the system drops the oldest user and assistant messages first, but keeps the system prompt and the most recent exchanges. That means the assistant remembers what it’s supposed to do (your initial instructions), but not why you made certain decisions earlier in the session.
This is why an assistant might contradict itself after a long thread. It still knows it’s helping you write a welcome sequence, but it no longer remembers that you explicitly rejected a friendly tone in message twelve.
Some platforms handle this better than others. Claude‘s Projects feature lets you pin reference material—brand guidelines, product specs, style rules—so it doesn’t count against the per-conversation limit. That buys you more room for the actual back-and-forth. If you’re using the standard chat interface, though, everything counts.
How to structure prompts so you don’t lose context
The simplest fix: start a new conversation before you hit the limit. If you’re working on a complex project, break it into discrete sessions. Draft the outline in one chat. Write the first section in another. Edit in a third. Paste the relevant output from the previous session into the new one as reference material.
It’s more manual, but it forces you to compress and curate what the assistant actually needs to remember. Instead of a 50-message thread where half the exchanges are clarifications and dead ends, you give it a clean summary and move forward.
If you need to preserve context across a long session, front-load your instructions. Put your brand voice, audience details, and key constraints in the first message. If the model starts forgetting, it’ll drop the middle of the conversation first—your initial setup stays intact longer.
Another option: export the conversation transcript periodically and save it as a text file. If the assistant loses context, you can paste relevant sections back in as a reminder. It’s not elegant, but it works when you’re deep in a research or drafting session and don’t want to start over.
When to check token usage
Most AI platforms don’t surface token counts prominently. Claude’s interface shows approximate usage if you hover over certain UI elements, but it’s not always visible. The OpenAI Playground displays token counts directly; the ChatGPT web interface does not.
If you’re using the API, you can track tokens in the response metadata. For the web interfaces, assume that any conversation longer than 30 back-and-forth messages is approaching limits—especially if you’re pasting documents, code, or formatted text.
A rough heuristic: if you’ve scrolled more than three screen heights in a single conversation, consider starting fresh or summarizing.
The platform doesn’t owe you a warning. Token limits are documented, but the interface won’t stop you from continuing a conversation that’s functionally useless because the model can’t remember the beginning. It’ll keep responding. It just won’t be accurate.
If you’re running into this regularly, reply and tell us what you’re working on. We’re cataloging the specific workflows where context limits cause the most friction—and which tools handle it best.
Heads up — some links in this article are affiliate links. If you sign up through them, we may earn a small commission at no extra cost to you. We only recommend tools we use ourselves.