How Chatbot Memory Works: A Beginner’s Playbook on Long Conversations
Learn how chatbot memory works in plain words: context windows, trimming, summaries and saved notes, plus five simple plays to build a bot that remembers.

Meera, a developer in Pune, built a small trip-planning chatbot for her cousin’s travel agency. On day one it was charming. A customer said “I am vegetarian and I hate early flights,” and the bot planned a lovely Goa weekend around that.
Forty messages later, the same bot suggested a seafood dinner and a 5:40 am flight. Had the bot “forgotten”? To fix it, she had to learn how chatbot memory works: what the AI really remembers, what it never remembered at all, and how apps fake a good memory with a few simple tricks.
This chapter follows Meera as she fixes her bot in five plays.
Last verified: 16 November 2026. AI tools change fast; check the linked sources for anything newer.

Who this is for: beginners who use or build chatbots and want to know why they forget, and how to make them remember the right things. Time needed: about 30-40 minutes. Helpful but not required: our chapter on how LLMs work (tokens and the context window).
How chatbot memory works, in simple words
Here is the surprising part about how chatbot memory works: the AI model itself has no memory between messages. Each time you send a message, the app sends the model the whole conversation again, from the top. The model reads it all, writes a reply, and then forgets everything.
OpenAI’s own docs say it plainly: each request is “independent and stateless”. Stateless means “keeps nothing from last time”.
The analogy: think of a very smart waiter with no memory at all. To serve you, the restaurant hands him a notepad with every order you have made tonight. He reads the whole notepad before each visit to your table. The notepad is the chat history. The waiter is the model.
The notepad has a fixed number of pages. That limit is the context window: the maximum amount of text (measured in tokens, small chunks of words) the model can read in one go, including its own reply. Anthropic’s docs call it the model’s “working memory”.
Why bots seem to forget
- The notepad got cut. When a chat grows too long, many apps quietly drop the oldest messages to make room. If “I am vegetarian” was message 2, it is gone.
- The notepad got too long to read well. Even with a huge window, Anthropic notes that accuracy and recall get worse as the token count grows. They call this “context rot”. Important details get lost in the crowd.
- It was a new chat. A fresh conversation starts with an empty notepad, unless the app saved notes somewhere else.
Two kinds of memory
Short-term memory is the current conversation inside the context window. Long-term memory is a separate notebook the app keeps outside the model, such as “prefers window seats” or “vegetarian”. The app looks up the relevant notes and slips them into the next request. The model never stores anything itself; the app does.
What’s new (November 2026)
Memory is a hot topic this month, with chat apps competing on how well they remember you. For builders, the useful change is that the big AI APIs now include tools for long chats. Anthropic’s docs list context windows of up to 1 million tokens on current Claude models, and server-side compaction (automatic summarising of older turns) in beta.
OpenAI’s docs describe a Conversations API that stores a chat for you, plus compaction options for its Responses API. A bigger window and built-in summaries change the tools, not how chatbot memory works, so the plays below still apply: you decide what is worth remembering.
The playbook at a glance
- Measure what you send on every turn.
- Keep a sliding window of recent messages.
- Summarise the older part of the chat.
- Save key facts as long-term notes.
- Give users control and protect privacy.
Play 1: Measure what you send on every turn
Goal: see how fast your notepad fills up, before you fix anything. This is how chatbot memory works in practice: a list that grows every turn.
- Log the token count of every request. Most APIs return it in a “usage” field with each reply.
- Note your model’s context window size from its docs.
- Have one long test chat (40-60 messages) and plot the token count per turn.
Meera found her requests grew by roughly 300 tokens per turn (illustrative). She also noticed her own code dropped old messages after 30 turns. That was the vegetarian note vanishing.
You are done when: you know how many turns fit, and what your app does when the limit is near.
Play 2: Keep a sliding window of recent messages
Goal: keep requests small and fast without losing the rules.
- Always keep the system prompt (your bot’s instructions) at the top. Never trim it.
- Keep only the last N messages (for example the last 20) after it.
- Trim whole turns, never half a message, and keep a tool call together with its result.
Think of it as a train that only carries its newest coaches. It is cheap and simple, but anything in an old coach is gone. That is why Play 3 exists.
You are done when: long chats never hit the limit, and the bot’s rules still apply at message 100.
Play 3: Summarise the older part of the chat
Goal: keep the meaning of old messages without keeping every word.
- When the chat passes a size you choose, ask the model to summarise the older messages in a few lines.
- Tell it what to keep: names, preferences, decisions, open questions.
- Send the summary plus the recent window on the next turn. Or use your provider’s built-in compaction if it has one.
Here is the idea as a short illustration. Real APIs differ, so treat it as pseudo-code:
MAX_RECENT = 20
def build_request(system_prompt, summary, history, notes):
recent = history[-MAX_RECENT:]
older = history[:-MAX_RECENT]
if older:
summary = summarise(summary, older) # one extra model call
context = system_prompt + "\nKnown facts: " + notes
context += "\nEarlier in this chat: " + summary
return context, recentIn plain steps: keep the newest 20 messages, fold anything older into a running summary, add saved facts, and send that bundle. The model sees a short, tidy notepad instead of a giant one.
Meera’s summary now starts with “Customer: vegetarian, no flights before 9 am, budget trip to Goa.” The seafood suggestions stopped.
You are done when: a fact from message 2 still shapes the answer at message 60.
Play 4: Save key facts as long-term notes
Goal: remember the important things across separate chats, not just inside one.
- After each chat, ask the model to pull out lasting facts as short sentences (“Prefers window seats”).
- Store them per user in a normal database. Update a fact when it changes; do not just pile up copies.
- At the start of a new chat, add the relevant notes to the system prompt. With many notes, find the relevant ones using embeddings and vector search.
This is like the cupboard and notebook from the RAG chapter. The notebook holds facts about the person; RAG holds facts from your documents. Both work the same way: look up, then paste in.
A returning customer told Meera’s bot “Plan Kerala this time.” It already knew: vegetarian, late flights. The customer did not have to repeat a thing.
You are done when: a brand new chat uses the right saved facts, and an outdated fact gets replaced, not duplicated.
Play 5: Give users control and protect privacy
Goal: a bot that remembers helpfully, not creepily.
- Tell users what the bot remembers, and let them see, edit and delete their notes.
- Never save passwords, card numbers, health details or ID numbers as memory. Filter them out before storing.
- Offer a “don’t remember this chat” option, and set how long old chats are kept.
- Treat saved notes as untrusted text. A sneaky message could plant a fake “fact”; our prompt injection chapter explains why.
Meera added a small “What I remember about you” page. One customer deleted an old address. Trust went up, not down.
You are done when: any user can view and wipe their memory in two clicks, and no sensitive data is stored.
Which memory approach do you need?
| Approach | Good at | Trade-off | Choose when |
|---|---|---|---|
| Send full history | Simplest, nothing lost | Cost and slowness grow every turn | Short chats |
| Sliding window | Cheap and predictable | Old details vanish | Casual chat, quick help |
| Summary or compaction | Keeps meaning of long chats | Summaries can drop details | Long sessions, agents |
| Long-term notes | Remembers across chats | Privacy work, stale facts | Returning users |
Once you see how chatbot memory works, the pattern is clear: most real apps combine the last three. Note that resending history is not free: OpenAI’s docs say that when you chain responses, all earlier input tokens are still billed. Our LLM cost playbook covers ways to keep that bill down.
Mistakes to avoid
- Trimming the system prompt. If the rules fall off the notepad, the bot changes personality. Fix: pin it at the top.
- Assuming a bigger window solves it. Long contexts cost more and recall gets worse. Fix: curate what goes in.
- Vague summaries. “The user discussed travel” is useless. Fix: tell the summariser exactly which facts to keep.
- Saving everything forever. Old facts conflict with new ones and raise privacy risk. Fix: update, expire and let users delete.
How to know it is working
- The message-2 test: state a preference early, chat 50 more turns, then ask a question that depends on it.
- The new-chat test: open a fresh chat and check the bot uses saved facts correctly.
- The change test: change a preference and confirm the old one is replaced.
- The cost check: tokens per turn stay flat instead of climbing forever.
These four tests show you how chatbot memory works in your own app. Turn them into a small test set, as in our AI evals chapter, and rerun it after every change.
Your checklist
- ☐ I log tokens per request and know my context window size.
- ☐ The system prompt is pinned and never trimmed.
- ☐ Old turns are summarised with clear “what to keep” rules.
- ☐ Lasting facts are saved per user and updated, not duplicated.
- ☐ Users can see and delete what the bot remembers.
- ☐ Sensitive data is never stored as memory.
Frequently asked questions
Does a chatbot learn from my conversations?
Not during the chat. Remember how chatbot memory works: the model’s knowledge stays fixed; “memory” is the app resending history or pasting in saved notes. Whether a company later uses chats for training is a separate policy question, so check its settings.
Why does ChatGPT or another chatbot forget things in a long chat?
Usually because the conversation outgrew the context window and older parts were trimmed or summarised, or because very long contexts make details easier to miss. Restating key facts, or starting a new chat with a short recap, often helps.
Is the context window the same as memory?
It is short-term memory only: what the model can read right now. Long-term memory lives outside the model, in a database the app controls.
Do I need a vector database for chatbot memory?
Not at first. A plain list of short facts per user works well for small apps. Add vector search when you have many notes and need to find the relevant few.
Is chatbot memory safe for private data?
Only if you design it to be. Filter sensitive details, let users delete notes, and keep data no longer than needed.
Meera’s bot remembers now
Two weeks later, the same customer came back for a second trip. The bot greeted them with vegetarian food stops and a relaxed 11 am flight. Meera had not made the model smarter. She had just given the forgetful waiter a better notepad and a tidy notebook. That, in one line, is how chatbot memory works.
Your next step: run the message-2 test on your own bot today. Then read long-running AI agents to see the same memory ideas applied to agents that work for hours.
Sources: Anthropic, Context windows (working memory, context rot, 1M-token windows, compaction); OpenAI, Conversation state (stateless requests, Conversations API, billing of chained input tokens, compaction). Story and numbers about Meera are illustrative.



1 comment