A chatbot with memory
Language model APIs have no memory: every request is independent, and the model only knows what you send in that request. A chatbot “remembers” because your code sends the earlier conversation back each time. This lesson builds that memory step by step — a message list, a token budget that keeps it inside the context window, summaries of older turns, persistence to disk, and long-term facts that survive across sessions.
The API is stateless
An API that keeps no information between requests. If the user says “my name is Asha” in one request and asks “what's my name?” in the next, the model cannot know unless the first message is included in the second request.
The earlier messages (or a summary of them) that your application stores and sends with each new request so the model can continue the conversation.
The simplest memory is a list of messages. Each turn appends the user's message, sends the whole list, and appends the assistant's reply.
import re
def fake_llm(messages: list[dict]) -> str:
"""A tiny stand-in model that can only 'remember' what is in the messages it receives."""
question = messages[-1]["content"].lower()
if "my name" in question and "?" in question:
for m in messages:
found = re.search(r"my name is (\w+)", m["content"], re.IGNORECASE)
if m["role"] == "user" and found:
return f"Your name is {found.group(1)}."
return "I don't know your name."
return "Nice to meet you!" if "my name is" in question else "Okay."
class Chatbot:
def __init__(self, system: str, llm=fake_llm):
self.messages = [{"role": "system", "content": system}]
self.llm = llm
def send(self, text: str) -> str:
self.messages.append({"role": "user", "content": text})
reply = self.llm(self.messages)
self.messages.append({"role": "assistant", "content": reply})
return reply
bot = Chatbot("You are a friendly assistant.")
print(bot.send("Hi, my name is Asha."))
print(bot.send("What's my name?"))
print(len(bot.messages), "messages in memory")
print(fake_llm([{"role": "user", "content": "What's my name?"}])) # no history sentThe last line is the same question without history: the “model” has no way to know. The same is true of a real LLM — memory is entirely your application's job.
With the real API, only the llm function changes:
def openai_llm(messages: list[dict]) -> str:
response = client.chat.completions.create(model="gpt-4.1-mini", messages=messages, temperature=0.3)
return response.choices[0].message.content
bot = Chatbot("You are a friendly assistant.", llm=openai_llm)Memory must fit the context window
The maximum number of tokens a model can process in one request — input plus output. Every message you send as memory uses part of it, and you pay for those tokens on every turn.
History grows with every turn, so long conversations become slow, expensive, and eventually too large to send. The fix is a token budget: always keep the system message, keep the most recent turns, and drop the oldest ones when the total exceeds the budget.
def count_tokens(text: str) -> int:
"""Rough estimate: about 4 characters per token for English. Use tiktoken for exact counts."""
return max(1, len(text) // 4)
def trim_history(messages: list[dict], budget: int) -> list[dict]:
system, rest = messages[0], messages[1:]
kept, used = [], count_tokens(system["content"])
for message in reversed(rest): # newest first
cost = count_tokens(message["content"])
if used + cost > budget:
break
kept.append(message)
used += cost
return [system] + list(reversed(kept))
history = [{"role": "system", "content": "You are a helpful assistant."}]
for i in range(1, 7):
history.append({"role": "user", "content": f"Question {i}: " + "details " * 10})
history.append({"role": "assistant", "content": f"Answer {i}: " + "explanation " * 10})
trimmed = trim_history(history, budget=120)
print(len(history), "->", len(trimmed), "messages")
print([m["content"].split(":")[0] for m in trimmed])Trim whole turns, not single messages. Keeping an assistant reply without the question it answered — or, in tool-calling apps, a tool result without the tool call that requested it — confuses the model and can make the API reject the request.
Summary memory
Replacing older turns with a short model-written summary, so the conversation keeps the important facts without keeping every message. Recent turns stay word for word.
def fake_summarise(messages: list[dict]) -> str:
facts = [m["content"] for m in messages if m["role"] == "user" and "my" in m["content"].lower()]
return "Earlier the user said: " + " ".join(facts)
def compress(messages: list[dict], keep_recent: int = 2, summarise=fake_summarise) -> list[dict]:
system, rest = messages[0], messages[1:]
if len(rest) <= keep_recent:
return messages
old, recent = rest[:-keep_recent], rest[-keep_recent:]
summary = {"role": "system", "content": "Conversation summary: " + summarise(old)}
return [system, summary] + recent
chat = [
{"role": "system", "content": "You are a travel assistant."},
{"role": "user", "content": "My budget is 2000 dollars."},
{"role": "assistant", "content": "Great, noted."},
{"role": "user", "content": "My dates are 10-17 March."},
{"role": "assistant", "content": "Got it."},
{"role": "user", "content": "Suggest a destination."},
{"role": "assistant", "content": "How about Lisbon?"},
]
for m in compress(chat):
print(f"{m['role']:>9}: {m['content']}")In production the summary is written by the model itself with a prompt like “Summarise this conversation, keeping names, numbers, decisions, and open questions.” Re-summarise when the summary plus recent turns grows past your budget.
Persisting conversations
A web server handles many users and may restart at any time, so conversation state belongs in storage keyed by a session id — not in a Python variable.
import json
from pathlib import Path
STORE = Path("sessions")
STORE.mkdir(exist_ok=True)
def load_session(session_id: str, system: str) -> list[dict]:
path = STORE / f"{session_id}.json"
if path.exists():
return json.loads(path.read_text(encoding="utf-8"))
return [{"role": "system", "content": system}]
def save_session(session_id: str, messages: list[dict]) -> None:
(STORE / f"{session_id}.json").write_text(json.dumps(messages, indent=2), encoding="utf-8")
messages = load_session("user-42", "You are a friendly assistant.")
messages.append({"role": "user", "content": "My name is Asha."})
save_session("user-42", messages)
print(len(load_session("user-42", "unused")), "messages restored")JSON files are fine for learning. Production apps use a database (Postgres, Redis, DynamoDB) with a time-to-live so old sessions expire.
Long-term memory
Facts about a user or task that persist across separate conversations — preferences, profile details, past decisions. They are stored outside the chat history and inserted into the prompt when relevant.
| Memory type | What it stores | How it is used | Example |
|---|---|---|---|
| Short-term (buffer) | Recent messages, verbatim | Sent every turn | The last 10 turns |
| Summary | Compressed older turns | Sent every turn | “User's budget is $2,000…” |
| Long-term facts | Extracted key facts | Inserted when relevant | “Prefers vegetarian restaurants.” |
| Semantic (retrieval) | Many past messages or documents, embedded | Top matches retrieved per question | Search all past chats for “visa” |
profile: dict[str, str] = {}
def remember(key: str, value: str) -> None:
profile[key] = value
def system_prompt() -> str:
facts = "\n".join(f"- {k}: {v}" for k, v in sorted(profile.items()))
return "You are a travel assistant.\n\nKnown facts about the user:\n" + (facts or "- none yet")
remember("diet", "vegetarian")
remember("home_city", "Pune")
print(system_prompt())Long-term memory is personal data. Tell users what is remembered, let them view and delete it, and never store secrets such as passwords or card numbers.
- Combine
trim_historyandcompress: summarise only when trimming would drop messages. - Replace
count_tokenswithtiktokenand compare the counts. - Let the model call a
remember(key, value)tool (next lessons) to save facts itself.
Key takeaways
- LLM APIs are stateless; memory means sending earlier messages with every request.
- Keep history inside a token budget: always keep the system message, drop the oldest whole turns.
- Summarise older turns to keep important facts at a fraction of the tokens.
- Store conversations by session id in a database, not in process memory.
- Long-term memory stores facts across sessions — treat it as personal data.