Every multi-turn AI agent needs memory. The simplest implementation is obvious: load the last N turns of conversation before each call, then append the new turn after.
It works well enough to ship. But “last N turns” is a recency window—not a true memory system. This post explores where it falls short and what a more intelligent design looks like.
What Last N turns Pattern Does
On each agent invocation, the system loads recent conversation history:
recent_turns = memory_client.get_last_k_turns(
actor_id=actor_id,
session_id=session_id,
k=4,
)
context_messages = build_context_messages(recent_turns)
agent.messages = context_messages
This reconstructs the last few user/assistant exchanges and prepends them to the current prompt. After the response, the new turn is saved.
This is the classic sliding window approach—and it has predictable limitations.
The Problems with a Fixed Window
1. Recency ≠ Relevance
The most recent turns are not necessarily the most useful.
Example:
- “What’s the weather?”
- “Thanks”
- “Remind me what we discussed earlier”
- “Never mind”
None of these help answer:
“Summarize the risks we identified last week.”
Yet they are always included, while older—but relevant—context is dropped.
2. All Turns Are Treated Equally
A one-word reply (“ok”) occupies the same space as a detailed explanation.
The system ignores information density, wasting context on low-value content.
3. Token Usage Is Uncontrolled
Even with a fixed number of turns, token usage can vary widely.
- Short turns → small footprint
- Long turns → thousands of tokens
Without token-aware selection, context can become:
- Expensive
- Noisy
- Inefficient
4. Weak Cross-Session Memory
The same recent-window logic is applied regardless of time gaps.
- A message sent seconds ago = treated the same as one weeks ago
- Long-term continuity is effectively lost
5. No Persistent User Model
User preferences are not separated from conversation history.
If a user says:
“Always respond in bullet points”
That preference competes with normal messages and will eventually be forgotten.
A More Intelligent Design
The key shift:
Move from “what was said most recently” → “what is most useful right now”
This requires multiple memory layers with different roles.
Layer 1: Semantic Retrieval (Episodic Memory)
Instead of blindly loading recent turns, retrieve relevant ones based on meaning:
async def load_relevant_context(memory_client, query, token_budget=3000):
query_embedding = await embed(query)
candidates = memory_client.search_turns(
embedding=query_embedding,
limit=20,
)
selected = []
used_tokens = 0
for turn in candidates:
tokens = count_tokens(turn["text"])
if used_tokens + tokens > token_budget:
break
selected.append(turn)
used_tokens += tokens
return selected
This allows the system to pull in older but relevant context, regardless of when it occurred.
Layer 2: Working Memory (Short-Term Context)
You still need immediate continuity within a session.
Keep a small number of recent turns (e.g., last 2), then combine with semantic retrieval:
context = (
last_k_turns(k=2)
+ retrieve_relevant(query, remaining_token_budget)
)
This preserves conversational flow while adding deeper context only when needed.
Layer 3: User Profile (Persistent Memory)
Separate who the user is from what they said.
This layer is:
- Always loaded
- Never evicted
- Independent of conversation history
How It Updates
Explicit signals:
“Always use bullet points.”
Implicit signals:
- Repeated edits
- Frequent requests for shorter answers
Injecting the Profile
def build_system_prompt(base_prompt, profile):
if not profile:
return base_prompt
profile_block = f"""
## User Preferences
- Format: {profile.preferred_format}
- Verbosity: {profile.verbosity}
- Expertise: {profile.domain_expertise}
- Notes: {profile.disliked_patterns}
"""
return base_prompt + profile_block
This shapes responses without consuming conversation context.
Memory Architecture Summary
Each agent invocation:
├─ User Profile (always included)
│ Persistent preferences and traits
├─ Working Memory (last 2 turns)
│ Immediate conversational continuity
└─ Episodic Memory (semantic retrieval)
Relevant past interactions within token budget
Key Takeaway
A fixed window isn’t wrong—it’s a good starting point.
But over time, it breaks in predictable ways:
- Important context gets dropped
- Irrelevant context is included
- User preferences are forgotten
The core mistake is treating recency and relevance as the same thing.
A better system separates:
- Short-term memory (recent turns)
- Episodic memory (relevant past)
- User model (persistent preferences)
When these layers are handled independently, memory stops being a buffer—and starts becoming something closer to reasoning.
