Beyond Last-K Turns: Building Memory That Actually Thinks

Every multi-turn AI agent needs memory. The simplest implementation is obvious: load the last N turns of conversation before each call, then append the new turn after.

It works well enough to ship. But “last N turns” is a recency window—not a true memory system. This post explores where it falls short and what a more intelligent design looks like.

---

What Last N turns Pattern Does

On each agent invocation, the system loads recent conversation history:

recent_turns = memory_client.get_last_k_turns(
    actor_id=actor_id,
    session_id=session_id,
    k=4,
)

context_messages = build_context_messages(recent_turns)
agent.messages = context_messages

This reconstructs the last few user/assistant exchanges and prepends them to the current prompt. After the response, the new turn is saved.

This is the classic sliding window approach—and it has predictable limitations.

---

The Problems with a Fixed Window

1. Recency ≠ Relevance

The most recent turns are not necessarily the most useful.

Example:

• “What’s the weather?”
• “Thanks”
• “Remind me what we discussed earlier”
• “Never mind”

None of these help answer:
“Summarize the risks we identified last week.”

Yet they are always included, while older—but relevant—context is dropped.

---

2. All Turns Are Treated Equally

A one-word reply (“ok”) occupies the same space as a detailed explanation.

The system ignores information density, wasting context on low-value content.

---

3. Token Usage Is Uncontrolled

Even with a fixed number of turns, token usage can vary widely.

• Short turns → small footprint
• Long turns → thousands of tokens

Without token-aware selection, context can become:

• Expensive
• Noisy
• Inefficient

---

4. Weak Cross-Session Memory

The same recent-window logic is applied regardless of time gaps.

• A message sent seconds ago = treated the same as one weeks ago
• Long-term continuity is effectively lost

---

5. No Persistent User Model

User preferences are not separated from conversation history.

If a user says:

“Always respond in bullet points”

That preference competes with normal messages and will eventually be forgotten.

---

A More Intelligent Design

The key shift:

Move from “what was said most recently” → “what is most useful right now”

This requires multiple memory layers with different roles.

---

Layer 1: Semantic Retrieval (Episodic Memory)

Instead of blindly loading recent turns, retrieve relevant ones based on meaning:

async def load_relevant_context(memory_client, query, token_budget=3000):
    query_embedding = await embed(query)

    candidates = memory_client.search_turns(
        embedding=query_embedding,
        limit=20,
    )

    selected = []
    used_tokens = 0

    for turn in candidates:
        tokens = count_tokens(turn["text"])
        if used_tokens + tokens > token_budget:
            break
        selected.append(turn)
        used_tokens += tokens

    return selected

This allows the system to pull in older but relevant context, regardless of when it occurred.

---

Layer 2: Working Memory (Short-Term Context)

You still need immediate continuity within a session.

Keep a small number of recent turns (e.g., last 2), then combine with semantic retrieval:

context = (
    last_k_turns(k=2)
    + retrieve_relevant(query, remaining_token_budget)
)

This preserves conversational flow while adding deeper context only when needed.

---

Layer 3: User Profile (Persistent Memory)

Separate who the user is from what they said.

This layer is:

• Always loaded
• Never evicted
• Independent of conversation history

How It Updates

Explicit signals:

“Always use bullet points.”

Implicit signals:

• Repeated edits
• Frequent requests for shorter answers

---

Injecting the Profile

def build_system_prompt(base_prompt, profile):
    if not profile:
        return base_prompt

    profile_block = f"""
## User Preferences
- Format: {profile.preferred_format}
- Verbosity: {profile.verbosity}
- Expertise: {profile.domain_expertise}
- Notes: {profile.disliked_patterns}
"""
    return base_prompt + profile_block

This shapes responses without consuming conversation context.

---

Memory Architecture Summary

Each agent invocation:

├─ User Profile (always included)
│   Persistent preferences and traits

├─ Working Memory (last 2 turns)
│   Immediate conversational continuity

└─ Episodic Memory (semantic retrieval)
    Relevant past interactions within token budget

---

Key Takeaway

A fixed window isn’t wrong—it’s a good starting point.

But over time, it breaks in predictable ways:

• Important context gets dropped
• Irrelevant context is included
• User preferences are forgotten

The core mistake is treating recency and relevance as the same thing.

A better system separates:

• Short-term memory (recent turns)
• Episodic memory (relevant past)
• User model (persistent preferences)

When these layers are handled independently, memory stops being a buffer—and starts becoming something closer to reasoning.