🤖Organizing a Project for Coding Agents

Instructions Explain, Scripts Execute, Tests Prove

Coding agents can read a lot of code.

That usually isn't the hard part.

The harder part is everything an experienced engineer already knows but the repository doesn't clearly say. Organizing a project for coding agents is really about putting that knowledge in the right place.

A simple principle helps:

Instructions explain. Skills guide. Scripts execute. Tests prove.

---

🏗️ An Application Is Often Many Repositories

A modern application is rarely one repository.

It may be split across several:

app                → application code and build pipeline
database           → schema, migrations, seed data
app-infra          → cloud resources, IAM, DNS, search
cluster-infra      → Kubernetes clusters and shared services
deploy-manifests   → environment-specific deployment configuration

An agent looking at only one repository cannot infer that entire relationship.

That is the first thing we should make explicit:

What repositories make up the application, and what does each one own?

That gives the agent a map before it starts navigating.

---

🧠 Instructions Explain What the Code Cannot

Instruction files such as AGENTS.md or CLAUDE.md should contain information the agent cannot reliably discover from source code alone.

For example:

• Repository relationships

• Architecture boundaries

• How environments are organized

• How tests should be run

• Important operational constraints

• Critical guardrails

The goal is not to document the entire repository.

The agent can already read the repository.

A good rule is:

If the agent can reliably discover something from the code, don't repeat it in the instructions.

Keep instructions short and focused.

Long instruction files consume context and make the important rules harder to notice.

---

🧩 Skills Guide Repeatable Procedures

Some engineering tasks follow the same process again and again.

For example:

• Check deployment status

• Investigate an incident

• Compare environments

• Validate a release

• Inspect a scheduled job

Instead of expecting the agent to rediscover the procedure every time, package it as a skill.

Conceptually:

User Question
      ↓
Relevant Skill
      ↓
Known Procedure
      ↓
Tools / Scripts
      ↓
Result

A skill should tell the agent:

• When to use it

• What procedure to follow

• Which tools or scripts to run

• How to interpret the result

The advantage is that specialized instructions only need to load when they are relevant.

The main agent instructions stay small.

---

⚙️ Scripts Execute Deterministic Work

Suppose the procedure is:

Get deployments
Check pods
Check scheduled jobs
Compare versions
Identify failures

An agent could construct those commands every time.

But why make it reason through something deterministic?

Instead:

Agent
   ↓
Run Script
   ↓
Collect State
   ↓
Structured Result
   ↓
Agent Interprets Result

The script handles the repeatable mechanics.

The agent handles judgment.

This distinction is important:

Every step the agent doesn't have to reason through is one less step it can get wrong.

Use the model where reasoning adds value.

Use scripts where consistency matters more.

---

🛡️ Move Critical Guardrails Below the Prompt

Instructions are useful, but they are still guidance.

Consider a rule like:

Production cluster access is read-only.

You can put that in AGENTS.md.

But if the rule must always hold, don't rely on the model remembering it.

Move the constraint into deterministic controls:

Instruction
    ↓
Hook
    ↓
Permissions

For example:

• The instruction says access is read-only

• A hook blocks unsafe commands

• Cluster permissions prevent writes entirely

The deeper the control is enforced, the less the agent has to remember.

An instruction is a request. A policy or permission is a guarantee.

---

✅ Tests Prove the Work

A coding agent shouldn't stop because a change looks correct.

It should have a way to verify the result.

Without validation:

Change Code
    ↓
Looks Correct
    ↓
Stop

With validation:

Change Code
    ↓
Run Test
    ↓
Failure?
    ↓
Fix
    ↓
Run Again

Tests give the agent evidence.

That changes the role of the human.

Instead of being the verification loop, the human becomes the reviewer of a result that has already been checked.

The same idea applies to agent tooling itself.

Skills, scripts, hooks, and configuration should also be validated like production code.

---

🔄 Let Skills Improve Through Use

Skills shouldn't be treated as finished artifacts.

The most useful improvements often come from real usage.

You may discover that:

• The output is too verbose

• A status is ambiguous

• One workflow takes too long

• A special case needs a shortcut

• An important field is missing

Capture those learnings.

Then update the skill or script.

Use
 ↓
Observe
 ↓
Learn
 ↓
Improve
 ↓
Next Run Gets Better

Over time, operational knowledge moves out of people's heads and into reusable automation.

---

📚 Documentation Is Still for Humans

Agents need instructions, skills, scripts, and tests.

Humans still need architecture documentation.

A short README or architecture document should explain:

• What the major components are

• How they connect

• Where they run

• Which repositories own which responsibilities

The agent can be pointed to that documentation when needed.

There is no reason to copy all of it into the permanent instruction context.

---

🧭 Where Things Belong

A simple mental model:

Need	Put It Here
How do repositories relate?	Instructions
What should the agent know?	Instructions
What repeatable process should it follow?	Skill
What can execute deterministically?	Script / Tool
What must always be enforced?	Hook / Permission
How do we know it worked?	Test
How does the procedure improve?	Learnings
What helps humans understand the system?	Documentation
What is the implementation truth?	Source code

The general direction should be:

Move work toward the most deterministic layer possible.

If guidance can become a script, make it a script.

If a rule can become a permission, enforce it.

If an outcome can be tested, test it.

---

🎯 Final Thought

Making coding agents more effective isn't only about better models or larger context windows.

It's also about designing the engineering environment so the agent has less to infer, less to remember, and less to invent.

A well-organized project gives the agent:

• A map of the repositories

• Clear instructions

• Reusable skills

• Deterministic scripts

• Enforced guardrails

• Tests that prove the result

Instructions explain. Skills guide. Scripts execute. Tests prove.

The goal isn't to make the agent remember everything.

It's to organize the project so it knows where to look, what to run, what it is allowed to do, and how to prove the work is complete.

🧠 Context Offloading Isn't Always a Cost Optimization

 Many modern agent frameworks support automatic context offloading.

The idea is simple.

Instead of keeping large tool responses inside the model's context, the framework stores them externally, leaves behind a small preview, and retrieves the original only if the model needs it later.

On paper, it sounds like an obvious optimization.

Smaller prompts should mean lower token usage and lower cost.

I measured it on one of my production agents.

The result surprised me.

Token usage increased by 68%.

The interesting part wasn't the number.

It was why.

---

🤔 How Context Offloading Works

Imagine an agent receives a large tool response.

Without offloading:

Tool
   ↓
Large Result
   ↓
LLM

The entire payload remains in the model's context.

With offloading:

Tool
    ↓
Large Result
    ↓
Store externally
    ↓
Preview in context
    ↓
Retrieve only if needed

The expectation is simple:

• Smaller prompts
• Lower token usage
• Longer conversations

---

📊 What I Measured

I evaluated multiple context-management strategies against the same workload.

The baseline simply used a sliding context window.

Automatic offloading enabled the framework's built-in context manager.

Strategy	Model Calls	Input Tokens	Quality
Baseline	7	227K	9/9
Automatic Offloading	14	382K	8/9

Instead of reducing cost, offloading:

• increased model calls
• increased total token usage
• slightly reduced answer quality

---

🧩 The Hidden Cost

The reason became obvious after looking at the traces.

The framework successfully reduced the size of individual prompts.

But every time the model needed information that had been offloaded, it performed another retrieval.

That meant another model invocation.

The flow looked like this:

Tool
    ↓
Large Result
    ↓
Offload
    ↓
Preview
    ↓
Model requests original
    ↓
Retrieve
    ↓
Another model call

The content wasn't removed.

It simply came back later.

You paid for:

• the preview
• the retrieval
• another model call
• the conversation history again

The content was effectively processed twice.

---

📉 Compression Worked

One result initially confused me.

The average prompt actually became smaller.

Per-call context was reduced by roughly 16%.

So why did total cost increase?

Because billing depends on total tokens processed, not the size of an individual prompt.

Reducing the size of each request didn't help when the framework created many more requests.

In my case:

Smaller prompts
        +
More model calls
        =
More total tokens

The compression worked.

The retrieval round trips outweighed the savings.

---

📏 Context Window Matters

The most important number turned out not to be token usage.

It was context utilization.

My model supports a 1 million token context window.

The baseline execution peaked at approximately 55,000 tokens.

That's roughly 5.5% of the available context.

I had enabled a feature designed to prevent context overflow.

The problem was...

I was nowhere near overflowing.

I was paying the cost of protection against a problem that didn't exist.

---

✅ When Context Offloading Makes Sense

Context offloading isn't a bad feature.

It's solving a different problem.

It works well when:

• Long-running conversations
• Small context windows
• Very large tool responses
• Information that is unlikely to be referenced again

Those are situations where preventing context growth is more important than minimizing model calls.

---

❌ When It Doesn't

Be cautious when:

• The conversation easily fits inside the context window
• Tool responses are immediately reused
• Every reasoning step depends on previous tool output

In these cases, offloading often creates additional retrievals without providing meaningful savings.

---

🎯 Final Thought

Context offloading isn't primarily a cost optimization.

It's a context management strategy.

If your agent is approaching the context window, it can prevent important information from being discarded.

If your agent is nowhere near that limit, it may simply replace larger prompts with more model calls.

The lesson isn't:

Always enable context offloading.

It's:

Measure first. Optimize only for the bottleneck you actually have.

🔥 Hot, Warm, and Cold: Reducing Vector Search Costs with Tiered Storage

In previous articles, we looked at retrieval strategies and knowledge base architectures.

One assumption quietly made throughout was that every embedding lives in the same vector index.

That works well for small datasets.

At hundreds of millions of vectors, it becomes expensive.

The reason isn't the embeddings.

It's the index.

---

💰 The Real Cost of Vector Search

Unlike relational databases, vector search doesn't scan every embedding.

Instead, it traverses an Approximate Nearest Neighbor (ANN) index to quickly find similar vectors.

To achieve low-latency search, that ANN index is typically kept in memory.

As the corpus grows, so does the amount of memory—and infrastructure—needed to keep that index available.

The important observation is this:

Not every vector is queried equally.

---

📈 Most Corpora Follow a Long Tail

Think about any corpus with a time dimension:

• Support tickets

• Meeting notes

• Product documentation

• Incident reports

• Contracts

Recent information is queried constantly.

Older information is still valuable—but much less frequently.

A typical access pattern looks like this:

10% of documents
        ↓
90% of queries

90% of documents
        ↓
10% of queries

Yet many systems keep 100% of the vectors in the fastest—and most expensive—infrastructure.

---

🔥 Hot, 🌤️ Warm, ❄️ Cold

The idea is the same one object storage has used for years.

Not every document needs the same retrieval speed.

🔥 Hot Tier

Frequently accessed vectors.

Examples:

• OpenSearch k-NN

• pgvector (HNSW)

• Pinecone

• Weaviate

Characteristics:

• ANN index kept in memory

• Lowest latency

• Highest infrastructure cost

Typical data:

• Recent documents

• Frequently queried content

---

🌤️ Warm Tier

Occasionally accessed vectors.

The vectors live in low-cost object storage but are loaded into memory on demand and cached for subsequent searches.

Examples:

• Cached JSONL or Parquet embeddings in S3

• Amazon S3 Vectors

• Other managed vector services

Typical flow:

First Query

S3
   ↓
Load vectors
   ↓
Cache in memory
   ↓
Similarity search

Subsequent Queries

Memory
   ↓
Similarity search

Compared to the hot tier:

• No always-running ANN index

• Lower infrastructure cost

• Slightly higher latency

• Good for infrequently accessed data

---

❄️ Cold Tier

Rarely accessed vectors.

Vectors remain in object storage and are only read when a query requires them.

Typical flow:

User Query
      ↓
Read vectors from object storage
      ↓
Exact similarity search
      ↓
Return Top-K

Characteristics:

• No ANN index

• No persistent memory

• Lowest storage cost

• Highest latency

Cold storage is ideal for historical archives where occasional slower queries are acceptable.

---

🏗️ Hide the Complexity

The biggest mistake is exposing storage tiers to application code.

Instead of writing:

if document_is_hot:
    search_hot()
elif document_is_warm:
    search_warm()
else:
    search_archive()

Applications should simply ask:

Search documents

The retrieval layer decides which storage tier to query.

This keeps applications independent of storage decisions while allowing the storage architecture to evolve over time.

---

📉 Where the Savings Come From

Many people assume tiering reduces storage cost.

That's only part of the story.

The real savings come from reducing:

• Memory-resident ANN indexes

• Compute

• Always-on infrastructure

The fewer vectors you keep in expensive memory, the smaller your always-running infrastructure becomes.

Tiering isn't about storing vectors more cheaply.

It's about keeping only the right vectors in expensive infrastructure.

---

🤖 Intelligent Routing

As storage tiers grow, deciding where to search becomes another retrieval problem.

Simple rules such as document age often work.

Increasingly, AI agents can route queries based on user intent.

For example:

• "Show the latest incident." → Hot

• "Summarize the original agreement." → Cold

• "How has this evolved over time?" → Multiple tiers

Applications continue to issue a single retrieval request.

The retrieval layer decides which storage tier—or combination of tiers—to search.

---

📋 When Tiering Makes Sense

Tiering is valuable when:

• Very large vector collections

• Strongly skewed access patterns

• Historical data is queried infrequently

• Different latency requirements are acceptable

It adds little value when:

• Datasets are relatively small

• Access patterns are uniform

• Every query requires consistently low latency

As with most optimizations, measure your access patterns before introducing additional architectural complexity.

---

🎯 Final Thought

The biggest cost in vector search isn't storing embeddings.

It's maintaining large, memory-resident ANN indexes.

Just as object storage evolved into Standard, Infrequent Access, and Archive tiers, vector storage is beginning to follow a similar path.

The goal isn't to store vectors more cheaply.

It's to keep only the vectors that require low-latency access in your fastest—and most expensive—infrastructure, while allowing the long tail to move to lower-cost storage without changing how applications retrieve information.

Deep Dive into a Strands Agent Trace

When working with AI agents, one of the most useful ways to understand what is happening behind the scenes is through tracing.

A seemingly simple user request can result in multiple model invocations, tool executions, and event-loop cycles before the agent produces its final response.

In this post, I'll walk through a simple Strands Agent example:

greet rahul using tool

---

🧠 What Actually Happens?

The resulting trace looks roughly like this:

agent.run
|
+-- invoke_agent
    |
    +-- execute_event_loop_cycle #1
    |   |
    |   +-- chat #1
    |   |
    |   +-- execute_tool greet
    |
    +-- execute_event_loop_cycle #2
        |
        +-- chat #2

Although the user made only one request, the agent performed:

• Two event-loop cycles

• Two model calls

• One tool execution

The obvious question is:

Why is a second model call needed?

---

⚙️ Event-Loop Cycle #1: Decide and Act

The first chat span represents a model invocation.

The model receives:

• The user prompt

• Conversation history

• Available tools

It decides that the appropriate action is:

greet(name="rahul")

Strands then executes the tool.

chat #1
      |
      | decides to call
      v
execute_tool greet
      |
      v
"Hello, Rahul"

At this point the tool has completed.

But the agent has not.

---

🧩 Why Another Model Call?

Tools execute code.

They don't decide what should be shown to the user.

Once the tool returns its result, the model must reason over that result and determine:

• Is another tool required?

• Is the task complete?

• What should the final response be?

That requires another model invocation.

---

⚙️ Event-Loop Cycle #2: Observe and Respond

The updated context now contains:

User:
greet rahul using tool

Assistant:
toolUse -> greet("rahul")

Tool:
toolResult -> "Hello, Rahul"

The agent enters another event-loop cycle:

execute_event_loop_cycle #2
        |
        +-- chat #2
               |
               v
         Final Response

The model observes the tool result, determines that no further actions are required, and generates the final response.

The complete execution becomes:

User Prompt
      |
      v
Cycle #1
      |
      +-- Model decides
      |
      +-- Tool executes
      |
      v
Tool Result
      |
      v
Cycle #2
      |
      +-- Model reasons over result
      |
      v
Final Response

Notice that the first model call decides what to do.

The second model call decides what to say.

---

🧠 Why This Looks Like ReAct

This execution naturally follows the ReAct pattern:

Reason
   |
   v
Act
   |
   v
Observe
   |
   v
Reason
   |
   v
Answer

For our example:

Reason  → Decide to call greet()
Act     → Execute greet()
Observe → Receive "Hello, Rahul"
Reason  → Determine no further actions are required
Answer  → Return final response

ReAct describes how an agent behaves.

OpenTelemetry describes how that behavior is observed.

For example:

Agent Execution              OpenTelemetry Representation

Model invocation       →     chat span
Tool execution         →     execute_tool span
Tool result            →     span output / attributes
Next model invocation  →     next chat span
Agent invocation       →     trace containing these spans

A ReAct iteration is therefore a logical agent concept—not an OpenTelemetry primitive.

---

📂 Where Does a Session Fit?

A session can contain multiple user interactions.

Each interaction creates a new trace.

Session
|
+-- Prompt #1
|     +-- Trace #1
|
+-- Prompt #2
|     +-- Trace #2
|
+-- Prompt #3
      +-- Trace #3

Within each trace, the agent may execute multiple event-loop cycles.

Session
     |
     | 1:N
     v
Trace
     |
     | 1:N
     v
Event-Loop Cycles
     |
     | captured as
     v
Spans

The exact relationship between sessions and traces depends on the application's instrumentation, but this is a useful mental model.

---

🧩 Two Views of the Same Execution

When debugging an agent, think about two different perspectives.

Execution

Event Loop
     |
     +-- Reason
     +-- Act
     +-- Observe
     +-- Repeat

This explains how the agent works.

---

Observability

Trace
   |
   +-- Span
   +-- Span
   +-- Span

This explains what happened while the agent was working.

---

🎯 Final Thought

For our simple example:

One User Request
      |
      v
One Agent Execution / Trace
      |
      +-- Event-Loop Cycle #1
      |      +-- chat
      |      +-- execute_tool
      |
      +-- Event-Loop Cycle #2
             +-- chat
      |
      v
Final Response

Understanding this distinction makes agent traces much easier to read.

The event loop explains how the agent reasons.

The trace records that reasoning as observable spans.

OpenTelemetry doesn't change how the agent works.

It simply makes that work visible—making it easier to debug behavior, optimize execution, analyze tool usage, and evaluate agent performance.

🏗️ When a Managed Knowledge Base Isn't Enough

In the previous article, we looked at how a knowledge base works—from connectors and indexing to retrieval and generation.

The next question is usually:

Should I use a managed knowledge base or build my own?

For most projects, the answer is simple:

Start with a managed service.

Custom architectures only become worthwhile when your requirements exceed what managed knowledge bases were designed to solve.

---

☁️ Why Managed Knowledge Bases Exist

Managed knowledge bases package the entire retrieval pipeline into a single service.

They typically provide:

• Connectors
• Document ingestion
• Chunking
• Embedding generation
• Indexes
• Retrieval
• Synchronization

You configure the pipeline.

The platform operates it.

For many RAG applications, that's exactly what you need.

Typical use cases include:

• Internal documentation
• Product manuals
• Customer support
• Enterprise search
• AI assistants

---

🔧 When Managed Starts to Break Down

Managed services are intentionally opinionated.

They optimize for common retrieval problems.

Eventually, some teams discover that retrieval isn't their biggest challenge anymore.

The challenge becomes everything that happens before retrieval.

For example:

• Custom chunking strategies
• Rich metadata extraction
• External data enrichment
• Specialized embedding models
• Multiple indexing pipelines
• Custom ranking logic

These requirements often don't fit naturally into a managed pipeline.

---

🚀 When Custom Makes Sense

Building your own knowledge base gives you complete control over every stage of the pipeline.

You can customize:

• Connectors
• Ingestion
• Chunking
• Metadata extraction
• Enrichment
• Embedding generation
• Retrieval

That flexibility comes at a cost.

You now own:

• Infrastructure
• Scaling
• Monitoring
• Synchronization
• Upgrades
• Operational support

The question isn't whether you can build it.

The question is whether you need to.

---

📋 Decision Guide

Requirement	Managed	Custom
Standard document search	✅
Built-in connectors	✅
Semantic search	✅
Basic metadata filtering	✅
Low operational overhead	✅
Custom chunking		✅
Rich metadata extraction		✅
External data enrichment		✅
Multiple indexing pipelines		✅
Specialized retrieval or ranking		✅
Full control over the pipeline		✅

---

⚠️ Don't Build Custom Too Early

One of the biggest misconceptions is that a vector database is a knowledge base.

It isn't.

The real complexity isn't storing vectors.

It's everything around them:

• ingestion
• synchronization
• metadata management
• enrichment
• indexing
• retrieval

Building those components yourself is a long-term engineering commitment.

---

🎯 Final Thought

Managed knowledge bases solve the infrastructure problem.

Custom knowledge bases solve specialized business problems.

Start with a managed knowledge base whenever possible.

Move to a custom architecture only when your requirements clearly exceed what managed services were designed to support.

In the next article, we'll look at one of the biggest reasons teams outgrow managed knowledge bases:

The real challenge isn't retrieval—it's the ingestion and enrichment pipeline.

📚 Understanding Knowledge Bases: From Documents to Retrieval

In the previous article, we looked at different retrieval strategies—BM25, Vector Search, Hybrid Search, SQL, and LLM-powered retrieval. The next logical question is:

Where does the information actually come from?

The answer is a Knowledge Base.

Knowledge bases have become the foundation of many RAG and AI applications. Every major cloud provider now offers a managed knowledge base service, but regardless of the implementation, they all follow a very similar architecture.

At a high level, a knowledge base consists of six stages:

Data Sources
      ↓
Connectors
      ↓
Indexing Pipeline
      ↓
Indexes
(Vector • Metadata • Keyword)
      ↓
Retrieval
      ↓
LLM

---

🔌 Connectors

The first step is getting data into the knowledge base.

Common connectors include:

• S3

• SharePoint

• Confluence

• Google Drive

• Salesforce

• Web crawlers

• Custom APIs

The connector continuously discovers new or updated documents and feeds them into the indexing pipeline.

---

🏗️ Indexing Pipeline

Before documents become searchable, they pass through an indexing pipeline.

Typical steps include:

• Text extraction

• Chunking

• Metadata extraction

• Embedding generation

• Index creation

This is where raw documents are transformed into searchable knowledge.

---

📚 Indexes

Most knowledge bases maintain multiple indexes, each optimized for a different retrieval strategy.

• Vector Index for semantic search

• Metadata Index for filtering

• Keyword Index (BM25) for exact matching

Different queries may use one index—or combine multiple indexes—depending on the retrieval strategy.

---

🔍 Retrieval

When a user submits a query, the retrieval layer determines the best way to find relevant information.

Depending on the use case, it may use:

• BM25

• Vector Search

• Hybrid Search

Metadata filters are often applied before returning the most relevant chunks.

(If you're interested in when to use each retrieval strategy, see my previous article on BM25, Vector Search, Hybrid Search, SQL, and LLM-powered retrieval.)

---

🤖 Generation

The retrieved chunks are passed to an LLM as context.

The LLM doesn't search your documents directly—it generates an answer using the retrieved context.

This is the Generation in Retrieval-Augmented Generation (RAG).

---

☁️ Managed Knowledge Bases

Most cloud providers package this entire pipeline into a managed service.

You typically configure:

• Connectors

• Chunking strategy

• Embedding model

• Indexes

• Retrieval settings

The platform manages ingestion, indexing, synchronization, and retrieval.

Although services differ in their connectors, indexing options, and extensibility, the underlying architecture remains largely the same.

---

🎯 Final Thought

A knowledge base is much more than a vector database.

It's an end-to-end pipeline that ingests content, transforms it into searchable indexes, retrieves relevant context, and provides it to an LLM.

Understanding this architecture makes it much easier to understand where managed knowledge bases fit—and why some organizations eventually choose to build their own.

🔍 Choosing the Right Retrieval Strategy: BM25, Vector, Hybrid, SQL, or LLM?

 

One of the most common questions when building AI applications is:

Should I use BM25, Vector Search, Hybrid Search, SQL, or an LLM?

The answer is almost always:

It depends on what your users are trying to do.

Retrieval isn't a single problem.

Users have different intents, and each intent favors a different retrieval strategy.

Let's use a simple pizza menu to illustrate.

---

🍕 The Menu

• Margherita Pizza
• Pepperoni Special
• Veggie Supreme
• Cheese Lovers

Now imagine users searching in different ways.

Some know exactly what they want.

Others describe what they want.

Some ask questions.

Each requires a different retrieval strategy.

---

🎯 BM25: Exact Lookup

BM25 is traditional keyword search.

It excels when users already know what they're looking for.

Examples

• Pizza #3
• Margherita Pizza
• Order #12345
• Error Code 500

Strengths

• Fast
• Simple
• Highly precise

Limitation

BM25 understands words—not meaning.

Searching for "cheese pizza" won't necessarily find pizzas that only mention mozzarella or parmesan.

The good news is that BM25 is often enhanced with features such as:

• Fuzzy matching
• Prefix matching
• Stemming
• Synonym expansion
• Phonetic search

These improvements make keyword search much more forgiving, but they still don't provide true semantic understanding.

Best for

• IDs
• Product names
• Error codes
• Exact matches

---

🧠 Vector Search: Semantic Discovery

Vector search understands meaning rather than keywords.

Instead of matching words, it matches concepts.

For example:

"Something with cheese but no meat."

A vector search understands that mozzarella, parmesan, provolone, and cheddar are all forms of cheese.

Likewise,

"Vegetarian options"

finds pizzas without meat, even if the word vegetarian isn't explicitly present.

It also handles many spelling variations naturally.

"Margarita" → "Margherita"

Unlike BM25, these capabilities don't require manually defining synonyms or fuzzy rules.

Vector search is also commonly combined with metadata filtering to narrow results.

Best for

• Natural language
• Synonyms
• Concept search
• Discovery

---

⚖️ Hybrid Search: Best of Both

Real-world users don't all search the same way.

Some search:

Pizza #3

Others search:

Something spicy

Hybrid search combines BM25 with vector search.

BM25 provides precision for exact matches.

Vector search provides semantic understanding.

Together they produce better results than either approach alone.

For many production systems, hybrid search delivers the best overall user experience.

Best for

• Customer-facing search
• E-commerce
• Enterprise knowledge bases
• Mixed search behavior

---

🤖 LLM-Powered Retrieval

Sometimes retrieval isn't the problem.

Reasoning is.

Instead of querying an index directly, an LLM understands the user's request, expands it into multiple search strategies, executes those searches, and synthesizes the results.

For example:

"Find all places where authentication is implemented."

The model may search for:

• login
• authentication
• OAuth
• JWT
• identity
• authorization

and combine the results into a single answer.

This is similar to how tools like Claude Code search large codebases.

The trade-off is cost and latency.

Each query requires one or more LLM calls, making this approach significantly slower and more expensive than indexed retrieval.

Best for

• Developer tools
• Code exploration
• Internal documentation
• Research workflows

---

🗄️ Don't Forget SQL

Not every retrieval problem needs vectors.

If a user asks:

"Show all orders placed yesterday."

or

"List customers in California."

that's a structured query.

A relational database or metadata filter is often simpler, faster, and more accurate than semantic search.

One of the biggest mistakes in AI systems is using vector search where SQL is the better solution.

---

📋 Choosing the Right Retrieval Strategy

Example Query	Best Approach	Why	Helpful Features
Pizza #3	BM25	Exact identifier lookup where precision matters most	Fuzzy search, prefix matching
Margherita Pizza	BM25	User knows the exact name and expects an exact match	Synonyms, stemming
Vegetarian options	Vector Search	User is searching by concept rather than keywords	Metadata filtering
Something with cheese but no meat	Vector Search	Requires semantic understanding of ingredients and constraints	Metadata filtering
Margarita pizza	BM25 + Fuzzy or Vector Search	Needs typo tolerance while preserving relevance	Fuzzy matching or semantic similarity
Pizza under $15 with mushrooms	SQL + Vector Search	Price is structured data; description is unstructured	Structured filters + semantic search
Pizza #3 or recommend something similar	Hybrid Search	Combines exact lookup with semantic recommendations	BM25 + Vector fusion
Find all places where authentication is implemented	LLM-Powered Retrieval	Requires reasoning, query expansion, and synthesis across multiple sources	Multi-step reasoning

---

⚠️ Every Approach Has Trade-offs

There isn't a perfect retrieval strategy.

BM25

• Doesn't understand meaning.
• Requires additional features like fuzzy search and synonym expansion for better recall.

Vector Search

• Understands semantics but may rank conceptually similar results above exact matches.

Hybrid Search

• Delivers excellent results but requires tuning and balancing.

LLM-Powered Retrieval

• Powerful reasoning but higher cost and latency.

SQL

• Excellent for structured data but poor for semantic discovery.

Understanding where each approach fails is just as important as understanding where it succeeds.

---

🎯 Final Thought

The question shouldn't be:

"Which retrieval technology is the best?"

The better question is:

"What kind of retrieval problem am I trying to solve?"

Sometimes the answer is BM25.

Sometimes it's vector search.

Sometimes it's SQL.

Sometimes it's an LLM.

And increasingly, the best AI applications combine multiple retrieval strategies, using the right tool for the right job rather than forcing every query through the same pipeline.

🔍 Building Trust in AI: Reasoning Visibility and On-Demand Validation

One of the biggest challenges with AI isn't generating answers.

It's trusting them.

Most AI systems behave like black boxes—they provide an answer without showing how they arrived at it.

That leaves users with two choices:

• Trust the answer blindly.

• Verify everything manually.

Neither is a great experience.

We approached this with two complementary features:

• Reasoning Visibility — show how the AI arrived at its answer.

• On-Demand Validation — let users independently verify the answer.

Together they change the experience from:

"Trust me."

to

"Here's how I got there. Verify it if you'd like."

---

🧠 Feature 1: Reasoning Visibility

Instead of hiding execution, we make it available through a collapsible Show Reasoning section.

Users can see:

• The tools the agent invoked

• The actual requests that were executed

• The reasoning between each step

For example, instead of simply saying:

"I checked the status of your order."

The UI shows the actual execution:

GET /orders?customer=Acme&status=pending
GET /shipments?customer=Acme

Along with the reasoning:

"I retrieved all pending orders, then checked their shipment status before generating the summary."

Users can immediately understand what the agent did, which data it used, and why it reached its conclusion. Of course, this must be balanced with security by exposing only what is appropriate and redacting sensitive implementation details.

---

📚 Think of It Like Showing Your Work

When we were in school, teachers didn't just grade the final answer.

They asked us to show our work.

Not because the final answer wasn't important, but because the reasoning revealed whether we actually understood the problem.

AI systems should work the same way.

The goal isn't to expose every internal token the model generates. It's to provide enough transparency that users can understand, debug, and trust the result.

---

✅ Feature 2: On-Demand Validation

Sometimes seeing the work isn't enough.

You still want to know:

"Is the answer actually correct?"

Think back to school.

Showing your work helped the teacher understand how you solved the problem.

But for important exams, your work might also be reviewed by another teacher or an independent grader.

The reason is simple:

You don't grade your own homework.

We apply the same principle to AI.

When users click Justify, a second independent AI model reviews the answer.

Instead of trusting the first model, it:

• Re-queries the same data sources

• Verifies the facts independently

• Returns a verdict:

  • ✅ Valid

  • ⚠️ Partially Valid

  • ❌ Invalid

• Provides a confidence score

• Explains any discrepancies

The second model isn't grading its own work.

It's independently verifying the answer before giving its opinion.

That additional layer of validation builds confidence, especially for high-impact decisions.

---

🤝 Why They Work Together

These two features solve different problems.

Reasoning Visibility answers:

"How did the AI arrive at this answer?"

On-Demand Validation answers:

"Is the answer actually correct?"

One provides transparency.

The other provides confidence.

Together they allow users to inspect the reasoning when they're curious and independently validate the answer when accuracy really matters.

---

🎯 Final Thought

AI systems shouldn't ask users to trust them blindly.

They should make it easy to understand how an answer was produced and simple to verify whether it's correct.

Reasoning visibility and independent validation don't eliminate mistakes.

They make mistakes visible, explainable, and verifiable.

That's how trust is built.

🧪 Testing AI Agents: Multi-Layer Evaluation Strategy

Testing AI agents is fundamentally different from testing traditional software.

Traditional systems follow a simple contract:

input → expected output → assertion

Agents don't.

They reason dynamically, choose tools, compose responses, and operate on data that changes constantly. Hardcoded expected answers quickly become stale and brittle.

To address this, we use a three-layer evaluation strategy that combines deterministic validation, semantic evaluation, and model migration testing.

---

🚫 No Hardcoded Expected Results

One of the biggest challenges in agent testing is avoiding stale expectations.

Market data changes.
Accounts change.
Reference data evolves.

A hardcoded answer is often outdated the moment it's written.

Instead of storing static expected responses, each test case points to one or more live reference APIs.

Before evaluation runs, the framework fetches current ground-truth data directly from upstream services and evaluates the agent response against that live data.

This shifts the goal from validating snapshots to validating reasoning and communication against current reality.

---

⚡ Layer 1: Deterministic Validation

The first layer performs fast, reproducible checks against the agent response.

Examples include:

• Required values present
• Expected identifiers returned
• Missing fields detected
• Error or fallback messages detected
• Structured formats validated

A key capability is validating values against live reference data.

Expected values are extracted from the reference payload and normalized into multiple formats before checking whether they appear in the response.

For example:

659161818

might be recognized as:

659,161,818
$659.2M
659.2 million

This layer catches:

• Empty responses
• Missing information
• Incorrect values
• Malformed output
• Tool execution failures

Fast, deterministic, and easy to debug.

---

🤖 Layer 2: LLM-as-Judge

Deterministic checks verify facts.

They cannot determine whether an answer is complete, coherent, or grounded.

For that, a second LLM acts as a judge.

The judge evaluates:

• Completeness — Did the response answer everything that was asked?
• Coherence — Is the response clear and logically structured?
• Groundedness — Are all claims supported by reference data?

The judge receives:

• The original question
• The agent response
• The live reference data

It reasons step-by-step before assigning scores.

Importantly, the judge does not validate numeric accuracy. That responsibility remains with the deterministic layer.

Both layers use the same reference data but in different ways:

• Layer 1 verifies that required values appear in the response
• Layer 2 verifies that claims made in the response are supported by the reference

This separation prevents overlap and conflicting evaluations.

---

🔄 Layer 3: Model Migration Testing

Sometimes the question isn't:

Is this answer correct?

It's:

Can I safely switch from one model to another?

For migration testing, each test case runs twice:

• Baseline model
• Candidate model

A judge then compares the two responses and classifies the candidate

This mode answers a fundamentally different question from the first two layers.

The first two layers compare responses against objective ground truth.

The migration layer compares a candidate model against the current production baseline.

---

🏗️ Evaluation Pipeline

Ground Truth Evaluation

invoke_agent
    ↓
fetch_live_reference
    ↓
deterministic_validation
    ↓
(optional) llm_judge

---

Model Migration Evaluation

baseline_model
        ↓
candidate_model
        ↓
pairwise_judge

This path focuses solely on migration safety and does not run the ground-truth evaluation layers.

---

🎯 Why Multiple Layers?

No single evaluation method is sufficient.

Layer 1 provides:

• Speed
• Exactness
• Reproducibility

Layer 2 provides:

• Semantic validation
• Contextual reasoning
• Groundedness checks

Layer 3 provides:

• Safe model migration
• Regression detection
• Comparative evaluation

Together they provide a practical framework for testing AI agents without relying on brittle hardcoded outputs.

As agents become more autonomous and business-critical, having a robust evaluation strategy becomes just as important as the agent itself.Testing AI agents is fundamentally different from testing traditional software.

---

🔥 Chaos Engineering in Production: The Challenges Nobody Talks About

Part 2 of our multi-region failover series

Last week I wrote about running monthly chaos engineering exercises in production to validate our disaster recovery architecture.

The obvious follow-up:

"How do you get there safely?"

That's the right question.

Because chaos engineering in production is not where you start.

It's where you arrive — after building the operational maturity that makes failure survivable.

Here are the challenges you need to solve before you get there.

---

🔍 Challenge 1: Observability Gaps Will Expose You

Before triggering your first failover exercise, ask yourself:

If failover started right now, could you tell exactly what was happening?

Not after the fact.

In real time.

Can you see:

• traffic shifting between regions?
• application health in both regions?
• user impact during the transition?

Most dashboards are built for normal operations. Failover creates a completely different signal spanning infrastructure, DNS, networking, and applications simultaneously.

If you can't observe the recovery process, you can't safely test it.

Maturity bar: Build dashboards specifically for failover scenarios, not just general infrastructure health.

---

❤️ Challenge 2: Health Checks That Lie

Does your health check confirm the application is ready to serve traffic — or just that the process is running?

Applications can report healthy while:

• connection pools are still initializing
• caches are cold
• downstream services are unavailable

A failover mechanism that trusts shallow health checks can route traffic to a region that isn't actually ready.

Maturity bar: Validate readiness, not existence.

A lying health check is worse than no health check at all.

---

🌐 Challenge 3: DNS TTL Is Not Your Friend

Route 53 failover is not a switch.

Traffic does not instantly move from one region to another.

DNS caches.
Clients cache.
Resolvers cache.

Even with aggressive TTLs, some traffic continues flowing to the original region during transition.

Maturity bar: Understand your propagation behavior before running production exercises.

What looks like a failure may simply be DNS doing exactly what DNS does.

---

🚀 Challenge 4: Cold Start Reality vs. Cold Start Assumption

Recovery timelines often look great on architecture diagrams.

What they rarely account for:

• connection pool initialization
• cache warmup
• downstream dependency stabilization
• application readiness under real load

Container startup is only the beginning.

Maturity bar: Measure end-to-end recovery under load and let observed behavior define your recovery objectives.

---

🛑 Challenge 5: Blast Radius Without Boundaries

Chaos engineering gives automation authority over production infrastructure.

Without guardrails, a controlled exercise can become an actual incident.

Every exercise should have:

• hard limits
• abort conditions
• rollback procedures
• a designated kill switch

Maturity bar: Define the boundaries before the exercise starts.

---

💓 Challenge 6: The Traffic Shift Window Is Invisible Without a Heartbeat

During failover:

• the primary region is degrading
• the secondary region is coming online
• DNS is propagating

The question isn't:

"Did failover work?"

The real question is:

"Did users experience downtime?"

To answer that, we run an external synthetic heartbeat every 30 seconds through the same public endpoint users access.

The heartbeat records:

• success/failure
• latency
• timestamps

After every exercise we have evidence.

Not:

"We think there was no downtime."

But:

"Every heartbeat succeeded during the entire failover window."

The heartbeat doesn't prevent outages.

It proves the absence of them.

Maturity bar: Build an independent synthetic monitor before attempting production chaos.

---

✅ The Maturity Stack Before Production Chaos

Before running chaos engineering in production, you should have:

• Full observability
• Deep readiness-based health checks
• External heartbeat monitoring
• Tested recovery automation
• Practiced failback procedures
• Defined blast-radius controls
• Team readiness and communication plans
• Successful non-production validation

---

🎯 Final Thought

Chaos engineering in production is not about proving you're brave.

It's about proving your recovery process works.

The architecture matters.

The automation matters.

But confidence comes from continuous validation.

The heartbeat monitor proves users weren't impacted.

The recovery tests prove automation still works.

Together they turn disaster recovery from a theoretical capability into a continuously validated one.

That's not chaos.

That's engineering.

🌪️ Chaos Engineering for Disaster Recovery: Proving Multi-Region Failover Actually Works

 Most disaster recovery architectures are built with a hidden assumption:

The failover process will work when we need it.

The problem is that assumptions don't survive outages.

Infrastructure changes.
Deployments drift.
Permissions break.
Health checks evolve.
Automation silently fails.

A disaster recovery strategy is only as good as the last time it was tested.

That's why we built chaos engineering directly into our multi-region architecture.

---

The Architecture

Like many organizations, we run a primary AWS region that handles all production traffic.

The secondary region is fully provisioned but runs with zero application tasks during normal operation.

When the primary region becomes unhealthy:

• CloudWatch detects degradation
• Lambda initiates recovery
• The secondary region scales up
• Health checks begin passing
• Route 53 shifts traffic

The entire process is automated and completes in roughly 10 minutes.

This isn't designed for instant failover.

It's designed to provide a balance between resilience and cost efficiency.

---

The Real Challenge Isn't Failover

The real challenge is confidence.

Most teams test disaster recovery once during implementation and then assume it continues to work forever.

But recovery paths are software.

And software breaks.

The critical question becomes:

How do you know your failover automation still works six months from now?

---

Enter Chaos Engineering

Once a month we intentionally trigger a failover event in production.

Not a simulation.

A real failover.

We reduce capacity in the primary region and allow the system to respond naturally.

Alarms fire.
Recovery automation executes.
The secondary region activates.
Route 53 redirects traffic.
Production traffic runs from the backup region.

Several hours later we restore the primary region and validate failback behavior.

---

What Gets Validated

Each exercise validates the entire recovery chain:

✅ CloudWatch alarms

✅ Lambda execution

✅ Auto-scaling behavior

✅ Route 53 failover

✅ Application startup

✅ Service dependencies

✅ Recovery procedures

Instead of testing components individually, we're testing the complete system under real conditions.

---

The Detail That Prevents Downtime

One implementation detail made these exercises much safer.

During chaos testing, we don't scale the primary region to zero.

Instead, we reduce capacity by a single task.

That leaves enough healthy capacity to continue serving traffic while the secondary region comes online.

As DNS transitions occur, users continue receiving responses.

The recovery path is exercised without creating customer-visible downtime.

---

Why This Matters

The biggest risk in disaster recovery isn't infrastructure failure.

It's recovery procedures that haven't been tested recently.

A recovery plan sitting in a wiki isn't resilience.

A recovery plan executed successfully every month is.

---

🎯 Final Thought

Most organizations invest heavily in disaster recovery infrastructure.

Far fewer invest in continuously validating it.

Our multi-region architecture is intentionally cost-optimized, with the secondary region sitting idle most of the time.

But the real value isn't the architecture.

It's the confidence that comes from proving every month that failover still works.

Because in disaster recovery, the question isn't:

"Do we have a failover plan?"

It's:

"When was the last time we proved it actually works?"

🚦 Controlling Tool Output: Response Field Projection in Agent Workflows

One of the less obvious performance problems in agentic systems isn’t which tool gets called — it’s how much data comes back from it.

As agent workflows become more sophisticated, context growth can quietly become one of the biggest drivers of:

• latency
• token cost
• reasoning instability

---

🧠 The Problem: Context Growth Across Cycles

In chained workflows, tool responses accumulate across reasoning cycles:

Cycle 1 → tool response (2,000 tokens)
Cycle 2 → tool response + prior context (4,500 tokens)
Cycle 3 → accumulated context (8,000+ tokens)

Most enterprise APIs are designed for systems integration, not LLM efficiency.

A financial data endpoint may return:

• dozens of fields per record
• nested metadata
• audit attributes
• internal identifiers
• unused fields

But the agent may only need two or three fields to answer the user’s question.

When raw responses flow into the model unfiltered:

• 📈 token usage grows every cycle
• 🐢 latency increases as context expands
• ⚠️ field-selection mistakes become more common
• 🧾 prompt-level filtering becomes ineffective because tokens are already consumed before the instruction executes

A simple lookup can easily turn into thousands of unnecessary tokens.

---

🚀 The Fix: Projection at the Tool Layer

Instead of relying on the LLM to discard unnecessary fields after receiving the response, we moved the optimization into the tool layer itself.

We added a response_fields parameter to the HTTP request tool.

The agent specifies exactly which fields it needs before making the request, and the tool filters the response before returning it to the model.

Instead of:

Tool → large raw JSON → LLM → filter + reason + respond

We now use:

Tool → projected response → LLM → respond

The projection supports:

• arrays and nested objects
• dot-notation field selection
• graceful fallback to full responses when projection is unavailable

✅ Minimal payloads
✅ Smaller context
✅ Faster reasoning

---

🧩 Closing the Loop

Projection only works if the agent knows which fields to request.

That knowledge can come from:

• system prompts
• tool metadata
• endpoint descriptions
• execution guidance
• field-level documentation

The important part is that the agent identifies required fields before making the tool call instead of reasoning over a large payload afterward.

This shifts optimization from:

• post-processing responses
  to:
• controlling responses at the source

---

🔄 Execution Pattern

Execution guidance
    → agent identifies required fields
        → tool-level response projection
            → minimal structured output into LLM context

Rather than continuously expanding context across cycles, the agent keeps context compact and purpose-driven.

---

📊 Results

Queries that previously returned thousands of tokens per cycle now return only a fraction of that.

For multi-step workflows, this:

• 💰 reduces token consumption
• ⚡ lowers latency
• 📏 stabilizes context growth
• 🎯 improves reasoning reliability

The pattern is similar to GraphQL:
the client declares what it needs, and only that data comes back.

In this case, the “client” is the LLM itself.

---

🎯 Final Thought

Efficient agents don’t just call the right tools.

They also control what data comes back from them.

In many production systems, optimizing tool output has a larger impact on performance and reliability than changing the model itself.