🤖Organizing a Project for Coding Agents

Instructions Explain, Scripts Execute, Tests Prove

Coding agents can read a lot of code.

That usually isn't the hard part.

The harder part is everything an experienced engineer already knows but the repository doesn't clearly say. Organizing a project for coding agents is really about putting that knowledge in the right place.

A simple principle helps:

Instructions explain. Skills guide. Scripts execute. Tests prove.

---

🏗️ An Application Is Often Many Repositories

A modern application is rarely one repository.

It may be split across several:

app                → application code and build pipeline
database           → schema, migrations, seed data
app-infra          → cloud resources, IAM, DNS, search
cluster-infra      → Kubernetes clusters and shared services
deploy-manifests   → environment-specific deployment configuration

An agent looking at only one repository cannot infer that entire relationship.

That is the first thing we should make explicit:

What repositories make up the application, and what does each one own?

That gives the agent a map before it starts navigating.

---

🧠 Instructions Explain What the Code Cannot

Instruction files such as AGENTS.md or CLAUDE.md should contain information the agent cannot reliably discover from source code alone.

For example:

• Repository relationships

• Architecture boundaries

• How environments are organized

• How tests should be run

• Important operational constraints

• Critical guardrails

The goal is not to document the entire repository.

The agent can already read the repository.

A good rule is:

If the agent can reliably discover something from the code, don't repeat it in the instructions.

Keep instructions short and focused.

Long instruction files consume context and make the important rules harder to notice.

---

🧩 Skills Guide Repeatable Procedures

Some engineering tasks follow the same process again and again.

For example:

• Check deployment status

• Investigate an incident

• Compare environments

• Validate a release

• Inspect a scheduled job

Instead of expecting the agent to rediscover the procedure every time, package it as a skill.

Conceptually:

User Question
      ↓
Relevant Skill
      ↓
Known Procedure
      ↓
Tools / Scripts
      ↓
Result

A skill should tell the agent:

• When to use it

• What procedure to follow

• Which tools or scripts to run

• How to interpret the result

The advantage is that specialized instructions only need to load when they are relevant.

The main agent instructions stay small.

---

⚙️ Scripts Execute Deterministic Work

Suppose the procedure is:

Get deployments
Check pods
Check scheduled jobs
Compare versions
Identify failures

An agent could construct those commands every time.

But why make it reason through something deterministic?

Instead:

Agent
   ↓
Run Script
   ↓
Collect State
   ↓
Structured Result
   ↓
Agent Interprets Result

The script handles the repeatable mechanics.

The agent handles judgment.

This distinction is important:

Every step the agent doesn't have to reason through is one less step it can get wrong.

Use the model where reasoning adds value.

Use scripts where consistency matters more.

---

🛡️ Move Critical Guardrails Below the Prompt

Instructions are useful, but they are still guidance.

Consider a rule like:

Production cluster access is read-only.

You can put that in AGENTS.md.

But if the rule must always hold, don't rely on the model remembering it.

Move the constraint into deterministic controls:

Instruction
    ↓
Hook
    ↓
Permissions

For example:

• The instruction says access is read-only

• A hook blocks unsafe commands

• Cluster permissions prevent writes entirely

The deeper the control is enforced, the less the agent has to remember.

An instruction is a request. A policy or permission is a guarantee.

---

✅ Tests Prove the Work

A coding agent shouldn't stop because a change looks correct.

It should have a way to verify the result.

Without validation:

Change Code
    ↓
Looks Correct
    ↓
Stop

With validation:

Change Code
    ↓
Run Test
    ↓
Failure?
    ↓
Fix
    ↓
Run Again

Tests give the agent evidence.

That changes the role of the human.

Instead of being the verification loop, the human becomes the reviewer of a result that has already been checked.

The same idea applies to agent tooling itself.

Skills, scripts, hooks, and configuration should also be validated like production code.

---

🔄 Let Skills Improve Through Use

Skills shouldn't be treated as finished artifacts.

The most useful improvements often come from real usage.

You may discover that:

• The output is too verbose

• A status is ambiguous

• One workflow takes too long

• A special case needs a shortcut

• An important field is missing

Capture those learnings.

Then update the skill or script.

Use
 ↓
Observe
 ↓
Learn
 ↓
Improve
 ↓
Next Run Gets Better

Over time, operational knowledge moves out of people's heads and into reusable automation.

---

📚 Documentation Is Still for Humans

Agents need instructions, skills, scripts, and tests.

Humans still need architecture documentation.

A short README or architecture document should explain:

• What the major components are

• How they connect

• Where they run

• Which repositories own which responsibilities

The agent can be pointed to that documentation when needed.

There is no reason to copy all of it into the permanent instruction context.

---

🧭 Where Things Belong

A simple mental model:

Need	Put It Here
How do repositories relate?	Instructions
What should the agent know?	Instructions
What repeatable process should it follow?	Skill
What can execute deterministically?	Script / Tool
What must always be enforced?	Hook / Permission
How do we know it worked?	Test
How does the procedure improve?	Learnings
What helps humans understand the system?	Documentation
What is the implementation truth?	Source code

The general direction should be:

Move work toward the most deterministic layer possible.

If guidance can become a script, make it a script.

If a rule can become a permission, enforce it.

If an outcome can be tested, test it.

---

🎯 Final Thought

Making coding agents more effective isn't only about better models or larger context windows.

It's also about designing the engineering environment so the agent has less to infer, less to remember, and less to invent.

A well-organized project gives the agent:

• A map of the repositories

• Clear instructions

• Reusable skills

• Deterministic scripts

• Enforced guardrails

• Tests that prove the result

Instructions explain. Skills guide. Scripts execute. Tests prove.

The goal isn't to make the agent remember everything.

It's to organize the project so it knows where to look, what to run, what it is allowed to do, and how to prove the work is complete.

🧠 Context Offloading Isn't Always a Cost Optimization

 Many modern agent frameworks support automatic context offloading.

The idea is simple.

Instead of keeping large tool responses inside the model's context, the framework stores them externally, leaves behind a small preview, and retrieves the original only if the model needs it later.

On paper, it sounds like an obvious optimization.

Smaller prompts should mean lower token usage and lower cost.

I measured it on one of my production agents.

The result surprised me.

Token usage increased by 68%.

The interesting part wasn't the number.

It was why.

---

🤔 How Context Offloading Works

Imagine an agent receives a large tool response.

Without offloading:

Tool
   ↓
Large Result
   ↓
LLM

The entire payload remains in the model's context.

With offloading:

Tool
    ↓
Large Result
    ↓
Store externally
    ↓
Preview in context
    ↓
Retrieve only if needed

The expectation is simple:

• Smaller prompts
• Lower token usage
• Longer conversations

---

📊 What I Measured

I evaluated multiple context-management strategies against the same workload.

The baseline simply used a sliding context window.

Automatic offloading enabled the framework's built-in context manager.

Strategy	Model Calls	Input Tokens	Quality
Baseline	7	227K	9/9
Automatic Offloading	14	382K	8/9

Instead of reducing cost, offloading:

• increased model calls
• increased total token usage
• slightly reduced answer quality

---

🧩 The Hidden Cost

The reason became obvious after looking at the traces.

The framework successfully reduced the size of individual prompts.

But every time the model needed information that had been offloaded, it performed another retrieval.

That meant another model invocation.

The flow looked like this:

Tool
    ↓
Large Result
    ↓
Offload
    ↓
Preview
    ↓
Model requests original
    ↓
Retrieve
    ↓
Another model call

The content wasn't removed.

It simply came back later.

You paid for:

• the preview
• the retrieval
• another model call
• the conversation history again

The content was effectively processed twice.

---

📉 Compression Worked

One result initially confused me.

The average prompt actually became smaller.

Per-call context was reduced by roughly 16%.

So why did total cost increase?

Because billing depends on total tokens processed, not the size of an individual prompt.

Reducing the size of each request didn't help when the framework created many more requests.

In my case:

Smaller prompts
        +
More model calls
        =
More total tokens

The compression worked.

The retrieval round trips outweighed the savings.

---

📏 Context Window Matters

The most important number turned out not to be token usage.

It was context utilization.

My model supports a 1 million token context window.

The baseline execution peaked at approximately 55,000 tokens.

That's roughly 5.5% of the available context.

I had enabled a feature designed to prevent context overflow.

The problem was...

I was nowhere near overflowing.

I was paying the cost of protection against a problem that didn't exist.

---

✅ When Context Offloading Makes Sense

Context offloading isn't a bad feature.

It's solving a different problem.

It works well when:

• Long-running conversations
• Small context windows
• Very large tool responses
• Information that is unlikely to be referenced again

Those are situations where preventing context growth is more important than minimizing model calls.

---

❌ When It Doesn't

Be cautious when:

• The conversation easily fits inside the context window
• Tool responses are immediately reused
• Every reasoning step depends on previous tool output

In these cases, offloading often creates additional retrievals without providing meaningful savings.

---

🎯 Final Thought

Context offloading isn't primarily a cost optimization.

It's a context management strategy.

If your agent is approaching the context window, it can prevent important information from being discarded.

If your agent is nowhere near that limit, it may simply replace larger prompts with more model calls.

The lesson isn't:

Always enable context offloading.

It's:

Measure first. Optimize only for the bottleneck you actually have.

🔥 Hot, Warm, and Cold: Reducing Vector Search Costs with Tiered Storage

In previous articles, we looked at retrieval strategies and knowledge base architectures.

One assumption quietly made throughout was that every embedding lives in the same vector index.

That works well for small datasets.

At hundreds of millions of vectors, it becomes expensive.

The reason isn't the embeddings.

It's the index.

---

💰 The Real Cost of Vector Search

Unlike relational databases, vector search doesn't scan every embedding.

Instead, it traverses an Approximate Nearest Neighbor (ANN) index to quickly find similar vectors.

To achieve low-latency search, that ANN index is typically kept in memory.

As the corpus grows, so does the amount of memory—and infrastructure—needed to keep that index available.

The important observation is this:

Not every vector is queried equally.

---

📈 Most Corpora Follow a Long Tail

Think about any corpus with a time dimension:

• Support tickets

• Meeting notes

• Product documentation

• Incident reports

• Contracts

Recent information is queried constantly.

Older information is still valuable—but much less frequently.

A typical access pattern looks like this:

10% of documents
        ↓
90% of queries

90% of documents
        ↓
10% of queries

Yet many systems keep 100% of the vectors in the fastest—and most expensive—infrastructure.

---

🔥 Hot, 🌤️ Warm, ❄️ Cold

The idea is the same one object storage has used for years.

Not every document needs the same retrieval speed.

🔥 Hot Tier

Frequently accessed vectors.

Examples:

• OpenSearch k-NN

• pgvector (HNSW)

• Pinecone

• Weaviate

Characteristics:

• ANN index kept in memory

• Lowest latency

• Highest infrastructure cost

Typical data:

• Recent documents

• Frequently queried content

---

🌤️ Warm Tier

Occasionally accessed vectors.

The vectors live in low-cost object storage but are loaded into memory on demand and cached for subsequent searches.

Examples:

• Cached JSONL or Parquet embeddings in S3

• Amazon S3 Vectors

• Other managed vector services

Typical flow:

First Query

S3
   ↓
Load vectors
   ↓
Cache in memory
   ↓
Similarity search

Subsequent Queries

Memory
   ↓
Similarity search

Compared to the hot tier:

• No always-running ANN index

• Lower infrastructure cost

• Slightly higher latency

• Good for infrequently accessed data

---

❄️ Cold Tier

Rarely accessed vectors.

Vectors remain in object storage and are only read when a query requires them.

Typical flow:

User Query
      ↓
Read vectors from object storage
      ↓
Exact similarity search
      ↓
Return Top-K

Characteristics:

• No ANN index

• No persistent memory

• Lowest storage cost

• Highest latency

Cold storage is ideal for historical archives where occasional slower queries are acceptable.

---

🏗️ Hide the Complexity

The biggest mistake is exposing storage tiers to application code.

Instead of writing:

if document_is_hot:
    search_hot()
elif document_is_warm:
    search_warm()
else:
    search_archive()

Applications should simply ask:

Search documents

The retrieval layer decides which storage tier to query.

This keeps applications independent of storage decisions while allowing the storage architecture to evolve over time.

---

📉 Where the Savings Come From

Many people assume tiering reduces storage cost.

That's only part of the story.

The real savings come from reducing:

• Memory-resident ANN indexes

• Compute

• Always-on infrastructure

The fewer vectors you keep in expensive memory, the smaller your always-running infrastructure becomes.

Tiering isn't about storing vectors more cheaply.

It's about keeping only the right vectors in expensive infrastructure.

---

🤖 Intelligent Routing

As storage tiers grow, deciding where to search becomes another retrieval problem.

Simple rules such as document age often work.

Increasingly, AI agents can route queries based on user intent.

For example:

• "Show the latest incident." → Hot

• "Summarize the original agreement." → Cold

• "How has this evolved over time?" → Multiple tiers

Applications continue to issue a single retrieval request.

The retrieval layer decides which storage tier—or combination of tiers—to search.

---

📋 When Tiering Makes Sense

Tiering is valuable when:

• Very large vector collections

• Strongly skewed access patterns

• Historical data is queried infrequently

• Different latency requirements are acceptable

It adds little value when:

• Datasets are relatively small

• Access patterns are uniform

• Every query requires consistently low latency

As with most optimizations, measure your access patterns before introducing additional architectural complexity.

---

🎯 Final Thought

The biggest cost in vector search isn't storing embeddings.

It's maintaining large, memory-resident ANN indexes.

Just as object storage evolved into Standard, Infrequent Access, and Archive tiers, vector storage is beginning to follow a similar path.

The goal isn't to store vectors more cheaply.

It's to keep only the vectors that require low-latency access in your fastest—and most expensive—infrastructure, while allowing the long tail to move to lower-cost storage without changing how applications retrieve information.

Deep Dive into a Strands Agent Trace

When working with AI agents, one of the most useful ways to understand what is happening behind the scenes is through tracing.

A seemingly simple user request can result in multiple model invocations, tool executions, and event-loop cycles before the agent produces its final response.

In this post, I'll walk through a simple Strands Agent example:

greet rahul using tool

---

🧠 What Actually Happens?

The resulting trace looks roughly like this:

agent.run
|
+-- invoke_agent
    |
    +-- execute_event_loop_cycle #1
    |   |
    |   +-- chat #1
    |   |
    |   +-- execute_tool greet
    |
    +-- execute_event_loop_cycle #2
        |
        +-- chat #2

Although the user made only one request, the agent performed:

• Two event-loop cycles

• Two model calls

• One tool execution

The obvious question is:

Why is a second model call needed?

---

⚙️ Event-Loop Cycle #1: Decide and Act

The first chat span represents a model invocation.

The model receives:

• The user prompt

• Conversation history

• Available tools

It decides that the appropriate action is:

greet(name="rahul")

Strands then executes the tool.

chat #1
      |
      | decides to call
      v
execute_tool greet
      |
      v
"Hello, Rahul"

At this point the tool has completed.

But the agent has not.

---

🧩 Why Another Model Call?

Tools execute code.

They don't decide what should be shown to the user.

Once the tool returns its result, the model must reason over that result and determine:

• Is another tool required?

• Is the task complete?

• What should the final response be?

That requires another model invocation.

---

⚙️ Event-Loop Cycle #2: Observe and Respond

The updated context now contains:

User:
greet rahul using tool

Assistant:
toolUse -> greet("rahul")

Tool:
toolResult -> "Hello, Rahul"

The agent enters another event-loop cycle:

execute_event_loop_cycle #2
        |
        +-- chat #2
               |
               v
         Final Response

The model observes the tool result, determines that no further actions are required, and generates the final response.

The complete execution becomes:

User Prompt
      |
      v
Cycle #1
      |
      +-- Model decides
      |
      +-- Tool executes
      |
      v
Tool Result
      |
      v
Cycle #2
      |
      +-- Model reasons over result
      |
      v
Final Response

Notice that the first model call decides what to do.

The second model call decides what to say.

---

🧠 Why This Looks Like ReAct

This execution naturally follows the ReAct pattern:

Reason
   |
   v
Act
   |
   v
Observe
   |
   v
Reason
   |
   v
Answer

For our example:

Reason  → Decide to call greet()
Act     → Execute greet()
Observe → Receive "Hello, Rahul"
Reason  → Determine no further actions are required
Answer  → Return final response

ReAct describes how an agent behaves.

OpenTelemetry describes how that behavior is observed.

For example:

Agent Execution              OpenTelemetry Representation

Model invocation       →     chat span
Tool execution         →     execute_tool span
Tool result            →     span output / attributes
Next model invocation  →     next chat span
Agent invocation       →     trace containing these spans

A ReAct iteration is therefore a logical agent concept—not an OpenTelemetry primitive.

---

📂 Where Does a Session Fit?

A session can contain multiple user interactions.

Each interaction creates a new trace.

Session
|
+-- Prompt #1
|     +-- Trace #1
|
+-- Prompt #2
|     +-- Trace #2
|
+-- Prompt #3
      +-- Trace #3

Within each trace, the agent may execute multiple event-loop cycles.

Session
     |
     | 1:N
     v
Trace
     |
     | 1:N
     v
Event-Loop Cycles
     |
     | captured as
     v
Spans

The exact relationship between sessions and traces depends on the application's instrumentation, but this is a useful mental model.

---

🧩 Two Views of the Same Execution

When debugging an agent, think about two different perspectives.

Execution

Event Loop
     |
     +-- Reason
     +-- Act
     +-- Observe
     +-- Repeat

This explains how the agent works.

---

Observability

Trace
   |
   +-- Span
   +-- Span
   +-- Span

This explains what happened while the agent was working.

---

🎯 Final Thought

For our simple example:

One User Request
      |
      v
One Agent Execution / Trace
      |
      +-- Event-Loop Cycle #1
      |      +-- chat
      |      +-- execute_tool
      |
      +-- Event-Loop Cycle #2
             +-- chat
      |
      v
Final Response

Understanding this distinction makes agent traces much easier to read.

The event loop explains how the agent reasons.

The trace records that reasoning as observable spans.

OpenTelemetry doesn't change how the agent works.

It simply makes that work visible—making it easier to debug behavior, optimize execution, analyze tool usage, and evaluate agent performance.