In previous articles, we looked at retrieval strategies and knowledge base architectures.
One assumption quietly made throughout was that every embedding lives in the same vector index.
That works well for small datasets.
At hundreds of millions of vectors, it becomes expensive.
The reason isn't the embeddings.
It's the index.
💰 The Real Cost of Vector Search
Unlike relational databases, vector search doesn't scan every embedding.
Instead, it traverses an Approximate Nearest Neighbor (ANN) index to quickly find similar vectors.
To achieve low-latency search, that ANN index is typically kept in memory.
As the corpus grows, so does the amount of memory—and infrastructure—needed to keep that index available.
The important observation is this:
Not every vector is queried equally.
📈 Most Corpora Follow a Long Tail
Think about any corpus with a time dimension:
Support tickets
Meeting notes
Product documentation
Incident reports
Contracts
Recent information is queried constantly.
Older information is still valuable—but much less frequently.
A typical access pattern looks like this:
10% of documents
↓
90% of queries
90% of documents
↓
10% of queries
Yet many systems keep 100% of the vectors in the fastest—and most expensive—infrastructure.
🔥 Hot, 🌤️ Warm, ❄️ Cold
The idea is the same one object storage has used for years.
Not every document needs the same retrieval speed.
🔥 Hot Tier
Frequently accessed vectors.
Examples:
OpenSearch k-NN
pgvector (HNSW)
Pinecone
Weaviate
Characteristics:
ANN index kept in memory
Lowest latency
Highest infrastructure cost
Typical data:
Recent documents
Frequently queried content
🌤️ Warm Tier
Occasionally accessed vectors.
The vectors live in low-cost object storage but are loaded into memory on demand and cached for subsequent searches.
Examples:
Cached JSONL or Parquet embeddings in S3
Amazon S3 Vectors
Other managed vector services
Typical flow:
First Query
S3
↓
Load vectors
↓
Cache in memory
↓
Similarity search
Subsequent Queries
Memory
↓
Similarity search
Compared to the hot tier:
No always-running ANN index
Lower infrastructure cost
Slightly higher latency
Good for infrequently accessed data
❄️ Cold Tier
Rarely accessed vectors.
Vectors remain in object storage and are only read when a query requires them.
Typical flow:
User Query
↓
Read vectors from object storage
↓
Exact similarity search
↓
Return Top-K
Characteristics:
No ANN index
No persistent memory
Lowest storage cost
Highest latency
Cold storage is ideal for historical archives where occasional slower queries are acceptable.
🏗️ Hide the Complexity
The biggest mistake is exposing storage tiers to application code.
Instead of writing:
if document_is_hot:
search_hot()
elif document_is_warm:
search_warm()
else:
search_archive()
Applications should simply ask:
Search documents
The retrieval layer decides which storage tier to query.
This keeps applications independent of storage decisions while allowing the storage architecture to evolve over time.
📉 Where the Savings Come From
Many people assume tiering reduces storage cost.
That's only part of the story.
The real savings come from reducing:
Memory-resident ANN indexes
Compute
Always-on infrastructure
The fewer vectors you keep in expensive memory, the smaller your always-running infrastructure becomes.
Tiering isn't about storing vectors more cheaply.
It's about keeping only the right vectors in expensive infrastructure.
🤖 Intelligent Routing
As storage tiers grow, deciding where to search becomes another retrieval problem.
Simple rules such as document age often work.
Increasingly, AI agents can route queries based on user intent.
For example:
"Show the latest incident." → Hot
"Summarize the original agreement." → Cold
"How has this evolved over time?" → Multiple tiers
Applications continue to issue a single retrieval request.
The retrieval layer decides which storage tier—or combination of tiers—to search.
📋 When Tiering Makes Sense
Tiering is valuable when:
Very large vector collections
Strongly skewed access patterns
Historical data is queried infrequently
Different latency requirements are acceptable
It adds little value when:
Datasets are relatively small
Access patterns are uniform
Every query requires consistently low latency
As with most optimizations, measure your access patterns before introducing additional architectural complexity.
🎯 Final Thought
The biggest cost in vector search isn't storing embeddings.
It's maintaining large, memory-resident ANN indexes.
Just as object storage evolved into Standard, Infrequent Access, and Archive tiers, vector storage is beginning to follow a similar path.
The goal isn't to store vectors more cheaply.
It's to keep only the vectors that require low-latency access in your fastest—and most expensive—infrastructure, while allowing the long tail to move to lower-cost storage without changing how applications retrieve information.
No comments:
Post a Comment