Retrieval Engineering: Beyond Vector Search for Production RAG

There is a persistent myth in AI development: vector search is a silver bullet for grounding agents in private knowledge. While vector search is excellent for conceptual, semantic queries, it creates massive gaps when building production-grade systems. These gaps lead to hallucinations, incomplete answers, and unreliable results.

To build reliable agents, you must move from simple RAG (Retrieval Augmented Generation) to Retrieval Engineering. This discipline involves designing specific retrieval strategies based on the scope and capabilities of your system. This tutorial analyzes nine distinct failure points of vector search and demonstrates the specific engineering patterns required to solve them.

Prerequisites

  • Basic understanding of RAG (Retrieval Augmented Generation) architectures.
  • Familiarity with Python and vector databases (e.g., Pinecone, Chroma, or Weaviate).
  • Knowledge of LLM orchestration concepts (Agents, Tools).

The Limitations of Vector Search

Vector search relies on similarity algorithms. When you query a vector store, you receive the most similar chunks, not necessarily the most relevant ones. This lack of exactness is a feature for fuzzy searching but a critical bug when exactness is required.

Retrieval Engineering treats retrieval as a “tool call” within a decision loop. The agent must decide: Do I have enough info? Or do I need to query a SQL database, scan a file, or perform a keyword search?

# Conceptualizing the Retrieval Decision Loop
def retrieval_router(query):
if requires_exact_match(query):
return keyword_search(query)
elif requires_aggregation(query):
return sql_tool(query)
elif requires_concept(query):
return vector_search(query)
else:
return agent_reasoning(query)

Strategy 1: Full Document Processing

Vector search fails when an answer requires synthesizing information scattered across an entire document rather than a single chunk.

Case Study: Meeting Decisions

Question: “What decisions were made in the leadership meeting?”

Decisions are often sprinkled throughout a transcript and rarely labeled explicitly. A standard vector search for “decision” will retrieve random context or miss implicit agreements entirely.

Case Study: Comprehensive Summaries

Question: “Summarize the Q3 Internal Report.”

Vector databases retrieve the “top K” chunks. If a document has 50 chunks, retrieving the top 5 results in a partial, potentially misleading summary.

The Solution: Context Expansion & Hierarchical Summarization

Don’t rely on chunks alone. Use Context Expansion (or Parent Document Retrieval). When a chunk is matched, retrieve the full parent section or the entire document if it fits the context window.

For large documents, implement a Map-Reduce strategy or delegate to a sub-agent responsible for document processing.

# Pseudocode for Context Expansion Pattern
def retrieve_with_context(query):
# 1. Find relevant small chunks via vector search
relevant_chunks = vector_store.similarity_search(query)

# 2. Map chunks back to their Parent Document IDs
doc_ids = [chunk.metadata[‘parent_id’] for chunk in relevant_chunks]

# 3. Retrieve full documents/sections from a doc store (e.g., S3, SQL)
full_contexts = doc_store.get_documents(doc_ids)

return full_contexts

Strategy 2: High-Precision Retrieval

Embedding models are trained on general internet data. They often fail to represent domain-specific codes, internal acronyms, or rare project names.

Case Study: The “Blue Sheet” Project

Question: “Who created the Blue Sheet system?”

If “Blue Sheet” is an internal term absent from the model’s training data, its vector representation will be weak or generic. Semantic search may return results about “blueprints” or “sheets,” failing to find the exact match.

Case Study: Specific Error Codes

Question: “Explain error 15-CFR-744.”

Vector search struggles with alphanumeric codes. It sees “15”, “CFR”, and “744” as separate tokens, often returning similar but incorrect codes (e.g., error 742 or 745).

Combine semantic vectors with Lexical Search (BM25) or keyword matching. This ensures that if an exact keyword exists in the corpus, it boosts the ranking of that document.

# Example: Hybrid Search Configuration (Conceptual)
search_results = vector_store.search(
query=”Blue Sheet system”,
search_type=”hybrid”, # Combines Sparse (Keyword) + Dense (Vector)
alpha=0.5 # Weighting: 50% keyword, 50% semantic
)

# For codes, use Regex/Pattern matching tools
import re
if re.match(r’\d{2}-[A-Z]{3}-\d{3}’, query):
tool = “precise_code_lookup”

Strategy 3: Metadata & Conditional Filtering

Vector databases generally do not understand time or recency unless explicitly instructed.

Case Study: “Who is the CEO?”

Question: “Who is the CEO of our company?”

If your company has had three CEOs in 20 years, vector search will find documents about all of them. It might rank a document from 2015 higher than one from 2024 if the semantic match is stronger, leading to an obsolete answer.

The Solution: Metadata & Promptable Reranking

Enrich your chunks with metadata (dates, document types). Use an agent to extract temporal constraints from the user’s query and apply them as filters.

# 1. Agent extracts filters from the query
# Query: “Who is the CEO?” -> Implicitly current
filters = {
“doc_type”: “org_chart”,
“year”: {“$gte”: 2024}
}

# 2. Apply filters during retrieval
docs = vector_store.similarity_search(
“CEO”,
filter=filters
)

# 3. Use a Reranker to sort by recency if needed
docs.sort(key=lambda x: x.metadata[‘date’], reverse=True)

Strategy 4: Structured Data & Aggregation

Standard RAG struggles with quantitative questions that require calculation or tabular lookup.

Case Study: Tabular Data

Question: “What was our revenue in Q2 2024?”

Financial figures are often embedded in PDF tables. Vector search treats rows and columns as unstructured text, often losing the relationship between the header “Q2” and the value “$1.5M”.

Case Study: Aggregation

Question: “How many support tickets were closed last month?”

This requires counting across documents. An LLM cannot “count” documents in a vector database; it can only read the text provided in the context window.

The Solution: Text-to-SQL and OCR

For tabular data in PDFs, use Markdown-friendly OCR tools (like LlamaParse or Mistral) to preserve structure. For aggregation, route these queries to a Text-to-SQL tool or a DataFrame agent.

Strategy 5: GraphRAG & Multi-Hop Reasoning

Some questions require “global” knowledge or connecting multiple isolated entities.

Case Study: Global Patterns

Question: “What are the recurring operational challenges across all team retrospectives?”

This requires reading all documents to find themes. Vector search only sees a subset.

Case Study: Multi-Hop Logic

Question: “What projects are affected if Sarah goes on leave?”

To answer this, the system must chain facts:

1. Identify Sarah’s Role.

2. Identify Projects linked to that Role.

3. Check deadlines/dependencies for those projects.

The Solution: Knowledge Graphs (GraphRAG)

Knowledge graphs model entities (People, Projects) and relationships (WORKS_ON, DEPENDS_ON). GraphRAG pre-computes communities and summaries, allowing you to query the relationships rather than just the text.

# Conceptual Graph Query (Cypher)
MATCH (p:Person {name: “Sarah”})-[:WORKS_ON]->(proj:Project)
MATCH (proj)-[:HAS_STATUS]->(s:Status)
RETURN proj.name, s.deadline, s.risk_level

Advanced Edge Cases

Visual Retrieval

Question: “How do I change the toner? Show me the diagram.”

Text-only RAG fails here. Use Multimodal RAG. Store image embeddings alongside text, or keep image URLs in metadata to inject visual assets into the chat interface.

False Premises

Question: “Which VP led the Berlin office before it closed?” (Premise: The company never had a Berlin office).

Vector search will find the closest match (e.g., “Berlin expansion plans” or “VP of Operations”), potentially causing the LLM to hallucinate that the office existed. Use a Verification Step where the agent cross-references the retrieved answer against a ground-truth check before responding.

Next Steps

Building production-grade agents requires moving beyond the “vector search is enough” mindset. By implementing a robust routing layer, you can direct semantic queries to vector stores, precise queries to keyword search, and analytical queries to SQL databases.

Start by auditing your failed queries. Categorize them into the buckets above (Aggregation, Exact Match, Multi-hop) and engineer the specific retrieval path that solves that category.