Most developers build their first Retrieval-Augmented Generation (RAG) system using a simple template. You load some text files, split them into fixed-size chunks of 500 characters, embed them using an API, and dump them into a vector database. When a user asks a question, you perform a similarity search, grab the top three chunks, stuff them into the prompt, and hope the LLM gives a sensible answer.
This basic approach works fine for simple demos. But when you deploy it to production, things fall apart. Users ask vague questions. Documents contain tables, cross-references, and long-tail context that gets lost when you slice text into arbitrary blocks.
The core issue is a fundamental conflict in chunk size. For vector search to work well, chunks need to be small and focused. A small chunk contains a clean, specific semantic signal. If you make the chunk too large, the vector embedding becomes a muddy average of multiple topics, and search accuracy drops.
But for the generator (the LLM) to write a correct answer, it needs broad context. If the LLM only receives a tiny, isolated sentence, it misses the surrounding logic, the section headers, and the overall meaning. It ends up hallucinating or claiming it cannot find the answer.
To build a system that actually works in production, you have to break this link. You must separate the text you use for vector search from the text you pass to the LLM. You also need to fix user queries before they ever hit your database. Let us look at how to implement two patterns that solve these issues, parent-document retrieval and query rewriting.
The Mechanics of Parent-Document Retrieval
Instead of indexing the exact text blocks you plan to send to the LLM, you split your documents into two distinct layers.
First, you define parent documents. These can be entire pages, chapters, or large paragraphs of 1000 to 2000 tokens. These parent documents contain the complete context, details, and logical flow.
Next, you split each parent document into several smaller child documents, perhaps 100 to 200 tokens each.
You embed only the child documents and store their vectors in your database. Crucially, each child vector points to the ID of its parent document.
When a user query comes in, you search the child vectors. Because the child chunks are small and focused, the similarity search is highly accurate. It finds the exact sentence or paragraph relevant to the query.
But instead of passing that tiny child chunk to the LLM, you use the parent ID to fetch the entire parent document from a document store. You feed the parent document to the LLM.
This gives you the best of both worlds. You get the high precision of small-chunk vector search, and the rich context of large-chunk generation.
Here is a practical Python implementation using LangChain and an in-memory document store. You can easily swap these for production-grade databases like PostgreSQL or Pinecone.
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.storage import InMemoryStore
from langchain.retrievers import ParentDocumentRetriever
from langchain_text_splitters import RecursiveCharacterTextSplitter
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=50)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma(collection_name="split_parents", embedding_function=embeddings)
docstore = InMemoryStore()
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=docstore,
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)
docs = [
Document(page_content="Your long document text goes here...", metadata={"source": "doc1"})
]
retriever.add_documents(docs, ids=None)
query = "What is the specific metric used for evaluation?"
retrieved_docs = retriever.invoke(query)Behind the scenes, the retriever performs a similarity search on the child chunks. It collects the parent document IDs associated with the matching children, dedupes them, and fetches the full parent texts.
If you prefer not to use LangChain abstractions, you can build this manually. In your vector database schema, add a metadata field called parent_id. When you split and embed your documents, write the parent text to an external database (like Redis, MongoDB, or Postgres) with a unique key. Write the child chunks to your vector store, ensuring the parent_id metadata field matches the key in your document database.
During retrieval, run your vector search, extract the parent_id values from the top results, query your document store for those IDs, and pass the retrieved parent text to your LLM prompt.
Query Rewriting: Fixing the Input
Even with a perfect retrieval setup, your system will fail if the user writes a bad query.
Real users do not write search-optimized queries. They write things like, "Why did the numbers drop last quarter?" or "How do I fix that error we saw yesterday?"
These queries lack context. They contain pronouns, vague references, and conversational fluff. If you convert "Why did the numbers drop last quarter?" directly into a vector, the embedding will be dominated by words like "numbers" and "drop." The vector database will search for literal occurrences of those words, completely missing the actual documents containing the quarterly financial reports.
Query rewriting solves this by placing an LLM before the retrieval step. The LLM acts as a translator. It takes the user's messy query, analyzes the conversation history, and outputs a clean, search-optimized query.
Let us look at a simple query rewriter implementation.
import openai
def rewrite_query(user_query: str, chat_history: list) -> str:
prompt = f"""
You are an AI assistant helping optimize search queries for a vector database.
Your task is to rewrite the user's latest query to make it self-contained and search-friendly.
Guidelines:
- Replace pronouns (it, they, that, this) with the actual entities mentioned in the chat history.
- Strip out conversational phrases like "can you tell me" or "please explain".
- Keep the core intent of the user's question intact.
- Output only the rewritten query, nothing else.
Chat History:
{chat_history}
User Query: {user_query}
Optimized Query:"""
response = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0.0
)
return response.choices[0].message.content.strip()If the user previously talked about "the Q3 marketing campaign" and then asks "Why did it fail?", the rewriter outputs "Reasons for Q3 marketing campaign failure." This rewritten query contains the exact keywords needed to find the relevant child chunks in your vector database.
Query Expansion: Searching from Multiple Angles
Sometimes, a single query is not enough to find all the necessary information. The user might ask a complex question that requires pulling data from different parts of your documentation.
Query expansion addresses this by generating multiple variations of the user's query. You run vector searches for all these variations, combine the results, and deduplicate them before sending them to the LLM.
For example, if the query is "How do we secure our API?", the expansion step might generate:
- "API authentication and authorization best practices"
- "Securing API endpoints against unauthorized access"
- "API rate limiting and firewall configuration"
By searching for all three variations, you gather a much more complete set of context documents than you would with the original query alone.
Building the Combined Pipeline
To get the best results, you should chain these techniques together.
First, the user's raw query passes through the query rewriter.
Second, the optimized query goes through a query expansion step, generating three distinct search queries.
Third, you query the vector database using all three search queries.
Fourth, the vector database returns the top child chunks for each query.
Fifth, you collect the unique parent document IDs from all retrieved child chunks.
Sixth, you fetch the complete parent documents from your document store.
Seventh, you pass these parent documents to the LLM to generate the final response.
This pipeline adds some latency, but the improvement in context relevance is significant. You avoid the classic RAG failure modes where the LLM misses the point because it only received a fragment of a paragraph, or because the user used a pronoun that confused the vector search.
Managing Latency and Costs
Adding LLM steps before your retrieval phase naturally increases response times. To keep your application responsive, you need to optimize this pipeline.
Use smaller, faster models for query rewriting. A model like gpt-4o-mini or a fine-tuned open-source model like Llama 3 8B is perfect for this task. You do not need a massive model to rewrite a sentence.
Run your search queries in parallel. If you generate three expanded queries, send them to your vector database simultaneously using asynchronous calls.
Cache the results of your query rewriting step. If a user asks the exact same question, skip the LLM call entirely.
Also, keep an eye on your storage costs. Storing both child vectors and parent documents means you are keeping two copies of your data. The parent documents can live in a cheap, fast key-value store or a relational database, while only the child embeddings need to occupy expensive RAM in your vector database.
Evaluating the Improvements
Do not just take it on faith that these techniques are working. You need to measure the impact on your specific dataset.
Use evaluation frameworks like Ragas to track key metrics.
Look closely at context precision. This measures whether the retrieved context contains only relevant information. If you retrieve too much irrelevant parent text, your precision drops, which can clutter the LLM prompt and increase token costs.
Monitor context recall. This measures whether all the information needed to answer the question was successfully retrieved. If your child chunks are too small or your queries are not expanded enough, your recall will suffer.
By tracking these metrics as you tune your chunk sizes and query expansion prompts, you can find the sweet spot for your specific documents and user patterns.
