Retrieval Augmented Generation (RAGs)
An introduction to RAGs.
Foundation models are powerful, but they still make a very human kind of mistake that is: “they answer confidently when they do not have enough information.” Aka, what we call “Hallucination”
That is the core problem RAG solves.
First, let’s get something out of the way Retrieval-augmented generation (RAG) is not magic, that is to say it does not make a weak model “brilliant”, it just does something practical.
It gives the model the specific context it needs for a specific query, instead of just forcing it to rely on whatever it remembers from training. That simple shift changes a lot. Responses become more detailed. Hallucinations can drop entirely. User-specific and company-specific data become far more usable. And suddenly, a general-purpose model starts to behave like it actually knows your business inside out.
To me, the cleanest way to think about RAG is this: it is basically like doing feature engineering for foundation models. Classical ML systems needed carefully constructed features before they could make good predictions. Modern language models need carefully constructed context before they can generate good answers.
That sounds less glamorous than “AI agent” or “long-context reasoning.” But in production, it is often the difference between a demo and a system people can trust & buy.
What RAG actually is
A RAG system has two main parts.
The first is a retriever, which finds information relevant to the query. The second is a generator, which uses that retrieved information to produce the final answer.
The external memory source can be almost anything: internal documents, meeting notes, product manuals, previous chat history, an SQL database, or the public internet. The user asks a question, the retriever pulls the most relevant context, and the model answers using that context.
A model by itself only has its weights and the current prompt. A RAG system gives it access to fresh, query-specific knowledge. That matters because most real applications are not failing because the model lacks general intelligence. They fail because the model does not have the right facts at the right moment.
Why RAG still matters in the age of long context
A lot of people assume larger context windows will eventually make RAG unnecessary… Not true.
First, the amount of available data grows faster than the amount of context you can reasonably shove into a prompt. Even if a model can technically accept a very long context, that does not mean you should always give it one.
Second, models do not always use long context well. More tokens do not automatically mean more signal. In the real world, longer prompts can make the model focus on the wrong section, increase latency, and drive up cost. Every extra token has both a financial cost and an attention cost.
Third, many applications need different context for different users and different queries. If one user asks about printer specs and another asks about refund policy, they should not both drag around the same giant context blob. RAGs lets you construct context per query, which is much cleaner and much cheaper.
So the real competition is not “RAG versus long context.” It is “relevant context versus bloated context.” And relevant context wins more often than people expect.
The importance of the retriever
When people talk about RAGs, they usually just focus on the model. But the retriever is often the real bottleneck.
If the retriever finds weak context, the generator is boxed in. Even a strong model cannot answer well if it is handed the wrong documents.
There are two broad ways to retrieve information.
Term-based retrieval
This is the old-school approach. Search is based on matching terms in the query to terms in the documents. Systems like TF-IDF, BM25, and inverted indexes live here.
This approach is fast, mature, cheap, and still very useful, even today. It is especially strong when exact keywords matter. Product names, error codes, IDs, and weird strings are classic examples. If a user searches for something like:
PRODUCTID (99), you really do not want your retriever smoothing that into some vague semantic neighborhood.
Term-based retrieval is not flashy, but it works. There is a reason systems like Elasticsearch became so dominant…
Embedding-based retrieval
This is the semantic version. Instead of matching exact terms, you convert documents and queries into vector embeddings and retrieve the nearest neighbors in embedding space.
This is much better when the wording changes but the meaning stays the same. A user might ask “I can’t log in,” while the document is titled “How to reset your password.” Term matching can miss that. Embeddings often catch it.
But embeddings can also blur important keywords… That is the trade-off. They understand the underlying meaning better, but they are not always great at exact strings.
This is why many strong production RAG systems end up being hybrid systems. They combine term-based and embedding-based retrieval instead of pretending one method solves everything.
Sparse versus dense retrieval
Another useful distinction is sparse versus dense representations.
Term-based methods are usually sparse. Most entries in the vector are zero, and only the terms that appear matter. Embedding-based methods are usually dense, where every dimension carries some value.
Dense retrieval is more expressive, but sparse retrieval can be easier to interpret, cheaper to run, and more reliable for exact matching. This is one of those areas where the boring answer is often the right one: use the representation that matches your failure modes.
If your users constantly search by specific model numbers, dense retrieval alone is probably not enough. If they ask fuzzy semantic questions, pure keyword search will likely feel brittle.
A RAG system should be evaluated like a retrieval system
One mistake I see often is evaluating only the final answer while ignoring the retrieval step. By that point, it is too late.
A retriever has its own metrics, and they matter. Two of the most useful ones are:
Context precision: out of the retrieved documents, what percentage is actually relevant?
Context recall: out of all the relevant documents that exist, what percentage did you retrieve?
These two pull in different directions. A retriever can have high recall by bringing back a giant pile of documents, but then precision drops and the model has to sort through noise. Or it can be very precise but miss key evidence entirely.
If your RAG system feels inconsistent, do not just blame the model. Sometimes the answer quality problem is really a retrieval quality problem in disguise.
3 optimization tactics that make RAG better
Once the basic retriever works, three improvements tend to matter a lot.
1. Query rewriting
Users ask messy questions. Search systems prefer clean ones.
If a user asks, “How about Emily Doe?” after a previous question about John Doe, the retriever should not search that follow-up literally. It should rewrite it into the actual query: “When was the last time Emily Doe bought something from us?”
That sounds simple, but it matters enormously. A lot of retrieval errors come from the fact that user input is conversational while search works best on explicit intent.
2. Re-ranking
A cheap retriever can fetch a candidate set, then a more precise but more expensive model can rerank those candidates. This is often one of the best trade-offs in RAG.
You do not need the expensive model to look at everything. You just need it to sort the shortlist better.
Reranking is especially helpful when you want to reduce the number of chunks before passing them into the final model.
3. Contextual retrieval
Sometimes a chunk is hard to retrieve because, on its own, it lacks enough context. One useful trick is to augment each chunk with metadata: titles, summaries, tags, entities, keywords, or even example questions it can answer.
That makes retrieval much stronger.
A support article about password resets can be augmented with related phrasings like “I forgot my password,” “I can’t log in,” or “Help, I can’t access my account.” Suddenly, the retriever has multiple ways to find the same underlying answer.
This is one of those ideas that looks obvious after you see it, but it can materially improve recall.
RAG for SQL
A lot of RAG discussions act like external memory means text documents. That is far too narrow. RAG can also work with tables, images, and other structured sources.
A great example is text-to-SQL. Suppose a user asks, “How many units of Fruity Feddys were sold in the last 7 days?” That answer is not sitting in a paragraph somewhere. It lives inside a SQL table.
In that setting, the workflow changes:
Translate the natural language question into SQL.
Execute the SQL query.
Feed the result into the generator and produce the final answer.
This is still RAG. The model is still augmenting itself with external context before responding. The difference is that the context comes from a database query rather than document retrieval.
That matters because many real business problems are tabular. If your mental model of RAG is “PDF chatbot,” you are leaving a lot on the table.