What is RAG (retrieval-augmented generation)?
By Yassine Seghrouchni, founderUpdated
In short
RAG (retrieval-augmented generation) is a technique in which an AI assistant first looks up the relevant passages in your own documents or systems, and only then answers based on what it found. The language model doesn’t have to draw on its memory. As a result, answers are more up to date, more often correct, and you can show the source for each one.
Also known as: retrieval-augmented generation, retrieval augmented generation
How it works
The name describes the three steps:
- Retrieval: the question first goes to a search function over your documents, such as handbooks, contracts, collective agreements or your customer service knowledge base. It finds the passages that are relevant.
- Augmentation: the passages found go to the language model together with the question, with the instruction to answer only on that basis.
- Generation: the model writes the answer and refers to the passage it is based on.
To search well, documents are split into chunks beforehand and stored in a way that lets the search function also find passages that cover the same thing in different words. A question about holiday allowance then also finds the passage about holiday pay.
RAG or training your own model?
People who want an AI assistant on their own knowledge often assume the model needs to be trained for it. For questions about facts and rules, RAG is usually the better choice:
- Up to date: if a document changes, you replace it in the search index. No retraining needed.
- Verifiable: you can see which passage an answer rests on, and check it.
- Permissions: the assistant can search within what the person asking is allowed to see.
Further training a model, fine-tuning, is mainly useful for teaching it a style or fixed behaviour, such as a particular way of writing or a set structure for answers. For knowledge that changes, it’s a detour.
Why it matters in practice
The quality of a RAG assistant depends more on the sources and the search than on the model. If the index contains outdated or contradictory documents, you get a convincing wrong answer, complete with a source. So tidy up the sources first and assign someone to keep them current.
A common misconception: that RAG rules out hallucinations. It greatly reduces the chance, but a model can still misread a passage or combine two sources wrongly. So also test with questions whose answer isn’t in the documents. A good assistant then says it doesn’t know, instead of guessing.
That’s how we built it for an organisation in the staffing industry: an assistant that pre-fills a form from the collective agreement and the company’s own policies. It only answers if it finds the answer in the documents provided, and then shows the exact excerpt with the file name and page. If it finds nothing, the field stays empty.
Frequently asked questions
What is the difference between RAG and fine-tuning?
With RAG, the assistant looks up the relevant passages in your documents for every question and answers on that basis. With fine-tuning, you train the model itself further, so it picks up different behaviour or a different style. For knowledge that changes, such as policies, prices or procedures, RAG is usually the better fit: you update the document instead of the model.
Does RAG prevent hallucinations?
It greatly reduces the chance, because the model answers from your own sources instead of from memory. It doesn’t rule them out. Have the assistant show the source with every answer and give no answer when it finds nothing. Then a member of staff can quickly check whether it’s right.
