How RAG works, in plain terms
RAG splits the job in two:
- Retrieval. When someone asks a question, the system searches your own documents and pulls out the few passages most likely to contain the answer.
- Generation. The language model receives the question together with those passages and is instructed to answer only from them, citing where each statement comes from.
The model does not "learn" your documents. They are not used to train it. They are looked up at the moment of the question, which is why RAG can work with documents that change every day and why access rules can be enforced on every request.
The parts that decide quality
Ingestion: turning documents into searchable passages
Documents are converted to text, split into passages ("chunks") and indexed. This step is unglamorous and decisive:
- Scans need OCR, and tables need to survive the conversion.
- Chunks should follow the structure of the document — articles of a contract, sections of a procedure — not an arbitrary number of characters.
- Every chunk keeps its metadata: source file, page, date, version, and who is allowed to see it.
Retrieval: finding the right passages
Most systems combine two kinds of search. Semantic search uses embeddings — numeric representations of meaning — to find passages that say the same thing in different words. Keyword search finds exact terms such as product codes, article numbers or names, where semantic search can be vague. A re-ranking step then orders the candidates. Getting retrieval right matters more than the choice of model: if the right passage is not retrieved, no model can cite it.
Generation: answering with citations
The prompt instructs the model to answer from the retrieved passages only, to cite each source, and to say so when the passages do not contain the answer. "I could not find this in the documents available to you" is a feature, not a failure.
Permissions: answers must respect access rights
If a user cannot open a document in the repository, the assistant must not quote it either. The reliable way is to filter retrieved passages by the user's permissions, read from the company directory, on every query — not once at indexing time.
Evaluating a RAG system
A RAG system should be tested like any other software, with a fixed set of questions whose correct answers and sources are known. For each question, check:
- Retrieval: was the right passage among those retrieved?
- Faithfulness: is every statement in the answer supported by a cited passage?
- Completeness: did the answer cover what the question asked?
- Refusal: when the answer is not in the documents, did the system say so?
Fifty to a hundred representative questions, written with the people who will use the system, are enough to compare versions and catch regressions when documents, prompts or models change.
Common mistakes
- Indexing everything at once. Start with one well-curated collection — the procedures of one department, the contracts of one business line — and expand.
- Ignoring document versions. Two versions of the same policy in the index produce contradictory answers. Retire superseded documents.
- No feedback loop. Let users flag wrong or incomplete answers, and review those flags.
- Treating the model as the product. The model can be swapped. The ingestion pipeline, the permissions model and the evaluation set are what you build once and keep.
Where RAG fits
RAG is the right tool when people need answers from a body of documents that is too large to read and changes over time: policies and procedures, technical documentation, contract archives, case files, support knowledge. It is also the knowledge layer behind AI agents that prepare cases in your ERP.
It needs two things to be in place: documents that are reasonably clean and current — often the first task of a data platform project — and a clear decision about where the data is processed. When confidential documents may not leave the company, the whole system can run on private, on-premise models.
Where to start
Pick one collection of documents and one group of users with a real, recurring need. Write the evaluation questions with them before building anything. Our applied AI service runs exactly this kind of proof of concept, on your documents and with your access rules, at a fixed price.