What Is Retrieval-Augmented Generation (RAG)?

What Is Retrieval-Augmented Generation (RAG)?

Suppose you ask an AI assistant, “Can I book the library meeting room for free, and for how long?” The answer depends on a policy that might have changed since the model was trained. Retrieval-augmented generation, or RAG, is a way to bring relevant documents into the answer when you ask the question.

Think of it as an open-book step added to a writing step: find passages, give them to the model, then generate an answer. The book can still be wrong, outdated, or badly searched. Let's use a tiny fictional library to see both the benefit and the failure points.

Three passages, one question

Imagine the library has these made-up records:

For the question above, A is the relevant current source. B talks about the same subject but is old; C is current but does not answer the question. A careful answer is: “Under the current fictional room policy, booking is free for up to two hours [A].” The citation is a pointer to A, which a person can open and verify.

The retrieval step is doing more than finding matching words. It must bring back the right version of the right document. A system that retrieves B because it says “meeting room” may confidently produce the wrong price and duration.

The chart below uses illustrative scores, not measured search results. B has more words in common with the question, so a simple search ranks it first. Filtering out archived policies before ranking lets A become the best eligible source. C stays out because it concerns printing.

What happens behind the answer

A common RAG application prepares documents for search, often splitting long files into smaller passages and storing them in an index. At question time it searches that index, selects useful passages, and puts them in the model's context with the question. The model then writes a reply. Search may use keywords, vector similarity, or both; RAG does not require one particular database. Microsoft's RAG architecture guide describes retrieval, generation, citations, and the importance of document preparation.

The model's weights do not automatically change because a document was retrieved. The passage is supplied as context for this answer. That is different from retraining the model, though teams may use both techniques in other projects. The 2020 RAG research paper by Lewis and colleagues combined a model's learned knowledge with retrieved external material; today's applications use varied architectures under the same broad name.

Build a tiny RAG request in Node.js

Here is the library example as an application flow. Before requests arrive, an administrator stores passages with source IDs and status metadata. For each question, the app filters out archived passages, searches the remaining text, sends the selected passages with the question to a model, and displays the answer with its sources. Azure AI Search's RAG overview describes the same separation between retrieval and answer generation, including query-time security filtering.

Install the official OpenAI JavaScript SDK, set OPENAI_API_KEY in your environment, save this as rag-demo.mjs, then run node rag-demo.mjs. The example calls the Responses API for generation; its search code is plain JavaScript so you can see the retrieval step.

npm install openai
node rag-demo.mjs

1import OpenAI from "openai";
2
3// These passages stand in for a prepared, searchable document index.
4const documents = [
5  { id: "A", status: "current", text: "Meeting rooms are free for up to two hours." },
6  { id: "B", status: "archived", text: "A meeting room costs $10 for up to three hours." },
7  { id: "C", status: "current", text: "Printing costs 10 cents per page." },
8];
9const question = "Is the meeting room free, and for how long?";
10
11// A deliberately simple keyword scorer for this three-passage demo.
12const words = (text) => new Set(
13  (text.toLowerCase().match(/\p{L}+/gu) ?? []).filter((word) => word.length > 3)
14);
15const queryWords = words(question);
16const hits = documents
17  .filter((doc) => doc.status === "current") // Exclude B before ranking.
18  .map((doc) => ({
19    ...doc,
20    score: [...queryWords].filter((word) => words(doc.text).has(word)).length,
21  }))
22  .filter((doc) => doc.score > 0)
23  .sort((a, b) => b.score - a.score)
24  .slice(0, 2);
25
26if (hits.length === 0) throw new Error("No supporting passage found");
27console.log("Retrieved:", hits.map(({ id, score }) => ({ id, score })));
28
29const sources = hits.map(({ id, text }) => `[${id}] ${text}`).join("\n");
30const response = await new OpenAI().responses.create({
31  model: "gpt-6-astra",
32  instructions: "Answer only from the supplied sources. Cite source IDs in brackets. If the sources do not answer the question, say you do not know.",
33  input: `Question: ${question}\nSources:\n${sources}`,
34});
35console.log(response.output_text);

For this question, the retrieval log prints [{ id: 'A', score: 2 }]: it excludes archived B and unrelated C. These token counts are separate from the illustrative 92/76/14 chart above. The generated wording may vary, so compare any claimed price or duration with A rather than assuming a citation makes it correct. This small scorer misses synonyms and variations such as “room” versus “rooms”; a production system also needs document updates, permission checks, retrieval tests, and citation validation.

A second example: a return-policy question

Imagine a store assistant asked, “Can I return headphones bought 20 days ago?” Its search finds a current general return policy allowing 30 days and a current opened electronics exception allowing only 14 days. A reply based on the general policy alone would be wrong if the headphones were opened. A useful answer would say: “The general window is 30 days, but opened electronics have a 14-day limit. Were the headphones opened?” It should point to both policy passages, then apply the relevant one after the customer clarifies. If the documents do not state whether headphones count as electronics, the assistant should say so instead of guessing.

This example shows why RAG is more than attaching a citation to a fluent sentence: the system must retrieve all relevant exceptions, notice missing facts, and avoid claiming that a source says more than it does.

Where RAG can still go wrong

Even a perfect link is not proof that the claim beside it is right. Microsoft's guide warns that incomplete or irrelevant retrieval can lead to inaccurate answers, and that access controls matter for indexed content. A retrieved page can also contain a prompt-injection attempt; RAG does not make outside text trustworthy.

When to use the pattern

RAG is useful when the answer depends on a document collection that changes, is too large to paste into every prompt, or needs a traceable source: internal manuals, product documentation, or policy pages. For a small stable fact, direct search and reading may be simpler. For a calculation, retrieval can supply inputs, but you still need to check the arithmetic.

The practical question is not “does it have RAG?” but did it find the right source, use it faithfully, and show enough evidence for me to check? Ask those questions when you evaluate a system. For the model underneath, read What Is an LLM?; for a repeatable check, see How to Test an AI Assistant.

Sources and further reading

• Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Microsoft Learn: RAG and indexes
• OWASP: Prompt injection risks in retrieved material