How RAG Works: From Document Indexing to Generated Answers

How documents become searchable context for a language model

A language model’s trained knowledge and an application’s source collection are separate. A model may contain broad general knowledge, but it does not automatically have access to an organization’s private documents or to material added after its training data was collected.

Retrieval-augmented generation (RAG) gives an application a way to retrieve relevant passages from a selected collection and include them in the model’s input for a particular request. The response is then generated from the application’s instructions, the user’s question, and the retrieved text.

A basic RAG system therefore has two connected paths. The indexing path prepares source material for search. The question-answering path retrieves context and supplies it to the model. The rest of this article traces both paths and shows how they combine to produce an answer.

The name describes three actions. Retrieval finds material related to a question. Augmentation adds selected material to the model’s input. Generation produces a response from the instructions, the question, and the retrieved context.

“Retrieve, then generate” captures the request-time flow, but it hides the preparation required beforehand. Before retrieval can happen, documents must be parsed, divided into smaller units, represented in searchable form, and stored in an index. That preparation is what makes the later search possible.

RAG does not permanently teach the model the retrieved material or update the model’s parameters. It supplies selected evidence for the current request.

Preparing documents for retrieval

Before users ask questions, the system prepares source material for search. It first collects the documents, records, or other approved sources the application is allowed to use. Parsers then extract usable text and relevant metadata, such as the source, title, document type, or access-related attributes.

Long documents are usually divided into smaller retrievable units called chunks. Each chunk is converted into a searchable representation. In a vector-based system, that representation is often an embedding: a numeric vector intended to capture aspects of meaning. The application stores the representation alongside the chunk text and metadata in an index.

That distinction matters. The embedding helps the system find a chunk, but the model normally needs the chunk’s readable text. When a query matches a stored representation, the application retrieves the associated text to build the model’s context.

This path is often called offline because it happens before a particular request, not because it necessarily runs only once. For now, the important idea is simple: a RAG system can retrieve only from material that has already been made searchable.

What happens when a question arrives

When a user submits a question, the application begins the second path. Depending on the retrieval method, it may convert the question into a representation compatible with the index. The retriever searches the indexed material and returns candidate chunks judged relevant to the query.

The application then selects and formats the context. It may include chunk text, source metadata, and instructions describing how the model should use the evidence. The final model input is therefore more than the user’s sentence: it commonly contains application instructions, the question, and retrieved passages.

A simplified model input combines application instructions, retrieved context, and the user’s question.

The language model generates a response from that input. If the product supports citations, the application can also expose links or source labels associated with the retrieved chunks.

This separation is important. Retrieval determines what evidence reaches the model; generation determines how the model interprets and expresses that evidence. A fluent answer can still be wrong if retrieval selected irrelevant or incomplete material. Conversely, useful evidence can be retrieved but misread or ignored during generation.

The model does not search the entire collection directly. It sees only the material the application selected for that request.

One question through the complete system

Consider a fictional company called Northstar Labs. Its employee handbook states that departing employees must return company laptops within five business days of their final working day.

During indexing, the handbook is parsed and divided into chunks. One chunk contains the equipment-return policy, while its metadata identifies the handbook and the relevant section. The system stores both the chunk text and its searchable representation.

Later, an employee asks, “How soon must I return my laptop after leaving?” The retriever selects the policy chunk. The application places that text beside the question and an instruction to answer from the supplied handbook. The model responds that the laptop must be returned within five business days and may point to the policy section.

Three different things occurred: the handbook contained the policy, retrieval selected the relevant passage, and the model turned that passage into an answer. If the wrong chunk had been selected, confident generation would not repair the missing evidence. This example illustrates the flow; it does not demonstrate the reliability of every RAG system.

What RAG changes—and what it does not guarantee

RAG changes the information available to an application at request time. Instead of relying only on knowledge represented in the model from training, the application can provide source material from an external collection. That collection can be maintained separately from the model, and retrieved passages can be connected to identifiable documents.

RAG does not by itself guarantee correctness. It does not guarantee that the right document entered the index, that retrieval found the best passage, or that the selected context was complete. It also does not guarantee that the model followed the evidence faithfully or that a citation supports every statement in the answer.

Consequently, knowing that a system uses RAG is not enough to assess answer quality. Quality depends on the source collection, document preparation, retrieval, context construction, generation, and presentation of the final response.

The complete flow

RAG involves two linked paths. The indexing path converts source documents into searchable chunks and stores the text, representations, and metadata needed for retrieval. The question-answering path uses a query to retrieve selected chunks, assemble the model input, and generate a response.

This complete flow determines the answer. The source collection controls what information is available, retrieval controls what reaches the model, and generation controls how that material is expressed. Examining all three is more informative than evaluating the language model alone.

The next article will focus on one of the first design choices in that pipeline: why RAG systems divide documents into chunks. We will look at what chunking changes mechanically and why document boundaries influence what a retriever can find—without yet trying to prescribe a universally “best” chunk size.