I Didn't Understand RAG Until I Built One

Over the past year, Retrieval-Augmented Generation, better known as RAG, has become one of the most commonly discussed concepts in Generative AI. Every tutorial explains the same pipeline: load documents, generate embeddings, store them in a vector database, retrieve relevant chunks, and send them to a language model. Individually, each step seems straightforward. Together, however, the entire system often feels like a black box.

That was exactly my experience.

I watched numerous videos explaining embeddings, vector databases, chunking strategies, semantic search, and prompt engineering. I could explain each component in isolation, but if someone had asked me how they worked together inside a complete system, I wouldn't have had a satisfying answer. Everything made sense in theory, yet I still lacked an intuitive understanding of why every stage existed.

Eventually, I stopped watching tutorials and decided to build the simplest possible RAG pipeline I could run locally. My goal wasn't to build a production-ready application or compete with enterprise frameworks. I simply wanted to understand the moving parts by implementing them myself.

That decision turned out to teach me far more than any tutorial I had watched.

What Problem Does RAG Actually Solve?

Language models are incredibly good at generating text, but they have an important limitation. Their knowledge comes from the data they were trained on. They cannot automatically access your documents, company knowledge base, meeting notes, PDFs, or personal files unless that information is provided during inference.

Imagine uploading your company's documentation and asking a language model questions about internal policies. Without additional context, the model has never seen those documents. Even if it generates an answer, there's no guarantee that answer is grounded in your data.

This is the problem RAG was designed to solve.

Instead of expecting the language model to remember everything, RAG retrieves only the pieces of information that are relevant to the user's question and provides them as context before generating a response. The language model is no longer relying solely on what it learned during training. It is reasoning over information retrieved specifically for that question.

Why Can't We Just Give The Entire Document?

This was one of the first questions I had.

If I have a PDF, why not simply send the entire document to the language model?

The answer becomes obvious once documents become larger.

A single technical manual, research paper, or employee handbook can easily contain tens or hundreds of pages. Feeding everything into the prompt would quickly exceed the model's context window, increase inference costs, and make it harder for the model to focus on the information that actually matters.

Instead, RAG retrieves only a handful of relevant sections. Rather than reading an entire book, the model is given only the few paragraphs most likely to answer the question.

That simple idea is what makes RAG practical.

Understanding The Pipeline

When I finally built the system myself, the pipeline became surprisingly easy to understand.

It looks something like this:

PDF Documents
      │
      ▼
Text Extraction
      │
      ▼
Chunking
      │
      ▼
Embedding Generation
      │
      ▼
Vector Database
      │
      ▼
Question
      │
      ▼
Semantic Retrieval
      │
      ▼
Prompt Construction
      │
      ▼
Language Model
      │
      ▼
Answer

Each stage has a single responsibility.

The documents are first converted into plain text. That text is then divided into smaller chunks that can later be retrieved independently. Each chunk is converted into a numerical embedding representing its semantic meaning and stored inside a vector database.

Later, when a user asks a question, the same embedding process is applied to the query itself. The vector database searches for chunks that are semantically similar to the question, and only those retrieved chunks are inserted into the prompt that is finally sent to the language model.

Building every stage myself made this flow far easier to understand than any diagram ever had.

The Biggest Surprise Wasn't The Language Model

Before starting the project, I assumed that the language model would have the greatest influence on answer quality.

I was wrong.

The biggest improvements came from the retrieval pipeline.

My initial implementation used a very simple chunking strategy. Text was extracted from the PDF, flattened into a single stream, and divided using a fixed character count. Although the implementation worked, unrelated topics frequently ended up inside the same chunk. Retrieval became noisy because a single chunk could contain information about completely different sections of the document.

After redesigning the chunker to preserve paragraph boundaries and merge related content more intelligently, answer quality improved immediately.

Nothing else changed.

• The embedding model stayed the same.

• The language model stayed the same.

• Only the chunking strategy changed.

That experiment completely changed my understanding of where the real complexity in RAG lies.

Retrieval Matters More Than I Expected

As I continued experimenting, another pattern became obvious.

Whenever the retriever selected relevant chunks, Mistral 7B usually produced accurate and grounded answers.

Whenever retrieval included unrelated context, answer quality dropped noticeably.

One particularly interesting example came from my resume.

When I asked:

"Where does Srivatsav currently work?"

The pipeline correctly answered:

Cognizant.

However, changing the wording slightly to:

"Where does Srivatsav work?"

sometimes caused retrieval to select different chunks, leading the model to respond that the information could not be determined from the provided context.

• The language model hadn't suddenly become less capable.

• The retrieved context had changed.

• That was a much bigger lesson than I expected.

Building It Changed How I Think About RAG

Before this project, I thought of RAG primarily as a way of connecting documents to language models.

After building one, I started thinking about it very differently.

A RAG system is really a retrieval problem.

The language model only answers using the information it receives. If retrieval finds the wrong context, even an excellent language model will struggle to produce the correct answer. Conversely, when retrieval works well, surprisingly small local models can generate high-quality responses.

That completely shifted where I think optimization effort should be spent.

Final Thoughts

Building this project didn't just teach me how to implement a RAG pipeline. It gave me a much clearer mental model of why each stage exists and how they work together. Concepts like embeddings, vector databases, and semantic retrieval stopped feeling like isolated buzzwords and became parts of a system that I could reason about end to end.

More importantly, it reminded me that sometimes the fastest way to understand a concept isn't by consuming another tutorial. It's by building the simplest possible version yourself, making mistakes, observing unexpected behavior, and gradually refining your understanding through experimentation.

If you'd like to explore the implementation itself, I published a detailed Local RAG Lab covering the architecture, implementation, experiments, and lessons learned while building the system from scratch.