Local RAG Lab

For the longest time, RAG felt like a black box to me.

I watched countless tutorials explaining embeddings, vector databases, chunking, retrieval, prompts, and generation. While I understood each concept individually, I still couldn't explain how they worked together inside a complete Retrieval-Augmented Generation pipeline.

Eventually, I stopped watching tutorials and decided to build the simplest possible RAG system from scratch.

This project wasn't intended to become a production-ready application. It was built as a learning exercise to understand every stage of a RAG pipeline by implementing each component myself, without relying on frameworks that abstract away the complexity.

You can explore the complete source code on GitHub.

The Goal

The objective was straightforward.

Can a fully local RAG system running entirely on consumer hardware answer questions over private documents without relying on cloud APIs or heavyweight orchestration frameworks?

To answer that question, I intentionally kept the stack as minimal as possible.

The project uses:

• Python
• Ollama
• Mistral 7B
• nomic-embed-text
• ChromaDB
• pypdf

No LangChain, LlamaIndex, Haystack, CrewAI, or cloud APIs were used.

System Architecture

I wanted every stage of the pipeline to be explicit and easy to understand.

PDF Documents
      │
      ▼
Text Extraction
      │
      ▼
Chunking
      │
      ▼
Embedding Generation
      │
      ▼
ChromaDB
      │
      ▼
Question
      │
      ▼
Semantic Retrieval
      │
      ▼
Prompt Construction
      │
      ▼
Mistral 7B
      │
      ▼
Answer

Rather than hiding complexity behind frameworks, every transformation is visible. From extracting text to retrieving context and generating the final answer, each stage has a single responsibility.

Building the Pipeline

The application consists of two independent workflows: indexing and question answering.

Indexing

During indexing, every PDF is processed locally. The text is extracted using pypdf, divided into smaller chunks, converted into embeddings using nomic-embed-text, and stored inside a local ChromaDB database.

This step only needs to run when new documents are added.

Question Answering

When a question is asked, the same embedding model converts the query into a vector. ChromaDB retrieves the most semantically similar chunks, which are then inserted into a prompt before being sent to Mistral 7B running locally through Ollama.

The language model never reads the original PDF directly. It only answers using the retrieved context.

Experiments

To evaluate the pipeline, I indexed two different documents.

The first was a sample PDF containing refund policies, warranty information, customer support details, and product inventory.

The second was my personal resume.

This allowed me to test both factual document retrieval and personalized question answering using the same pipeline.

Some example questions included:

• What is the refund period?
• How long is the battery warranty?
• Which laptop has 32 GB RAM?
• Where does Srivatsav currently work?
• What programming languages are mentioned?
• What impact did Srivatsav deliver in the recommerce platform?

Biggest Learning

Before building this project, I assumed the language model would have the biggest impact on answer quality.

I was wrong. The biggest improvement came from redesigning the chunking strategy.

My initial implementation simply split extracted text using a fixed character count. While this was easy to implement, it frequently mixed unrelated sections together, broke sentences apart, and produced noisy retrieval results.

After rewriting the chunker to preserve paragraph boundaries, merge related paragraphs, and fall back to sentence-based splitting only when necessary, retrieval quality improved significantly without changing the language model.

That experiment completely changed how I think about RAG systems.

What Worked Well

The experiment successfully demonstrated that a complete Retrieval-Augmented Generation pipeline can run entirely on modest consumer hardware.

Some of the observations that stood out were:

• Indexing completed successfully using local embeddings.
• ChromaDB consistently retrieved relevant chunks.
• Mistral generated grounded responses when relevant context was available.
• Hallucinations were significantly reduced by restricting answers to the retrieved context.

Perhaps the biggest takeaway was that retrieval quality influenced answer quality far more than the language model itself.

Current Limitations

Although the system performs well as a learning project, there are several areas that can still be improved.

• Retrieved chunks occasionally contain unrelated context.
• Similarity filtering should be investigated.
• Prompt construction could better instruct the model to ignore irrelevant information.
• PDF extraction loses some structural information such as headings and tables.

Most of the remaining issues originate from retrieval rather than generation.

Future Work

The next improvements I plan to explore are:

• Similarity threshold based retrieval
• Better prompt engineering
• Benchmarking different chunk sizes and overlap values
• Evaluating alternative local models such as Qwen, Gemma, and Llama
• Adding support for TXT, DOCX, JSON, and CSV documents

This project achieved exactly what I set out to accomplish. More than building a RAG pipeline, it helped me understand why each stage exists and how seemingly small implementation details, especially chunking and retrieval, can significantly influence the quality of the final answer.

Repository

The complete implementation, along with setup instructions and source code, is available on GitHub.