The Problem: Hallucination & Staleness

LLMs (GPT-4, Claude) are frozen in time.

1. Staleness: They don't know today's news.
2. Private Data: They don't know your company's internal wiki.
3. Hallucination: They confidently make up facts.

RAG solves this by fetching relevant data before meaningful generation.

The Architecture

1. Ingestion (Offline)

• Load: Read PDFs, Slack history, Notion.
• Split: Break text into chunks (e.g., 500 tokens).
• Embed: Convert text to vectors using an Embedding Model (OpenAI text-embedding-3).
• Store: Save vectors + text in a Vector DB (Pinecone, Milvus, pgvector).

2. Retrieval (Online)

• User asks: "What is the vacation policy?"
• Convert question to vector: [0.1, 0.5, -0.9...]
• Perform Semantic Search (Cosine Similarity) in Vector DB.
• Get top 3 chunks: "Policy 2024: 20 days off..."

3. Generation (Online)

• Prompt Engineering:
  text

  System: Answer using only the Context below.
  Context: "Policy 2024: 20 days off..."
  User: What is the vacation policy?
  

• LLM generates accurate answer grounded in fact.

Vector Search Internals

How do we search 1 billion vectors in milliseconds? We can't compare every vector ($O(N)$). We use Approximate Nearest Neighbor (ANN) algorithms.

HNSW (Hierarchical Navigable Small World)

Think of it like a skip-list for graphs.

• Data Structure: Multi-layered graph.
• Search: Start at top layer (sparse), drill down to bottom layer (dense).
• Complexity: $O(\log N)$.

Hybrid Search

Semantic search (Vectors) is great for concepts ("dog" matches "puppy"), but bad for keywords ("Error 504" might match "Error 404").

Solution: Combine Vector Search + Keyword Search (BM25).

• RRF (Reciprocal Rank Fusion) merges the two result sets.

Code Example: Simple RAG pipeline

import os
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.chat_models import ChatOpenAI
from langchain.prompts import PromptTemplate

# 1. Setup Retrieval
embeddings = OpenAIEmbeddings()
db = Chroma(persist_directory="./db", embedding_function=embeddings)
retriever = db.as_retriever(search_kwargs={"k": 3})

# 2. Define Chain
llm = ChatOpenAI(model_name="gpt-3.5-turbo")
prompt = PromptTemplate.from_template(
    "Context: {context}\n\nQuestion: {question}\nAnswer:"
)

def ask(question):
    # A. Retrieve
    docs = retriever.get_relevant_documents(question)
    context_text = "\n\n".join([d.page_content for d in docs])
    
    # B. Augment
    final_prompt = prompt.format(context=context_text, question=question)
    
    # C. Generate
    return llm.predict(final_prompt)

# print(ask("How do I reset my password?"))

Advanced Techniques

Corrective RAG (CRAG)

If vector search returns low confidence score:

1. Fallback: Use Web Search (Google API).
2. Filter: LLM grades retrieved documents for relevance. Discard irrelevant ones.

Multi-Query Retrieval

User asks complex question?

1. LLM rewrites query into 3 sub-queries.
2. Execute all 3.
3. Deduplicate results.

Chunking: The Underrated Make-or-Break Decision

Retrieval quality is decided at ingestion time, long before any query arrives. The chunking questions that matter:

• Size: 200–500 tokens is the common sweet spot. Smaller chunks embed more precisely (one idea per vector) but lose surrounding context; bigger chunks dilute the embedding across many topics, so queries match them weakly.
• Boundaries beat sizes: splitting mid-sentence or mid-table destroys meaning. Structure-aware splitters (by heading, paragraph, or Markdown/HTML section) consistently outperform fixed-size windows. For code, split by function/class.
• Overlap (10–20%): ensures a fact straddling a boundary lands whole in at least one chunk.
• Contextual enrichment: a chunk saying "It increased by 40%" is unanswerable in isolation. Prepending a generated one-line context ("From: Q3 finance report, section on cloud spend...") before embedding dramatically improves retrieval of pronoun-heavy text — this is the idea behind contextual retrieval techniques.

The meta-lesson: garbage chunks, garbage answers. Teams debugging "the LLM ignored the right document" usually discover the right document was chunked so poorly it never ranked.

Evaluation: How Do You Know Your RAG Works?

A RAG system has two failure points, and they need separate measurement:

1. Retrieval metrics — did the right chunks come back?
   • Recall@k: is the gold-answer chunk in the top k?
   • MRR (Mean Reciprocal Rank): how high does it rank? Build a test set of (question → known source passage) pairs; even 50 pairs catches most regressions.
2. Generation metrics — did the model use them honestly?
   • Faithfulness/groundedness: is every claim in the answer supported by the retrieved context? (Judged by a second LLM pass.)
   • Answer relevance: does it actually address the question?

Measure separately, because the fixes differ: poor recall → better chunking, hybrid search, or query rewriting; poor faithfulness → tighter prompting, citations-required output format, or a stronger generator model. Frameworks like RAGAS package these metrics, but the discipline matters more than the tooling: evaluate on every change to chunking, embeddings, or prompts — RAG systems regress silently.

Production Concerns Beyond the Demo

• Freshness & incremental sync: documents change. Re-embedding the whole corpus nightly is wasteful; production pipelines hash content per chunk and re-embed only what changed, deleting vectors for removed documents (orphaned vectors are a classic source of confidently outdated answers).
• Access control: the vector store must respect document permissions. Filter at retrieval time by the user's ACLs (metadata filtering), never after generation — an LLM that has already read a confidential chunk can leak it in a paraphrase.
• Latency budget: a typical breakdown — query embedding ~20ms, ANN search ~10–50ms, reranking ~50–200ms, generation 1–5s. The LLM dominates; this is why retrieval sophistication (rerankers, multi-query) is usually "free" from the user's perspective, and why streaming the answer matters more than shaving retrieval milliseconds.
• Reranking: ANN search optimizes for speed, not final precision. A cross-encoder reranker scoring the top-50 candidates and keeping the best 5 typically adds more answer quality per millisecond than any other single component.
• When fine-tuning instead? RAG injects knowledge; fine-tuning shapes behavior (format, tone, domain vocabulary). They compose: fine-tune for style and citation discipline, RAG for facts. Fine-tuning alone is the wrong tool for factual grounding — the model still can't cite sources or reflect yesterday's update.

Interview Tips 💡

• "Context Window Limit" — LLMs can process 128k+ tokens now, so why RAG?
  • Cost: Processing 1M tokens per query is expensive ($$).
  • Latency: Takes seconds to process huge context.
  • Accuracy: "Lost in the Middle" phenomenon.
• "Chunking Strategy" — Too small? Missing context. Too big? Noise. Overlapping chunks helps.
• "How do you evaluate it?" — Separate retrieval (recall@k on a labeled set) from generation (faithfulness judged by an LLM). Interviewers reward candidates who treat RAG as a measurable IR system, not vibes.
• "Security?" — Permission-filtered retrieval; prompt-injection defense for content ingested from untrusted sources (a malicious document can carry instructions to the model).

Related Concepts

• Vector Databases
• Database Indexing (HNSW vs B-Tree)