RAG Architecture (LLMs)

The Problem: Hallucination & Staleness

A language model's learned parameters do not automatically contain every recent event or private document.

1. Staleness: They don't know today's news.
2. Private Data: They don't know your company's internal wiki.
3. Hallucination: They confidently make up facts.

RAG supplies retrieved material before generation. It can improve grounding, but irrelevant retrieval, stale sources, and unsupported generation can still produce wrong answers.

The Architecture

1. Ingestion (Offline)

• Load: Read PDFs, Slack history, Notion.
• Split: Break text into chunks (e.g., 500 tokens).
• Embed: Convert text to vectors using an Embedding Model (OpenAI text-embedding-3).
• Store: Save vectors + text in a Vector DB (Pinecone, Milvus, pgvector).

2. Retrieval (Online)

• User asks: "What is the vacation policy?"
• Convert question to vector: [0.1, 0.5, -0.9...]
• Perform Semantic Search (Cosine Similarity) in Vector DB.
• Get top 3 chunks: "Policy 2024: 20 days off..."

3. Generation (Online)

• Prompt Engineering:
  text

  System: Answer using only the Context below.
  Context: "Policy 2024: 20 days off..."
  User: What is the vacation policy?
  

• The model proposes an answer; verify that its claims are supported by the retrieved passages and allow it to abstain when evidence is missing.

Vector Search Internals

Exact search compares a query with the candidate vectors. For large collections, Approximate Nearest Neighbor (ANN) algorithms trade some retrieval recall for lower search work; latency depends on the index, hardware, filters, and recall target.

HNSW (Hierarchical Navigable Small World)

Think of it like a skip-list for graphs.

• Data Structure: Multi-layered graph.
• Search: Start at top layer (sparse), drill down to bottom layer (dense).
• Behavior: Often efficient in practice, but logarithmic search is not a universal worst-case guarantee. Measure recall and latency with your own embeddings and filters.

Hybrid Search

Semantic search (Vectors) is great for concepts ("dog" matches "puppy"), but bad for keywords ("Error 504" might match "Error 404").

Solution: Combine Vector Search + Keyword Search (BM25).

• RRF (Reciprocal Rank Fusion) merges the two result sets.

Provider-independent retrieval flow

The following is pseudocode: the retrieval and generation functions are application-specific adapters, not a library API.

receive question and authenticated user
retrieve only documents that user is permitted to read
combine keyword and vector candidates; rerank if useful
keep source identifiers, document versions, and passage boundaries
if evidence is insufficient: ask for clarification or abstain
provide passages as untrusted evidence, separate from instructions
generate an answer with passage-level citations
check that cited passages support the answer's key claims

Do not retrieve every tenant's data and rely on the model to hide unauthorized passages. Apply access controls before passages enter model context. Documents can contain hostile instructions; treat their content as evidence, not as authority to use tools or disclose information.

Advanced Techniques

Corrective RAG (CRAG)

A retrieval score is not a calibrated probability of correctness. If evaluation shows weak evidence, an application can use a guarded fallback:

1. Fallback: Use Web Search (Google API).
2. Filter: LLM grades retrieved documents for relevance. Discard irrelevant ones.

Multi-Query Retrieval

User asks complex question?

1. LLM rewrites query into 3 sub-queries.
2. Execute all 3.
3. Deduplicate results.

Interview Tips 💡

• "Context Window Limit" — Even with long-context models, why retrieve selectively?
  • Cost: Processing 1M tokens per query is expensive ($$).
  • Latency: Takes seconds to process huge context.
  • Accuracy: "Lost in the Middle" phenomenon.
• "Chunking Strategy" — Too small? Missing context. Too big? Noise. Overlapping chunks helps.

Related Concepts

• Vector Databases
• Database Indexing (HNSW vs B-Tree)

Sources and a practical evaluation

The original RAG paper studies retrieval-augmented generation; it does not guarantee factual correctness. Build a test set containing answerable questions, missing answers, outdated policies, and unauthorized documents. Score passage recall and answer support separately: a fluent answer can hide a retrieval failure.