A RAG prototype can be built in a few lines of code. A production-grade RAG system is a very different engineering problem.

Architecture

As discussed in above blog, the architecture of modern RAG systems looks something like this -

However, many critical nuances are hidden in this simplified diagram, and these details can directly impact system quality, accuracy and performance. To understand these trade-offs, let's look at a much deeper view of Production Grade RAG Systems.

Below is a comprehensive RAG architecture which covers most of the components generally used in Production use cases -

The most critical components in the above architecture which directly impact response relevancy and system performance are -

1. Chunking

2. Indexing

3. Search

Let's understand the challenges and importance of each as well as the techniques used to improve overall system performance.

Note: This article mostly focusses on Dense Retrieval, while Sparse Retrieval is extensively covered in previous article.

Query Re-writing (Optional)

Users interact with LLMs in a very naive/human manner. It can happen that the prompts entered by them can be very verbose and only a small part of the prompt is actually relevant enough to generate the right response.

For e.g. "I went to the garden with my baby and he played a lot with his friends. They played football, played on swings and ran a lot. He is generally fussy about eating. Later we went to the mall and played there as well. Now, he is tired and I don't know what to cook for him quickly".

Such a prompt, if sent for retrieval, will not only cost a lot but will also have higher probability of returning irrelevant results since the response should essentially be about quick, healthy food for the baby and not about games, playing etc.

Therefore, it is sometimes important to use another LLM to first re-write the query before submitting it to the retriever so that things like database search are optimized and high response relevancy is maintained.

Some useful techniques include -

1. Named Entity Recognition (NER) - NER Models can easily identify entities in prompts like person, locations, time, objects etc., which can then be passed as structural signals to the LLM to re-write the prompt. For example, GLINER supports zero-shot NER and can be used when entity extraction is required.

2. Hypothetical Document Embeddings (HYDE) - In this technique, we first generate hypothetical documents which would ideally answer the query. Then we generate its embedding and finally send that embedding to the retriever for finding relevant documents.

Chunking

Chunking is the process of splitting a document into smaller, semantically meaningful units that can be independently embedded and retrieved. It also helps keep each unit within the embedding model's input limits.

Now, if we create too big chunks then there will be too many information within a single chunk. If Embedding Model's context window is small, then a significant part of the chunk can be lost, otherwise the embedding will be too generic since a lot of information will be there in the chunk.

If we create too small chunks, we run the risk of losing surrounding context for each chunk and hence impacting relevance accuracy since few relevant documents might come out as irrelevant because of disjoint chunks.

In short, Chunk Size affects -

1. Retrieval Granularity

2. Embedding Quality

3. Context Preservation

4. Number of retrieved chunks

5. Downstream Context/Token Cost

Hence, right chunking strategy is critical to overall RAG accuracy !

Following are some of the possible chunking strategies which can be selected depending upon the use case -

Fixed Size Chunking

Generally done at a word level or character level, this strategy provides fixed size chunks, even if deriving proper meaning of a text required multiple words within the same chunk.

Tip: Retrieval Quality can be significantly improved with Fixed Size Chunking by adding an overlap of 'n' characters between 2 consecutive chunks, to each chunk. However, optimal overlap depends on the document structure and should be determined through empirical validations.

Recursive Character Text Splitting

This strategy recursively splits text using a hierarchy of separators—for example, paragraphs, newlines and spaces—while attempting to keep related content together. This produces variable-sized chunks and often preserves semantic structure better than blindly splitting at a fixed character count. The only downside is that it can result in too many chunks and also some of the chunks can be small and irrelevant when viewed in isolation.

In most of the practical cases, Fixed Size & Recursive Text Splitting are both used in conjunction. For e.g. texts like heading and follow-up paragraph are first split by recursive character splitting but then headings are merged because of fixed size splitting, thereby, making them more meaningful.

There are also more advanced chunking strategies which help improve accuracy significantly but come with higher cost as well

Semantic Chunking

1. Chunk at sentence level and create vector for each chunk

2. Calculate cosine distance between 2 consecutive chunks

3. If cos-distance > threshold, merge chunks to create bigger chunk

LLM Based Chunking

Use another LLM in the RAG pipeline to create chunks of the source document by including instructions like 'keep concepts together', 'add breaks when new topic starts' etc.

Context-Aware Chunking

Use another LLM to add additional context to every single chunk (e.g. summary text) and re-create the chunk. This strategy is flexible enough to be applied on top of any other previous strategies.

Based on my experience, here is a quick suggestion on when to apply which strategy -

Indexing & Search

Once we have document embeddings (N) and incoming query's embedding (Q), we can find K-Nearest Neighbors for Q using any similarity measure but this will have a high complexity of O(N*D), where

N = number of documents & D = dimension size

In Production Systems, N can run into millions & D can be in thousands, hence, such an exact KNN search approach becomes infeasible.

Since, the way we store our vectors (Indexing) hugely impacts the subsequent search operation (Search), most of the Vector DB providers today provide integrated Indexing & Search functionality.

Inverted Files (IVF)

This solves the problem of 'too many vectors to check' by -

1. Grouping documents into multiple groups (nlist) using clustering algorithms like K-Means and each group's centroid vector is calculated

2. Each group is then organized as Inverted Lists (remember BM25!) labelled by group's centroid vector

3. For given Q, we first run similarity search with all the centroid vectors

4. Then, find most similar documents only from within the groups whose centroid vectors were the closest.

This can dramatically reduce the number of database vectors that need to be compared, trading some recall for lower search cost.

Product Quantization (PQ)

In this approach, original vector is partitioned into 'm' sub-vectors/sub-spaces and a separate codebook is learned for each sub-space independently. Each sub-vector is then represented by the ID of its nearest codeword, significantly reducing the memory required to store the vector and enabling efficient approximate distance computation. Thus, basically providing a compression technique to efficiently run 'semantic search over large vectors'.

Q is also then partitioned into sub-vectors and cosine distance is calculated across centroids for each sub-vector (m*K combinations). Documents (represented by list of centroid-ids) are fetched based on the shortest cumulative distance across Q's sub-vectors by matching centroid-ids.

IVF+PQ

The more popular technique here is the one which brings the best of both worlds - first grouping document vectors using K-Means & finding top-k centroid vectors. Then, residual vectors are computed (original - centroid) and partitioning is done on these residual vectors.

Tip: With PQ based approaches, we end up effectively doing an Approximate Nearest Neighbor (ANN) Search instead of Exact Search. For most practical purposes, ANN Search is good enough especially when dealing with millions of documents and latency is critical. Exact Search based techniques are generally reserved for Post-Retrieval/Re-Ranking.

Navigable Small Worlds (Hierarchical - HNSW)

Most Vector DBs today implement a different strategy for indexing & search which leverages the flexibility of graphs. In NSW, a graph is built connecting close vectors (nodes) with each other but limiting the number of connections -- every node is connected to maximum 'k' other nodes, based on cost vs accuracy analysis.

In Hierarchical NSW, multiple graphs are built (imagine vertically), each connected with another graph below it using common nodes. Number of neighbors of each node increase as we go towards lower graphs. A node present in upper graph will always be present in lower graphs.

Search starts by finding the most similar node in top most graph and its similar neighbors are added to relevant documents list as we traverse downwards, with each similar document becoming the parent node for next search. This enables highly efficient approximate nearest-neighbor search with approximately logarithmic scaling in many practical settings.

Choosing an Index

Here is a comparative analysis of HNSW and IVF (+PQ):

As you can see, there is no universal winner here.

Tip: My personal experience tells me that while the right choice depends on the required recall, latency, memory budget, update pattern and filtering requirements, however, if accuracy and latency are paramount, then HNSW is an attractive default, otherwise if Cost is critical and marginally lower accuracy and latency is acceptable, then IVF+PQ is particularly attractive.

System Optimization

While Data Scientists remain mostly concerned about getting the architecture right as it will directly impact accuracy and response relevancy, AI engineers need to also look at implementing an optimized system with considerable focus on cost optimization - both memory and compute, as well as latency (response time).

Following are some of the best techniques to explore in finding the right balance of accuracy, cost and latency -

Cost Optimization

1. Smaller models for retrieval lead to less memory footprint, less compute cost and better latency, but suffer from lower accuracy. Bigger models can be saved for Re-ranking

2. If, however, smaller models is not an option, Model Quantization generally incurs only marginal drop in accuracy for most use cases while reducing compute cost significantly. Model quantization represents model weights and/or activations using lower-precision numerical formats, such as INT8 or lower-bit formats instead of FP16/FP32.

3. In addition to compressing the model, Vector Quantization can also be done to reduce compute cost -

   1. Use lightweight embeddings, possibly generated using a smaller embedding models, for retrieval and full-size embeddings (from bigger models) for re-ranking

   2. Matryoshka Embedding Learning - Some embedding models are trained so that useful information is preserved in progressively smaller prefixes of the embedding vector. This allows applications to use a lower-dimensional representation for fast retrieval and the full representation when higher accuracy is required.

4. Smaller, optimized prompts can help save a lot on token/compute costs at response generation time. Some techniques to include -

   1. Retrieve less number of top-k documents

   2. Output tokens are commonly charged higher than input tokens, so set a limit on maximum number of output tokens

   3. Include system prompts to encourage LLM to generate shorter responses

5. Memory Cost - Production systems typically need persistent storage and an index optimized for vector retrieval. While this can be provided by a dedicated vector database or by an existing database/search platform with vector capabilities, these are essential for scalability, redundancy, failure handling etc. So using the right memory for right purpose often leads to significant cost savings. One such option can be -

   1. RAM (Faster, Expensive) - Ideal for HSNW Indexes for fast retrieval

   2. Disk (Slower, Cheaper) - Mostly for infrequently accessed vectors

   3. Cloud Object (Slowest, Cheapest) - Ideal for storing Raw Documents

Latency

Defined as Response Time or Turnaround Time taken by an AI system to generate response for an input query, Latency is one of the most important system performance metric which is used in AI Systems across domains. Higher latency leads to end user frustration and depletion of trust in system usability.

In order to optimize overall latency of a RAG System, it is important to breakdown and measure latency of each individual component, in other words, measure & optimize separately -

Also, most of the techniques mentioned for cost optimization also help directly in improving latency of the system like smaller models, optimized prompts, quantization etc.

There are also some additional latency specific optimization techniques which may prove to be beneficial for certain large-scale use cases -

1. Using a small Router LLM to first determine if Retrieval is required or is query can be answered directly by the LLM model itself. And if Retrieval is required, then it can also determine complexity of the query so as to route to bigger or smaller LLMs for response generation.

2. Keep frequently submitted prompts and their responses in cache memory. Here, first match incoming query with previously answered query and if match is found, send cache content to a smaller LLM for response generation. Use the typical flow only if there is no match found cache.

Note: Caching requires careful invalidation when source documents or permissions change.

Evaluation

AI-based systems require extensive evaluation as it is the single most important component which can help ensure if the system will fail in Production or not.

Generally in Production RAG Systems, evaluation is done in 2 Stages -

1. Retrieval Evaluation

2. Generation/RAG Evaluation

Retrieval Evaluation

Are we retrieving the right information ?

There are some standard evaluation metrics widely used to measure retrieval accuracy -

$$Precision @ K = \frac {Number\ of\ relevant\ documents\ in\ K} {K}$$

$$ Recall @ K = \frac {Number\ of\ relevant\ documents\ in\ K} {Total\ Relevant\ Documents}$$

$$ MeanAveragePrecision\ (MAP) = Average\ [\frac {1}{|Q|}\sum_{q\in Q} AP(q)]$$

$$ AveragePrecision(q) = \frac {1}{R_q}\sum_{k=1}^K Precision@k\ *\ rel_k$$

$$ MeanReciprocalRank\ (MRR) = \frac {1}{|Q|}\sum_{i=1}^{|Q|} \frac {1}{Rank_i}$$

$$ NormalizedDiscountedCumulativeGain (NDCG@K) = \frac {DCG@K}{IDCG@K}$$

$$ DCG@K = \sum_{i=1}^K \frac {rel_i}{log_2(i+1)}$$

$$rel_i = relevant\ score\ of\ item\ at\ position\ i$$

$$ IDCG@K = Max\ DCG\ score.\ Calculated\ by\ sorting\ all\ items\ by\ relevance\ in\ descending\ order$$

Where,

R_q = number of relevant items for query q; Q = collection of total queries

Generation/RAG Evaluation

Given the retrieved context, did the system generate a good answer?

Here, I will mention two most common RAG evaluation techniques used in Production -

1. Human-as-a-Judge - AI Engineers curate a large set of prompts covering as much diversity as possible including document-related prompts to evaluate relevance accuracy as well as other prompts to evaluate guardrails and security. Each one of the prompts is then manually evaluated by humans for pre-decided criteria

2. LLM-as-a-Judge - Most of the time AI systems are first evaluated by another LLM(s) which act a Judge against a pre-decided rubric of evaluation criteria. 'RAGAS' is one such library which provides this functionality. Here, the rubric generally comprises of metrics like -

   1. Response Relevancy - Evaluates relevance of response regardless of factual accuracy. Here input prompt is compared, in terms of similarity, with synthetic prompts which could have led to the same response.

   2. Faithfulness - It determines factual accuracy by making additional LLM calls to determine if the response claim is factually supported by the retrieved information.

   3. Correctness - When an Expected/Reference Answer exists, it determines if the answer matches the same.

There are also additional metrics like Noise Sensitivity and Citation Ability which are sometimes used to evaluate effectiveness of the RAG system.

Tip: RAGAS is a great library for RAG evaluation. Reading about certain metrics and definitions (not just their implementation) brings better clarity on applicability of certain metrics for robustness of your specific use case.

Before we end this article, I wanted to share my personal thoughts on debugging RAG Pipelines.

Debugging

Understanding points of failure in a Production RAG Pipeline and their possible reasons is one of the most underrated skill in my opinion. If mapped well, this can save a lot of valuable engineering time and also help maintain user trust.

Some of the most common failure nodes in a RAG system and their potential causes are -

That's all on RAG for now. Feel free to comment below if there are some other possibilities in the architecture or optimization or evaluation which helped you improve your specific use cases.

Fine-tuning or retraining can help adapt a model to a specific use case, but neither is an ideal solution when the underlying information changes frequently or exists outside the model's training data.

This is where Retrieval-Augmented Generation (RAG) comes in.

RAG

In simple words, give model access to external source of information before generating any response, thereby grounding the response in factual data. From this very sentence, we can see that RAG consists of 2 stages -

1. Access to secondary information

2. Response Generation (or Text Generation)

While Transformer-based language models are exceptionally good at text generation, Information Retrieval (IR)—the technology underlying search engines for decades—provides a natural way to retrieve relevant external information before generation.

Hence, RAG architecture became -

Retrieval

Text Retrieval is one of the most critical part of any RAG system. While there are many complexities involved in implementing the same, one of the easiest, fastest implementation came from Information Retrieval (IR) systems.

IR systems have mostly been based on Keyword-Search, i.e. documents are represented using term-based representations and are scored based on the occurrence and importance of query terms. This has been a fast and efficient technique which enabled search engines return matching links/documents based on search queries.

Because the total vocabulary across a corpus can be very large, each document typically contains only a small subset of those terms. As a result, the corresponding vector representation contains mostly zeros and is therefore called a sparse representation. Hence, this approach is commonly referred to as Sparse Retrieval.

Sparse Retrieval

$$Each\ document\ d_i:\ [c_1,c_2,....,c_k]\ -\ vector\ of\ dimension\ k$$

$$k\ =\ total\ unique\ words\ across\ corpus$$

$$c_j\ =\ count\ of\ word\ j\ in\ the\ document\ (simplified\ version)$$

In Sparse retrieval, Each document is saved as a vector representation of word counts. For any incoming user query -

1. Query is tokenized into words

2. For each word in the query, a score is computed against each document

3. Scores for all query words are added as a document score for that query

4. Documents are then returned as sorted list based on these scores - higher the score, relevant the document

Score Computation

Multiple techniques have been used for computing this 'relevance score', but the 2 main techniques used widely were -

1. TF-IDF

2. BM25 (modified TF-IDF)

TF-IDF

Term Frequency (TF) represents frequency (count) of term (word) in the document. Idea: Higher Count --> More Relevant Document

Inverted Document Frequency (IDF) represents inverted count of documents containing the term (word). It is used to give additional importance to rare, meaningful words

Formally defined as -

$$TF(t,d) = 1+log_{10}Count(t,d),\ if\ Count(t,d)\ >\ 0$$

$$TF(t,d) = 0,\ if\ Count(t,d)\ =\ 0$$

$$IDF(t)\ =\ log_{10}(N/DF_t),\ N=total\ docs,\ DF_t=docs\ containing\ word\ t$$

TF will tend to bias towards longer documents, hence, TF is modified as -

$$TF(t,d) = 1+log_{10}(Count(t,d)/total\ words\ in\ doc\ d),\ if\ Count(t,d)\ >\ 0$$

$$Score(q,d)=\sum_{t\in q\cap d} TF_{t,d}\ .\ IDF_{t}$$

Note: A common alternative is to normalize term frequency by document length (as above). The exact TF normalization varies across implementations but the important intuition is that raw term counts can favor longer documents.

Best Matching 25 (BM25)

BM25 can be viewed as an evolution of TF-IDF that introduces tunable controls for 'term-frequency saturation' and 'document-length normalization'. This flexibility has made BM25 one of the most widely used scoring functions for Sparse Retrieval.

Term Frequency Saturation - TF-IDF will uniformly increase the score of documents based on the count of a given keyword in the document. This may or may not be desirable depending upon use case. BM25 introduces a factor, 'k', which can control the effect of term count

Document Length Normalization - TF-IDF either places a full penalty on the length of the document or does not place a penalty at all, based on the TF definition used. BM25 makes it configurable using a factor 'b'.

$$IDF(t) = log_{10}[(N-DF_t+0.5)/(DF_t+0.5)]$$

$$Score(q,d)=\sum_{t\in q\cap d} IDF_{t}\ .\ [(TF_{t,d}.(k+1))\ /\ (TF_{t,d}+k(1-b+b(|d|/avg|d|)))]$$

Here,

|d| = length of document d; Avg|d| = Avg. length of all documents in corpus,

k = commonly between 0.5 to 2; and 0 <= b <= 1

Tuning 'b' - Increase to apply higher normalization like in cases where document sizes are of varying length and longer ones should not dominate.

Tuning 'k' - Increase if term repetition is actually important like legal documents, scientific journals, tech manuals etc.

Indexing

A core data structure behind efficient sparse retrieval is Inverted Index. It maps each term to a postings list containing document IDs and, typically, term-frequency information. This allows the search engine to efficiently retrieve documents containing query terms.

Dense Retrieval

Since Sparse Retrieval is only concerned with occurrence of words, it can only help find documents containing the exact words. Therefore, retrieval accuracy will be low if -

1. Documents contain synonyms or

2. Meaning of the document is completely different based on order of words but the words are essentially same as in query

3. Documents contain few common words but mostly different words but overall context or meaning is similar

4. Query is specific to rare words or words not present in memory (corpus)

But thanks to Transformer architecture and subsequent explosion of Language Models - big and small, it is now possible to encode the meaning of a document in an embedding (another name for vector representation).

An embedding model is commonly a fine tuned model trained on retrieval objectives, which encodes text into a dense vector representation designed to capture useful semantic information. These vectors are much smaller than a vocabulary-sized sparse representation, hence the term dense vectors. Searching this embedding space is commonly referred to as semantic search.

Here's a quick comparison between Sparse & Dense Retrieval

There are primarily 2 ways of implementing Dense Retrieval -

Cross Encoder

Both query and document are combined into a single input, separated by special tokens, and fed as input to the model. Model is trained to predict a score, representative of the relevance of the document for the given query.

Cross-encoders can model query-document interactions more directly and often provide stronger relevance judgments, but they are substantially more expensive because the query and each candidate document must be processed together

Bi-Encoder

A bi-encoder independently encodes the query and document into vector representations. The model is trained so that relevant query-document pairs have similar representations for e.g. a Siamese Network.

The biggest advantage of Bi-Encoders is that embeddings for all documents can be pre-computed and stored in a vector database (databases specialized for vector storage & retrieval). Whenever a query comes, its embedding is generated using the same model (hence, query & document embeddings will be in same latent space) and is then used to search similar documents using a similarity metric on the embeddings. This makes this approach trade some interaction modeling for superior scalability.

In Production RAG Systems, there are generally 2 stages of dense retrieval -

1. Retrieval - Bi-Encoders are generally employed for fast retrieval when size of data (documents) is huge. This will help reduce search set to a limited set of relevant documents (commonly called 'candidate set').

2. Post-Retrieval (Re-Ranking) - Once we get our candidate set, it becomes feasible to run our cross-encoder model to return a more accurate relevance score for each of the candidate documents. Final response is then generated based on the rank of each document, calculated based on cross-encoder's similarity score.

Some common similarity measures used are -

1. Euclidean Distance - Distance between 2 vectors (L2 Distance)

2. Dot Product - Sum of element-wise products of two vectors; equivalently, the product of their magnitudes and the cosine of the angle between them

3. Cosine Similarity (most common) - Measures the angular similarity between two vectors, independent of their magnitude

Hybrid Retrieval

Many Production RAG systems today don't use only one retrieval technique, instead they deploy a Hybrid RAG system where both dense retrieval and sparse retrieval work in tandem. This is then complemented by 'Metadata Filtering' which can be applied alongside retrieval as a business-logic-based constraint—for example, filtering documents by tenant, date, document type, access permissions, or geography.

Production RAG Architecture

Re-Ranking

While above Architecture depicts a Sequential RAG - first Dense, then Sparse and finally Filtering, there are multiple ways of defining this architecture depending upon which leads to highest accuracy -

1. Sequential

   1. Dense Retrieval as Candidate Selector

   2. Sparse Retrieval for Ranking

2. Parallel

   1. Dense Retrieval as Ranker 1

   2. Sparse Retrieval as Ranker 2

   3. Combined Ranking Score (Reciprocal Rank Fusion) =

$$Rank = [w_1\ /\ (K+Rank_{Ranker1})]\ +\ [w_2\ /\ (K+Rank_{Ranker2})]$$

$$w_1, w_2 = Weights\ of\ each\ ranker\ (configurable)$$

Metadata Filtering can be used across any design as business-logic based document filter.

Note: To improve final ranking of relevant documents in RAG, sometimes a secondary LLM is also used in Re-Ranking to provide relevance scores.

This concludes the deep dive into the retrieval layer. If you'd like to explore how these concepts come together in a production-grade RAG architecture, check out my next article: Practical Considerations While Designing a Production-Grade RAG System.