RAG chatbot development is the practice of building a chatbot that answers from your own documents rather than from the model's memory. At query time, the system searches a knowledge base, pulls the most relevant passages, and puts them into the model's prompt along with the user's question. The model then answers using that retrieved context instead of whatever it happens to remember.
The term comes from the 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Patrick Lewis and colleagues, accepted at NeurIPS 2020 (arXiv:2005.11401, retrieved 25 September 2026). The paper explains why RAG is common: large pre-trained models "store factual knowledge in their parameters" but "their ability to access and precisely manipulate knowledge is still limited," while "providing provenance for their decisions and updating their world knowledge remain open research problems."
Key Takeaways
• RAG combines a model's parametric memory (what it learned in training) with non-parametric memory (your documents, retrieved at query time). The original paper set state-of-the-art results on three open-domain QA tasks and found RAG models generate "more specific, diverse and factual language" than a parametric-only baseline (arXiv:2005.11401).
• RAG is about grounding. Agency — deciding what to do next and taking action — is a different concern; we separate the two in AI agents vs chatbots.
• The pipeline is five stages: ingest, chunk, embed, retrieve, generate. Chunking and retrieval are common failure points and should be evaluated before changing the generator.
• Chunking has a usable rule of thumb: if a chunk does not make sense to a human without the surrounding context, it will not make sense to the model either (Pinecone, published 28 June 2025, retrieved 25 September 2026).
• For many knowledge-base workloads, hybrid retrieval and reranking can outperform vector-only retrieval; Anthropic reports substantial gains in its own tests.
• Retrieval, not the model, is where you should measure first.
What RAG is for
The problem RAG solves is simple: you know things the model does not, and those things change.
A model trained on public data cannot know your return policy, your SLA credits, last quarter's pricing approval or the state of your order. Fine-tuning is primarily used to change behaviour, style, or task performance. RAG more directly supports updatable knowledge and passage-level attribution, although neither approach automatically guarantees either property. The original paper lists provenance and knowledge updating as open problems that RAG is designed to address.
RAG offers four practical benefits:
• Updatability: change a document and re-index without retraining the model.
• Provenance: return the passage behind a claim for human review.
• Access control: filter retrieval by tenant, region, or role before generation.
• A shared source: support, sales, and operations can use the same approved material.
The five stages of a RAG pipeline
The five stages are:
• Ingest: load PDFs, web pages, help-centre articles, or database rows, then strip noise and boilerplate. Navigation text can pollute the index.
• Chunk: split documents into retrievable pieces with overlap. Chunks can be too large, too small, or cut mid-idea.
• Embed: convert chunks to vectors and store them with metadata. The embedding model must fit the domain and languages.
• Retrieve: find the top-k chunks for the query. Pure vector search can miss exact terms.
• Generate: put the query and chunks in the prompt, answer with citations, and hand off when the context is insufficient.
Everything above is a separate engineering decision with its own evaluation method. If retrieval and generation are evaluated together, a bad retrieval result can be mistaken for a generation failure.
Stage 1 and 2: ingestion and chunking
Chunking is the step most often underestimated. Pinecone's guide frames it clearly: chunks must be "big enough to contain meaningful information, while small enough to enable performant applications and low latency responses," and offers a rule of thumb worth adopting — if the chunk of text makes sense without the surrounding context to a human, it will make sense to the language model as well (Pinecone, retrieved 25 September 2026).
The same guide's advice is to start with fixed-size chunking and iterate only when it proves insufficient, then move up a ladder:
• Fixed-size: split on a token count with overlap; a sensible baseline to try first.
• Content-aware: split on headings, tables, and list items; useful for policies, specifications, and catalogue pages.
• Recursive character-level: try separators such as paragraph, sentence, then word; useful for mixed prose with uneven paragraphs.
• Semantic: group adjacent sentences by embedding similarity and cut at topic shifts; useful when topic boundaries are clear.
• Contextual: prepend a short generated description of where the chunk sits in the source; useful for long or frequently topic-switching documents.
Chunk expansion retrieves a chunk's neighbours so the model receives surrounding context without the whole document in the prompt.
One finding shapes all of this. The "Lost in the Middle" study found that model performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models" (Liu et al., arXiv:2307.03172, TACL 2023, retrieved 25 September 2026). Dumping a 200-page document into a large context window is not a substitute for good retrieval.
Stage 3 and 4: embeddings, hybrid search and reranking
Embeddings capture semantic similarity, so "refund policy" can match "returning an item". They can miss exact strings such as SKUs, order numbers, error codes, and names.
BM25 is the older lexical technique that handles exactly that case. Anthropic's explanation is a good illustration: a support agent asking about "Error code TS-999" might not be matched by an embedding model looking for error-code content generally, but BM25 finds that exact string. Anthropic's reference pipeline therefore runs BM25 and embeddings, combines and de-duplicates the two result sets with rank fusion, and only then adds the top-k chunks to the prompt (Anthropic, Introducing Contextual Retrieval, published 19 September 2024, retrieved 25 September 2026).
Reranking is the final filter: retrieve many candidates, score them against the query, then keep the most relevant chunks. Anthropic reports using the top 150 candidates and top 20 final chunks.
What Anthropic's numbers actually say
Anthropic published retrieval-failure measurements for a method it calls Contextual Retrieval, which prepends a short generated description of the chunk's place in the source document before embedding and indexing it. Measured as 1 − recall@20, the share of relevant documents that failed to appear in the top 20 retrieved chunks, across their own test corpora and with the top-performing embedding configuration they tested:
• Baseline (embeddings only): 5.7% failure rate.
• Contextual Embeddings: 3.7%, a 35% reduction.
• Contextual Embeddings plus Contextual BM25: 2.9%, a 49% reduction.
• The above plus reranking: 1.9%, a 67% reduction.
These are Anthropic's own experimental results on its own datasets and setup, not a guarantee for your project. The results support evaluating hybrid retrieval and reranking in that order, but they do not guarantee the same gains on another corpus. Anthropic's summary lists six findings, including that embeddings plus BM25 outperformed embeddings alone in its tests, that top-20 chunks performed better than top-10 or top-5, and that adding context improved retrieval accuracy substantially.
Stage 5: generation
A grounded prompt should tell the model to:
• Answer only from the supplied context
• State when the context lacks the answer
• Cite a checkable source
• Avoid inventing URLs, policy numbers, prices, or dates
Anthropic also makes a specific generation point worth testing: distinguish between what is context and what is the retrieved chunk in the prompt rather than merging them, because this separation may improve downstream response quality. A retrieval system that finds nothing should route to a human, not to a fluent guess.
How to evaluate a RAG system
Evaluation has to be split, because a bad answer has two possible causes and they need different fixes.
• Retrieval: Did the right chunks come back? Use Recall@k on a labelled question set; 1 − recall@k is the failure rate.
• Generation: Given the right chunks, is the answer correct and faithful? Use ground-truth answers and faithfulness checks against the context.
• End to end: Does the user get a correct answer? Test a real question set and review a sample manually.
• Operations: Is the system fast, cheap, and stable? Track latency per stage, tokens and cost per query, and error rate.
Build the question set from real transcripts — actual support tickets and sales questions — not only from documents. If customers use Hinglish and the evaluation set is English-only, the results may not reflect real use.
The order of work matters: fix retrieval first. A better prompt cannot rescue chunks that were never retrieved, and no amount of model quality fixes a corpus that does not contain the answer.
When RAG is the wrong choice
Two situations call for something else.
Your knowledge base is small enough to fit in the prompt. Anthropic notes that if the knowledge base is under roughly 200,000 tokens (about 500 pages), you can include it in the prompt directly, and that prompt caching makes this approach significantly faster and cheaper. Retrieval machinery is worth it when the corpus outgrows the context window, not before.
The answer lives in a system, not a document. Order status, live inventory, account balance and refund eligibility change constantly. Retrieving a stale PDF about them is worse than useless. Those cases belong in a tool call against the system of record — which is an agent pattern, not a RAG pattern. We map that boundary in AI agents vs chatbots.
India-specific considerations
Language. If your customer transcripts mix English, Hindi, regional languages, and Hinglish, test retrieval in those languages. Performance varies by model and language, so benchmark the languages in your own corpus and consider routing queries to a suitable index or model rather than assuming one handles everything.
Where the data sits. A RAG index built from order history, invoices, support tickets, and internal policy documents may contain personal data. A commencement notification brought specified provisions of the Digital Personal Data Protection Act, 2023 into effect, and MeitY notified the Digital Personal Data Protection Rules, 2025 on 13 November 2025. DLA Piper's India overview, last modified 13 February 2026, notes that implementation is phased and the IT Act and Privacy Rules continue to govern until the Act's core operational provisions are fully effective in May 2027. Decisions about embedding storage, inference location, log retention, and index access should be assessed with a qualified adviser.
Document formats. For corpora containing spreadsheets, invoice PDFs, WhatsApp exports, or email threads, treat parsing quality as an explicit evaluation area. Include extraction, table reconstruction, and layout handling in the estimation before selecting a model. We cover the systems side in database solutions and custom application development.
A build checklist
• Define the scope. Which questions must the bot answer well, and which must it refuse?
• Assemble and clean the corpus. Remove navigation, headers, footers and duplicates.
• Chunk with a baseline. Fixed-size with overlap, plus metadata (source, date, tenant, category) on every chunk.
• Embed with a model suited to your languages and domain.
• Retrieve with hybrid search — embeddings plus BM25 — and add a reranker.
• Generate with grounding instructions and citations, and a defined "I don't know" path.
• Evaluate retrieval separately from generation, using real questions.
• Add the operational layer: logs, per-query cost and latency tracking, access control, human handoff.
• Plan for re-indexing. Decide who updates the corpus when a policy changes.
Frequently asked questions
Is RAG the same as fine-tuning?
No, and they solve different problems. Fine-tuning primarily changes the model's behaviour, style, or task performance; RAG supplies an external knowledge source at query time. The original RAG paper was explicitly about giving a model "a differentiable access mechanism to explicit non-parametric memory." RAG can usually be updated by changing documents and re-indexing rather than retraining a model, but ingestion, storage, retrieval, and evaluation costs depend on corpus size and architecture.
Does RAG stop hallucinations?
It reduces them; it does not eliminate them. Grounding gives the model something true to work from, but a model can still misread the passage it was given. That is why retrieval evaluation, faithfulness checks, explicit "not in the context" behaviour and human handoff all still matter. Any claim that RAG makes a chatbot hallucination-free should be treated as a sales statement, not an architecture.
How large does the knowledge base need to be before RAG is worth it?
Enough that it will not fit comfortably in the chosen context window, or that you need citations, access control, or frequent updates. Anthropic uses roughly 200,000 tokens (about 500 pages) as an example threshold where the whole knowledge base may be included directly with prompt caching. Model limits, permissions, and evaluation results determine the actual choice.
Which vector database should I use?
For many initial projects, corpus quality, chunking, retrieval, and evaluation matter more than the brand of vector database selected. Infrastructure choice becomes more important as scale, filtering, latency, and operations grow. Evaluate the simplest store that meets measured requirements, then add complexity when a test or operational limit justifies it.
Start with retrieval, not with the model
RAG chatbot development fails in predictable places: chunks that mean nothing alone, exact terms that vector search cannot match, and no way to tell whether a bad answer came from bad retrieval or bad generation. Build the pipeline so you can measure each stage, ground the answer in cited context, and hand off to a human when the evidence is not there.
Want a scoped plan for your own documents and channels? Share your requirements with GrowMyStore and we will map the corpus, the retrieval strategy and the evaluation set before anyone writes a prompt. If the system needs to do more than answer — take actions, run workflows, escalate — that is the adjacent problem we cover in agentic AI setup.