RAG Development: AI That Answers From Your Own Data

What is RAG? A plain-English explanation

RAG stands for retrieval-augmented generation. A general model like GPT or Claude has read much of the public internet, yet it has never seen your return policy or your 2024 supplier agreement. RAG fills that gap at question time instead of training time.

The loop has three steps. First, retrieve: search your indexed content for the handful of passages most likely to contain the answer. Second, augment: place those passages into the prompt along with the user's question and instructions such as 'answer only from the sources below'. Third, generate: the model writes an answer and cites the passages it used.

Because the model reads your content fresh on every question, updating a policy document updates the answers. Nothing has to be retrained. And because each answer carries citations, a reviewer can click through and check it in seconds.

What production RAG development involves

The three-step loop hides most of the work. When we scope a RAG system, these are the parts that take the hours:

• Connectors to each data source, such as SharePoint, Google Drive, Confluence, Zendesk, a product database, or a document management system, with scheduled or event-based syncing
• Parsing that turns PDFs, scanned forms, slide decks, spreadsheets, and HTML into clean text while keeping headings, tables, and page numbers
• Chunking that splits documents into passages small enough to search precisely but large enough to keep their meaning
• Embeddings and a vector store, plus a keyword index for hybrid search
• A reranking step that reorders search results before they reach the model
• Permission filtering so each user only retrieves content they are allowed to open
• Citations that link to the source document, section, or page
• An evaluation set of real questions, used to test every change
• Index refresh jobs, monitoring, and feedback capture once people start using it

Parsing and chunking decide most of the answer quality

Teams tend to spend their energy picking a model. The bigger lever is usually the ingestion pipeline. A model cannot cite a table it never saw because the parser flattened it into a jumble of numbers.

Good parsing keeps structure. A benefits handbook should come through with its section headings intact, so a passage about dental coverage still knows it belongs under 'Part-time employees'. Scanned documents need OCR. Spreadsheets often need to be turned into short sentences or kept as structured rows that the system queries directly.

Chunking is the next decision. Split by fixed character count and you cut procedures in half. Split by headings and sections, with a little overlap and the parent heading attached to each passage, and retrieval gets much sharper. Contracts, manuals, and support articles each want a slightly different strategy, and we test several against your real questions before settling on one.

Embeddings, vector stores, and hybrid search

Each chunk is converted into an embedding, a list of numbers that captures its meaning. Questions get the same treatment, and the system finds chunks whose embeddings sit closest to the question. This is why a search for 'can I bring my dog to the office' can find a policy titled 'Animals on company premises'.

Those embeddings live in a vector store. For many companies, pgvector inside an existing PostgreSQL database is the sensible default: one less service to run, and document metadata sits next to the vectors. Hosted vector databases such as Pinecone, Weaviate, or Qdrant make sense at larger scale or when you want managed infrastructure. One thing to plan for either way: if you switch embedding models later, every document has to be embedded again.

Vector search alone misses exact terms like part numbers, SKUs, error codes, and people's names. Hybrid search runs a keyword search alongside the vector search and merges the results. A reranker, a smaller model that scores each candidate passage against the question, then picks the best few to send to the language model. These two additions fix a large share of the 'it found the wrong document' complaints that come up in early prototypes.

Permissions: users should only retrieve what they can open

This is where internal prototypes most often fail a security review. If the index holds payroll files and board minutes alongside the employee handbook, a plain RAG system will happily quote any of them to anyone who asks the right question.

The fix is to carry access rights into the index. Each chunk stores the groups or users allowed to see its source document, synced from the source system's permissions. At query time, the search filters by the current user's identity before anything reaches the model. Filtering after generation is too late, because the model has already read the restricted text.

Permission sync is real engineering. Folder sharing in SharePoint or Google Drive changes daily, and a stale permission is a data leak. We cover the wider controls, including logging, data retention, and provider terms, in our guide to secure enterprise AI integration.

Citations, evaluation, freshness, and monitoring

Citations do two jobs. They let users verify an answer, and they let your team spot bad sources. When the assistant keeps citing a 2019 pricing sheet, the problem is the document library, not the AI.

Evaluation is what separates a RAG system you can improve from one you can only hope about. We build a set of 100-300 real questions with known correct answers and source documents, drawn from support tickets, help desk logs, and interviews with the people who answer these questions today. Every change to chunking, prompts, or models gets scored against that set before release.

Freshness needs a plan too. New and edited documents should reach the index within minutes or hours, deleted ones must disappear, and archived versions should not compete with current ones. After launch, monitoring tracks questions with no good answer, thumbs-down feedback, latency, and model cost per query. The unanswered questions are often the most useful output of the whole project, because they show exactly which documentation is missing.

RAG vs fine tuning: which one fits your problem

People often ask whether they should train a model on their documents instead. Fine tuning adjusts a model's weights using example inputs and outputs. It is good at teaching behavior and weak at teaching facts that change. Here is how the two compare for business use.

Choose RAG when the questions are about your content and that content changes. Choose fine tuning when the model already knows enough but keeps formatting output wrong, or when you need a smaller, cheaper model to do one narrow task at high volume. The two can be combined, but we rarely recommend fine tuning as the first step. Most teams get further with better retrieval and clearer prompts.

Internal assistant vs customer-facing RAG chatbot

The same architecture serves both, but the requirements shift once the public can type into it. An internal assistant can afford an occasional 'I'm not sure, here are the closest documents'. A customer-facing RAG chatbot needs tighter guardrails: strict instructions to stay on topic, refusal of questions outside the knowledge base, no exposure of internal-only content, and a clean handoff to a human agent with the conversation attached.

Customer bots also tend to need live data, such as order status or account details, which means calling your APIs as well as searching documents. That starts to look like an agent, and our guide on how to build an AI agent for business covers that step. If you are weighing a bot against your current help desk setup, see AI chatbots vs traditional support ticketing, and for a full support build, our AI chatbot for customer service page explains what we deliver.

What RAG development costs

These are planning estimates, calculated as engineering hours multiplied by $100-$175 an hour, a typical range for a US development team. They cover discovery, ingestion, search, the user interface, evaluation, and deployment. Your actual figure depends on the number of sources, document quality, and how strict the permissions and accuracy targets are.

Running costs sit on top of the build. Model providers charge per token, both for the question plus retrieved passages and for the generated answer, so longer context means a higher bill per query. Embedding the initial document set is a one-time charge with smaller ongoing charges as content changes. Add hosting, the vector store, and monitoring. Provider prices move often, so check each vendor's current pricing page, and we model expected monthly cost from your query volume during scoping.

Before any build, we review your sources, sample questions, and access rules, then hand you a fixed-scope estimate at no charge. That way the number you approve is tied to a defined set of sources and features.

Getting a RAG project right from the start

The best RAG projects start small and specific. Pick one audience, such as support agents or new hires, and one body of content they search every day. Collect fifty questions they ask often. Measure how well the first version answers them, then add sources and users once the numbers hold up.

Our AI development team builds RAG systems end to end, from connectors and permission-aware search to evaluation and monitoring, on the cloud account you control. If you have documents your team or customers keep digging through, book a free consultation and we will map out the sources, the risks, and a scoped estimate for your RAG development project.