Large language models are good at language but know nothing about your contracts, product specs or internal policies. Retrieval-augmented generation, usually shortened to RAG, is the most common way to close that gap. This article explains RAG for business teams without the jargon: what happens under the hood, where it works well, where it disappoints and what you should prepare before anyone writes code.

The idea in one paragraph

Instead of hoping the model “knows” the answer, a RAG system first searches your own content for the passages most relevant to the question. It then hands those passages to the model along with the question and an instruction such as “answer using only this material and cite your sources”. The model writes the answer; your documents supply the facts. When the documents change, the answers change, with no retraining.

What happens behind the scenes

A RAG system has two phases. The first runs whenever content is added or updated; the second runs on every question.

Phase one: preparing your content

  1. Collect the sources: help articles, PDFs, wiki pages, tickets, product data.
  2. Clean them by stripping navigation, duplicate boilerplate and outdated versions.
  3. Split each document into chunks, typically a few paragraphs long, keeping headings and metadata such as title, date and department.
  4. Embed each chunk: an embedding model turns the text into a list of numbers that captures its meaning.
  5. Store the chunks and their embeddings in a vector-capable database. PostgreSQL with the pgvector extension is a popular option, and it is available on Supabase, which we compare in Supabase vs Firebase for startups. Dedicated vector databases are another route.

Phase two: answering a question

  1. The question is embedded the same way.
  2. The system retrieves the chunks whose meaning is closest, often combined with ordinary keyword search for names, codes and part numbers.
  3. Optionally, a reranking step reorders the results so the most useful passages come first.
  4. The top passages go into the prompt, and the model writes an answer with references.

RAG compared with the alternatives

ApproachWhat it doesGood forWeak spot
Plain LLMAnswers from general training dataDrafting, brainstorming, general questionsKnows nothing about your business; may invent details
Long prompt with documents pasted inSends whole documents every timeSmall, stable sets of contentGets slow and expensive as content grows
RAGRetrieves relevant passages per questionLarge or changing knowledge basesAnswer quality depends on retrieval quality
Fine-tuningAdjusts the model on examplesConsistent format, tone or classificationPoor way to teach changing facts

A common misunderstanding is that fine-tuning is how you “teach the model your data”. For facts that change, RAG is usually the better tool. Fine-tuning is more about behavior and style.

Where RAG for business pays off

  • Customer-facing help that should only use approved content.
  • Internal assistants that answer policy, HR or process questions.
  • Sales enablement: finding the right case detail, spec or clause quickly.
  • Technical support over manuals and past tickets.

The common thread is a body of written knowledge that people currently search by hand or ask colleagues about.

Why RAG projects disappoint

When a RAG system gives poor answers, the model is rarely the main culprit. The usual causes are upstream:

  • Messy sources. Three versions of the same policy, and the oldest one gets retrieved.
  • Bad chunking. A table split in half, or an answer separated from the heading that gives it meaning.
  • Missing metadata. Without dates, regions or product lines, the system cannot filter to what is relevant.
  • Questions the content cannot answer. If nobody wrote it down, retrieval cannot find it.
  • No evaluation. Teams tweak settings by gut feeling instead of measuring against a fixed set of real questions.

What to prepare before building

You can do a lot of useful groundwork without any technical skills:

  • Make a list of the questions people actually ask, ideally pulled from emails, tickets and chat logs.
  • Identify the source of truth for each topic and archive outdated duplicates.
  • Decide who is allowed to see what. If sales should not see HR documents, the system must enforce that at retrieval time, not just in the interface.
  • Choose an owner for the content, because the assistant will only be as current as the documents behind it.
  • Agree what a good answer looks like: length, tone, whether it must cite sources, and when it should say “I don’t know”.

A worked example

Imagine a manufacturer whose support team answers technical questions from a library of product manuals, installation guides and service bulletins. Today, an agent searches a shared drive, opens several PDFs and pieces an answer together. With RAG, the agent types the customer’s question and gets a short answer with links to the exact manual sections it came from. If a service bulletin has replaced an older instruction, metadata such as the publication date lets the system prefer the newer document. The agent still checks the answer before sending it, but the searching and reading step shrinks considerably. The same index can later power a customer-facing help assistant, limited to the documents that are approved for public use.

Questions to ask any RAG vendor or developer

  • How do you handle access control per user or per team?
  • Can every answer show which documents it used?
  • How is new or updated content synced, and how quickly?
  • What happens when nothing relevant is found?
  • How will we measure answer quality before and after launch?

Clear answers to these questions are a good sign that the system is being designed for real use, not a demo.

Building it with SiteWright Technologies

RAG sits where data engineering, application development and AI meet, which is why we treat it as a software project rather than a plug-in. If you want an assistant that answers from your own content, respects permissions and shows its sources, see our custom software development work or contact us to talk through your documents and use case.