RAG and Enterprise Knowledge: Making Models Answer From Your Data
RAG is the most over-sold and under-specified term in enterprise AI. What it genuinely fixes, and the part that decides whether it works.
RAG retrieves relevant passages from your own content and puts them in front of the model along with the question. It is not fine-tuning, it does not teach the model anything permanently, and it does not fix bad source documents — it makes them findable, which is a different problem and usually the one you actually have.
What the evidence says about RAG
Both findings point the same way: output people cannot verify does not get used. Citing the retrieved source alongside every answer is the cheapest available trust mechanism, and it is a design decision rather than a model capability.
What makes retrieval actually answer correctly
Source curation before indexing
Decide which documents are authoritative and remove the superseded ones. A retrieval system over three versions of the same policy will confidently answer from the wrong one.
This is where most RAG disappointment originates, and it is a content governance task rather than an engineering one.
Chunking that respects meaning
Split documents on semantic boundaries — sections, clauses, procedures — rather than fixed character counts. A chunk that ends mid-clause retrieves badly and reads worse.
Retrieval evaluation with a real question set
Assemble 50 to 100 real questions with known correct sources, then measure whether retrieval surfaces them. This is the single highest-value test in the whole pipeline and it is routinely skipped.
If the right passage is not retrieved, the model cannot answer from it. Model upgrades will not fix a retrieval failure.
Citations in every answer
Show which document and section an answer came from. It converts an unverifiable claim into a checkable one, and it is what makes people willing to act on the output.
Permission-aware retrieval
Retrieval must respect who is asking. An assistant that surfaces HR or legal content to the wrong person is a data-access incident, not a bug.
Freshness and re-indexing policy
Decide how quickly a changed document must appear in answers, and build to that. Stale retrieval erodes trust faster than no system at all.
Where RAG is the wrong answer
- Your source documents are contradictory or out of date. Fix the content first — retrieval makes bad content more findable, not more correct.
- The questions have precise, structured answers that live in a database. That is a query, not a retrieval problem, and a report will serve you better.
- You need the model to reason across an entire corpus rather than answer from a passage. Retrieval surfaces fragments; it does not summarise everything you own.
- Nobody owns document governance. Without an owner, the index degrades and trust goes with it.
Green light, red light
| Dimension | Go ahead when | Do not when |
|---|---|---|
| Source content | The answer genuinely exists in your documents and a person could find it | The answer is not written down anywhere — retrieval cannot surface what does not exist |
| Content freshness | Documents have owners and are maintained | The corpus is full of superseded versions with no way to tell which is current; RAG will cite the wrong one confidently |
| Question shape | Questions are answerable from a passage or two of source material | Questions require reasoning across many documents, or arithmetic over records — that is a query problem, not a retrieval one |
| Evaluation | You have a set of real questions with known-correct answers to test against | Quality is assessed by asking it a few things and being impressed |
| Access control | Retrieval respects the same permissions as the source systems | Documents with different audiences sit in one index — the fastest route to a confidentiality incident |
How to test RAG on your own content
-
01
Pick one document set with a clear owner and a real question volume — support articles or internal policy are common good candidates.
-
02
Assemble 50 to 100 real questions with the correct source noted for each. This becomes your evaluation set.
-
03
Remove superseded documents before indexing anything.
-
04
Measure retrieval first: for each question, is the correct passage in the top results? Fix that before evaluating any model.
-
05
Ship with citations visible from day one, and watch which answers people click through to verify.
Questions buyers ask about RAG
They solve different problems. Fine-tuning changes how a model behaves; RAG changes what it knows at question time. For content that changes regularly — policies, documentation, support history — RAG is almost always the right tool, and it is far cheaper to keep current.
Model choice matters less than retrieval quality. Teams routinely upgrade the model hoping to fix answers that were failing because the right passage was never retrieved. Measure retrieval first.
Classify content before indexing and route by classification: regulated and confidential classes to a model inside your boundary, everything else to whichever frontier model is best value. That routing decision is architectural and hard to retrofit.
For a single well-owned document set with an evaluation question list ready, typically six to ten weeks to a usable internal system. The variable is almost always source curation, not engineering.
Continue the kit
Want an honest read on your readiness?
Six evidence tests, four minutes, no email required. It will tell you plainly if the foundations are not there yet.
Run the assessment →Read the rest of the kit.
Five capability guides, every figure cited to a named source and year.
Open the AI Resource Kit →