Enterprise RAG Assistants That Answer From Your Documents, Cited
An enterprise RAG (retrieval-augmented generation) assistant answers employee questions by first retrieving passages from approved internal documents and then having a language model compose an answer that cites those passages. NTSPL designs, builds and evaluates these assistants on AWS, Azure and Google Cloud for policy, clinical, credit, engineering and compliance teams that need answers they can verify.
Delivered on
In scope
How a Grounded Answer Is Produced: Retrieve, Re-rank, Compose, Cite
Most organisations already tried a pilot: a chatbot pointed at a SharePoint library that answers fluently and is wrong about one question in five. The failures are predictable. Circulars get superseded but the old PDF still ranks first. A 140-page manual is chunked mid-table, so the model sees half a rate card. Retrieval ignores who is asking, so a branch user sees board notes. Nobody built a test set, so there is no number to show the risk committee, and the pilot stalls between demo and production.
NTSPL treats RAG as a retrieval engineering problem first and a prompt problem second. We start with 150-300 real questions from the people who will use the assistant, build a golden answer set with them, and tune chunking, metadata, hybrid retrieval and re-ranking until citation precision clears an agreed bar. Only then do we wire the chat surface, document-level permissions, refusal behaviour for out-of-scope questions and a feedback loop owned by the content team.
Signals That Your Team Needs a Cited Knowledge Assistant
Signals you need Enterprise RAG & Knowledge Assistants
- Staff spend 20-40 minutes per query searching circulars, SOPs or manuals, and the same 50 questions reach a central desk every week.
- A pilot chatbot exists but nobody can state its accuracy, and at least one wrong answer has already been escalated.
- The corpus is 5,000 to 2 million pages across SharePoint, file shares, a DMS or Confluence, with versions and supersessions that matter.
- Answers must be traceable to a clause or page because an auditor, regulator or clinician will check them.
- Different user groups may only see different documents, so a single shared index would leak content.
When to choose something else
- If users mostly need to find a document rather than get a composed answer, an AI-enhanced search experience is cheaper and safer; see Enterprise Search with AI.
- If the questions are about numbers in a warehouse (sales by region, yesterday's NPA), RAG over PDFs is the wrong tool; Conversational Analytics over a governed semantic layer fits.
- If the source documents are mostly scanned images with no clean text yet, start with AI Data Preparation or Intelligent Document Processing before building a chat layer.
What We Build Inside a RAG Assistant Engagement
Question bank and golden set
Workshops with 8-15 real users to collect 150-300 representative questions, expected answers and the exact source clause, tagged by difficulty and audience.
Corpus onboarding
Connectors to SharePoint, S3, Blob, Cloud Storage, Confluence or a DMS, with version, effective date, owner and access-group metadata captured per document.
Retrieval tuning
Structure-aware chunking, hybrid keyword plus vector search, metadata filters and a cross-encoder or managed re-ranker, tuned against the golden set rather than by eye.
Answer composition and citation
System prompts, answer templates and citation rendering that link to the page or paragraph, plus a refusal path when retrieved evidence is too weak.
Permission trimming
Security filters that pass the user's Entra ID, IAM Identity Center or Google group claims into retrieval so restricted documents never reach the prompt.
Feedback and content operations
Thumbs and comment capture routed to document owners, a weekly unanswered-questions report and a process to retire superseded documents from the index.
Reference Design for a Permission-Aware RAG Pipeline
Permission-Aware RAG Pipeline with Evaluation Gate
Documents are parsed, chunked with structure and access metadata, embedded and stored in a hybrid index; at query time the user's group claims filter retrieval, a re-ranker selects the top passages and the model answers only from them with citations. Every release of prompts, chunking or model must pass the golden-set evaluation before it reaches users.
Select any component
Permission-Aware RAG Pipeline with Evaluation Gate: click any of its 10 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
Building RAG on Bedrock Knowledge Bases, Foundry and Agent Search
Knowledge Bases handles parsing, chunking and sync from S3 or SharePoint; we add metadata filters for access groups and use Guardrails contextual grounding checks to suppress unsupported answers, running in Mumbai where the chosen models are offered.
Azure AI Search provides hybrid retrieval with semantic ranking and security trimming on group IDs; Foundry hosts the model, prompt flow and evaluations, with regional deployments in an Indian region when residency matters.
Agent Search gives managed connectors and grounded generation; for custom chunking we use RAG Engine with a vector store and the Ranking API, and keep data stores in asia-south1 or asia-south2 where supported.
Chunking, Retrieval and Refusal Choices That Decide Answer Quality
Chunking strategy
Fixed 512-token chunks are a starting point, not a design. For policy and SOP corpora we chunk by heading with 10-15% overlap and attach the parent section title to every chunk. Tables stay whole up to roughly 1,500 tokens. Long manuals get a two-level scheme: small chunks for retrieval, the parent section passed to the model. We decide using recall@10 on the golden set, not intuition.
Hybrid retrieval and re-ranking
Pure vector search misses circular numbers, part codes and clause references such as 4.2(b). We default to hybrid retrieval with reciprocal rank fusion, then re-rank the top 40-50 to 5-8 passages. If re-ranking adds more than 400 ms at p95, we cache embeddings for frequent questions rather than drop the re-ranker, because it is usually the largest single quality lever.
When to refuse
An assistant that always answers will eventually invent policy. We set a minimum re-ranker score and require at least one supporting passage; below that, the assistant says it cannot find guidance and offers the owning desk's contact. Target refusal rate on in-scope golden questions is under 5%, and on deliberately out-of-scope questions above 90%.
Version and supersession handling
Superseded documents cause the most damaging wrong answers. Every chunk carries effective-from and superseded-by metadata; retrieval filters to the current version by default and shows older versions only when a user asks about a historical date. Content owners get a monthly list of documents older than 24 months that are still being cited.
Golden-Set-First Delivery: From 200 Questions to Production
Question harvesting and golden set
Collect 150-300 real questions from users and the help desk, agree expected answers and source clauses with subject experts, and define pass thresholds.
Corpus audit and connector set-up
Inventory libraries, owners, ACLs and version practices; remove duplicates and drafts; configure incremental sync into a non-production index.
Retrieval tuning sprints
Iterate chunking, embeddings, hybrid weights and re-ranking against the golden set until recall@10 and citation precision meet target.
Answer layer, permissions and guardrails
Build prompts, citation UI, refusal path, identity-scoped filters and content-safety policies; run a permission-leak test with users from each access group.
Pilot with 50-100 users
Release to one department, capture feedback, add failed questions to the golden set and re-run evaluation before each prompt change.
Production hand-over
Release manifest, runbooks, content-owner process and evaluation pipeline handed to the internal team or NTSPL managed support.
What you receive
Answer Quality Numbers We Report Before and After Launch
| KPI | Typical target |
|---|---|
| Citation precision (cited passage actually supports the claim) | 92-97% on golden set |
| Faithfulness / groundedness score | 0.90 or higher on automated evaluation |
| Retrieval recall@10 for in-scope questions | 90% or higher |
| Correct refusal on out-of-scope questions | Above 90% |
| Median answer latency including retrieval | 2.5-5 seconds |
Where RAG Assistants Go Wrong and How We Prevent It
| Risk | How NTSPL prevents it |
|---|---|
| Restricted documents leak to users through a shared index | Group-claim filters enforced at retrieval, a nightly ACL reconciliation job and a pre-launch leak test using accounts from each access group. |
| Superseded circulars produce confident wrong answers | Effective-date and superseded-by metadata on every chunk, default current-version filter and visible document dates in the answer. |
| Quality silently degrades after a model or prompt change | Golden-set regression in the release pipeline; no promotion if faithfulness or citation precision drops more than 2 points. |
| Users stop trusting the tool after early errors | Pilot with a small cohort, publish accuracy numbers openly and fix the top 20 failed questions weekly during the first two months. |
RAG vs fine-tuning: which should we use for an internal knowledge assistant?
Use RAG. Fine-tuning changes how a model writes, not what it reliably knows, and it cannot cite a source, respect document permissions or forget a superseded circular without retraining. RAG keeps knowledge in an index you can update in minutes, filter by user and audit by citation. Fine-tuning earns a place later only for narrow style or format needs, such as producing a fixed report layout, or for a small model that must run cheaply at high volume. Even then it sits on top of retrieval, not instead of it. For policy, clinical, credit or engineering content, we have not seen a case where fine-tuning alone was the right first step.
Key facts: Enterprise RAG & Knowledge Assistants
Enterprise RAG & Knowledge Assistants in Practice
2 Enterprise RAG & Knowledge Assistants scenarios, from Healthcare (multi-specialty hospital group) and Banking (trade finance operations), plus related platform case studies.
Hospital group gives night-shift nurses cited protocol answers in under 5 seconds
Bank trade-finance desk cuts LC discrepancy research from 35 minutes to 6
Wealth firm's research assistant cuts RM client-prep time from 50 to 12 minutes
Infrastructure company's legal team reviews contracts in hours, not days
Enterprise RAG & Knowledge Assistants: Frequently Asked Questions
How accurate can an enterprise RAG assistant realistically be?
On a well-curated corpus, 90-95% of in-scope questions can be answered correctly with valid citations. The remaining errors usually trace to missing or contradictory source documents rather than the model. We measure accuracy on your own golden set before launch, so the number you see is specific to your content and users, not a vendor benchmark.
What does building a RAG assistant typically cost?
A first production assistant over one or two departments' content usually takes 8-10 weeks of NTSPL effort, with indicative fees from about ₹18-35 lakh depending on corpus size, connectors and identity complexity. Run cost is mostly model tokens and the search index, often ₹40,000 to ₹3 lakh a month for a few thousand users.
Can the assistant keep data inside India under the DPDP Act 2023?
Yes, in most designs. Indexes, document stores and logs can sit in AWS Mumbai or Hyderabad, Azure Central India or South India, or Google Cloud Mumbai or Delhi. The model endpoint is the variable: we choose models offered in an Indian region with regional processing and avoid global or cross-region routing options when the data includes personal data.
How do you stop the assistant showing documents a user should not see?
Permissions are enforced at retrieval, not in the prompt. Each chunk carries the access groups of its source document, and the user's identity-provider groups are applied as a mandatory filter. A restricted passage never reaches the model, so it cannot be summarised. We verify this with a leak test using accounts from every group.
Which documents should we index first?
Start with the 200-500 documents behind the most frequent questions to a central desk, such as HR, credit policy, clinical protocols or product manuals, that have a named owner and a clear version history. Broad crawls of every share drive produce noisy retrieval and permission risk; we widen scope only after the first corpus meets its accuracy target.
What happens when documents change?
Incremental sync picks up new and changed files every 15-60 minutes, and deleted or superseded files are removed within one cycle. Content owners get a weekly report of questions the assistant could not answer, which usually shows the gaps in documentation more clearly than any audit.
Scoping Enterprise RAG & Knowledge Assistants?
Start with the question buyers ask first: “RAG vs fine-tuning: which should we use for an internal knowledge assistant?” NTSPL will answer it for your estate and give you a fixed-scope proposal.