India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
AI Data Preparation: Turning Unstructured Content into AI-Ready Data
AI data preparation is the engineering work that turns raw unstructured content, such as PDFs, scans, manuals, emails and product data, into a clean, deduplicated, chunked, metadata-rich and privacy-safe corpus that retrieval and generative AI systems can use reliably. NTSPL builds these pipelines with lineage and refresh schedules, so AI applications stay accurate as content changes.
Delivered on
In scope
Why Retrieval Quality Starts with Corpus Engineering
Most GenAI quality problems blamed on the model are data problems. A plant's O&M library has the same OEM manual in six copies with different revision letters, 30-year-old logbooks scanned at 150 dpi, and drawings with text only in title blocks. A retailer's catalogue has 40 spellings of the same fabric, missing attributes for a third of SKUs and supplier descriptions copied from competitors. Feed that to an embedding model and retrieval returns duplicates, contradictions and noise, however good the prompt.
NTSPL treats the corpus as a product with an owner, a schema and quality metrics. We profile content first, then build a pipeline that parses layout, cleans OCR, removes exact and near duplicates, resolves versions, normalises vocabulary, enriches metadata, redacts personal data and chunks by structure before embedding. Every chunk carries lineage back to its source file, page and pipeline version. Refreshes are incremental, and a quality report on each run tells content owners what changed and what failed.
Content Conditions That Call for a Dedicated Preparation Pipeline
Signals you need AI Data Preparation
- A planned or live RAG or search system draws on more than 10,000 documents or 50,000 product records from mixed sources.
- Retrieval tests show duplicates or outdated versions in more than 10% of top-5 results.
- More than a fifth of content is scanned, handwritten, or held in drawings, tables or slides.
- Documents contain personal data such as Aadhaar, PAN, phone numbers or health details that must not reach embeddings or prompts.
- Product or asset data has inconsistent attributes and vocabulary across suppliers, plants or business units.
When to choose something else
- If the source is structured transactional data flowing into a warehouse, that is data integration and ingestion work, not AI data preparation.
- If the corpus is a few hundred clean, born-digital documents, managed connectors with default chunking are usually sufficient.
- If the goal is capturing specific fields from incoming forms for a business process, intelligent document processing fits better.
Preparation Stages NTSPL Builds for Unstructured Content
Corpus profiling
Inventory by source, format, size, language, scan quality, duplication rate, version practice and personal data presence.
Parsing and OCR clean-up
Layout-aware parsing to Markdown or JSON with headings, tables and figures preserved, plus OCR correction for low-quality scans.
Deduplication and version resolution
Exact hashing and near-duplicate detection, with rules to select the authoritative current version and retire the rest.
Normalisation and enrichment
Controlled vocabularies, unit normalisation, entity extraction and metadata such as asset, product, owner, effective date and access group.
Privacy redaction
Detection and masking of personal and sensitive data before embedding, with reversible tokenisation only where justified.
Chunking, embedding and lineage
Structure-aware chunks, embedding with pinned model versions, and lineage from every vector back to source, page and pipeline run.
Parse, Clean, Deduplicate, Enrich, Redact, Chunk, Embed
Unstructured Content Preparation Pipeline with Lineage
Raw content lands in a staging bucket, is parsed and cleaned, deduplicated and version-resolved, enriched and redacted, then chunked and embedded into a vector or hybrid store. Each run emits a quality report and lineage records, and changes are processed incrementally.
Select any component
Unstructured Content Preparation Pipeline with Lineage: click any of its 9 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
Document Parsing, PII Detection and Vector Stores on Each Cloud
Bedrock Data Automation and Textract parse documents, Comprehend detects PII, Glue jobs run deduplication and enrichment at scale, and vectors land in S3 Vectors or OpenSearch for Bedrock Knowledge Bases.
The Document Intelligence layout model outputs Markdown with tables and sections for chunking; Fabric pipelines handle cleaning and deduplication, and Purview tracks classification and lineage.
Layout Parser produces structure-aware chunks, Sensitive Data Protection de-identifies content, Dataflow runs large-scale cleaning, and Dataplex records metadata and lineage across the corpus.
Parsing Depth, Deduplication, Redaction and Embedding Choices
How deeply to parse
Plain text extraction is enough for born-digital prose. Use layout-aware parsing wherever tables, multi-column pages or headings carry meaning, which covers most manuals, specifications and policies. Reserve generative parsing with a vision model for complex drawings, forms and charts, at roughly 5-20 times the cost per page. Profile the corpus and apply each tier only where needed.
Deduplication thresholds
Exact hashes catch identical files; near-duplicate detection catches revisions, re-scans and copies with different headers. We typically flag pairs above 0.85-0.9 similarity for version resolution rather than deleting automatically. Resolution rules should be explicit: latest effective date, preferred source system, or owner-approved list. Expect 15-40% duplication in uncurated enterprise shares.
Redaction approach
Mask personal data before embedding, because vectors can leak information through retrieval. Use pattern and checksum validation for Aadhaar, PAN, GSTIN and phone numbers, and model-based detection for names and addresses. Where users legitimately need identifiers, store them in the source system and fetch at answer time under access control, rather than placing them in the index.
Embedding model and re-embedding
Choose the embedding model by recall on your own test queries, including Indian-language and code-like queries, not by public leaderboard. Pin the version. Re-embedding a corpus of 5 million chunks is a planned project with cost and downtime implications, so we budget for it and run old and new indexes side by side until retrieval tests confirm the switch.
Corpus Profiling to Scheduled Incremental Refresh
Corpus profiling
Profile sources, formats, scan quality, languages, duplication and PII, and agree priority sources with content owners.
Parsing and cleaning design
Select parsers per content type, tune OCR correction and benchmark extraction quality on a stratified sample.
Deduplication, versions and vocabulary
Implement duplicate detection, version rules and controlled vocabularies with owner sign-off.
Enrichment and redaction
Build metadata extraction and PII redaction, and validate both on labelled samples.
Chunking, embedding and retrieval tests
Tune chunking and embedding against retrieval test queries and publish the first indexed corpus.
Incremental refresh and reporting
Schedule incremental runs, deletion propagation and per-run quality reports to owners.
What you receive
Corpus Quality Indicators We Measure on Every Run
| KPI | Typical target |
|---|---|
| Duplicate or superseded content in top-5 retrieval results | Under 2% |
| Parsing success rate per content type | Above 98% for born-digital, above 90% for scans |
| PII recall on labelled validation set | 98% or higher for structured identifiers |
| Metadata completeness for key facets | Above 95% |
| Source change to index latency | Under 24 hours, or under 1 hour for priority sources |
Data Preparation Shortcuts That Degrade AI Systems
| Risk | How NTSPL prevents it |
|---|---|
| Over-aggressive deduplication removes legitimately different versions | Near-duplicates are flagged for rule-based resolution, not deleted, and owners approve the rules. |
| Personal data embedded in vectors and exposed through retrieval | Redaction before embedding, validated recall on labelled samples and periodic scans of the index. |
| Poor OCR silently degrades retrieval for old documents | Per-page OCR confidence scoring, enhanced OCR for low scores and reports listing pages below threshold. |
| Embedding model change breaks retrieval without warning | Pinned versions, side-by-side indexes during migration and retrieval regression tests before cut-over. |
Managed RAG connectors vs a custom data preparation pipeline: which do we need?
Start with managed connectors and default chunking if the corpus is small, born-digital and well organised; they are quick and cheap to run. Move to a custom pipeline when profiling shows high duplication, many scans, complex tables, personal data or inconsistent vocabulary, because managed defaults cannot resolve versions, normalise terms or redact Indian identifiers to your rules. In practice many programmes use both: custom preparation writes a clean, enriched corpus to storage, and the managed knowledge base or search service indexes that corpus. This keeps managed retrieval features while controlling what goes in.
Key facts: AI Data Preparation
AI Data Preparation in Practice
2 AI Data Preparation scenarios, from Power generation (thermal power plant operations and maintenance) and Retail (fashion and home furnishings omnichannel), plus related platform case studies.
Thermal power station turns 31 years of O&M records into a clean AI corpus
Retailer normalises 2.4 lakh SKUs so its shopping assistant stops guessing
D2C beauty brand reallocates 18 percent of ad spend after unifying attribution in BigQuery
Dairy cooperative sees milk procurement by collection centre within 15 minutes
AI Data Preparation: Frequently Asked Questions
How much does data preparation affect RAG accuracy?
Usually more than model choice. In our projects, deduplication, version resolution and structure-aware chunking often raise retrieval recall by 15-30 points on the same queries and model. A strong model cannot recover an answer from a passage that retrieval never surfaces, or choose correctly between three conflicting copies.
What does an AI data preparation pipeline cost?
A pipeline for one to three major sources typically takes 8-10 weeks with indicative NTSPL fees of ₹14-30 lakh, depending on scan share, formats and redaction needs. Processing costs vary widely: born-digital text costs very little per page, while vision-based parsing of drawings can cost ₹1-5 per page.
How do you handle Aadhaar, PAN and other personal data?
We detect structured identifiers with patterns and checksum validation, and names, addresses and contact details with model-based detection, then mask them before embedding. Under the DPDP Act 2023, personal data should be processed only for a specified purpose, so most AI corpora do not need identifiers at all.
Can old scanned documents and logbooks be used?
Often yes, with enhanced OCR and clean-up. Scans below about 200 dpi, faded carbon copies and handwriting reduce accuracy, so we score OCR confidence per page and report pages below threshold. Owners then decide whether to re-scan, transcribe critical pages or exclude them.
Where is content processed and stored?
Staging, processing and vector stores can run in AWS Mumbai or Hyderabad, Azure Central India or South India, or Google Cloud Mumbai or Delhi. We confirm that the chosen parsing, PII and embedding services are available in-region, because sending content to an out-of-region endpoint moves the data.
How often should the corpus be refreshed?
Match refresh to how often content changes and how costly stale answers are. Policy and protocol libraries usually need hourly or daily incremental runs; archives may need weekly runs. Deletions and supersessions must propagate within one run, or stale content keeps appearing in answers.