India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
CDC, Batch and Event Ingestion Pipelines into Cloud Analytics
Data integration and ingestion is the engineering that moves structured data from operational systems, SaaS apps and event sources into a cloud lake or warehouse reliably, through change data capture, scheduled batch loads or streams. NTSPL builds these pipelines with replay, schema-drift handling and data contracts, so analytics teams receive complete, on-time tables.
Delivered on
In scope
Why Nightly Extracts Break as Source Systems Grow
Nightly full extracts were fine when the core system held 2 crore rows. Now the extract runs past 7 a.m., locks tables during branch opening, and silently drops records whenever a vendor adds a column. Meanwhile 40-odd SaaS tools each export CSVs on their own schedule, and nobody can say whether yesterday's collections file was complete. When ingestion is fragile, every downstream dashboard inherits the doubt. Fixing reports one at a time never ends.
NTSPL replaces full extracts with log-based change data capture where the source allows it, managed connectors for SaaS, and events for anything that must arrive in minutes. Every pipeline lands raw, immutable data first, records row counts and watermarks, and has a written contract with the source owner covering schema changes and freshness. Freshness, completeness and drift are monitored from day one, not added after the first incident.
Symptoms That Your Ingestion Layer Needs Rebuilding
Signals you need Data Integration & Ingestion
- Nightly extracts overrun the batch window or lock production tables during business hours
- Reconciled row counts between source and warehouse differ on more than 1 in 20 loads
- A vendor schema change has broken a pipeline silently at least twice in a quarter
- Business wants loan, order or shipment data within 15 minutes instead of next morning
When to choose something else
- You are moving a database to the cloud and switching applications over: that is Database Migration in the migration family, not analytics ingestion.
- Your content is PDFs, emails and scanned forms for an AI assistant: AI Data Preparation in our Generative AI family handles parsing and chunking.
Sources, Patterns and Controls We Engineer
Database CDC
Log-based capture from Oracle, SQL Server, PostgreSQL and MySQL using redo logs, CDC tables or binlogs, with initial snapshot and ongoing change streams.
SaaS and file connectors
Managed connectors for Salesforce, SAP, Zoho, Google Ads and similar, plus governed SFTP drops with checksum and manifest validation.
Event ingestion
Kafka, Kinesis, Event Hubs or Pub/Sub topics for app and device events, with schema registry and dead-letter handling.
Raw landing and replay
Immutable, partitioned raw zone with 30-90 day replay so any downstream table can be rebuilt without touching the source.
Contracts and observability
Per-source data contracts, freshness and volume monitors, and reconciliation reports sent to source owners.
How Data Integration & Ingestion Works, Step by Step
Log-Based CDC and Event Ingestion Pipeline
Database logs, SaaS APIs and event topics feed a capture tier that writes immutable raw files with watermarks. A merge tier applies changes to bronze tables while monitors reconcile counts against the source.
Select any component
Log-Based CDC and Event Ingestion Pipeline: click any of its 8 components to see its role and the design choice NTSPL makes for it, or trace the 6-step primary path.
Native Ingestion Services on AWS, Azure and Google Cloud
DMS handles Oracle and SQL Server CDC into S3; we prefer Redshift zero-ETL integrations for supported Aurora and RDS sources to avoid running pipelines at all.
Fabric Mirroring covers Azure SQL, Cosmos DB and Snowflake with near-real-time replication into OneLake; Data Factory and Debezium cover on-prem Oracle and SAP.
Datastream streams Oracle, MySQL, PostgreSQL and SQL Server changes directly into BigQuery; Data Transfer Service covers Google Ads and many SaaS sources without code.
Choosing CDC, Batch or Streaming per Source
CDC versus scheduled batch
Use log-based CDC when a table changes more than about 5% a day, exceeds 50 million rows or must be fresh within an hour. Use incremental batch on an updated-at column for smaller tables with reliable timestamps. Never use trigger-based CDC on a busy OLTP system; it adds write latency that application teams will notice.
Managed service or Debezium
Managed CDC (DMS, Datastream, Fabric Mirroring) wins when sources are supported and volumes are moderate. Choose Debezium on Kafka when you need sub-minute latency, many consumers of the same change stream, or sources such as Db2 that managed tools handle poorly. Debezium needs Kafka skills on call; budget for it.
Schema drift policy
Agree with each source owner: new nullable columns flow through automatically, type widening is allowed, renames and drops block the pipeline and alert. Store the schema version with every raw file. This single policy prevents most silent data loss we see in inherited pipelines. Review the policy whenever the source vendor issues a major release.
Where to capture from
Capture from a read replica, Data Guard standby or Always On secondary rather than the primary. Confirm log retention covers at least 24-48 hours of pipeline downtime, so an outage over a weekend can resume from the log instead of forcing a full re-snapshot. Test this by deliberately pausing capture.
Source-by-Source Pipeline Rollout
Source inventory and contract drafting
Rank sources by freshness need and volume; agree contracts and drift policy with owners.
Landing zone and capture setup
Raw zone, network paths, replica access and the first CDC tasks with initial snapshots.
Merge logic and reconciliation
Idempotent merges into bronze, soft-delete handling and daily count and checksum reports.
SaaS and event sources
Connector and topic onboarding in priority order, each with monitors before go-live.
Runbooks and handover
Restart, replay and re-snapshot runbooks, alert routing and a failure drill with your team.
What you receive
Pipeline Freshness and Completeness Targets
| KPI | Typical target |
|---|---|
| Freshness of CDC-fed bronze tables | Under 15 minutes p95 |
| Source-to-target row count match | 100% daily, tolerance zero for financial tables |
| Silent schema-drift incidents | Zero; all drift either flows or alerts |
| Load on source primary from extraction | Near zero (capture from replica) |
Ingestion Failure Modes and Our Safeguards
| Risk | How NTSPL prevents it |
|---|---|
| Redo or binlog retention too short, forcing full re-snapshots after outages | Retention is sized to 48 hours of downtime and tested by pausing a task for a day. |
| Hard deletes in the source vanish from analytics | CDC captures delete operations as soft-delete flags, reconciled against source counts. |
| Late-arriving events skew daily totals | Event-time partitioning with a defined lateness window and an automatic restatement of affected days. |
Log-based CDC vs nightly ETL extracts: which is better for analytics?
For tables above roughly 50 million rows or with high daily churn, log-based CDC is better on every count: it reads the transaction log instead of querying tables, adds almost no source load, captures deletes and delivers changes in minutes. Nightly extracts remain fine for small reference tables and sources without accessible logs, such as some SaaS apps. The cost of CDC is operational: log retention, replica access and someone who understands restarts. Most estates we build end up 70-80% CDC by volume and batch for the long tail.
Key facts: Data Integration & Ingestion
Data Integration & Ingestion in Practice
2 Data Integration & Ingestion scenarios, from Two-wheeler finance (NBFC) and Port operator (multi-purpose cargo port), plus related platform case studies.
Two-wheeler lender sees EMI bounces by 11 a.m. instead of the next morning
Port operator unifies TOS, gate and customs feeds to track vessel turnaround hourly
General insurer cuts motor-claim intake from 2 days to 25 minutes with serverless
Dealership group cuts OEM warranty claim rejections by integrating DMS through Service Bus
Data Integration & Ingestion: Frequently Asked Questions
Can you capture changes from our Oracle core system without slowing it down?
Yes. We read redo logs from a Data Guard standby or use supplemental logging on a replica, so the primary sees negligible additional load. We measure source CPU and I/O before and after go-live and share the comparison with your DBA team.
How much does an ingestion build cost?
Effort scales with source count and type. A typical first release of 8-12 sources, mixing CDC and SaaS connectors, takes 8-10 weeks and costs INR 25-50 lakh. Cloud run cost for managed CDC is usually INR 40,000-2 lakh a month depending on change volume.
Does data have to stay in India during ingestion?
For RBI-regulated payment data and most personal data under DPDP-driven policies, we keep capture, landing and processing in Indian regions such as AWS Mumbai, Azure Central India or Google Cloud Mumbai. Pipeline logs are retained 180 days to align with CERT-In directions.
What happens when a source adds or renames a column?
Under the agreed drift policy, new nullable columns flow automatically and widened types are accepted. Renames and drops stop that table's pipeline and alert the owner before bad data lands, so downstream reports never silently lose a field. Every block is logged with the schema version.
Do we need Kafka?
Only if several systems consume the same change stream or you need sub-minute latency at scale. For a single warehouse target with 15-minute freshness, managed CDC straight into the lake or warehouse is simpler and needs fewer specialist skills. We add Kafka later if consumers multiply.
Can you take over pipelines another vendor built?
Yes. We start with a two-week audit of existing jobs, reconciling counts for every table, then migrate fragile jobs first. Pipelines that already reconcile cleanly are documented and kept rather than rewritten for the sake of it. Audit findings are shared table by table.