India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
Cloud Native Integration: Replacing ESBs and File Drops with Events
Cloud native integration connects applications through managed event buses, queues, change data capture and serverless integration flows instead of a central ESB or nightly file transfers. NTSPL replaces brittle point-to-point links, retires ageing ESB and file-transfer platforms, and designs event contracts, replay and tracking for enterprises where data moving between systems arrives late, twice or not at all.
Delivered on
In scope
Why central ESBs and file drops became the bottleneck
Ask how an order reaches the warehouse system and the answer is often a chain. A trigger writes to a staging table; an Oracle SOA or TIBCO flow polls it every five minutes and transforms it in XSLT nobody can read; a CSV lands on SFTP; a Windows scheduled task in the warehouse picks it up. When anything stalls, nobody knows which hop failed until a customer calls. ESB licences and specialist skills are expensive, and every new consumer means another flow queued behind the same small integration team.
NTSPL moves integration to events owned by the systems that produce them. Producers publish business events such as order placed, stock adjusted or lead assigned, through an outbox or CDC, onto a managed event bus. Each consumer subscribes with its own queue, so adding one never touches the producer. Transformations live with consumers or in small serverless flows, not a central hub. Every message carries a correlation ID, every queue has a dead-letter queue and replay, and ESB flows are retired domain by domain until the hub is switched off.
Integration symptoms that justify a rebuild
Signals you need Cloud Native Integration
- Data between core systems is 4-24 hours late because integrations run as hourly or nightly batches.
- An ESB or managed file transfer platform such as Oracle SOA Suite, TIBCO, webMethods or BizTalk faces end of support or a costly renewal.
- Adding one consumer of order or customer data takes six weeks or more because the integration team is a bottleneck.
- Business users, not monitoring, discover failed messages several times a month.
- Dozens of partner EDI or file exchanges are each scripted differently.
When to choose something else
- External partners needing request-response access to your systems are served by API Enablement, with a gateway and developer portal.
- Moving data into a lake or warehouse for analytics belongs to Data Integration & Ingestion pipelines.
- With fewer than five stable integrations, simple scheduled jobs may cost less than an event backbone.
Integration modernization components
Integration inventory and catalogue
Every flow, file drop, DB link and ESB service listed with producer, consumer, volume, latency need and failure history.
Event design
Business event definitions, schemas, versioning rules and an owning team per event type.
Event capture
Outbox in applications we can change; CDC through Debezium, Datastream or DMS for those we cannot.
Routing and transport
Event bus, topics and per-consumer queues with ordering and dead-lettering.
Transformation and B2B
Serverless or iPaaS flows for mapping, enrichment, EDI and partner file exchange.
Tracking and replay
Correlation IDs, message tracking, DLQ dashboards and replay tooling for operators.
How Cloud Native Integration Works, Step by Step
Event Backbone with CDC, Routing and Replay
Producer systems emit business events through an outbox or CDC onto a managed event bus, which routes a copy to each consumer's own queue. Consumers transform and apply events independently, while failed messages land in dead-letter queues that operators can inspect and replay after a fix.
Select any component
Event Backbone with CDC, Routing and Replay: click any of its 10 components to see its role and the design choice NTSPL makes for it, or trace the 6-step primary path.
Managed messaging and iPaaS by cloud
EventBridge routes events with content-based rules and archive-and-replay; each consumer gets an SQS queue with a DLQ. Amazon MQ bridges legacy JMS producers during migration, and Transfer Family replaces self-hosted SFTP.
Logic Apps Standard is Microsoft's migration path for BizTalk workloads, including EDI and AS2. Service Bus topics with sessions give ordered delivery per entity, and Event Grid handles lightweight fan-out.
Pub/Sub carries events with ordering keys and dead-letter topics, Eventarc routes them to Cloud Run, Application Integration provides iPaaS flows to SaaS and databases, and Datastream captures CDC.
Integration design choices to settle early
Events, commands or queries
Publish events for facts other systems may react to, such as order placed. Send commands through a queue when exactly one system must act, such as create shipment. Use APIs for queries that need a current answer. Mixing these on one topic is the usual source of new coupling, because consumers start depending on fields meant for one recipient.
Fat or thin events
Carry enough data for most consumers to act without calling back when payloads stay under roughly 64-256 KB and hold no unnecessary personal data. Use thin events carrying a reference when data is large or sensitive and consumers can fetch details through an authorised API. Decide per event type and record the reason in the catalogue.
Ordering and delivery guarantees
Guarantee ordering only per business key, such as order or account ID, never globally; use FIFO or session queues or ordering keys. Assume at-least-once delivery everywhere and make every consumer idempotent with a processed-message table. Exactly-once features in brokers are narrower than they sound and rarely cover your own database writes.
iPaaS or code
Use iPaaS tools such as Logic Apps, Application Integration or AppFlow for SaaS connectors, EDI and simple mappings maintained by an integration team. Write code in functions or containers for flows with complex logic, sustained volumes above a few hundred messages a second, or heavy testing needs. Never rebuild a central ESB inside an iPaaS.
Retiring the ESB one domain at a time
Integration inventory and catalogue
List every flow, file and DB link with owners, volumes and failure history.
Event backbone and standards
Stand up the bus, queues, DLQs, archive and naming, schema and correlation standards.
First domain migration
Rebuild one domain's flows as events and run them in parallel with the ESB.
Cutover and flow retirement
Reconcile outputs, switch consumers and switch off the old flows.
Remaining domains and hub shutdown
Repeat per domain until the ESB and file-transfer servers can be decommissioned.
What you receive
Integration reliability targets
| KPI | Typical target |
|---|---|
| End-to-end latency of migrated flows | Seconds to minutes instead of hours |
| Messages lost | Zero; every failure held in a DLQ |
| Time to add a new consumer | Under 1 week |
| Time to detect a failed message | Alert within 5 minutes |
| ESB flows remaining | Falling to zero on plan |
Integration modernization pitfalls
| Risk | How NTSPL prevents it |
|---|---|
| Rebuilding the ESB in the cloud | Transformations stay with consumers or domain-owned flows; the backbone only routes and stores. |
| Drift during parallel running | Old and new paths run side by side with automated reconciliation of counts and values before any cutover. |
| Schema changes breaking consumers | A schema registry with compatibility checks; breaking changes require a new event version. |
| Personal data spreading through events | Data minimisation is reviewed per event, and sensitive fields are tokenised or replaced by references. |
Is an iPaaS a like-for-like replacement for our ESB?
No, and treating it as one repeats the original problem. An ESB centralises routing, transformation and orchestration in one hub run by one team. Cloud native integration splits those concerns: a managed bus or queue for routing, transformation close to consumers, and workflow orchestration only where a business process needs it. An iPaaS such as Logic Apps or Application Integration is the right tool for SaaS connectors, EDI and partner mappings, but high-volume domain events belong on EventBridge, Service Bus, Pub/Sub or Kafka. Plan the ESB exit by domain and decommission each flow as it moves.
Key facts: Cloud Native Integration
Cloud Native Integration in Practice
2 Cloud Native Integration scenarios, from Agritech produce marketplace and Online property portal, plus related platform case studies.
Agritech marketplace moves 14 file-based integrations to events and speeds farmer payouts
Property portal delivers builder leads in 20 seconds, not 4 hours, after leaving BizTalk
General insurer cuts motor-claim intake from 2 days to 25 minutes with serverless
Dealership group cuts OEM warranty claim rejections by integrating DMS through Service Bus
Cloud Native Integration: Frequently Asked Questions
What does ESB-to-cloud integration modernization cost?
Plan a five-week foundation, then roughly 3-5 weeks per integration domain such as orders, inventory or customer, typically INR 15-30 lakh per domain. Managed messaging usually costs far less than ESB licences and support; a mid-size estate often runs INR 30,000-1.5 lakh a month in messaging and iPaaS charges.
How do we keep integration messages and logs compliant in India?
Run brokers and flows in Indian regions, encrypt messages at rest, and keep message-tracking logs for 180 days to meet CERT-In directions. Under the DPDP Act 2023, minimise personal fields in events and set retention per event type so archives do not keep data longer than the purpose requires.
Can BizTalk or Oracle SOA flows be migrated automatically?
Partially. Logic Apps Standard can reuse BizTalk XSLT maps and schemas, and Microsoft publishes migration guidance, but orchestrations usually need redesign. For Oracle SOA and TIBCO we re-implement flows, using automated extraction of XSLT and endpoint lists to speed up the work.
Should we use Kafka or a cloud event bus?
Use Kafka, as Amazon MSK, Event Hubs' Kafka endpoint or Google Cloud Managed Service for Apache Kafka, for high-volume streams, long retention and stream processing. Use EventBridge, Service Bus or Pub/Sub for business events at moderate volume with routing rules and lower operating effort. Many estates run both.
What happens to partners who still exchange files over SFTP?
They keep SFTP if they need it, but on managed endpoints such as AWS Transfer Family or Azure Blob Storage SFTP, where a file's arrival triggers an event and a processing flow. Partners able to move to APIs or AS2 are migrated over time, starting with the highest-volume ones.
How do operators find a lost message?
Every message carries a correlation ID from the source transaction. The tracking dashboard shows its path across bus, queues and flows. Failures sit in dead-letter queues with the error reason attached, and can be replayed individually or in bulk once the fault is fixed.