India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
Lakehouse Architecture on Apache Iceberg and Delta Lake Tables
A lakehouse architecture stores data once in cloud object storage using an open table format (Apache Iceberg, Delta Lake or Hudi) that adds ACID transactions, schema evolution and time travel, so SQL analytics, BI and machine learning share the same tables. NTSPL designs medallion layers, catalogues and engine choices so one governed copy serves every workload.
Delivered on
In scope
The Cost of Keeping Lake, Warehouse and ML Copies in Sync
Many enterprises now pay to store the same data three times: raw in the lake, modelled in the warehouse, and exported again into feature files for data scientists. Each copy has its own pipeline, its own lag and its own access rules. When a return, cancellation or price correction arrives, the lake has to rewrite whole partitions, the warehouse reloads overnight and the ML team trains on numbers finance has already restated.
NTSPL designs lakehouses around one table per business entity in Iceberg or Delta, with MERGE-based upserts, bronze-silver-gold layers and one catalogue enforcing access across Spark, SQL and BI engines. We choose the table format and catalogue from the engines you will actually run, and prove interoperability before committing. Table maintenance, snapshot expiry and erasure across snapshots are automated from the first release, because these are the jobs most lakehouse teams forget until storage doubles.
When One Open-Format Copy Should Serve Every Workload
Signals you need Lakehouse Architecture
- The same entities are copied into a lake, a warehouse and ML feature files, each refreshed on different schedules
- Late updates such as returns, cancellations or quality re-grades force full partition rewrites in the lake
- Spark data engineering and SQL or BI users need the same tables with one set of permissions
- Data volume exceeds 50 TB or includes large semi-structured or time-series datasets alongside ERP tables
When to choose something else
- Your data is under about 10 TB, almost entirely structured, and used only for BI: a Cloud Data Warehouse is simpler and needs fewer skills.
- You lack any Spark or Python engineering capacity and do not plan to build it: prefer a managed warehouse and revisit later.
Table Formats, Layers and Catalogues We Implement
Table format and catalogue
Iceberg or Delta selection, catalogue choice (Unity Catalog, AWS Glue/S3 Tables, BigLake metastore, OneLake) and cross-engine read and write tests.
Medallion layers
Bronze for raw ingested tables, silver for cleaned and conformed entities, gold for business aggregates and features, with promotion rules.
Upsert and history handling
MERGE patterns for late and corrected records, change data feeds, snapshot retention and time-travel windows.
Table maintenance
Compaction, Z-order or clustering, snapshot expiry and orphan-file clean-up scheduled per table by write pattern.
Serving engines
SQL warehouse endpoints for BI, Spark for engineering and ML, and direct BI reads such as Power BI Direct Lake.
How Lakehouse Architecture Works, Step by Step
Medallion Lakehouse on Open Table Formats
Ingested data lands in bronze Iceberg or Delta tables, is merged into conformed silver entities, and is aggregated into gold tables and ML features. A single catalogue governs every engine that reads or writes.
Select any component
Medallion Lakehouse on Open Table Formats: click any of its 8 components to see its role and the design choice NTSPL makes for it, or trace the 6-step primary path.
Lakehouse Engines and Catalogues by Cloud
S3 Tables manage Iceberg compaction and snapshot expiry automatically; Athena, Redshift and EMR read the same tables through the Glue catalogue.
Databricks suits heavy engineering and ML; Fabric suits Power BI-centric teams. OneLake shortcuts let Fabric read Databricks Delta tables without copying.
BigQuery can manage Iceberg tables in Cloud Storage, while Spark on Dataproc or Serverless for Apache Spark reads and writes them through the BigLake metastore.
Iceberg or Delta, Catalogue and Engine Choices
Iceberg or Delta Lake
Choose Delta when Databricks or Fabric is the primary engine; it is native there and best optimised. Choose Iceberg when several engines from different vendors must write, such as Athena, Spark on EMR, Snowflake and BigQuery, because its catalogue specification is widely supported. Avoid mixing formats for one entity; interoperability features exist but add a second failure point.
Catalogue as the control point
Pick the catalogue before the engine: Unity Catalog for Databricks-centred estates, Glue Data Catalog or S3 Tables on AWS, BigLake metastore on Google Cloud, OneLake for Fabric. Every engine must read permissions from it. If a tool can only bypass the catalogue with storage keys, keep it off production tables.
Merge-on-read or copy-on-write
Use copy-on-write for gold tables that are read hundreds of times a day and written hourly or less. Use merge-on-read or deletion vectors for silver tables with frequent small updates, such as order status changes, then compact nightly. Measure: if reads slow by more than 30% between compactions, compact more often.
Snapshot retention
Keep 7 days of snapshots for most silver and gold tables, which covers rollback after a bad load, and 30-90 days only where audit needs point-in-time queries. Longer retention keeps old files alive and quietly doubles storage. Erasure requests under DPDP must also expire old snapshots, or deleted personal data remains readable.
Lakehouse Rollout from Bronze to Gold
Engine and format proof
Test read and write of sample tables from every planned engine through the chosen catalogue.
Catalogue, storage and security foundation
Set up catalogue, storage per domain, identity integration and permission model.
Bronze and silver for the first domain
Ingest into bronze, build MERGE logic and quality gates for core entities.
Gold tables and BI serving
Publish gold tables, connect BI through SQL endpoints and tune clustering.
Maintenance automation and next domains
Automate compaction and snapshot expiry, then onboard further domains.
What you receive
Lakehouse Freshness, Efficiency and Reuse Measures
| KPI | Typical target |
|---|---|
| Copies of each core entity across platforms | One governed table |
| Silver freshness after source change | Under 30 minutes |
| Gold table query p95 for dashboards | Under 5 seconds |
| Storage overhead from snapshots and orphan files | Below 20% of live table size |
Lakehouse Design Traps and Countermeasures
| Risk | How NTSPL prevents it |
|---|---|
| Small files and unexpired snapshots inflate storage and slow reads | Per-table compaction and expiry jobs with alerts on file count and snapshot age. |
| An engine writes to tables outside the catalogue and corrupts metadata | Only catalogue-registered engines receive write permissions; storage keys are not issued. |
| Format features differ across engines, breaking reads | CI tests every engine against a reference table whenever versions or table features change. |
Lakehouse vs data warehouse plus data lake: is one platform really cheaper?
Usually yes once data passes about 50 TB or ML is a real workload, because you stop paying for duplicate storage, duplicate pipelines and the reconciliation between them. A separate warehouse can still be faster for highly concurrent BI on small data, and managed warehouses need fewer specialist skills. The honest test is your own workload: we run the top 20 queries and one training pipeline on both designs for two weeks and compare cost, latency and engineering effort before recommending.
Key facts: Lakehouse Architecture
Lakehouse Architecture in Practice
2 Lakehouse Architecture scenarios, from Steel (flat products) and E-commerce (horizontal marketplace), plus related platform case studies.
Steel plant traces every coil defect back to heat chemistry within 20 minutes
E-commerce marketplace keeps one Iceberg order table for finance, growth and ML
Agri-inputs company sees dealer secondary sales daily instead of 45 days late
Dairy cooperative sees milk procurement by collection centre within 15 minutes
Lakehouse Architecture: Frequently Asked Questions
How much does a lakehouse implementation cost?
A first domain with bronze, silver and gold layers and BI serving typically takes 12-16 weeks and costs INR 45 lakh to 1 crore, depending on source count and ML scope. Run cost depends mostly on compute; storage in Indian regions is a minor share for most clients.
Can Power BI read lakehouse tables directly?
Yes. On Fabric, Power BI uses Direct Lake mode on Delta tables in OneLake, avoiding import refreshes. On other platforms Power BI connects to a SQL warehouse endpoint, such as Databricks SQL or Athena, which reads the same tables. Permissions come from the same catalogue.
Is Iceberg or Delta better for Indian enterprises?
Neither is universally better. Delta is the native choice on Databricks and Fabric; Iceberg has the broadest multi-engine support across AWS, Google Cloud and Snowflake. We decide by your primary engines and prove reads and writes before committing. Switching later is possible but costly.
How do we delete personal data under the DPDP Act in a lakehouse?
Erasure needs three steps: delete or update rows, expire the snapshots that still reference old files, and remove orphaned files. We automate all three on a schedule and keep a deletion log as evidence for the Data Protection Board if needed.
Do we still need a separate warehouse?
Usually not for new builds. Gold tables and serverless SQL endpoints serve most BI needs. Some clients keep an existing warehouse for a defined period while dashboards are re-pointed, then retire it. Re-pointing normally takes one or two reporting cycles, after which the separate warehouse licence and its pipelines can be removed.
Can a lakehouse span two clouds?
Technically yes, with Iceberg tables and engines in each cloud reading a shared catalogue, but cross-cloud reads incur egress and latency. We recommend one primary storage location in India and sharing curated gold tables, not raw data, across clouds. Each cloud then serves its own users.