India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
Governed Cloud Data Lake Design on S3, ADLS Gen2 and Cloud Storage
A cloud data lake is low-cost object storage (Amazon S3, Azure Data Lake Storage Gen2 or Google Cloud Storage) that holds raw and refined data of any shape in open file formats, catalogued and access-controlled for analytics and data science. NTSPL designs zones, partitioning, catalogues, lifecycle rules and permissions so the lake stays queryable and affordable as it grows.
Delivered on
In scope
Why Lakes Turn Into Unqueryable Archives
Data lakes rarely fail on day one. They fail in year two, when 600 TB of JSON logs, CSV dumps and Parquet files sit in folders named final_v3, nobody knows which prefix is authoritative, Athena or Synapse queries scan the whole bucket, and storage is billed at hot-tier rates for data last read 14 months ago. Security teams then lock the bucket down and the lake becomes a write-only archive.
NTSPL builds lakes as products with rules. Every dataset lands in a named zone with an owner, a partition scheme agreed against real query patterns, a catalogue entry and a lifecycle policy. Small files are compacted, raw JSON is converted to Parquet within a day, and access is granted through tags rather than bucket policies written by hand. Lifecycle rules are tested against real access logs.
When a Governed Object-Storage Lake Is the Right Base
Signals you need Cloud Data Lake
- Semi-structured logs, clickstream or IoT data exceed 1 TB a day and are too costly to load into a warehouse raw
- Data scientists need file-level access to years of history the warehouse has aggregated away
- An existing bucket or container exceeds 100 TB with no catalogue, owners or lifecycle rules
- Retention rules (CERT-In 180-day logs, IRDAI or SEBI record-keeping) require cheap, immutable long-term storage
When to choose something else
- Your data is mostly structured ERP and CRM tables under 10-20 TB: a Cloud Data Warehouse is simpler to run and govern.
- You need ACID updates, time travel and BI on the same tables from day one: go to Lakehouse Architecture, which adds open table formats over the lake.
Zones, Formats and Controls in a Lake Build
Zone design
Raw, standardised, curated and sandbox zones with naming standards, ownership and promotion rules between them.
Formats and partitioning
JSON or CSV to Parquet conversion, compression, partition keys chosen from query logs and compaction to 128-512 MB files.
Catalogue and discovery
Glue Data Catalog, Purview or Dataplex registration with schema, owner, classification and freshness for every dataset.
Access model
Tag-based or attribute-based grants by domain and sensitivity, replacing per-user bucket policies and shared keys.
Lifecycle and retention
Tiering to infrequent, archive or cold classes by last access, with immutability where regulation demands it.
How Cloud Data Lake Works, Step by Step
Zoned Object-Storage Lake with Tag-Based Access
Data lands unchanged in a raw zone, is converted and partitioned into a standardised zone, then shaped into curated datasets. A catalogue and tag-based policies govern who can read each zone, while lifecycle rules tier ageing data.
Select any component
Zoned Object-Storage Lake with Tag-Based Access: click any of its 8 components to see its role and the design choice NTSPL makes for it, or trace the 6-step primary path.
Lake Building Blocks on Each Hyperscaler
Lake Formation tag-based access control (LF-TBAC) replaces bucket policies for analytics users; Athena and Redshift Spectrum query the same catalogue.
Hierarchical namespace with POSIX-style ACLs on ADLS Gen2; OneLake shortcuts expose existing containers to Fabric without copying data.
BigLake tables give BigQuery-style row and column security over Parquet in Cloud Storage; Autoclass handles tiering without hand-written rules.
Partitioning, Format and Tiering Choices
Partition keys
Partition on the column used in more than 80% of query filters, usually event date. Avoid high-cardinality keys such as customer ID, which create millions of tiny folders. Target partitions of at least 256 MB; if daily partitions are smaller, partition monthly and sort within files. Re-check against query logs after three months of real use.
File format
Store raw data in source format for replay, but serve everything else as Parquet with Zstd or Snappy compression. Parquet typically cuts scan volume by 70-90% versus JSON for column-selective queries. Use Avro only on the streaming edge where schema evolution per record matters, and convert within 24 hours. Keep one format per zone.
Storage tiering
Let access logs decide. Data untouched for 30 days moves to infrequent access, 180 days to archive, unless a dashboard reads it monthly. Use automatic tiering (S3 Intelligent-Tiering, Autoclass) for mixed or unpredictable prefixes. Watch retrieval fees: archive tiers are cheap to store and expensive to scan in bulk. Model retrieval before archiving.
Single lake or per-domain storage
Use one storage account, bucket set or project per domain and environment, joined through a shared catalogue. This keeps blast radius small, makes cost allocation trivial and lets domains own lifecycle rules. A single giant bucket is easier on day one and much harder to govern by year two. Shared datasets cross domains by catalogue grants.
Lake Build Sequence from Landing to Consumption
Query and retention discovery
Profile existing buckets, query logs and retention obligations; agree zones and owners.
Storage, keys and access foundation
Create per-domain storage, customer-managed keys, catalogue and tag taxonomy.
Conversion and compaction pipelines
Parquet conversion, partitioning and compaction for the priority datasets.
Catalogue and access cutover
Register datasets, move users to tag-based grants and remove legacy keys.
Lifecycle rollout and clean-up
Apply tiering, delete orphaned prefixes with owner sign-off and publish a storage report.
What you receive
Signals of a Healthy, Affordable Lake
| KPI | Typical target |
|---|---|
| Datasets catalogued with an owner | 100% of readable datasets |
| Average file size in standardised zone | 128-512 MB |
| Data in hot tier untouched for 90 days | Under 10% of total |
| Bytes scanned per typical query | 70-90% lower than raw JSON baseline |
Lake Anti-Patterns We Design Out
| Risk | How NTSPL prevents it |
|---|---|
| Small-file explosion from streaming writes slows every query | Scheduled compaction with file-size monitoring and alerts when median size falls below 64 MB. |
| Personal data lands in raw zone with broad read access | Raw zone is readable only by pipelines; classification scans tag PII before promotion. |
| Archive retrieval bills spike when someone scans old data | Archive-tier prefixes are excluded from default catalogue views and need an approved restore. |
Data lake vs data warehouse: when should an Indian enterprise build a lake?
Build a lake when a large share of your data is semi-structured or high-volume (logs, clickstream, sensor readings, documents), when data scientists need raw history, or when long retention must be cheap. Build a warehouse when data is mostly structured, users are mainly BI analysts and volumes are moderate. Many organisations need both, which is why most new builds in 2026 use a lakehouse: lake storage with open table formats and warehouse-style SQL on top. A plain lake without a table format is now mainly justified for raw retention and data science.
Key facts: Cloud Data Lake
Cloud Data Lake in Practice
2 Cloud Data Lake scenarios, from OTT streaming (subscription video) and Agri commodity trading (pulses, oilseeds and spices), plus related platform case studies.
OTT platform cuts lake query scans 86% and decides content renewals on watch data
Agri commodity trader puts mandi, weather and satellite data in one lake for buying calls
Regional-language OTT service halves video start time and cuts origin egress by 82%
News channel moves 1.4 PB of footage from LTO tape to tiered Blob Storage
Cloud Data Lake: Frequently Asked Questions
How much does it cost to store data in a cloud lake?
Standard object storage in Indian regions costs roughly INR 1.8-2.2 per GB per month, and archive classes a small fraction of that. A 200 TB lake with sensible tiering usually runs INR 2-3 lakh a month for storage. Our build engagements typically cost INR 20-40 lakh over 8-10 weeks.
Can a data lake stay entirely in India?
Yes. We pin storage, catalogue and query services to Indian regions such as AWS Mumbai or Hyderabad, Azure Central India and Google Cloud Mumbai or Delhi, and use policies that block replication outside India for datasets tagged as regulated. Access logs stay in India too.
Should raw data ever be deleted?
Yes, on a schedule. Raw zones exist for replay, not for ever. We keep raw data 30-90 days for most sources, longer only where a regulator or contract requires it, and delete personal data when its DPDP purpose ends. Deletion jobs are logged as evidence for audits.
Which query engine should we use on the lake?
Use the engine native to your primary cloud: Athena or Redshift Spectrum on AWS, Synapse serverless SQL, Fabric or Databricks on Azure, and BigQuery over BigLake tables on Google Cloud. All of them read Parquet from the same catalogue. Pick one primary engine to keep skills focused.
Can you fix an existing messy lake rather than starting again?
Usually. We profile access logs to find what is actually read, register and convert those datasets first, then archive or delete the rest with owner sign-off. Clients typically reclaim 30-50% of stored volume in the process. Consumers are moved to the cleaned datasets before anything is deleted.
How do you stop the lake becoming a data swamp?
Nothing becomes readable without a catalogue entry, owner and sensitivity tag, and sandbox prefixes expire after 90 days. A monthly report lists uncatalogued prefixes and orphaned data, and owners must act on it. Uncatalogued data older than 60 days is quarantined, then removed.