India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
AWS Data Lake, Analytics & Redshift Services for Indian Enterprises
AWS Data and Analytics is a lakehouse pattern that stores raw and curated data in Amazon S3, catalogues and transforms it with AWS Glue, governs access with Lake Formation, and queries it through Athena, Redshift Serverless and Amazon Quick Sight. NTSPL builds these platforms in AWS Mumbai for Indian enterprises that need governed, cost-predictable reporting on sales, operations and IoT data.
AWS services in scope
The Problem Amazon S3 with Apache Iceberg (including Amazon S3 Tables) and AWS Glue Solve
Analytics in most Indian mid-size companies still means an ERP extract, a dozen Excel files emailed every Monday, and a BI tool pointing at a reporting replica that slows down every month-end. Data from DMS, CRM, IoT devices and e-commerce marketplaces sits in silos, each with different product codes and dealer IDs. Queries that join two years of transactions time out, and nobody trusts the numbers because two dashboards show different sales for the same territory. Adding a new data source takes a quarter, not a week.
NTSPL builds a lakehouse on Amazon S3 with open Apache Iceberg tables, so storage is cheap, durable and readable by several engines. AWS Glue ingests and conforms data into raw, cleaned and curated zones, Lake Formation enforces who sees which rows and columns, and we pick the engine per workload: Athena for ad hoc and occasional queries, Redshift Serverless for heavy concurrent BI. Every pipeline ships with data-quality rules and a cost estimate per query pattern.
When an S3 Data Lake with Athena and Redshift Is the Right Analytics Platform (and When It Is Not)
Signals that point to data & analytics on AWS
- Month-end reporting queries on the ERP or a read replica take more than 10 minutes or slow down transactions.
- Sales, dealer or channel data comes from three or more systems with conflicting master data and manual Excel reconciliation.
- You hold IoT or event data above 100 million rows a month that current databases cannot retain or query affordably.
- Different teams publish different numbers for the same KPI, and leadership has stopped trusting dashboards.
- A DPDP Act review asks where personal data sits in analytics copies and who can see it.
- Your BI licence bill exceeds INR 30 lakh a year mainly for read-only viewers.
When a different approach is better
- If your data is under 50 GB in one PostgreSQL or SQL Server database and a dozen users run reports, a lake adds moving parts; use a read replica with Amazon RDS and Quick Sight directly.
- If you need sub-second operational lookups or transactional updates per record, a lake or warehouse is the wrong store; use Amazon Aurora, DynamoDB or OpenSearch Service for that path.
- If your company is standardizing on a single cross-cloud platform such as Databricks or Snowflake for all analytics, build the S3 storage and Lake Formation layer, but do not duplicate the engine with Redshift.
Analytics Building Blocks: S3 and Iceberg Tables, Glue, Lake Formation, Athena, Redshift Serverless and Quick Sight
| Service | Role in the architecture | Choose it when |
|---|---|---|
| Amazon S3 with Apache Iceberg (including Amazon S3 Tables) | Holds raw, cleaned and curated data as Parquet files under Iceberg tables, giving ACID updates, schema evolution and time travel, with S3 Tables offering managed Iceberg storage with automatic compaction. | You need one durable copy of data that Athena, Redshift, Glue and EMR can all read, with updates and deletes for corrections or DPDP erasure requests. |
| AWS Glue | Provides the Data Catalog, crawlers, serverless Spark and Python ETL jobs, and Glue Data Quality rules that run inside pipelines. | Data arrives from ERP, DMS, flat files and databases on schedules, and you want serverless transformation without running Spark clusters. |
| AWS Lake Formation | Centralizes permissions on catalogue databases, tables, columns and rows using LF-Tags and data filters, enforced across Athena, Redshift Spectrum, Glue and EMR. | Different users must see different slices, such as regional managers seeing only their states or analysts seeing masked personal data. |
| Amazon Athena | Runs serverless SQL directly on S3 data, billed per terabyte scanned, with federated connectors for operational databases. | Query volume is moderate or spiky, analysts run exploratory queries, or you want SQL on the lake without a running warehouse. |
| Amazon Redshift Serverless | Provides a columnar warehouse that scales capacity in Redshift Processing Units, with Spectrum to query the lake and zero-ETL integrations from Aurora and RDS. | Dozens of concurrent dashboard users run repeated joins over hundreds of millions of rows and need sub-second to few-second response. |
| Amazon Quick Sight (formerly QuickSight) | Delivers dashboards with the SPICE in-memory engine, row-level security, embedded analytics and natural-language querying. | You need dashboards for hundreds of field and management users, including on mobile, without per-desktop BI licences. |
| Amazon Kinesis Data Streams, Amazon MSK and Amazon Data Firehose | Ingest event and telemetry streams, with Firehose delivering buffered, partitioned data into S3 or Iceberg tables. | Data arrives continuously from meters, apps or order systems and must be queryable within minutes rather than the next morning. |
Three-zone S3 Iceberg lakehouse with governed Athena, Redshift and Quick Sight access
Three-zone S3 Iceberg lakehouse with governed Athena, Redshift and Quick Sight access
Batch extracts from ERP and DMS, and streams from devices or apps, land in an S3 raw zone, where Glue jobs validate, conform and write Iceberg tables into cleaned and curated zones. Lake Formation governs every read, Athena and Redshift Serverless serve queries, and Quick Sight dashboards apply row-level security for business users.
Select any component
Three-zone S3 Iceberg lakehouse with governed Athena, Redshift and Quick Sight access: click any of its 9 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
Data Platform Engineering Decisions: Table Format, Partitioning, Query Engine and Access Model
Open table format and file layout
We default to Apache Iceberg on Parquet with Zstandard compression, because Athena, Redshift, Glue and EMR all read and write it and it supports row-level deletes for corrections and erasure requests. Target file sizes of 128 to 512 MB and schedule compaction; streaming ingestion without compaction produces millions of small files that make Athena slow and expensive. Use S3 Tables when you want AWS to run compaction and snapshot management for you; use self-managed Iceberg in general-purpose buckets when you need custom maintenance or cross-account bucket policies.
Partitioning for how people actually filter
Partition on the columns that appear in most WHERE clauses, usually business date and sometimes region or state, not on high-cardinality IDs. Iceberg hidden partitioning lets us partition by day or month of a timestamp without users writing partition predicates. A sales fact table partitioned by month and state typically reduces Athena scan per dashboard query by 90 percent or more compared with an unpartitioned table, which directly cuts cost.
Athena versus Redshift Serverless
Athena wins when queries are occasional or exploratory and data is well partitioned: you pay only per TB scanned. Redshift Serverless wins when 30 or more users hit dashboards through the working day, joins are complex, or you need consistent sub-5-second response; you pay RPU-hours while queries run. A common NTSPL pattern is curated data in Iceberg, a Redshift star schema for the top 20 dashboards, and Athena for everything else.
Access model: Lake Formation first, BI second
Permissions defined only in the BI tool leak the moment someone queries Athena directly. We grant access in Lake Formation using LF-Tags for domain and sensitivity, apply data filters for row-level restrictions such as territory, and mirror the same rule in Quick Sight row-level security through a shared permissions table. Use IAM Identity Center trusted identity propagation where supported so queries run as the end user and appear in CloudTrail under their identity.
Streaming freshness versus cost
Real-time is expensive when nobody looks at the dashboard every minute. For most sales and meter analytics, Firehose buffering of 5 minutes and hourly Iceberg compaction give 10 to 15 minute freshness at a fraction of the cost of Managed Service for Apache Flink. We reserve true stream processing for alerts that act within seconds, such as tamper or theft detection, and keep everything else micro-batch.
How NTSPL Builds an S3 Lakehouse from Source Systems to Dashboards
KPI and source mapping
Agree the first 15 to 25 KPIs with business owners, trace each to source tables, and document conflicting definitions and master data gaps.
Lake foundation and catalogue
Create raw, cleaned and curated buckets or S3 table buckets, KMS keys, Glue databases and Lake Formation admin settings with LF-Tag taxonomy.
Ingestion pipelines
Build Glue jobs, DMS change data capture tasks or Firehose streams per source, with Data Quality rules and quarantine paths from day one.
Curated models and engine choice
Design conformed dimensions and facts in Iceberg, load high-concurrency marts into Redshift Serverless, and set Athena workgroups with scan limits.
Governed dashboards
Build Quick Sight dashboards with row-level security mirroring Lake Formation filters, and run user acceptance against the old Excel numbers.
Cost and freshness tuning
Review query logs after four weeks, add partitions or materialized views where scans are heavy, and adjust Redshift base and maximum RPU.
What you receive
Governing the Lake: Lake Formation Tags, Row-Level Filters and Encryption of Personal Data
Analytics platforms concentrate data that was spread across many systems, which makes them a single high-value target and a DPDP exposure. The risks are broad read access by default, personal data copied into curated tables nobody needs, and BI-only security bypassed through direct SQL. Controls must operate at the catalogue and storage layer, with personal data minimized before it reaches analysts, and every query attributable to a user.
- Lake Formation LF-Tag grants so new tables tagged sensitivity=pii are restricted automatically
- Lake Formation data filters for row and cell-level restriction by territory or customer segment
- Tokenization or hashing of phone, email and customer IDs in Glue before data reaches curated zones
- Separate KMS keys for raw and curated zones, with raw-zone decrypt limited to pipeline roles
- Iceberg row-level deletes and snapshot expiry to honour DPDP erasure requests in analytics copies
Analytics Pricing: Athena Bytes Scanned, Glue DPU-Hours, Redshift RPU-Hours and BI Seats
Each engine has its own meter. Athena bills per TB scanned with a 10 MB minimum per query, so file format and partitioning set the bill. Glue bills DPU-hours per second for ETL. Redshift Serverless bills RPU-hours per second with a 60-second minimum while queries run, plus managed storage. Quick Sight bills per author and reader or by capacity, plus SPICE storage. S3 storage itself is usually the smallest line item.
- Parquet with partitioning and compaction to cut Athena scan volume by 90 percent or more
- Glue Flex execution for non-urgent nightly jobs at a lower DPU-hour rate
- Redshift Serverless maximum RPU and usage limits to cap runaway query spend
- Athena workgroup per-query scan limits and query result reuse for repeated dashboard queries
- Quick Sight SPICE for heavily viewed datasets to avoid repeated warehouse queries
Athena vs Redshift Serverless vs Snowflake on AWS: which analytics engine should we choose?
Decide by concurrency, query pattern and platform strategy. Athena suits ad hoc and moderate-volume SQL directly on S3, with no capacity to manage and cost proportional to data scanned; it slows down with many concurrent heavy joins. Redshift Serverless suits repeated BI workloads with dozens to hundreds of concurrent users and complex joins, billed on RPU-hours, with zero-ETL from Aurora and native AWS integration, and data stays in Mumbai under your KMS keys. Snowflake on AWS suits organizations that want one engine across clouds, strong data sharing with partners and a credit-based model, but adds a separate vendor contract and governance plane. For most AWS-centric Indian enterprises, NTSPL recommends Iceberg on S3 with Athena plus Redshift Serverless for the hot dashboard layer.
Key facts about data & analytics on AWS
Data & Analytics on AWS in Practice
3 data & analytics scenarios from Agri-inputs (seeds and crop protection), Power distribution (DISCOM), Quick-commerce grocery: the starting position, the services chosen, the architecture, the rollout and the measured outcome.
Agri-inputs company sees dealer secondary sales daily instead of 45 days late
Power distribution company analyses 190 million smart-meter reads a day for loss detection
Quick-commerce grocer cuts dark-store stock-outs with hourly SKU demand analytics
Textile exporter consolidates three SAP landscapes from three hosts onto one AWS platform
Data & Analytics on AWS: Frequently Asked Questions
What does a data lake on AWS cost per month for a mid-size company?
For 5 to 20 TB of data, nightly Glue pipelines, Athena for analysts and Redshift Serverless for 50 to 100 dashboard users, monthly spend typically lands between INR 1.5 and 4 lakh, plus Quick Sight user or capacity pricing. The biggest variables are Redshift working hours and how well data is partitioned for Athena.
How do we move from our on-premises SQL Server data warehouse to AWS?
Usually in two tracks. First, replicate source systems into S3 with AWS DMS change data capture so the lake fills without touching the old warehouse. Then rebuild the highest-value marts in Iceberg and Redshift, validate numbers side by side for one or two reporting cycles, and retire SSIS jobs and cubes in waves. AWS Schema Conversion Tool helps translate stored procedures.
Can analytics data stay in India under the DPDP Act?
Yes. All lake buckets, Glue catalogue, Redshift and Athena run in ap-south-1, with a backup or replica in ap-south-2 if needed. The DPDP Act also expects purpose limitation and erasure, so NTSPL minimizes personal data in curated zones, tags it in Lake Formation, and uses Iceberg deletes to remove records when a data principal withdraws consent.
Is Redshift Serverless always cheaper than provisioned Redshift?
No. Serverless is cheaper for workloads with idle hours, such as dashboards used mainly during business hours. For a warehouse that runs heavy loads 18 to 24 hours a day, provisioned RA3 nodes with reserved pricing often cost less. We model both using one month of query history before deciding.
How fresh can dashboards be?
Batch pipelines from ERP and DMS typically refresh every one to four hours, and streaming pipelines through Kinesis or MSK with Firehose reach 5 to 15 minutes. Sub-minute freshness is possible with Managed Service for Apache Flink, but we recommend it only where someone acts on the data within that minute.
Can business users ask questions in plain language instead of building reports?
Yes, Amazon Quick Sight includes natural-language querying over curated datasets. It works well when datasets have clear field names, synonyms and defined measures, which is why we build a semantic layer first. It does not replace governed dashboards for statutory or board reporting, where exact definitions matter.