India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
Cloud Logging & Threat Detection Pipeline Design and Build
Cloud logging and threat detection is the engineered pipeline that collects security-relevant logs from cloud accounts and workloads, stores them for investigation and runs detection rules that raise alerts on attacker behaviour. NTSPL designs which sources to collect, which SIEM and storage tiers to use, and which ATT&CK-mapped rules to run, meeting CERT-In's 180-day retention for Indian organisations.
Delivered on
In scope
Too Many Logs, Too Few Detections: Fixing the Imbalance
Most cloud logging grows by accident. Someone enables every diagnostic setting into one expensive analytics workspace, the bill reaches lakhs a month, and logs are trimmed back until audit trails for half the subscriptions vanish. Elsewhere CloudTrail runs but nobody alerts on it, GuardDuty findings sit unread, and application logs that would show account takeover never leave the container. When CERT-In's six-hour reporting clock starts, the team discovers it cannot reconstruct what an attacker did last Tuesday.
NTSPL designs detection backwards from the threats that matter: identity compromise, privilege escalation, data exfiltration, crypto-mining and ransomware staging. For each, we list the log sources and fields needed, then choose storage tiers so high-value logs are searchable hot and bulk logs sit in cheaper long-term storage for 180 days or more. Detection rules are written as code, mapped to MITRE ATT&CK, tested with attack simulations and tuned until each produces fewer than a handful of false positives a week.
Signs Your Cloud Logs Would Not Survive an Investigation
Signals you need Logging & Threat Detection
- Security log spend exceeds INR 3 lakh a month but fewer than 30 custom detection rules exist
- Any account, subscription or project lacks audit logs retained for 180 days in India
- GuardDuty, Defender or Security Command Center threat findings are not routed to anyone
- An investigation or tabletop failed because sign-in, network or application logs were missing
- You are adopting a SIEM and need to decide what to send before the licence is sized
When to choose something else
- You need people to watch and respond to alerts 24x7; that is managed security operations, which runs on the pipeline we build.
- You need incident playbooks, roles and regulator reporting procedures rather than detection: see Security Incident Readiness.
Log Sources, Storage Tiers and Detection Content We Engineer
Log source catalogue
Define which control-plane, identity, network, workload and application logs are collected per account, with required fields and expected volume.
Storage tiering
Place high-value logs in searchable analytics storage for 90 days and bulk logs in low-cost long-term storage, with 180 days minimum retained in Indian regions.
Native threat detection
Enable GuardDuty, Defender for Cloud workload plans or Security Command Center threat detection, and route their findings into the SIEM.
Detection engineering
Write and test 40-80 custom rules mapped to MITRE ATT&CK for cloud, stored in Git with test cases.
Pipeline health monitoring
Alert when a log source stops sending, volume drops sharply or parsing fails, so silent gaps are caught within hours.
Source to Alert: The Cloud Detection Pipeline
Tiered Security Log Pipeline with ATT&CK-Mapped Detections
Control-plane, identity, network and workload logs flow from every account into a central collection layer that normalises them and splits them by value. High-value data feeds the SIEM's analytics tier where tested detection rules run, while bulk data lands in cheaper retention storage that can still be searched during investigations.
Select any component
Tiered Security Log Pipeline with ATT&CK-Mapped Detections: click any of its 9 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
Sentinel, Security Lake and Google Security Operations Pipelines
Security Lake normalises CloudTrail, VPC Flow Logs, Route 53 and Security Hub findings into OCSF in your own S3 buckets in Mumbai or Hyderabad.
Data collection rules filter and transform logs before ingestion, which is the main lever for Sentinel cost; analytics retention and long-term retention are set per table.
Organisation-level log sinks route audit logs to a central project; Security Operations runs YARA-L detection rules over normalised data.
Hot Versus Cold Storage, Rule Coverage and SIEM Placement
What goes to the analytics tier
Keep identity sign-ins, control-plane audit logs, threat findings, WAF blocks and key vault access in the analytics tier for 90 days. Route flow logs, DNS queries and verbose application logs to long-term storage unless a rule needs them; typically 60-75% of volume. Filtering at source with data collection rules or sink exclusions saves more than any licence discount.
Where to run the SIEM
If your users and endpoints live in Microsoft 365 and Defender, run Microsoft Sentinel even for AWS and GCP logs. If most workloads are on AWS with a small team, Security Lake plus OpenSearch Security Analytics or a partner SIEM is lighter. Google Security Operations suits high-volume estates already standardised on Google Cloud. Avoid two SIEMs.
How many custom rules
Start with 40-60 rules covering the top cloud techniques: console sign-in anomalies, MFA changes, new access keys, role assumption chains, logging disabled, security group opened to the internet, snapshot sharing and mass downloads. Add rules only when each existing one has fewer than five false positives a week. A hundred noisy rules detect less than forty tuned ones.
Retention length
CERT-In requires 180 days of logs within India, so treat 180 days as the floor for all security-relevant sources. Banks, brokers and insurers often keep one to two years on audit and identity logs for regulatory investigations. Use cheaper long-term tiers beyond 90 days; make sure searching them is tested, not assumed.
Building Detection Coverage Threat by Threat
Threat model and source catalogue
Pick 8-12 priority threats and list the sources and fields each needs.
Collection and tier routing
Deploy organisation-wide collection, filtering and tier routing with 180-day retention in India.
Native detector enablement
Enable provider threat detection in every account and region and route findings centrally.
Custom rule engineering and testing
Write rules as code, validate with attack simulations and tune to target false-positive rates.
Health monitoring and handover
Add source health alerts and hand rules, tests and coverage map to the SOC.
What you receive
Coverage, Fidelity and Retention Measures for Detection
| KPI | Typical target |
|---|---|
| Accounts with audit logs retained 180 days in India | 100% |
| Priority ATT&CK techniques with a tested detection | 80% or more |
| False positives per custom rule per week | Under 5 |
| Time to detect a silent log source | Under 2 hours |
Log Cost Blowouts, Silent Sources and Rule Rot
| Risk | How NTSPL prevents it |
|---|---|
| Ingestion costs rise unpredictably after enabling verbose sources | Volume budgets per source, filtering at collection and daily cost alerts with a named owner. |
| A log source silently stops and detections go blind | Heartbeat and volume-drop alerts for every source feed, checked within two hours. |
| Rules decay as cloud services and attacker techniques change | Monthly simulation runs verify each rule fires and a quarterly coverage review adds or retires rules. |
Do we need a SIEM, or are GuardDuty, Defender and Security Command Center enough?
Native threat detectors are essential but not sufficient once you have more than a handful of accounts or any compliance obligation. They detect known bad patterns well, such as crypto-mining, malicious IPs and credential exfiltration, but cannot correlate a suspicious sign-in in Entra ID with a key creation in AWS and a bulk download from GCP, and they do not hold your 180-day searchable trail. A small single-cloud start-up can begin with native detectors plus central log retention. Anyone regulated by RBI, SEBI or IRDAI, or running two clouds, should add a SIEM with custom rules.
Key facts: Logging & Threat Detection
Logging & Threat Detection in Practice
2 Logging & Threat Detection scenarios, from Financial services (online mutual fund distributor) and Fintech (prepaid payment instrument wallet, crypto-free), plus related platform case studies.
Mutual fund distributor halves Sentinel spend while tripling tested detections
Prepaid wallet detects a leaked access key in 4 minutes with Security Lake detections
Telecom VAS provider cuts daily pages from 2,300 alerts to 9 actionable incidents
IT services firm brings 43 Azure subscriptions under one policy hierarchy
Logging & Threat Detection: Frequently Asked Questions
What does building a cloud detection pipeline cost?
NTSPL's 12-week design and build typically costs INR 14-28 lakh, depending on the number of clouds, SIEM choice and custom rules. Run costs depend on volume; with tiering and filtering, mid-size Indian estates commonly spend INR 1.5-6 lakh a month on ingestion and retention.
What exactly does CERT-In require for logs?
CERT-In's directions of April 2022 require organisations to enable logs of ICT systems, maintain them securely for a rolling 180 days within Indian jurisdiction, and provide them when required for incident reporting. They also require clocks synchronised to NIC or NPL NTP servers or traceable sources.
Which logs matter most for detecting cloud attacks?
Identity sign-in logs and control-plane audit logs come first, because most cloud attacks begin with stolen credentials and continue through API calls. Next come threat detector findings, key and secret access logs, WAF logs and DNS logs. Flow logs are valuable for investigation but rarely the first signal.
Can Microsoft Sentinel monitor AWS and Google Cloud?
Yes. Sentinel has connectors for AWS CloudTrail, GuardDuty and other AWS logs through S3 and SQS, and for Google Cloud audit logs through Pub/Sub. It works well for organisations whose identity and endpoints are already in Microsoft, though cross-cloud data transfer costs should be estimated.
How do you test that detections work?
Each rule ships with a test case. We run attack simulations with open-source tools such as Stratus Red Team and Atomic Red Team in a controlled account, confirm the rule fires with the right entities, and repeat monthly so regressions are caught when services or log schemas change.
Should logs be stored in a separate account?
Yes. Security logs belong in a dedicated log archive account, subscription or project with restricted access, so an attacker who compromises a workload account cannot delete evidence. Object lock or immutable storage policies add further protection for the 180-day trail.