India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
Google Cloud Observability: Monitoring, Logging, Tracing & SLOs
Google Cloud observability is the native telemetry stack of Cloud Monitoring, Cloud Logging with Log Analytics, Cloud Trace, Managed Service for Prometheus and Error Reporting that measures what users experience and why it fails. NTSPL instruments services with OpenTelemetry, defines SLOs with burn-rate alerts and cuts log spend for Indian product and platform teams running on GKE, Cloud Run and Compute Engine.
Google Cloud services in scope
The Problem Cloud Monitoring with SLO monitoring and Google Cloud Managed Service for Prometheus Solve
The typical Google Cloud estate we inherit has thousands of default metrics, a Cloud Logging bill that grew 4x in a year, and an on-call rota that still learns about outages from customer tweets. Alerts fire on CPU above 80 percent while checkout errors go unnoticed because nobody alerts on the ratio of 5xx responses. Logs are text blobs with no trace ID, so linking a slow API call to the Cloud SQL query behind it takes an hour of grep. The real problem is not a lack of telemetry; it is telemetry that is not tied to user journeys, not correlated across signals and not priced deliberately.
NTSPL starts from user journeys, not dashboards. For each critical journey, such as login, search, payment or data ingestion, we define a service level indicator from load balancer, Prometheus or custom metrics, set an SLO in Cloud Monitoring and alert only on error-budget burn. Services emit structured JSON logs carrying the trace ID, OpenTelemetry spans go to Cloud Trace, and log routing is redesigned with exclusion filters and Log Analytics buckets so spend tracks value.
When Your Team Needs SLOs, Log Analytics and Managed Prometheus on Google Cloud
Signals that point to operations & observability on Google Cloud
- Mean time to detect user-facing incidents exceeds 15 minutes, or more than one in four incidents is first reported by customers.
- Cloud Logging is among your top five Google Cloud SKUs and nobody can say which services generate the volume.
- On-call engineers receive more than 50 pages a week and fewer than a third need action.
- You are moving from monoliths to microservices on GKE or Cloud Run and cannot follow a request across services.
- Leadership wants reliability targets in board or customer contracts, such as a 99.9 percent monthly availability commitment.
- A self-managed Prometheus, ELK or Grafana stack needs a dedicated engineer just to keep it running.
When a different approach is better
- If your workloads are spread evenly across AWS, Azure and on-prem and you need one pane for all, a vendor-neutral platform such as Grafana Cloud or an existing APM may be a better centre, with Google Cloud telemetry exported to it.
- If the requirement is security event correlation and threat detection, Cloud Logging is the source, not the analysis engine; use Google Security Operations or your SIEM.
- If you need product analytics such as funnels and cohorts, use BigQuery with Google Analytics exports rather than application logs in Log Analytics.
Google Cloud Observability Stack: Cloud Monitoring, Cloud Logging, Cloud Trace, Managed Prometheus and Error Reporting
| Service | Role in the architecture | Choose it when |
|---|---|---|
| Cloud Monitoring with SLO monitoring | Stores platform and custom metrics, runs uptime checks and alert policies, and computes service level objectives with error-budget and burn-rate alerting on request-based or windows-based SLIs. | Alerts are threshold-based on infrastructure metrics and on-call cannot say whether users are actually affected. |
| Google Cloud Managed Service for Prometheus | Collects Prometheus metrics from GKE, Compute Engine and other environments using managed or self-deployed collectors and stores them in Monarch, queryable with PromQL in Cloud Monitoring and Grafana. | You run self-managed Prometheus with Thanos or Cortex and spend engineering time on storage, retention and high-availability of the metrics backend. |
| Cloud Logging with log buckets and Log Analytics | Ingests platform, audit and application logs, routes them through sinks with inclusion and exclusion filters to log buckets, and lets you query upgraded buckets with SQL through Log Analytics or a linked BigQuery dataset. | Investigations require joining logs across services or counting error patterns, and the Logs Explorer query language is too limited. |
| Cloud Trace | Receives distributed traces from OpenTelemetry-instrumented services and Google-managed components such as load balancers, showing latency per span and linking to correlated logs. | p95 latency regresses and you cannot tell whether the time goes to your code, a downstream API, Cloud SQL or Memorystore. |
| Error Reporting | Groups exceptions and stack traces from logs into distinct error groups with first-seen and occurrence counts, and notifies on new error types. | Releases introduce new exceptions that are buried in log volume until a customer reports them. |
| Ops Agent and Cloud Profiler | The Ops Agent collects system and application metrics and logs from Compute Engine VMs; Cloud Profiler samples CPU and heap continuously in production with low overhead. | VM-based workloads report nothing beyond hypervisor metrics, or CPU cost rises without a code-level explanation. |
SLO-driven telemetry pipeline for GKE and Cloud Run services
SLO-driven telemetry pipeline for GKE and Cloud Run services
User requests enter through the external Application Load Balancer, whose request metrics define availability and latency SLIs, and flow into OpenTelemetry-instrumented services on GKE and Cloud Run. Metrics, traces and structured logs are collected into Cloud Monitoring, Cloud Trace and routed log buckets, and SLO burn-rate alerts notify on-call through incident channels with links to the correlated trace and log entries.
Select any component
SLO-driven telemetry pipeline for GKE and Cloud Run services: click any of its 9 components to see its role and the design choice NTSPL makes for it, or trace the 6-step primary path.
Observability Design Decisions: SLI Selection, Burn-Rate Windows, Log Routing and Metric Cardinality
Choosing SLIs per journey
For request-driven services we use load balancer metrics: good requests are non-5xx responses under a latency threshold, such as 400 ms for search and 1.5 s for payment. For asynchronous pipelines we measure freshness, such as Pub/Sub oldest unacknowledged message age under 60 seconds. For batch jobs we measure completion by a deadline. We avoid CPU or memory as SLIs entirely, and we keep each service to two or three SLOs so the error budget conversation stays readable for product owners.
Burn-rate alert windows
Single-threshold alerts on SLOs either page too late or flap. We implement multi-window burn-rate alerting: a fast-burn page when the one-hour burn rate exceeds about 14x and the five-minute rate confirms it, and a slow-burn ticket when the six-hour or three-day rate exceeds the budget pace. For a 99.9 percent SLO over 28 days, a 14.4x burn consumes about 2 percent of budget in one hour, which justifies waking someone. Low-traffic services get longer windows to avoid paging on three failed requests.
Log routing, retention and Log Analytics
The _Default bucket keeps 30 days by default and ingestion is billed per GiB beyond the free allotment of 50 GiB per project per month, so routing decides cost. We exclude load balancer health checks, Kubernetes info-level system logs and successful request logs from high-volume services, sampling the latter at 1 to 10 percent. Application logs go to Log Analytics-enabled buckets per environment, and we link a BigQuery dataset only for teams that run scheduled queries, avoiding duplicate exports.
Prometheus metric cardinality and scrape intervals
Managed Service for Prometheus bills per sample ingested, so a label carrying order ID or pod UID can multiply cost overnight. We enforce an allowlist of labels through metric relabelling, set scrape intervals to 30 or 60 seconds except for latency-critical services that need 15 seconds, and alert when any metric's active series count grows more than 50 percent week on week. Recording rules pre-aggregate expensive PromQL used on dashboards.
Trace sampling and log correlation
Full tracing on high-traffic services is costly and noisy. We use parent-based sampling at 5 to 10 percent, always sample requests that error or exceed the latency SLO threshold, and propagate W3C traceparent headers through Pub/Sub message attributes for async paths. Every log line carries logging.googleapis.com/trace and spanId fields in structured JSON so Logs Explorer and Cloud Trace link to each other without manual searching.
How NTSPL Instruments a Google Cloud Platform from Zero to SLO-Based On-Call
Telemetry and cost baseline
Pull Cloud Logging volume by log name and resource, active metric series counts and alert history for the last 90 days to see where noise and spend sit.
Journey and SLI workshop
Map top user journeys with product owners, choose request, freshness or completion SLIs, and agree initial SLO targets from measured performance.
Instrumentation and correlation
Roll out OpenTelemetry SDKs and structured logging with trace IDs, starting with the services on the most critical journey.
Log routing redesign
Apply exclusion filters, create Log Analytics buckets per environment and move audit logs into a locked retention bucket.
Burn-rate alerting cutover
Run new SLO alerts in parallel with legacy threshold alerts for two weeks, then retire alerts that never led to action.
Error-budget operating rhythm
Start a monthly SLO review where exhausted budgets pause feature releases for the affected service.
What you receive
Securing Telemetry: Log Bucket IAM, PII Redaction and Tamper-Resistant Audit Retention
Telemetry is a data leak channel. Application logs routinely capture phone numbers, tokens and full request bodies, and anyone with Logs Viewer on a project can read them. Audit logs are also evidence an attacker would like to erase. We treat the logging pipeline as a regulated data store: redact at source, restrict by log view, and keep audit logs in a separate bucket that application teams cannot modify.
- Log views on shared buckets so each team sees only its own services' logs
- Redaction of PII and secrets in the logging library before ingestion, with Sensitive Data Protection spot scans
- Separate audit-log bucket with bucket lock and retention of at least 180 days for CERT-In
- CMEK on log buckets holding regulated application logs
- Private access to Managed Prometheus and Grafana through Identity-Aware Proxy rather than public dashboards
Controlling Cloud Logging and Monitoring Spend: Ingestion Volume, Retention and Prometheus Sample Rates
Observability spend on Google Cloud has three main meters: Cloud Logging ingestion per GiB with retention beyond 30 days billed separately, Cloud Monitoring custom and Prometheus metrics billed by samples or bytes ingested, and Cloud Trace spans ingested beyond the free allotment. Platform metrics from Google services are largely free. In our audits, log ingestion from verbose GKE and load balancer logs is usually 60 to 80 percent of the total.
- Exclusion filters for health checks, kube-system info logs and repeated success logs
- Sampling load balancer request logging per backend service instead of logging every request
- Prometheus label allowlists and 60-second scrape intervals for non-critical workloads
- Retention set per bucket: 30 days for debug, 90 for application, 400 days only where audits require
- Trace sampling at 5 to 10 percent with forced sampling for errors and slow requests
Google Cloud Observability vs self-managed Prometheus and Grafana vs a third-party APM: which should we standardize on?
Decide on four criteria. Estate: if 80 percent or more of workloads run on Google Cloud, the native stack wins because load balancer, GKE and Cloud SQL telemetry arrive with no agents. Operations load: self-managed Prometheus, Thanos and Loki give full control but need an engineer to run storage and upgrades; Managed Service for Prometheus keeps PromQL and Grafana while removing the backend. Depth: third-party APMs still lead on code-level profiling for some languages and real-user monitoring, so add one only where those gaps hurt. Cost: native pricing is per GiB and per sample, while APMs often price per host, which is expensive on autoscaled GKE. Most of our clients standardize on the native stack plus Grafana.
Key facts about operations & observability on Google Cloud
Operations & Observability on Google Cloud in Practice
3 operations & observability scenarios from E-commerce marketplace, Logistics technology, fleet telematics, Education technology, live online classes: the starting position, the services chosen, the architecture, the rollout and the measured outcome.
Online marketplace halves customer-reported incidents by adopting SLOs and error budgets
Fleet telematics provider cuts GPS ingestion lag from 9 minutes to under 20 seconds
Live-classes EdTech traces session drops to a signalling bug and cuts them by 73 percent
Solar power producer cuts deviation charges with 15-minute generation analytics
Classifieds marketplace cuts scraper traffic 81 percent with Cloud Armor and reCAPTCHA
Payment gateway reaches a 22-minute regional failover between Mumbai and Delhi
Operations & Observability on Google Cloud: Frequently Asked Questions
Why is our Cloud Logging bill so high, and how much can be cut?
Usually because everything is ingested and kept by default. Cloud Logging bills per GiB ingested beyond 50 GiB per project per month, and the biggest sources are typically GKE container stdout, load balancer request logs and Data Access audit logs. NTSPL routinely cuts ingestion by 40 to 70 percent with exclusion filters and sampling without losing incident visibility, because most excluded lines were successful health checks and repeated info messages.
We already run Prometheus and Grafana on GKE. Do we have to rewrite dashboards to move?
No. Managed Service for Prometheus accepts the same scrape configurations through PodMonitoring resources and supports PromQL, so Grafana dashboards keep working by pointing the data source at the managed query endpoint. NTSPL runs both backends in parallel for two to four weeks, compares results for key dashboards, then retires the self-managed storage, typically freeing several nodes and an engineer's maintenance time.
Can logs and metrics stay in India for DPDP and RBI purposes?
Yes, with deliberate configuration. Log buckets can be created in asia-south1 or asia-south2, and an Organization Policy resource-location constraint plus the default storage location setting for Cloud Logging keeps new buckets in India. Because logs often contain personal data, NTSPL also redacts identifiers before ingestion, so even India-resident logs carry minimal DPDP exposure.
What is an error budget and how does it change how our team works?
An error budget is the amount of unreliability your SLO allows; a 99.9 percent monthly SLO allows about 43 minutes of bad service. When the budget is healthy, teams ship faster; when it is exhausted, the agreed policy shifts effort to reliability work. NTSPL sets up the SLOs, dashboards and a monthly review so this becomes a product decision, not an argument between developers and operations.
How long does it take to get useful tracing across our microservices?
For a typical 20 to 40 service platform, about four to six weeks. Week one covers the OpenTelemetry collector and SDK setup, then services are instrumented journey by journey, starting with the most critical. Auto-instrumentation for HTTP, gRPC and database clients covers most spans; NTSPL adds manual spans only where business steps such as payment authorisation need visibility.
Will we lose alerts during the move from threshold alerts to SLO alerts?
Not if both run in parallel. NTSPL keeps legacy alerts active for two weeks while SLO burn-rate alerts are introduced, then compares which alerts led to real action. Typically 60 to 80 percent of threshold alerts are retired, while hard-failure alerts, such as certificate expiry or a stopped Pub/Sub subscriber, stay as direct alerts alongside the SLOs.