{"id":2452,"date":"2021-01-08T13:01:21","date_gmt":"2021-01-08T13:01:21","guid":{"rendered":"https:\/\/ntsplhosting.com\/blog\/?p=2452"},"modified":"2021-12-21T05:27:34","modified_gmt":"2021-12-21T05:27:34","slug":"monitoring-query-language-mql-in-cloud-monitoring","status":"publish","type":"post","link":"https:\/\/www.ntsplhosting.com\/blog\/monitoring-query-language-mql-in-cloud-monitoring\/","title":{"rendered":"Introducing Monitoring Query Language, now GA in Cloud Monitoring"},"content":{"rendered":"<div class=\"block-paragraph\">\n<div class=\"rich-text\">\n<p>Developers and operators on IT and development teams want powerful metric querying, analysis, charting, and alerting capabilities to troubleshoot outages, perform root cause analysis, create custom SLI \/ SLOs, reports and analytics, set up complex alert logic, and more. So today we\u2019re excited to announce the General Availability of <a href=\"https:\/\/cloud.google.com\/monitoring\/mql\">Monitoring Query Language (MQL)<\/a> in <a href=\"https:\/\/cloud.google.com\/monitoring\">Cloud Monitoring<\/a>!<\/p>\n<p>MQL represents a decade of learnings and improvements on Google\u2019s internal metric query language. The same language that powers advanced querying for internal Google production users, is\u00a0 now available to Google Cloud users as well. For instance, you can use MQL to:<\/p>\n<ul>\n<li>Create ratio-based charts and alerts<\/li>\n<li>Perform time-shift analysis (compare metric data week over week, month over month, year over year, etc.)<\/li>\n<li>Apply mathematical, logical, table operations, and other functions to metrics<\/li>\n<li>Fetch, join, and aggregate over multiple metrics<\/li>\n<li>Select by arbitrary, rather than predefined, percentile values<\/li>\n<li>Create new labels to aggregate data by, using arbitrary string manipulations including regular expressions<\/li>\n<\/ul>\n<p>Let\u2019s take a look at how to access and use MQL from within Cloud Monitoring.<\/p>\n<h3>Getting started with MQL<\/h3>\n<p>It\u2019s easy to get started with MQL. To access the MQL Query Editor, just click on the button in <a href=\"https:\/\/console.cloud.google.com\/monitoring\/metrics-explorer\">Cloud Monitoring Metrics Explorer<\/a>:<\/p>\n<\/div>\n<\/div>\n<div class=\"block-image_full_width\">\n<div class=\"article-module h-c-page\">\n<div class=\"h-c-grid\">\n<figure class=\"article-image--large h-c-grid__col h-c-grid__col--6 h-c-grid__col--offset-3 \"><img src=\"https:\/\/storage.googleapis.com\/gweb-cloudblog-publish\/images\/MQL_Query_Editor.max-1000x1000.jpg\" alt=\"MQL Query Editor.jpg\" \/><\/figure>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"block-paragraph\">\n<div class=\"rich-text\">\n<p>Then, create a query in the Metrics Explorer UI, and click the Query Editor button. This converts the existing query into an MQL query:<\/p>\n<\/div>\n<\/div>\n<div class=\"block-image_full_width\">\n<div class=\"article-module h-c-page\">\n<div class=\"h-c-grid\">\n<figure class=\"article-image--large h-c-grid__col h-c-grid__col--6 h-c-grid__col--offset-3 \"><img src=\"https:\/\/storage.googleapis.com\/gweb-cloudblog-publish\/images\/Query_Editor.max-1000x1000.jpg\" alt=\"Query Editor.jpg\" \/><\/figure>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"block-paragraph\">\n<div class=\"rich-text\">\n<p>MQL is built using operations and functions. Operations are linked together using the common <a href=\"https:\/\/man7.org\/linux\/man-pages\/man2\/pipe.2.html\" target=\"_blank\" rel=\"noopener noreferrer\">\u2018pipe<\/a>\u2019 idiom, where the output of one operation becomes the input to the next. Linking operations makes it possible to build up complex queries incrementally. In the same way you would compose and chain commands and data via pipes on the Linux command line, you can fetch metrics and apply operations using MQL.<\/p>\n<p>For a more advanced example, suppose you&#8217;ve built a distributed web service that runs on Compute Engine VM instances and uses Cloud Load Balancing, and you want to analyze error rate\u2014one of the <a href=\"https:\/\/sre.google\/sre-book\/monitoring-distributed-systems\/\" target=\"_blank\" rel=\"noopener noreferrer\">SRE \u201cgolden signals\u201d<\/a>.<\/p>\n<p>You want to see a chart that displays the ratio of requests that return HTTP 500 responses (internal errors) to the total number of requests; that is, the request-failure ratio. The <i>loadbalancing.googleapis.com\/https\/request_count<\/i> metric type has a response_code_class label, which captures the class of response codes.<\/p>\n<p>In this example, because the numerator and denominator for the ratio are derived from the same time series, you can also <a href=\"https:\/\/cloud.google.com\/monitoring\/mql\/examples#qlx-group-ratio\">compute the ratio by grouping<\/a>. The following query shows this approach:<\/p>\n<div class=\"ng-star-inserted\">\n<div class=\"module--text h-c-page ng-star-inserted\">\n<div class=\"h-c-grid\">\n<div class=\"uni-paragraph h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6 h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3\">\n<pre><code class=\"ng-star-inserted\">fetch https_lb_rule::loadbalancing.googleapis.com\/https\/request_count\r\n<\/code><code class=\"ng-star-inserted\">| group_by [matched_url_path_rule],\r\n<\/code><code class=\"ng-star-inserted\">    sum(if(response_code_class = 500, val(), 0)) \/ sum(val())<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"ng-star-inserted\">\n<div class=\"module--text h-c-page ng-star-inserted\">\n<div class=\"h-c-grid\">\n<div class=\"uni-paragraph h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6 h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3\">\n<div class=\"rich-text\">\n<p>This query uses an aggregation expression built on the ratio of two sums:<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"block-paragraph\">\n<div class=\"rich-text\">\n<ul>\n<li>The first sum uses the if function to count 500-valued HTTP responses and a count of 0 for other HTTP response codes. The sum function computes the count of the requests that returned 500.<\/li>\n<li>The second sum adds up the counts for all requests, as represented by val().<\/li>\n<\/ul>\n<p>The two sums are then divided, resulting in the ratio of 500 responses to all responses.<\/p>\n<p>Now let\u2019s say that we want to create an alert policy from this query. You can go to <a href=\"https:\/\/console.cloud.google.com\/monitoring\/alerting\">Alerting<\/a>, click \u201cCreate Policy\u201d, click \u201cAdd Condition\u201d, and you\u2019ll see the same \u201cQuery Editor\u201d button you saw in Metrics Explorer.<\/p>\n<div class=\"ng-star-inserted\">\n<div class=\"module--text h-c-page ng-star-inserted\">\n<div class=\"h-c-grid\">\n<div class=\"uni-paragraph h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6 h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3\">\n<div class=\"rich-text\">\n<p>You can use the same query as above, but with a condition operator that provides the threshold for the alert:<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"ng-star-inserted\">\n<div class=\"module--text h-c-page ng-star-inserted\">\n<div class=\"h-c-grid\">\n<div class=\"uni-paragraph h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6 h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3\">\n<pre><code class=\"ng-star-inserted\">fetch https_lb_rule::loadbalancing.googleapis.com\/https\/request_count\r\n<\/code><code class=\"ng-star-inserted\">| group_by [matched_url_path_rule],\r\n<\/code><code class=\"ng-star-inserted\">    sum(if(response_code_class = 500, val(), 0)) \/ sum(val())\r\n<\/code><code class=\"ng-star-inserted\">| condition val() &gt; .50 '10^2.%'<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"ng-star-inserted\">\n<div class=\"module--text h-c-page ng-star-inserted\">\n<div class=\"h-c-grid\">\n<div class=\"uni-paragraph h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6 h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3\">\n<div class=\"rich-text\">\n<p>The condition tests each data point in the aligned input table to determine whether the ratio value exceeds the threshold value of 50%. The string &#8217;10^2.%&#8217; specifies that the value should be used as a percentage.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"block-paragraph\">\n<div class=\"rich-text\">\n<p>In addition to ratios, another common use case for MQL is <a href=\"https:\/\/cloud.google.com\/monitoring\/mql\/examples#qlx-timewarp\">time shifting<\/a>. For brevity, we won\u2019t cover this in our blog post, but the example documentation walks you through performing week-over-week or month-over-month comparisons. This is particularly powerful when coupled with long-term retention of 24 months of custom and Prometheus metrics.<\/p>\n<h3>Take monitoring to the next level<\/h3>\n<p>The sky\u2019s the limit for the use cases that MQL makes possible. Whether you need to perform joins, display arbitrary percentages, or make advanced calculations, we\u2019re excited to make this available to all customers and we are interested to see how you will use MQL to solve your monitoring, alerting, and operations needs. To learn more about MQL, check out the <a href=\"https:\/\/cloud.google.com\/monitoring\/mql\">documentation<\/a>, <a href=\"https:\/\/cloud.google.com\/monitoring\/mql\/quickstart\">quickstarts<\/a>, examples (<a href=\"https:\/\/cloud.google.com\/monitoring\/mql\/examples\">queries<\/a>, <a href=\"https:\/\/cloud.google.com\/monitoring\/mql\/alerts\">alerts<\/a>), <a href=\"https:\/\/cloud.google.com\/monitoring\/mql\/reference\">a language and function reference<\/a>, and more.<\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Developers and operators on IT and development teams want powerful metric querying, analysis, charting, and alerting capabilities to troubleshoot outages, perform root cause analysis, create custom SLI \/ SLOs, reports and analytics, set up complex alert logic, and more. So today we\u2019re excited to announce the General Availability of Monitoring Query Language (MQL) in Cloud [&hellip;]<\/p>\n","protected":false},"author":53,"featured_media":2499,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[445],"tags":[452,453],"_links":{"self":[{"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/posts\/2452"}],"collection":[{"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/users\/53"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/comments?post=2452"}],"version-history":[{"count":7,"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/posts\/2452\/revisions"}],"predecessor-version":[{"id":5216,"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/posts\/2452\/revisions\/5216"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/media\/2499"}],"wp:attachment":[{"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/media?parent=2452"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/categories?post=2452"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ntsplhosting.com\/blog\/wp-json\/wp\/v2\/tags?post=2452"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}