Introducing Amazon Lookout for Metrics: An anomaly detection service to proactively monitor the health of your business

by
Amazon AWS

Anomalies are unexpected changes in data, which could point to a critical issue. An anomaly could be a technical glitch on your website, or an untapped business opportunity. It could be a new marketing channel with exceedingly high customer conversions. As businesses produce more data than ever before, detecting these unexpected changes and responding in a timely manner is essential, yet challenging. Delayed responses cost businesses millions of dollars, missed opportunities, and the risk of losing the trust of their customers.

We’re excited to announce the general availability of Amazon Lookout for Metrics, a new service that uses machine learning (ML) to automatically monitor the metrics that are most important to businesses with greater speed and accuracy. The service also makes it easier to diagnose the root cause of anomalies like unexpected dips in revenue, high rates of abandoned shopping carts, spikes in payment transaction failures, increases in new user sign-ups, and many more. Lookout for Metrics goes beyond simple anomaly detection. It allows developers to set up autonomous monitoring for important metrics to detect anomalies and identify their root cause in a matter of few clicks, using the same technology used by Amazon internally to detect anomalies in its metrics—all with no ML experience required.

You can connect Lookout for Metrics to 19 popular data sources, including Amazon Simple Storage Solution (Amazon S3), Amazon CloudWatch, Amazon Relational Database Service (Amazon RDS), and Amazon Redshift, as well as software as a service (SaaS) applications like Salesforce, Marketo, and Zendesk, to continuously monitor metrics important to your business. Lookout for Metrics automatically inspects and prepares the data, uses ML to detect anomalies, groups related anomalies together, and summarizes potential root causes. The service also ranks anomalies by severity so you can prioritize which issue to tackle first.

Lookout for Metrics easily connects to notification and event services like Amazon Simple Notification Service (Amazon SNS), Slack, Pager Duty, and AWS Lambda, allowing you to create customized alerts or actions like filing a trouble ticket or removing an incorrectly priced product from a retail website. As the service begins returning results, you can also provide feedback on the relevancy of detected anomalies via the Lookout for Metrics console or the API, and the service uses this input to continuously improve its accuracy over time.

Digitata, a telecommunication analytics provider, intelligently transforms pricing and subscriber engagement for mobile network operators (MNOs), empowering them to make better and more informed business decisions. One of Digitata’s MNO customers had made an erroneous update to their pricing platform, which led to them charging their end customers the maximum possible price for their internet data bundles. Lookout for Metrics immediately identified that this update had led to a drop of over 16% in their active purchases and notified the customer within minutes of the said incident using Amazon SNS. The customer was also able to attribute the drop to the latest updates to the pricing platform using Lookout for Metrics. With a clear and immediate remediation path, the customer was able to deploy a fix within 2 hours of getting notified. Without Lookout for Metrics, it would have taken Digitata approximately a day to identify and triage the issue, and would have led to a 7.5% drop in customer revenue, in addition to the risk of losing the trust of their end customers.

Solution overview

This post demonstrates how you can set up anomaly detection on a sample ecommerce dataset using Lookout for Metrics. The solution allows you to download relevant datasets, set up continuous anomaly detection, and optionally set up alerts to receive notifications in case anomalies occur.

Our sample dataset is designed to detect abnormal changes in revenue and views for the ecommerce website across major supported platforms like pc_web, mobile_web, and mobile_app and marketplaces like US, UK, DE, FR, ES, IT, and JP.

The following diagram shows the architecture of our continuous detection system.

Building this system requires three simple steps:

  1. Create an S3 bucket and upload your sample dataset.
  2. Create a detector for Lookout for Metrics.
  3. Add a dataset and activate the detector to start learning and continuous detection.

Then you can review and analyze the results.

If you’re familiar with Python and Jupyter, you can get started immediately by following along with the GitHub repo; this post walks you through getting started with the service. After you set up the detection system, you can optionally define alerts that notify you when anomalies are found that meet or exceed a specified severity threshold.

Create an S3 bucket and upload your sample dataset

Download the sample dataset and save it locally. Then continue through the following steps:

  1. Create an S3 bucket.

This bucket needs to be unique and in the same Region where you’re using Lookout for Metrics. For this post, we use the bucket 059124553121-lookoutmetrics-lab.

  1. After you create the bucket, extract the demo dataset on your local machine.

You should have a folder named ecommerce.

  1. On the Amazon S3 console, open the bucket you created.

  1. Choose Upload.

  1. Upload the ecommerce folder.

It takes a few moments to process the files.

  1. When the files are processed, choose Upload.

Do not navigate away from this page while the upload is still processing. You can move to the next step when your dataset is ready.

Alternatively, you can use the AWS Command Line Interface (AWS CLI) to upload the file in just a few minutes using the following command:

!aws s3 sync {data_dirname}/ecommerce/ s3://{s3_bucket}/ecommerce/ --quiet

Create a detector for Lookout for Metrics

To create your detector, complete the following steps:

  1. On the Lookout for Metrics console, choose Create detector.

  1. For Name, enter a detector name.
  2. For Description, enter a description.
  3. For Interval, choose 1 hour intervals.
  4. Optionally, you can modify encryption settings.
  5. Choose Create.

Add a dataset and activate the detector

You now configure your dataset and metrics.

  1. Choose Add a dataset.

  1. For Name, enter a name for your dataset.
  2. For Description, enter a description.
  3. For Timezone, choose the UTC timezone.
  4. For Datasource, choose Amazon S3.

We use Amazon S3 as our data source in this post, but Lookout for Metrics can connect to 19 popular data sources, including CloudWatch, Amazon RDS, and Amazon Redshift, as well as SaaS applications like Salesforce, Marketo, and Zendesk.

You also have an offset parameter, for when you have data that can take a while to arrive and you want Lookout for Metrics to wait until your data has arrived to start reading. This is helpful for long-running jobs that may feed into Amazon S3.

  1. For Detector mode, select either Backtest or Continuous.

Backtesting allows you to detect anomalies on historical data. This feature is helpful when you want to try out the service on past data or validate against known anomalies that occurred in the past. For this post, we use continuous mode, where you can detect anomalies on live data continuously, as they occur.

  1. For Path to an object in your continuous dataset, enter the value for S3_Live_Data_URI.
  2. For Continuous data S3 path pattern, choose the S3 path with your preferred time format (for this post, we choose the path with {{yyyyMMdd}}/{{/HHmm}}), which is the third option in the drop down.

The options on the drop-down menu adapt for your data.

  1. For Datasource interval, choose your preferred time interval (for this post, we choose 1 hour intervals).

For continuous mode, you have the option to provide your historical data, if you have any, for the system to proactively learn from. If you don’t have historical data, you can get started with real-time or live data, and Lookout for Metrics learns on the go. For this post, we have historical data for learning, hence we check this box (Step 10)

  1. For Historical data, select Use historical data.
  2. For S3 location of historical data, enter the Amazon S3 URI for historical data that you collected earlier.

You need to provide the ARN of an AWS Identity and Access Management (IAM) role to allow Lookout for Metrics to read from your S3 bucket. You can use an existing role or create a new one. For this post, we use an existing role.

  1. For Service role, choose Enter the ARN of a role.
  2. Enter the role ARN.
  3. Choose Next.

The service now validates your data.

  1. Choose OK.

On the Map files page, you specify which fields you want to run anomaly detection on. Measures are the key performance indicators on which we want to detect anomalies, and dimensions are the categorical information about the measures. You may want to monitor your data for anomalies in number of views or revenue for every platform, marketplace, and combination of both. You can designate up to five measures and five dimensions per dataset.

  1. For Measures, choose views and revenue.
  2. For Dimensions, choose platform and marketplace.

Lookout for Metrics analyzes each combination of these measures and dimensions. For our example, we have seven unique marketplace values (US, UK, DE, FR, ES, IT, and JP) and three unique platform values (pc_web, mobile_web, and mobile_app), for a total of 21 unique combinations. Each unique combination of measures and dimension values is a metric. In this case, you have 21 dimensions and two measures, for a total of 42 time series metrics. Lookout for Metrics detects anomalies at the most granular level so you can pinpoint any unexpected behavior in your data.

  1. For Timestamp, choose your timestamp formatting (for this post, we use the default 24-hour format in Python’s Pandas package, yyyy-MM-dd HH:mm:ss).

  1. Choose Next.

  1. Review your setup and choose Save and activate.

You’re redirected to the detector page, where you can see that the job has started. This process takes 20–25 minutes to complete.

From here you could define the alerts. Lookout for Metrics can automatically send you alerts through channels such as Amazon SNS, Datadog, PagerDuty, Webhooks, and Slack, or trigger custom actions using Lambda, reducing your time to resolution.

Congratulations, your first detector is up and running.

Review and analyze the results

When detecting an anomaly, Lookout for Metrics helps you focus on what matters most by assigning a severity score to aid prioritization. To help you find the root cause, it intelligently groups anomalies that may be related to the same incident and summarizes the different sources of impact. In the following screenshot, the anomaly in revenue on March 8 at 22:00 GMT had a severity score of 97, indicating a high severity anomaly that needs immediate attention. The impact analysis also tells you that the Mobile_web platform in Germany (DE) saw the highest impact.

Lookout for Metrics also allows you to provide real-time feedback on the relevance of the detected anomalies, enabling a powerful human-in-the-loop mechanism. This information is fed back to the anomaly detection model to improve its accuracy continuously, in near-real time.

Clean up

To avoid incurring ongoing charges, delete the following resources created in this post:

  • Detector
  • S3 bucket
  • IAM role

Conclusion

Lookout for Metrics is available directly via the AWS Management Console, the AWS SDKs, the AWS CLI, as well as through supporting partners to help you easily implement customized solutions. The service is also compatible with AWS CloudFormation and can be used in compliance with the European Union’s General Data Protection Regulation (GDPR).

As of this writing, Lookout for Metrics is available in the following Regions:

  • US East (Ohio)
  • US East (N. Virginia)
  • US West (Oregon)
  • Asia Pacific (Singapore)
  • Asia Pacific (Sydney)
  • Asia Pacific (Tokyo)
  • Europe (Ireland)
  • Europe (Frankfurt)
  • Europe (Stockholm)

To get started building your first detector, adding metrics via various popular data sources, and creating custom alerts and actions via the output channel of your choice, see Amazon Lookout for Metrics Documentation.

About the Authors

Ankita Verma is the Product Lead for Amazon Lookout for Metrics. Her current focus is helping businesses make data driven decisions using AI/ ML. Outside of AWS, she is a fitness enthusiast, and loves mentoring budding product managers and entrepreneurs in her free time. She also publishes a weekly product management newsletter called ‘The Product Mentors’ on Substack.

 

 

Chris King is a Senior Solutions Architect in Applied AI with AWS. He has a special interest in launching AI services and helped grow and build Amazon Personalize and Amazon Forecast before focusing on Amazon Lookout for Metrics. In his spare time he enjoys cooking, reading, boxing, and building models to predict the outcome of combat sports.

Read More