Posts in category: Amazon AWS

We are excited to launch a causal contribution analysis capability in Amazon Lookout for Metrics that helps you to understand the potential root causes for the business-critical anomalies in the data. Previously, you were only given the root causes for a single anomaly per measure. You had to analyze to determine if causal relationships existed between the detected anomalies in different measures. When focusing on a single anomaly, you can easily miss the anomaly’s downstream (or upstream) impact. For example, you may see a spike in your checkout cart abandonment and know that your revenue will decrease. However, you may not know what caused the checkout carts to be abandoned at a higher rate. The causal contribution analysis feature can tell you that the spike in checkout cart abandonment may be due to spikes in transaction failures or sudden changes in prices due to promotion expiration.

Lookout for Metrics uses machine learning (ML) to automatically detect and diagnose anomalies in large datasets where deviations from normal are hard to detect and missed anomalies have business-critical impact. Lookout for Metrics reduces the time to implement AI/ML services for business-critical problems.

In this post, we discuss the new causal contribution analysis capability and its benefits.

Challenges in anomaly detection

Anomaly detection has two parts: detecting anomalies and identifying the root cause that triggered the anomalies so that team can take action to mitigate the problem.

Traditional business intelligence (BI) systems that use static threshold-based or rule-based anomalies have three problems. First, you might have millions of metrics to track across multiple data sources. Take digital advertisement, for example—you want to track metrics like impression, clicks, revenue, and shopping cart metrics across campaign IDs, product categories, geographies, and more. And it’s the same for any domain, be it retail, telecom, gaming, or financial services. With traditional BI tools, managing data across multiple sources, creating dashboards and reports, and adding alerts at a granular level requires a lot of manual work and isn’t scalable.

Second, these traditional BI tools work by setting up rules. You set up a range, and anything outside the range is an anomaly and you are alerted on those. If the range is too broad, you miss important alerts, and if it’s too narrow, you receive too many false alerts.

These ranges (upper bound and lower bound in image above) are also static, and don’t change based on the time of the day, day of the week, or seasons; they need to be manually updated. You’re likely to miss important anomalies and receive too many false alarms, or you lose trust in the tool and start ignoring these alerts altogether.

Lastly, BI reports and dashboards are often generated at the end of hour, end of day, or end of week, when it’s too late for you to act on a problem. And even when these results come, it doesn’t answer the why. So developers, analysts, and business owners can spend weeks trying to identify the root cause of the anomaly, delaying meaningful action even further.

Causal inference in Lookout for Metrics

Although asking for the root cause of an unexpected event seems to be at the heart of the human way of understanding the world, statistical associations are often misinterpreted as a causal influence. That is, correlation doesn’t imply causation, and discerning the causes of events from observational data requires specialized causal inference methods.

The root cause analysis in Lookout for Metrics uses causal inference techniques to increase the visibility and interpretability of anomalies across measures. Lookout for Metrics is capable of not only identifying causal drivers, but also quantitatively attributing the anomalous events to them, providing a percentage score of likelihood among the probable causal drivers of an anomalous event. For example, Lookout for Metrics can now draw causal links between a drop in advertisement views (anomaly) due to fewer clicks on your website, IOS, and Android (causation), leading to a decline in the revenue (downstream impact). Suppose one or more potential root causes occur (website, iOS, Android). In that case, Lookout for Metrics can identify the most likely cause (for example, website with a 90% likelihood) that led to a drop in advertisement views.

The scientific approach relies on a two-step procedure:

1. Infer the causal relations between the measures.
2. Based on the inferred causal structure, attribute the anomalies of the affected measure to the causing measures.

To infer the causal relations between the measures, we use a Granger causality method that takes the panel data structure of Lookout for Metrics into account. The existing Granger causality methods for panel data can’t deal with dependencies across dimension value combinations (for instance, dependencies of revenue across different countries that we typically have in real data). For example, events such as Black Friday increase the revenue of multiple countries and therefore there is an external source that renders the revenue of different countries dependent). We therefore had to develop our own Granger causality^[1] method on panel data that can deal with these types of dependencies.

Once the causal structure is available, we attribute the anomalies of the affected measure to its causing measures to quantify the cause-effect relationships.

Analyze anomalies on the Lookout for Metrics console

After Lookout for Metrics starts anomaly detection, you can look for the detected anomalies on the Anomalies page for the detector. When you choose an anomaly, you’re redirected to the details page for the observed anomaly.

The anomaly details page includes a Root cause analysis section. This section tries to explain this observed anomaly with respect to the other anomalies for the anomaly detector configured measures.

In the following example, “Revenue impacted” is the observed anomaly, and the potential causes include orders and non-configured measures. Orders contributes approximately 81.84% to the current anomaly, namely revenue that leads to a downstream impact on profit.

Choosing the potential cause orders takes us to the details of its observed anomaly. In this case, the possible causes for this anomaly are clicks and non-configured measures. Clicks could be one of the potential causes of this anomaly, but it gets a relatively low contribution score of 8.37%, and the detector doesn’t observe anything anomalous for it. In this case, Lookout for Metrics concludes that the orders anomaly is caused by external factors or measures that weren’t configured for monitoring during the detector setup phase. This anomaly in orders has a potential downstream impact on profit and revenue.

Choosing the potential downstream impact profit takes us to the details of its observed anomaly. In this case, the potential causes seem to be a mix of anomalies in revenue, orders, and non-configured measures, with respective contribution scores of 33%, 14%, and 53%. No downstream measures are affected by this anomaly.

For this example, the anomaly in profit can be partially explained by the anomalies in revenue and orders. Then the anomaly in revenue can be explained by the anomaly in orders with a high certainty.

Conclusion

The new causal contribution analysis capability in Lookout for Metrics detects the causal interaction between anomalies in your measures. To achieve this, the detector learns the causal relation between measures in your data fully self-supervised and uses this causal information to trace anomalies back to their root causes. This feature can help you causally connect anomalies across measures and provides you with a tool to quickly diagnose and subsequently fix any issues in your system.

[1] L. Minorics, C. Turkmen, P. Bloebaum, D. Kernert, L. Callot and D. Janzing. Testing Granger Non-Causality in Panels with Cross-Sectional dependencies. AISTATS, 2022.

About the Authors

Lenon Minorics is an Applied Scientist focusing on causal inference and anomaly detection. Prior to Amazon, Lenon was an academic researcher in mathematics. His personal research interests include machine learning, causal inference, stochastics, and fractal geometry. In his free time, Lenon enjoys practicing all kinds of sports, especially Brazilian Jiu-Jitsu.

Shashank Srivastava is Senior Product Manager for Amazon AI vertical services. He is passionate about solving problems in AI in NLP, novelty detection, and data scarcity. In his free time, Shashank enjoys playing tennis and golf.

Caner Türkmen is an Applied Scientist at Amazon Web Services, where he works on problems at the intersection of machine learning, forecasting, and anomaly detection. Before joining AWS, he worked in the management consulting industry as a data scientist, serving the financial services and telecommunications industries on projects across the globe. Caner’s personal research interests span a range of topics, including probabilistic and Bayesian ML, stochastic processes, and their practical applications.

Alex Kim is a Sr. Product Manager for AWS AI Services. His mission is to deliver AI/ML solutions to all customers who can benefit from it. In his free time, he enjoys all types of sports and discovering new places to eat.

Read More

AWS offers a broad set of artificial intelligence (AI) and machine learning (ML) services, including a suite of pre-trained, ready-to-use services for developers with no prior ML experience. In this post, we demonstrate how to use such services to build an application that fosters the inclusion of people with a visual or communication impairment, which includes difficulties in seeing, reading, hearing, speaking, or having a conversation in a foreign language. With services such as Amazon Transcribe, Amazon Polly, Amazon Translate, Amazon Rekognition and Amazon Textract, you can add features to your projects such as live transcription, text to speech, translation, object detection, and text extraction from images.

According to the World Health Organization, over 1 billion people—about 15% of the global population—live with some form of disability, and this number is likely to grow because of population ageing and an increase in the prevalence of some chronic diseases. For people with a speech, hearing, or visual impairment, everyday tasks such as listening to a speech or a TV program, expressing a feeling or a need, looking around, or reading a book can feel like impossible challenges. A wide body of research highlights the importance of assistive technologies for the inclusion of people with disabilities in society. According to research by the European Parliamentary Research Service, mainstream technologies such as smartphones provide more and more capabilities suitable for addressing the needs of people with disabilities. In addition, when you design for people with disabilities, you tend to build features that improve the experience for everyone; this is known as the curb-cut effect.

This post demonstrates how you can use the AWS SDK for JavaScript to integrate capabilities provided by AWS AI services into your own solutions. To do that, a sample web application showcases how to use Amazon Transcribe, Amazon Polly, Amazon Translate, Amazon Rekognition, and Amazon Textract to easily implement accessibility features. The source code of this application, AWS AugmentAbility, is available on GitHub to use as a starting point for your own projects.

Solution overview

AWS AugmentAbility is powered by five AWS AI services: Amazon Transcribe, Amazon Translate, Amazon Polly, Amazon Rekognition, and Amazon Textract. It also uses Amazon Cognito user pools and identity pools for managing authentication and authorization of users.

After deploying the web app, you will be able to access the following features:

• Live transcription and text to speech – The app transcribes conversations and speeches for you in real time using Amazon Transcribe, an automatic speech recognition service. Type what you want to say, and the app says it for you by using Amazon Polly text-to-speech capabilities. This feature also integrates with Amazon Transcribe automatic language identification for streaming transcriptions—with a minimum of 3 seconds of audio, the service can automatically detect the dominant language and generate a transcript without you having to specify the spoken language.
• Live transcription and text to speech with translation – The app transcribes and translates conversations and speeches for you, in real time. Type what you want to say, and the app translates and says it for you. Translation is available in the over 75 languages currently supported by Amazon Translate.
• Real-time conversation translation – Select a target language, speak in your language, and the app translates what you said in your target language by combining Amazon Transcribe, Amazon Translate, and Amazon Polly capabilities.
• Object detection – Take a picture with your smartphone, and the app describes the objects around you by using Amazon Rekognition label detection features.
• Text recognition for labels, signs, and documents – Take a picture with your smartphone of any label, sign, or document, and the app reads it out loud for you. This feature is powered by Amazon Rekognition and Amazon Textract text extraction capabilities. AugmentAbility can also translate the text into over 75 languages, or make it more readable for users with dyslexia by using the OpenDyslexic font.

Live transcription, text to speech, and real-time conversation translation features are currently available in Chinese, English, French, German, Italian, Japanese, Korean, Brazilian Portuguese, and Spanish. Text recognition features are currently available in Arabic, English, French, German, Italian, Portuguese, Russian, and Spanish. An updated list of the languages supported by each feature is available on the AugmentAbility GitHub repo.

You can build and deploy AugmentAbility locally on your computer or in your AWS account by using AWS Amplify Hosting, a fully managed CI/CD and static web hosting service for fast, secure, and reliable static and server-side rendered apps.

The following diagram illustrates the architecture of the application, assuming that it’s deployed in the cloud using AWS Amplify Hosting.

The solution workflow includes the following steps:

1. A mobile browser is used to access the web app—an HTML, CSS, and JavaScript application hosted by AWS Amplify Hosting. The application has been implemented using the SDK for JavaScript and the AWS Amplify JavaScript library.
2. The user signs in by entering a user name and a password. Authentication is performed against the Amazon Cognito user pool. After a successful login, the Amazon Cognito identity pool is used to provide the user with the temporary AWS credentials required to access app features.
3. While the user explores the different features of the app, the mobile browser interacts with Amazon Transcribe (StartStreamTranscriptionWebSocket operation), Amazon Translate (TranslateText operation), Amazon Polly (SynthesizeSpeech operation), Amazon Rekognition (DetectLabels and DetectText operations) and Amazon Textract (DetectDocumentText operation).

AWS services have been integrated in the mobile web app by using the SDK for JavaScript. Generally speaking, the SDK for JavaScript provides access to AWS services in either browser scripts or Node.js; for this sample project, the SDK is used in browser scripts. For additional information about how to access AWS services from a browser script, refer to Getting Started in a Browser Script. The SDK for JavaScript is provided as a JavaScript file supporting a default set of AWS services. This file is typically loaded into browser scripts using a <script> tag that references the hosted SDK package. A custom browser SDK was built with a specified set of services (for instructions, refer to Building the SDK for Browser).

Each service was integrated in the mobile web app following the guidelines and code samples available in the AWS SDK for JavaScript Developer Guide. The implementation of live transcription features required some additional steps because Amazon Transcribe Streaming WebSocket requires developers to encode the audio with event stream encoding and use the Signature Version 4 signing process for adding authentication information to AWS API requests sent by HTTP. For more information about this approach, refer to Transcribe speech to text in real time using Amazon Transcribe with WebSocket.

The user sign-in webpage has been implemented using authentication features of the AWS Amplify JavaScript library. For more details about the authentication and authorization flow, refer to Accessing AWS services using an identity pool after sign-in.

The following walkthrough shows how to deploy AugmentAbility by using AWS Amplify Hosting; it includes the following steps:

1. Create the Amazon Cognito user pool and identity pool, and grant permissions for accessing AWS AI services.
2. Clone the GitHub repository and edit the configuration file.
3. Deploy the mobile web app to the AWS Amplify console.
4. Use the mobile web app.

Create the Amazon Cognito user pool and identity pool, and grant permissions for accessing AWS AI services

The first step required for deploying the app consists of creating an Amazon Cognito user pool with the Hosted UI enabled, creating an Amazon Cognito identity pool, integrating the two pools, and finally granting permissions for accessing AWS services to the AWS Identity and Access Management (IAM) role associated with the identity pool. You can either complete this step by manually working on each task, or by deploying an AWS CloudFormation template.

The CloudFormation template automatically provisions and configures the necessary resources, including the Amazon Cognito pools, IAM roles, and IAM policies.

1. Sign in to the AWS Management Console and launch the CloudFormation template by choosing Launch Stack:
   
   The template launches in the EU West (Ireland) AWS Region by default. To launch the solution in a different Region, use the Region selector in the console navigation bar. Make sure to select a Region in which the AWS services in scope (Amazon Cognito, AWS Amplify, Amazon Transcribe, Amazon Polly, Amazon Translate, Amazon Rekognition, and Amazon Textract) are available (us-east-2, us-east-1, us-west-1, us-west-2, ap-south-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ca-central-1, eu-central-1, eu-west-1, eu-west-2).
2. Choose Next.
3. For Region, enter the identifier of the Region you want use (among the supported ones).
4. For Username, enter the user name you want to use to access the app.
5. For Email, enter the email address to which the temporary password for your first sign-in should be sent.
6. Choose Next.
7. On the Configure stack options page, choose Next.
8. On the Review page, review and confirm the settings.
9. Select the check box acknowledging that the template will create IAM resources and may require an AWS CloudFormation capability.
10. Choose Create stack to deploy the stack.

You can view the status of the stack on the AWS CloudFormation console in the Status column. You should receive a CREATE_COMPLETE status in a couple of minutes.

As part of the template deployment, the following permissions are granted to the IAM role that is assumed by the authenticated user:

• transcribe:StartStreamTranscriptionWebSocket
• translate:TranslateText
• comprehend:DetectDominantLanguage
• polly:SynthesizeSpeech
• rekognition:DetectText
• rekognition:DetectLabels
• textract:DetectDocumentText

Even though Amazon Comprehend is not explicitly used in this web application, permissions are granted for the action comprehend:DetectDominantLanguage. Amazon Translate may automatically invoke Amazon Comprehend to determine the language of the text to be translated if a language code isn’t specified.

Clone the GitHub repository and edit the configuration file

Now that access to AWS AI services has been configured, you’re ready to clone the GitHub repository and edit the configuration file.

1. In the AWS AugmentAbility GitHub repo, choose Code and Download ZIP.
   You’re either prompted to choose a location on your computer where the ZIP file should be downloaded to, or it will automatically be saved in your Downloads folder.
2. After you download the file, unzip it and delete the ZIP file.
   You should have obtained a folder named aws-augmentability-main with some files and subfolders in it.
3. Create a file named config.js with any text editor, and enter the following content in it:

   var appConfig = {
       "IdentityPoolId": "INSERT_COGNITO_IDENTITY_POOL_ID"
   }
   
   var amplifyConfig = {
       "Auth": {
           "region": "INSERT_AWS_REGION_ID",
           "userPoolId": "INSERT_COGNITO_USER_POOL_ID",
           "userPoolWebClientId": "INSERT_COGNITO_USER_POOL_CLIENT_ID",
           "mandatorySignIn": true,
           "cookieStorage": {
               "domain": window.location.hostname,
               "path": "/",
               "expires": 30,
               "secure": true
         }
       }
   }

4. In the config.js file you created, replace the four INSERT_ strings with the Amazon Cognito identity pool ID, identifier of your Region of choice, Amazon Cognito user pool ID, and user pool client ID.
   You can retrieve such values by opening the AWS CloudFormation console, choosing the stack named augmentability-stack, and choosing the Outputs tab.
   
5. Save the config.js file in the aws-augmentability-main folder, and zip the folder to obtain a new aws-augmentability-main.zip file.

Deploy the mobile web app to the Amplify console

Now that you have downloaded and edited the AugmentAbility project files, you’re ready to build and deploy the mobile web app using the Amplify console.

1. On the Get started with Amplify Hosting page, choose Deploy without Git provider.
2. Choose Continue.
3. In the Start a manual deployment section, for App name, enter the name of your app.
4. For Environment name, enter a meaningful name for the environment, such as development or production.
5. For Method, choose Drag and drop.
6. Either drag and drop the aws-augmentability-main.zip file from your computer onto the drop zone or use Choose files to select the aws-augmentability-main.zip file from your computer.
7. Choose Save and deploy, and wait for the message Deployment successfully completed.

Use the mobile web app

The mobile web app should now be deployed. Before accessing the app for the first time, you have to set a new password for the user that has been automatically created during Step 1. You can find the link to the temporary login screen in the Outputs tab for the CloudFormation stack (field UserPoolLoginUrl). For this first sign-in, you use the user name you set up and the temporary password you received via email.

After you set your new password, you’re ready to test the mobile web app.

In the General section of the Amplify console, you should be able to find a link to the app under the Production branch URL label. Open it or send it to your smartphone, then sign in with your new credentials, and start playing with AugmentAbility.

Animated screenshot showcasing the “Live transcription and text to speech” feature of AWS AugmentAbility.

Animated screenshot showcasing the “Object detection” feature of AWS AugmentAbility.

Animated screenshot showcasing the “Text recognition” feature of AWS AugmentAbility.

Next steps

If you want to make changes to the mobile web app, you can work on the files cloned from the repository, locally build the mobile web app (as explained in the README file), and then redeploy the app by uploading the updated ZIP file via the Amplify console. As an alternative, you can create a GitHub, Bitbucket, GitLab, or AWS CodeCommit repository to store your project files, and connect it to Amplify to benefit from automatic builds on every code commit. To learn more about this approach, refer to Getting started with existing code. If you follow this tutorial, make sure to replace the command npm run build with npm run-script build at Step 2a.

To create additional users on the Amazon Cognito console, refer to Creating a new user in the AWS Management Console. In case you need to recover the password for a user, you should use the temporary login screen you used for changing the temporary password. You can find the link on the Outputs tab of the CloudFormation stack (field UserPoolLoginUrl).

Clean up

When you’re done with your tests, to avoid incurring future charges, delete the resources created during this walkthrough.

1. On the AWS CloudFormation console, choose Stacks in the navigation pane.
2. Choose the stack augmentability-stack.
3. Choose Delete and confirm deletion when prompted.
4. On the Amplify console, select the app you created.
5. On the Actions menu, choose Delete app and confirm deletion when prompted.

Conclusion

In this post, I showed you how to deploy a code sample that uses AWS AI and ML services to put features such as live transcription, text to speech, object detection, or text recognition in the hands of everyone. Knowing how to build applications that can be used by people with a wide range of abilities and disabilities is key for creating more inclusive and accessible products.

To get started with AugmentAbility, clone or fork the GitHub repository and start experimenting with the mobile web app. If you want to experiment with AugmentAbility before deploying resources in your AWS account, you can check out the live demo (credentials: demo-user, Demo-password-1).

About the Author

Luca Guida is a Solutions Architect at AWS; he is based in Milan and supports Italian ISVs in their cloud journey. With an academic background in computer science and engineering, he started developing his AI/ML passion at university; as a member of the natural language processing (NLP) community within AWS, Luca helps customers be successful while adopting AI/ML services.

Read More

Today, customers can raise support tickets through multiple channels like – web, mobile, chat-bots, emails, or phone calls. When a support ticket is raised by a customer, it is processed and assigned to a category based on the information provided in the ticket. It is then routed to the support group for resolution according to the category of the ticket. It is estimated that a high number of support tickets are usually not routed to the right group due to incorrect ticket categorization. Incorrectly assigned tickets cause delay in overall resolution time, often resulting in severe customer dissatisfaction. It may also have other widespread impacts like financial, operational, or other business repercussions. Hence, ticket classification is an essential task for every organization these days. Although you may classify tickets manually, but it is prone to error, not cost-effective, and does not scale.

AWS Managed Services (AMS) uses Amazon Comprehend custom classifications to categorize inbound requests by resource and operation type based on how the customer described their issue. Amazon Comprehend is a natural language processing (NLP) service that uses machine learning (ML) to uncover valuable insights and connections in text. AMS utilizes custom classifiers to label customer requests with appropriate issue types, resource type, and resource action, thereby routing customer tickets to the SMEs. Amazon Comprehend classification is utilized to find opportunities for new internal automation tools that AMS engineers can use to fulfill customer requirements to reduce manual effort and chances of manual errors. The classification data is stored in an Amazon Redshift cluster and used to analyze customer requests and find new automation tool candidates. This automation results in increased operational efficiency and reduced cost.

In this post, we show how managed service providers can use Amazon Comprehend to classify and route the tickets, provide suggestions based on the classification, and utilize the classification data.

Solution overview

The following diagram shows the solution architecture.

The workflow is as follows:

1. A customer submits the ticket.
2. The ticket system receives the ticket from the customer, and invokes the ticket classifier AWS Lambda function with the ticket details. Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. Lambda is chosen for the solution to reduce cost and maintenance effort.
3. The ticket classifier Lambda function classifies the ticket with Amazon Comprehend using the ticket title and description. With Amazon Comprehend, you can train the NLP model and provide both batch and real-time classifiers without provisioning and maintaining infrastructure.
4. The ticket classifier Lambda function pushes the ticket classification data to the Amazon Redshift cluster via Amazon Kinesis Data Firehose. Kinesis Data Firehose is an extract, transform, and load (ETL) service that captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services. Amazon Redshift uses SQL to analyze structured and semi-structured data across data warehouses, operational databases, and data lakes, using AWS-designed hardware and ML to deliver the best price performance at any scale. Kinesis Data Firehose delivers data to an Amazon Simple Storage Service (Amazon S3) bucket first and then issues an Amazon Redshift COPY command to load the data into an Amazon Redshift cluster.
5. The ticket classifier Lambda function invokes the ticket handler Lambda function.
6. The ticket handler Lambda function runs code to help the ticket handling. In this example, it returns the recommended materials for handling the ticket based on the classification.
7. Ticket analysis can be done with Amazon QuickSight. From ticket analysis, you can find out the top requested ticket type. Based on the analysis, you can discover ticket trends and opportunities to automate top ticket types. QuickSight is a cloud-scale business intelligence (BI) service that you can use to deliver easy-to-understand insights to the people who you work with, wherever they are.

In the following sections, we walk you through the steps to implement the solution, integrate the ticket classification infrastructure with your ticketing system, and use the classification data with QuickSight.

Implement the solution

In this section, we walk through the steps to provision your solution resources and create the necessary infrastructure.

Configure Amazon Comprehend

In this step, we train two new Amazon Comprehend custom classification models: Operation and Resource, and create a real-time analysis endpoint for each model.

Upload the training data

To upload the training data, complete the following steps:

1. Download ticket_training_data.zip and unzip the file.
   This folder contains the following two files:
   • training_data_operations.csv – This file is a two-column CSV file that we use to train the Operation classification model. The first column contains class, and the second column contains document.
   • training_data_resources.csv – This file is a two-column CSV file that we use to train the Resource classification model. Like the training_data_operations.csv file, the first column contains class, and the second column contains document.
2. On the Amazon S3 console, create a new bucket for Amazon Comprehend. Because S3 bucket names are globally unique, you need to create a unique name for the bucket. For this post, we call it comprehend-ticket-training-data. Enable server-side encryption and block public access when creating the bucket.
3. Upload training_data_operations.csv and training_data_resources.csv to the new S3 bucket.

Create two new models

To create your models, complete the following steps:

1. On the Amazon Comprehend console, choose Custom classification in the navigation pane.
2. Choose Create new model.
3. Provide the following information:
   1. For Model name, enter ticket-classification-operation.
   2. For Language, choose English.
   3. For Classifier mode, select Using Single-label mode.
   4. For Data format, select CSV file.
   5. For Training dataset, enter the S3 path for training_data_operations.csv.
   6. For Test data source, select Autosplit.
      Autosplit automatically selects 10% of your provided training data to use as testing data.
   7. For IAM Role, select Create an IAM role.
   8. For Permissions to access, choose the training, test, and output data (if specified) in your S3 buckets.
   9. For Name suffix, enter ticket-classification.
4. Choose Create.
   
5. Choose Create new model again to create your resource classification model.
6. Provide the following information:
   1. For Model name, enter ticket-classification-resource.
   2. For Language, choose English.
   3. For Classifier mode, select Using Single-label mode.
   4. For Data format, select CSV file.
   5. For Training dataset, enter the S3 path for training_data_resources.csv.
   6. For Test data source, select Autosplit.
   7. For IAM Role, select Use an existing IAM role.
   8. For Role name, choose AmazonComprehendServiceRole-ticket-classification.
7. Choose Create.
   

Amazon Comprehend is now processing the CSV files and using them to train custom classifiers. We then use these to help classify customer tickets. The larger and more accurate our training data is, the more accurate the classifier will be.

Wait for the version status to show as Trained as below. It may take up to 1 hour to complete, depending on the size of the training data.

Create Amazon Comprehend endpoints

Amazon Comprehend endpoints are billed in 1-second increments, with a minimum of 60 seconds. Charges continue to incur from the time you start the endpoint until it’s deleted, even if no documents are analyzed. For more information, see Amazon Comprehend Pricing. To create your endpoints, complete the following steps:

1. On the Amazon Comprehend console, choose Endpoints in the navigation pane.
2. Choose Create endpoint to create your operation classification endpoint.
3. Provide the following information:
   1. For Endpoint name, enter ticket-classification-operation.
   2. For Custom model type, select Custom classification.
   3. For Classifier model, choose ticket-classification-operation.
   4. For Version, choose No Version Name.
   5. For Number of inference units (IUs), enter 1.
4. Choose Create endpoint.
   
5. Choose Create endpoint again to create the resource classification endpoint.
6. Provide the following information:
   1. For Endpoint name, enter ticket-classification-resource.
   2. For Custom model type, select Custom classification.
   3. For Classifier model, choose ticket-classification-resource.
   4. For Version, choose No Version Name.
   5. For Number of inference units (IUs), enter 1.
7. Choose Create endpoint.
   

After you create both endpoints, wait until the status for both shows as Active.

Test the Amazon Comprehend endpoints with real-time analysis

To test your endpoints, complete the following steps:

1. On the Amazon Comprehend console, choose Real-time analysis in the navigation pane.
2. For Analysis type¸ select Custom.
3. For Endpoint¸ choose ticket-classification-operation.
4. For Input text, enter the following:

   Hi support,
   Please update the timezone to UTC on i-abcd1234.
   Thanks.

5. Choose Analyze.
   The results show that the Update class has the highest confidence score.
6. Change Endpoint to ticket-classification-resource and choose Analyze again.

The results show that the EC2 class has the highest confidence score.

Create a secret for the Amazon Redshift cluster password

In this step, we create an AWS Secrets Manager secret for your Amazon Redshift cluster password. Secrets Manager helps you protect secrets needed to access your applications, services, and IT resources. The service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. In this post, we store the Amazon Redshift cluster password in a Secrets Manager secret.

1. On the Secrets Manager console, choose Secrets in the navigation pane.
2. Choose Store a new secret.
3. For Secret type, select Other type of secret.
   
4. Under Key/value pairs, set your key as password and value as your Amazon Redshift cluster password.
   The password must be between 8–64 characters in length and contain at least one uppercase letter, one lowercase letter, and one number. It can be any printable ASCII character except ‘ (single quote), “ (double quote), , /, @, or space.
5. Choose Next.
   
6. For Secret name, enter ClassificationRedshiftClusterPassword.
7. Choose Next.
8. In the Secret rotation section, choose Next.
9. Review your secret configuration and choose Store.

Provision your infrastructure with AWS CloudFormation

In this step, we provision the infrastructure for the solution using an AWS CloudFormation stack.

Upload the Lambda function code

Before launching the CloudFormation stack, upload your Lambda function code:

1. Download lambda_code.zip
2. On the Amazon S3 console, open the bucket that you created.
3. Upload lambda_code.zip.

Create your CloudFormation stack

To provision resources with AWS CloudFormation, complete the following steps:

1. Download cloudformation_template.json.
2. On the AWS CloudFormation console, choose Create stack.
3. Select With new resources (standard).
4. For Template source, choose Upload a template file.
5. Choose the downloaded CloudFormation template.
6. Choose Next.
7. For Stack name, enter Ticket-Classification-Infrastructure.
8. In the Parameters section, enter the following values:
   1. For ClassificationRedshiftClusterNodeType, enter the Amazon Redshift cluster node type. dc2.large is the default.
   2. For ClassificationRedshiftClusterPasswordSecretName, enter the Secrets Manager secret name that stores the Amazon Redshift cluster password.
   3. For ClassificationRedshiftClusterSubnetId, enter the subnet ID where the Amazon Redshift Cluster is hosted. The subnet must be within the VPC which you mentioned in the ClassificationRedshiftClusterVpcId parameter.
   4. For ClassificationRedshiftClusterUsername, enter the Amazon Redshift cluster user name.
   5. For ClassificationRedshiftClusterVpcId, enter the VPC ID where the Amazon Redshift cluster is hosted.
   6. For LambdaCodeS3Bucket, enter the S3 bucket name where you uploaded the Lambda code.
   7. For LambdaCodeS3Key, enter the Amazon S3 key of the deployment package.
   8. For QuickSightRegion, enter the Region for QuickSight. The Region for QuickSight should be consistent with the Region you’re using for Amazon Comprehend and the S3 bucket.
9. Choose Next.
10. In the Configure stack options section, choose Next.
11. In the Review section, select I acknowledge that AWS CloudFormation might create IAM resources.
12. Choose Create stack.

Configure your Amazon Redshift cluster

In this step, you enable audit logging and add the new table to the Amazon Redshift cluster created through the CloudFormation template.

Audit logging is not turned on by default in Amazon Redshift. When you turn on logging on your cluster, Amazon Redshift exports logs to Amazon CloudWatch, which capture data from the time audit logging is enabled to the present time. Each logging update is a continuation of the previous logs.

Enable audit logging

You can skip this step if you don’t need audit logging for your Amazon Redshift cluster.

1. On the Amazon Redshift console, choose Clusters in the navigation pane.
2. Choose the Amazon Redshift cluster starting with classificationredshiftcluster-.
3. On the Properties tab, choose Edit.
4. Choose Edit audit logging.
5. For Configure audit logging¸ choose Turn on.
6. For Log expert type, choose CloudWatch.
7. Select all log types.
8. Choose Save changes.

Create new table

To create a new table, complete the following steps:

1. On the Amazon Redshift console, choose Query data.
2. Choose Query in query editor v2.
3. On the Database page, choose your cluster.
   
4. For Database, enter ticketclassification.
5. Enter the user name and password you configured in the CloudFormation stack parameters.
6. Choose Create connection.
   
7. When the connection is made, choose the plus sign and open a new query window.
8. Enter the following query:

   CREATE TABLE tickets(
     id             VARCHAR(50)   NOT NULL,
     title          VARCHAR(1000) NOT NULL,
     description    VARCHAR(5000) NOT NULL,
     creation_time  TIMESTAMPTZ   NOT NULL,
     operation      VARCHAR(5000) NULL,
     resource       VARCHAR(5000) NULL
   );

9. Choose Run.

Test the classification infrastructure

Now the infrastructure for ticket classification is ready. Before integrating with your ticket system, let’s test the classification infrastructure.

Run the test

To run the test, complete the following steps:

1. On the Lambda console, choose Functions in the navigation pane.
2. Choose the function that starts with Ticket-Classification-Inf-TicketClassifier.
3. On the Test tab, choose Test event.
4. For Name, enter TestTicket.
5. Enter the following test data:

   {
     "TicketId": "00000001",
     "TicketTitle": "Update the timezone",
     "TicketDescription": "Hi support, Please update the timezone to UTC on i-abcd1234. Thanks.",
     "TicketCreationTime": "2020-12-04 03:09:00-08"
   }

6. Choose Test.
   

The ticket is classified, and the classification data is stored in the Amazon Redshift cluster. After the classification, the ticket handler Lambda function runs, which handles the ticket based on the classification, including recommending materials to support engineers.

Check the ticket classifier test log

To check the test log, complete the following steps:

1. In the result section of the test, choose Logs, or choose View logs in CloudWatch on the Monitor tab.
2. Choose the log stream.

You can view the logs in the following screenshot, which shows the output from Amazon Comprehend and the final top classification of the ticket. In this example, the test ticket is classified as Resource=EC2, Operation=Update.

Check the ticket classification output in the Amazon Redshift cluster

To validate the output in your cluster, complete the following steps:

1. On the Amazon Redshift query editor v2 console, choose the plus sign to open a new query window.
2. Enter the following query:

   SELECT * FROM "tickets";

3. Choose Run.

The following screenshot shows the ticket classification. If it’s not available yet, wait for a few minutes and retry (Kinesis Data Firehose needs some time to push the data). We can now use this data in QuickSight.

Check the ticket handler test log

After the ticket classifier pushes the classification data in the Amazon Redshift cluster, the ticket handler Lambda function runs, which handles the ticket based on the classification, including recommending materials to support engineers. In this example, the ticket handler returns recommended materials including the runbook, AWS documentation, and SSM documents so support can refer to them when handling the ticket. You can integrate the output with your ticket handling system, and you can customize the handling processes in the Lambda function code. In this step, we check what recommendations were made.

1. On the Lambda console, choose Functions in the navigation pane.
2. Choose the Lambda function that starts with Ticket-Classification-Inf-TicketHandlerLambdaFunct.
3. On the Monitor tab, choose View logs in CloudWatch.
4. Choose the log stream.

The following screenshot shows the logs. You can see the output from Amazon Comprehend and the list of recommended AWS documents and SSM documents for the ticket classified as Update EC2. You can add your own runbooks, documents, SSM documents, or any other materials in the Lambda function code.

Integrate the ticket classification infrastructure with your ticketing system

In this section, we walk through the steps to integrate your ticketing classification infrastructure with your ticketing system and customize your configuration.

Most ticketing systems have a trigger feature, which allows you to run code when the ticket is submitted. Set up your ticketing system to invoke the ticket classifier Lambda function with the following formatted input:

{
  "TicketId": "{{ Ticket ID }}",
  "TicketTitle": "{{ Ticket Title }}",
  "TicketDescription": "{{ Ticket Description }}",
  "TicketCreationTime": "{{ Ticket Creation Time. e.g. 2020-12-04 03:09:00-08 }}"
}

If you want to customize the input, modify the ticket classifier Lambda function code. You need to add or remove parameters (lines 90–105) and customize the input for Amazon Comprehend (lines 15–17).

You can customize the ticket handler Lambda function to run automation or edit the recommendations. For example, you can add the internal comment to the ticket with the recommendations. To customize, open the ticket handler Lambda code, and edit lines 68–70 and 75–81.

Use classification data with QuickSight

After you integrate the ticket classification infrastructure with your ticket system, the ticket classification data is stored in the Amazon Redshift cluster. You can use QuickSight to check this data and generate reports. In this example, we generate a QuickSight analysis with the classification data.

Sign up for QuickSight

If you don’t already have QuickSight, sign up with the following steps:

1. On the QuickSight console, choose Sign up for QuickSight.
2. Choose Standard.
3. Under QuickSight region, choose the Region you configured in the CloudFormation parameter QuickSightRegion.
4. Under Account info, enter your QuickSight account name and notification email address.
5. Under QuickSight access to AWS services, select Amazon Redshift.
6. If you want to allow access and autodiscovery for other resources, select them as well.
7. Choose Finish.
   
8. Choose Go to Amazon QuickSight after you’re signed up.

Connect your Amazon Redshift cluster to QuickSight

To connect your cluster to QuickSight as a data source, complete the following steps:

1. On the QuickSight console, choose Datasets in the navigation pane.
2. Choose New dataset.
3. Choose Redshift Auto-discovered.
4. Provide the following information:
   1. For Data source name, enter ticketclassification.
   2. For Instance ID, choose the Amazon Redshift cluster starting with classificationredshiftcluster-.
   3. For Connection type, choose Public network.
   4. For Database name, enter ticketclassification.
   5. Enter the Amazon Redshift cluster user name and password you configured in the CloudFormation stack parameters.
5. Choose Validate connection to see if the connection works.
   If it doesn’t work, this is likely due to using the wrong user name and password, or the QuickSight Region is different from what you specified in the CloudFormation stack.
6. Choose Create data source.
   
7. In the Choose your table section, select the tickets table.
8. Choose Select.
   
9. Select Import to SPICE for quicker analytics.
   SPICE is the QuickSight Super-fast, Parallel, In-memory Calculation Engine. It’s engineered to rapidly perform advanced calculations and serve data. Importing (also called ingesting) your data into SPICE can save time and money. For more information on SPICE, refer to Importing Data into SPICE. If you get the error “Not enough SPICE capacity,” purchase more SPICE capacity. For more information, refer to Purchasing SPICE capacity in an AWS Region.
10. Choose Visualize.
    

Create a ticket classification analysis report

Once you finish dataset creation, you can see the new QuickSight analysis. In this section, we walk through the steps to create a ticket classification analysis report, including a pivot table, pie charts, and line charts.

1. Choose AutoGraph.
2. Under Visual types, choose the pivot table.
3. Drag operation from Fields list to Rows.
4. Drag resource from Fields list to Columns.
5. On the Add menu, choose Add visual.
6. Under Visual types, choose the pie chart.
7. Drag operation from Fields list to Group/Color.
8. On the Add menu, choose Add visual again.
9. Under Visual types, choose the pie chart again.
10. Drag resource from Fields list to Group/Color.
11. On the Add menu, choose Add visual again.
12. Under Visual types, choose the line chart.
13. Drag creation_time from Fields list to X axis.
14. Drag operation from Fields list to Color.
15. On the Add menu, choose Add visual again.
16. Under Visual types, choose the line chart again.
17. Drag creation_time from Fields list to X axis.
18. Drag operation from Fields list to Color.
19. Resize and reorder the charts as needed.
20. Choose Save as.
21. Enter a name for your analysis and choose Save.

Congratulations! Your first ticket analysis is ready. Once you have more data, the analysis will look like the following screenshot.

Clean up

In this step, we clean up the resources we created with various services.

Amazon Comprehend

To delete your endpoints, complete the following steps:

1. On the Amazon Comprehend console, choose Endpoints in the navigation pane.
2. Select the endpoint ticket-classification-operation.
3. Choose Delete and follow the prompts.
4. Repeat these steps to delete the ticket-classification-resource endpoint.
   Next, delete the custom classifications you created.
5. Choose Custom classification in the navigation pane.
6. Select the classification ticket-classification-operation.
7. Select No Version Name.
8. Choose Delete and follow the prompts.
9. Repeat these steps to delete the ticket-classification-resource classification.

Amazon S3

Next, clean up the S3 bucket you created.

1. On the Amazon S3 console, select the bucket you created.
2. Delete all the objects in the bucket.
3. Delete the bucket.

Amazon QuickSight

Delete the QuickSight analyses and dataset you created.

1. On the QuickSight console, choose Analyses in the navigation pane.
2. Choose the options icon (three dots) on the analysis you created.
3. Choose Delete and follow the prompts.
4. Choose Datasets in the navigation pane.
5. Choose the tickets dataset.
6. Choose Delete dataset and follow the prompts.

AWS CloudFormation

Clean up the resources you created as part of the CloudFormation stack.

1. On the AWS CloudFormation console, choose Stacks in the navigation pane.
2. Choose the Ticket-Classification-Infrastructure stack.
3. On the Resources tab, choose the physical ID of ClassificationDeliveryStreamS3Bucket.
   The Amazon S3 console opens.
4. Delete any objects in this bucket.
5. Return to the AWS CloudFormation console, choose Delete, and follow the prompts.

AWS Secrets Manager

Lastly, delete the Secrets Manager secret.

1. On the Secrets Manager console, select the secret ClassificationRedshiftClusterPassword.
2. On the Actions menu, choose Delete secret.
3. Set the waiting period as 7 days and choose Schedule Delete.

Your secret will be automatically deleted after 7 days.

Conclusion

In this post, you learned how to utilize AWS services to create an automatic classification and recommendation system. This solution will help your organizations build the following workflow:

1. Classify customer requests.
2. Recommend automated solutions.
3. Analyze customer request classifications and discover top customer requests.
4. Release a new automated solution and increase the automation rate.

For more information about Amazon Comprehend, see Amazon Comprehend Documentation. You can also discover other Amazon Comprehend features and get inspiration from other AWS blog posts about using Amazon Comprehend beyond classification.

About the Authors

Seongyeol Jerry Cho is a Senior Systems Development Engineer at AWS Managed Services based in Sydney, Australia. He focuses on building highly scalable and automated cloud operations software using a variety of technologies, including machine learning. Outside of work, he enjoys travel, camping, reading, cooking, and running.

Manu Sasikumar is a Sr. Systems Engineer Manager with AWS Managed Services. Manu and his team focus on building powerful and easy-to-use automations to reduce manual effort, and build AI and ML-based solutions for managing customer requests. Outside of work, he loves spending his spare time with his family, as well as being part of various humanitarian and volunteer activities.

Read More

This is a guest post by Andrew Degenholtz, CEO and Founder of eMagazines, the parent company of ReadAlong.ai. eMagazines’ technology seamlessly transforms print products into premium digital and audio experiences. Leveraging Amazon technology, ReadAlong.ai offers a simple, turn-key way for publishers to add audio to their websites with a single line of code.

eMagazines supports publishers in bringing high-quality journalism content to readers across digital platforms. Our ReadAlong.ai brand allows our customers to deepen their connection to readers by adding audio to traditional text-first publishing formats. In March 2020, we helped TIME for Kids launch a digital version of its popular magazine for school-aged kids. This premium subscription product helped their users transition to digital when the pandemic forced schools to close and families needed high-quality educational tools to supplement classroom learning materials.

In this post, we share how we created an automated way for TIME for Kids to seamlessly add audio for early readers and pre-readers through ReadAlong.ai, which uses Amazon Polly technology.

Why did TIME for Kids decide to start creating audio narration of their articles?

The addition of audio with auto scrolling and highlighting of text supports pre-readers and those students still learning to read. Listening while reading supports vocabulary development and reading comprehension, and new words are more likely to be learned when both their oral and written forms are provided. A report from the National Center on Early Childhood Development, Teaching, and Learning states that developing brains need to hear language even before learning to talk, and that even infants’ brains are preparing to speak months before they say their first words. Not only that, but the report also revealed that listening to stories read aloud helps expand both the volume and variety of words entering young vocabularies and fields of comprehension. Experts at Scholastic report that being read to also helps early readers “focus on the sounds of words read without interruption and provides a model of fluent reading,” and also noted that resources like audio help children learn how to listen, a prerequisite to learning to read.

What was the business challenge we addressed?

TIME for Kids originally addressed pre-reader accessibility by hiring voice actors to record their stories. The earlier iteration of their audio play button used an HTML audio player without speed variation or the option to scroll the page or highlight the text. The experience was expensive and time-consuming, and the user experience wasn’t as engaging as it could be. TIME for Kids was also unable to see even basic data around play or completion rates.

Why Amazon Polly?

We chose Amazon Polly because its APIs and web services support our goal of automating processes and making things easy for our clients.

Amazon Polly’s neural text-to-speech synthesis does the best job of voicing words within the context of a sentence, and the consistency in speech quality allows for the automation of article rendering.

Additionally, Amazon Polly offers a responsive API and powerful SSML support. This offers support for those cases where more control is needed to change inflection, and in the event that text contains challenging names (people, brands, companies) or word and phrase replacements (reading out abbreviations or acronyms in a particular way).

Amazon Polly also supports speech marks, which are crucial for highlighting the text that is currently being read out.

For TIME for Kids, the Kevin voice was a clear winner. TIME for Kids loved the approachable sound of the Kevin voice—they wanted wanting a voice that sounded like a child’s in order to help establish a sense of connection with young readers. Hear an example of a TIME for Kids article using the Kevin voice.

The technical challenge

TIME for Kids needed an educational audio solution for their website. It needed to be a one-time setup that was highly automated and very low friction. The solution also needed to process new articles as they were added dynamically, on a daily basis. And when a user listens to the audio, the page needed to scroll along with the text and highlight the sentence currently being read out loud.

Part of our challenge was to reliably and programmatically identify which content should be read aloud. In a typical publishing context, the audio player needs to read the article title and content, but avoid reading the header and footer text, navigation bars, and certain kinds of ads or captions. Our page analysis solution combines positive and negative query selectors. For each configuration, defined by a set of articles that share the same structure and layout, the http://readalong.ai solution supports a set of allow list selectors and a set of deny list selectors that together capture the appropriate content for synthesizing speech.

Furthermore, the TIME for Kids website posed many technical challenges because some pages are available only for paying subscribers, whereas some are open to the public. TIME for Kids offers four grade-specific editions, teaching materials, curriculum guides, and weekly virtual learning plans for each issue, as well as worksheets and quizzes. Therefore, each article has multiple versions for different reading levels in both English and Spanish—some with as many as seven different reading levels in both languages.

Our solution

We created a simple drop-in script that allowed TIME for Kids to only add one line of code to the header of any page where they wanted to offer audio. The script automated everything from page content delivery to audio-synthesis to webpage integration. Since the start of the school year, we’ve added the Kevin and Lupe voices (for English and Spanish content, respectively) to thousands of articles on timeforkids.com.

Our solution allowed for automated content delivery and audio synthesizing, which meant no need to sign into a dashboard, FTP, Dropbox, or otherwise send new article content to ReadAlong.ai each time a new page was added. The user-friendly backend of the solution also allows TIME for Kids to easily make word replacements, including global rules, to give the audio synthesizer engine lexicon hints for context-based pronunciations and difficult names, brands, or acronyms.

In addition to positioning and styling the launcher and player to match the TIME for Kids site design, as part of the customization, we added functionality to highlight and scroll the text as the article is read aloud, which is another helpful tool to support children in learning to recognize words and connect them to sounds. We customized this feature to be visible but not distracting, so the audio and visual elements could work in tandem to aid young readers. To support this enhanced feature, we implemented the detailed word- and sentence-level metadata available in Amazon Polly to provide a fluid highlighting experience that helps readers follow along as they encounter new words and concepts. This allows the listener to identify what they’re hearing as they view the content as it’s highlighted on the browser.

We also created a default for the Amazon Polly Kevin and Lupe voices to start at a slower speed, so the default pacing is at .9x, rather than at 1x, as another way to help early readers and pre-readers better access the content. Listeners have the ability to lower the default voice speed to .75x or increase to 1.5x, in order to accommodate more reading levels.

Business benefits for the customer

With our product in place on their site, TIME for Kids was able to voice their content in a scalable way. They deliver content on an article-by-article basis in two different languages (English and Spanish) and in seven different reading levels.

They’re also now able to easily collect and analyze data in real time, including both play and completion rates, and view most popular articles as well as articles with the most audio engagement.

We now know that 55% of kids that click to listen to an article complete 100% of the article, and 66% of kids that listen to an article complete more than half of the article. These significant completion rates reinforce the benefit and confirm that listeners are comfortable with the technology and the voice is relatable. The ReadAlong.ai audio also helped TIME for Kids promote its advanced accessibility features, including key articles with Spanish translation and read-aloud functionality, because the presence of the audio is featured prominently on the preview of each article along with other benefits (such as Spanish translation).

Stacy Bien, Director of Curriculum for TIME for Kids, was impressed with both the solution and the engagement data, saying,

“This is really a thing of beauty. This solution will help so many early readers develop their reading skills and easily consume more content. For us, we’ve seen a huge lift in engagement. That, coupled with the ease of use and cost-effectiveness, makes this a slam dunk.”

Conclusion

ReadAlong.ai used Amazon Polly to help TIME for Kids streamline the process of adding high-quality audio voiceover content to its premium subscription product. Our solution enabled the customer to significantly improve product time, precision, and cost. For example, a voiceover artist typically spends 1 hour or more to record an article, edit the audio, and master the final audio output. Now, once the ReadAlong.ai script has been added to the site, when new articles are created, the content is automatically processed without any time spent by a voiceover artist, audio editor, or administrator. The audio reads articles precisely and rarely requires adjustments, creating a valuable and immeasurable savings of both time and cost.

Collected KPIs tell us that not only did this become an easy way for the TIME for Kids team to manage audio functionality, but that the end-users—children early in the development of their reading abilities—take to the functionality as another tool on their reading path.

About the Author

Andrew Degenholtz is CEO and Founder of eMagazines and ReadAlong.ai, and is President of ValueMags, which he founded in 1999. Degenholtz holds a master’s in marketing from Northwestern University and a B.A. from Muhlenberg College. Previously, he was a member of the Alliance for Audited Media digital edition task force, created to develop best practices for acquisition of digital magazine subscribers.

Read More

We are excited to announce that in Amazon Forecast, you can now start your forecast horizon at custom starting points, including on Sundays for weekly forecasts. This allows you to more closely align demand planning forecasts to local business practices and operational requirements.

Forecast is a fully managed service that uses statistical and machine learning (ML) algorithms to deliver highly accurate time series forecasts. It uses state-of-the-art algorithms to predict future time series data based on historical data, and requires no ML experience. Typical Forecast applications include resource planning for inventory, workforce staffing, and web traffic. In this post, we review a new option that allows you to align forecasts with business and demand cycles, while reducing operational cost by offloading aggregation workflows.

To optimize demand planning, forecasts need to align with business operations. Previously, starting points for forecasts were fixed: daily forecasts assumed demand starting at midnight each day, weekly predictions assumed Monday as the first day of the week, and monthly predictions started on the first day of each month. These predefined starting points presented two challenges. First, if your business cycle began at a different point than the fixed value, you had to manually aggregate forecasts to your required starting point. For example, if your business week began on a Sunday and you wanted to produce weekly forecasts, you had to manually aggregate daily forecasts to a Sunday–Saturday week. This additional work added cost and compute time, and presented opportunities for errors. Second, the training data and forecast periods weren’t consistent; if your data reflects a demand cycle that begins on Sundays, the predictor and forecast should also use Sunday as the starting point.

Custom forecast horizon starting points now align business operations and forecasts, eliminating the need for manual aggregation work and saving cost and compute. If you have a business week starting on Sundays, you can automatically aggregate daily data to generate weekly forecasts that begin on Sundays. Or you can begin daily forecasts starting at 9:00 AM. Predictors can now be aligned with your ground truth data, providing consistency between inputs and forecasts. Forecast horizon starting points are easily defined when training new predictors via the Forecast console or using Forecast APIs.

Define custom forecast horizon starting periods

The forecast horizon, also called frequency, is the length of time for which a forecast is made, and is bounded by a starting and ending point. In Forecast, you can now select specific starting points for daily, weekly, monthly, and yearly forecast horizons when training new predictors. These starting points—also called boundary values—are selected at one frequency unit finer than the forecast horizon, as shown in the following table.

With custom starting points, you can align forecasts to start at specific points in time that match your business processes and ground truth data, for example, the month of May, the 15th of the month, Sundays, or 15:00 hours. For forecast horizons coarser than the provided time series frequency, Forecast aggregates the time series data based on the custom starting point. For example:

• When generating daily forecasts from hourly data with a 9:00 AM starting period, forecasts are aggregated with hourly data each day between 9:00 AM to the following day at 8:00 AM
• When generating weekly forecasts from daily data with a Sunday starting period, forecasts are aggregated with daily data each week from Sunday to the following Saturday
• When generating monthly forecasts from daily data with a starting day of the 15th of the month, forecasts are aggregated with daily data from the 15th of the current month to the 14th of the next month
• When generating yearly forecasts from monthly data with a starting month of May, forecasts are aggregated with monthly data from May of the current year to April of next year

Available forecast frequencies

The following screenshots show examples of custom daily, weekly, monthly, and yearly forecast frequencies and starting points (the Time alignment boundary field on the Forecast console).

Specify custom forecast horizon starting points

You can define custom forecast horizon starting points when creating a new predictor. The following steps demonstrate how to do this using the Forecast console. We also offer a sample notebook that provides an example of how to integrate this new setting into your workflows.

1. On the Forecast console, choose View dataset groups, and then Create dataset group.
2. Create your dataset group, a target time series dataset, and load your data.
   You’re redirected to the Forecast console as your data is loaded.
3. After your target time series dataset is loaded into your dataset group and active, choose Start under Train a predictor.
4. In the Train predictor section, provide values for the Name, Forecast frequency, and Forecast horizon fields.
5. In the optional Time alignment boundary field, specify the starting point the predictor uses for the forecast.
   The values in this list depend on the Forecast frequency value you choose. In this example, we create weekly forecasts with a 1-week horizon, with Sunday as the starting day of the week and of the forecast.
   
6. Provide other optional configurations as needed and choose Create.
   
   After you create the predictor, you can create your forecast.
7. In the navigation pane, under your dataset group choose Predictors.
8. Select your new predictor.
9. Choose Create forecast.
   
10. Provide the necessary details and choose Start to create your forecast.
    
11. When the forecast is complete, choose Create forecast export to export the results.
    

The following screenshots are samples of the original input file (left) and the exported forecast results (right). The input file is at an hourly frequency, whereas the forecast is produced at a weekly frequency, beginning with Sunday as the first day of the week. This is an example of Forecast automatically aggregating over two levels of forecast frequencies (from hours to days).

Conclusion

Custom forecast horizon starting points in Forecast allow you to produce forecasts that align with your specific operational requirements. Work weeks start on different days in different regions, requiring forecasts that begin on days other than Mondays, and that are aligned with ground truth training and ongoing data. Or you may want to generate hourly forecasts that reflect a demand cycle beginning at 7:00 AM each day, for example.

Forecast also automatically aggregates fine-grained forecasts to higher-level frequencies (such as days into weeks). This allows you to produce forecasts aligned with your operations, and saves you costs by removing the need to stand up and manage aggregation workflows.

Custom starting points are optional. If you don’t provide specific starting points, forecasts start at default times. Specific forecast horizon starting points are only available with AutoPredictor. For more information, refer to New Amazon Forecast API that creates up to 40% more accurate forecasts and provides explainability and CreateAutoPredictor.

To learn more about forecast frequencies, refer to Data aggregation for different forecast frequencies. All these new capabilities are available in all Regions where Forecast is publicly available. For more information about Region availability, see AWS Regional Services.

About the Authors

Dan Sinnreich is a Sr. Product Manager for Amazon Forecast. He is focused on democratizing low-code/no-code machine learning and applying it to improve business outcomes. Outside of work, he can be found playing hockey, trying to improve his tennis serve, scuba diving, and reading science fiction.

Paras Arora is a Software Development Engineer in the Amazon Forecast Team. He is passionate about building cutting edge AI/ML solutions in the cloud. In his spare time, he enjoys hiking and traveling.

Chetan Surana is a Software Development Engineer in the Amazon Forecast team. His interests lie at the intersection of machine learning and software development, applying thoughtful design and engineering skills to solve problems. Outside of work, he enjoys photography, hiking, and cooking.

Read More

We’re excited to announce that you can now automatically monitor the accuracy of your Amazon Forecast predictors over time. As new data is provided, Forecast automatically computes predictor accuracy metrics, providing you with more information to decide whether to keep using, retrain, or create new predictors.

Monitoring predictor quality and identifying deterioration in accuracy over time is important to achieving business goals. However, the processes required to continuously monitor predictor accuracy metrics can be time-consuming to set up and challenging to manage: forecasts have to be evaluated, and updated accuracy metrics have to be computed. In addition, metrics have to be stored and charted to understand trends and make decisions about keeping, retraining, or recreating predictors. These processes can result in costly development and maintenance burdens, and place meaningful operational stress on data science and analyst teams. And for customers not willing to take on this time-consuming process (they would prefer to retrain new predictors even when not needed), this wastes time and compute.

With today’s launch, Forecast now automatically tracks predictor accuracy over time as new data is imported. You can now quantify your predictor’s deviation from initial quality metrics and systematically evaluate model quality by visualizing trends, and make more informed decisions about keeping, retraining, or rebuilding your models as new data comes in. Predictor monitoring can be enabled for new predictors at inception, or turned on for existing models. You can enable this feature with one click on the AWS Management Console or using Forecast APIs.

Predictor accuracy over time

A predictor is a machine learning model created at a point in time, using an original set of training data. After a predictor is created, it’s used on an ongoing basis over days, weeks, or months into the future to generate time series forecasts with new ground truth data generated through actual transactions. As new data is imported, the predictor generates new forecasted data points based on the latest data provided to it.

When a predictor is first created, Forecast produces accuracy metrics such as weighted quantile loss (wQL), mean absolute percentage error (MAPE), or root mean squared error (RMSE) to quantify the accuracy of the predictor. These accuracy metrics are used to determine whether a predictor will be put into production. However, the performance of a predictor will fluctuate over time. External factors such as changes in the economic environment or in consumer behavior can change the fundamental factors underlying a predictor. Other factors include new products, items, and services that may be created; changes in the financial or economic environment; or changes in the distribution of data.

For example, consider a predictor trained when a certain color of a product was popular. Months later, new colors may appear or become more popular and the distribution of values change. Or a shift occurs in the business environment that modifies long-standing purchasing patterns (such as from high-margin to low-margin products). All things considered, the predictor may need to be retrained, or a new predictor may need to be created to ensure highly accurate predictions continue to be made.

Automated predictor monitoring

Predictor monitoring is designed to automatically analyze your predictor’s performance as new ground truth time series data becomes available and is used to create new forecasts. This monitoring provides you with continuous model performance information, and saves you time so you don’t have to set up the process yourself.

If predictor monitoring is enabled in Forecast, each time you import new data and produce a new forecast, performance statistics are updated automatically. Until now, these performance statistics were only available when the predictor was initially trained; now these statistics are produced on a continuous basis using new ground truth data, and can be actively monitored to gauge predictor performance.

This allows you to use predictor performance statistics to decide when to train or retrain a new predictor. For example, as the average wQL metric deviates from the initial baseline values, you can determine whether to retrain a new predictor. If you decide to retrain a predictor or create a new one, you can begin generating new forecasted data points using the more accurate predictor.

The following graphs provide two examples of predictor monitoring. In the first chart, the average wQL metric is decreasing from the baseline (the initial value when the predictor was trained), indicating that forecast accuracy is increasing over time. The chart shows average wQL dropping from 0.3 to 0.15 over the course of a few days, meaning that forecast accuracy is increasing. In this case, there is no need to retrain the predictor because it’s producing more accurate forecasts than when it was first trained.

In the next figure, the opposite is true: the average wQL is increasing, indicating that accuracy is decreasing over time. In this case, you should consider retraining or rebuilding the predictor with new data.

In Forecast, you have the choice of retraining the current predictor or rebuilding it from scratch. Retraining is done with one click and incorporates more up-to-date data and any updates and improvements in the Forecast algorithms. Rebuilding the predictor allows you to provide new inputs (such as forecast frequency, horizon, or new dimension) to create a new predictor.

Enable predictor monitoring

You can enable predictor monitoring when creating a new predictor, or turn it on for existing predictors. The steps in this section demonstrate how to perform these steps using the Forecast console. There is also a Jupyter notebook that walks through a sequence of steps to enable predictor monitoring using APIs and generate predictor monitor results.

This example uses the time-sliced sample dataset available from the predictor monitoring notebook. In our example, we start with a 100,000-row dataset of New York City taxi pickups containing a timestamp, location ID, and target value (the number of pickups requested during the timestamp at the location ID).

Complete the following steps:

1. On the Forecast console, choose View dataset groups in the navigation pane.
2. Choose Create dataset group and provide your dataset group details.
   After you create the dataset group, you’re prompted to create a target time series dataset. You use this dataset to train the predictor and create forecasts.
3. On the Create target time series dataset page, provide your data’s schema, frequency, and location.
4. Choose Start to import your target dataset.
   Next, you build your predictor and train it using your initial dataset.
5. In the navigation pane, choose Predictors.
6. Choose Train new predictor.
   
7. In the Predictor settings section, enter a name for your predictor, how long in the future you want to forecast and at what frequency, and the number of quantiles you want to forecast for.
   
8. For Optimization metric, you can choose an optimization metric to optimize AutoPredictor to tune a model for a specific accuracy metric of your choice. We leave this as default for our walkthrough.
9. To get the predictor explainability report, select Enable predictor explainability.
10. To enable predictor monitoring, select Enable predictor monitoring.
11. Under the input data configuration, you can add local weather information and national holidays for more accurate demand forecasts.
12. Choose Start to start training your predictor.
    
    Forecast now trains the predictor with this initial dataset. With predictor monitoring enabled, every time new data is provided in this dataset group, Forecast is able to compute updated predictor accuracy metrics.
13. After the predictor has been trained, choose it to evaluate the initial accuracy metrics.
    
    The Metrics tab shows initial predictor quality metrics. Because you haven’t generated any forecasts from your predictor or imported any new ground truth data, there is nothing to show on the Monitoring tab.
    The next step is to generate a forecast using the new predictor.
14. Choose Forecasts in the navigation pane.
15. Choose Create forecast to create a new forecast based on the time series data you just imported and the predictor settings.
16. Provide the forecast name, predictor name, and any additional quantile metrics you wish to compute.

After you create the forecast, you can view and export its details and results on the Forecast details page.

Predictor monitoring: Evaluating accuracy over time

Through the passage of time, new ground truth data is created by your business processes, for example, updated sales figures, staffing levels, or manufacturing output. To create new forecasts based on that new data, you can import your data to the dataset you created.

1. On the Amazon Forecast console, on the Dataset groups page, choose your dataset group.
   
2. Choose your dataset.
   
3. In the Dataset imports section, choose Create dataset import.
   
4. Provide additional details about your updated data, including its location.
   
5. Choose Start.

With predictor monitoring, Forecast compares this new data to the previous forecast generated, and computes accuracy metrics for the predictor. Updated predictor quality metrics are computed on an ongoing basis as new data is added to the dataset.

You can follow these steps to import additional data, representing additional transactions that have occurred through time.

Evaluate predictor monitoring results

To see predictor monitoring results, you must add new ground truth data after generated the initial forecasts. Forecast compares this new ground truth data to the previous forecast, and produces updated model accuracy values for monitoring.

1. On the Dataset groups page, choose the relevant dataset groups and select the Target Time Series to update it with new ground truth data.
   
2. Choose Create Dataset Import and add your new ground truth data.
   
   After you provide the additional ground truth data, you can open your predictor and view initial predictor monitoring statistics.
3. Choose your predictor and navigate to the Monitoring tab.
   

You can follow these steps to run additional forecasts using this predictor and add further iterations of ground truth data. The progression of model accuracy statistics for your predictor are available on the Monitoring tab.

This example shows model accuracy statistics for a predictor that has been evaluated with four additional data updates. The predictor had an initial baseline MAPE of 0.55 when it was initially trained. As additional data was loaded, the MAPE dropped to .42 with the first additional dataset, indicating a more accurate predictor, and fluctuated within a tight range from .42 to .48 with subsequent datasets.

You can toggle the chart to view additional metrics. In the following examples, MASE and average wQL show similar fluctuations from the baseline over time.

The Monitoring History section at the bottom of the page provides full details on all predictor accuracy metrics tracked over time.

Set up prediction monitoring on an existing predictor

You can easily enable monitoring for existing predictors. To do so, complete the following steps:

1. In the navigation pane, under your dataset, choose Predictors.
2. From here there are two ways to enable monitoring:
   1. Choose Start monitoring under the Monitoring column.
      
   2. Choose your predictor and on the Monitoring tab, under Monitor details, choose Start monitor.
      
3. In the pop-up dialog, choose Start to start monitoring for the selected predictor.
   

The Monitoring tab now shows that predictor monitoring has started, and results are generated as you import more data.

Stop and restart predictor monitoring

You can also stop and restart predictor monitoring. Consider the following:

• Cost – Predictor monitoring consumes additional resources. With typical small datasets, the cost is minimal, but may increase with large datasets (number of items in the input dataset, and forecast horizon).
• Privacy – A copy of your forecast is stored during monitoring. If you don’t want to store this copy, you can stop monitoring.
• Noise – If you’re experimenting with a predictor and don’t want to see noise in your predictor monitor results, you can temporarily stop predictor monitoring and start it again when your predictor is stable again.

To stop predictor monitoring, complete the following steps:

1. Navigate to the Monitoring tab for a predictor where monitoring is enabled.
2. Choose Stop Monitor to stop the monitoring of the predictor.
   
3. Verify your choice when prompted.
   

A message shows on the next page to indicate that predictor monitoring is stopped.

You can restart predictor monitoring by choosing Resume monitor.

Conclusion

Monitoring the quality of your predictors over time is important to achieve your demand planning and forecasting objectives, and ultimately your business goals. However, predictor monitoring can be a time-consuming exercise, and the processes required to stand up and maintain the necessary workflows can lead to higher operational costs.

Forecast can now automatically track the quality of your predictors, allowing you to reduce operational efforts, while helping you make more informed decisions about keeping, retraining, or rebuilding your predictors. To enable predictor monitoring, you can follow the steps outlined in this post, or follow our GitHub notebook.

Please note that predictor monitoring is only available with AutoPredictor. For more information, refer to New Amazon Forecast API that creates up to 40% more accurate forecasts and provides explainability and CreateAutoPredictor.

To learn more, refer to Predictor Monitoring. We also recommend reviewing the pricing for using these new features. All these new capabilities are available in all Regions where Forecast is publicly available. For more information about Region availability, see AWS Regional Services.

About the Authors

Dan Sinnreich is a Sr. Product Manager for Amazon Forecast. He is focused on democratizing low code/no code machine learning and applying it to improve business outcomes. Outside of work he can be found playing hockey, trying to improve his tennis serve, and reading science fiction.

 Adarsh Singh works as a Software Development Engineer in the Amazon Forecast team. In his current role, he focuses on engineering problems and building scalable distributed systems that provide the most value to end users. In his spare time, he enjoys watching anime and playing video games.

Shannon Killingsworth is a UX Designer for Amazon Forecast. His current work is creating console experiences that are usable by anyone, and integrating new features into the console experience. In his spare time, he is a fitness and automobile enthusiast.

Read More