Optimizing your engagement marketing with personalized recommendations using Amazon Personalize and Braze

Optimizing your engagement marketing with personalized recommendations using Amazon Personalize and Braze

Today’s marketer has a wide array of channels to communicate with their customers. However, sending the right message to the right customer on the right channel at the right time remains the preeminent challenge marketers face. In this post, I show you how to combine Braze, a customer engagement platform built on AWS for today’s on-demand, always-connected customers, and Amazon Personalize to meet this challenge and deliver experiences that surprise and delight your customers.

Braze makes it easy to organize your customers into audiences, which update in real-time, based on their behavior and profile traits. Messaging campaigns are created to target audiences through messaging channels such as email, SMS, and push notifications. Multi-step and multi-channel engagement journeys can also be designed using Braze Canvas. Campaigns and Canvases are triggered manually, based on a schedule, or even due to customer actions. However, your ability to personalize messages sent to customers is limited to what is available in their profile. Including product and content recommendations based on the learned interests of each customer as they engage with your web and mobile application is needed to truly personalize each message.

Amazon Personalize is an AWS service that uses machine learning algorithms to create recommender systems based on the behavioral data of your customers. The recommenders are private to your AWS account and based only on the data you provide. Through the Braze Connected Content feature, you are able to connect Braze to the same Amazon Personalize recommenders used to power recommendations in your web and mobile application. Since Amazon Personalize is able to adjust recommendations for each customer based on their behavior in real-time, the messages sent through Braze reflect their current preferences and intent.

Overview of solutions

I present two architectures in this post: one that uses the real-time capabilities of Braze and Amazon Personalize, and another that trades some of the freshness of real-time recommendations for a more cost-effective batch approach. The approach you select should match the goals of your engagement strategy and the scale of your messaging needs. Fortunately, the features and integration options of Braze and Amazon Personalize provide the flexibility to suit your operational requirements.

Real-time integration

We start with a real-time integration architecture. The following diagram depicts the relevant components of a sample ecommerce application in which you use Amazon Personalize to provide machine learning (ML)-powered recommenders, referred to as solutions. The primary data used to build solutions is user-item interaction history. For an ecommerce application, this includes events such as viewing a product, adding a product to a shopping cart, and purchasing a product. When rich metadata on events, items, and users is available, you can incorporate it to further improve the relevance of recommendations from the recommender. Examples of metadata include device type, location, and season for events; category, genre, and price point for items; and users’ age, gender, and subscription tier. After you create solutions, you can create autoscaling API endpoints called campaigns with just a few clicks to retrieve personalized recommendations.

Later in this post, I show you how to deploy this application in your AWS account. A self-guided workshop is also packaged with the application that you use to walk through sending personalized emails with Braze.

Our example ecommerce application retrieves personalized recommendations from a Recommendations microservice that appends the recommended item IDs from Amazon Personalize with rich product information from a Products microservice. As users engage with the application and indicate interest by viewing a product, adding a product to their shopping cart, or purchasing a product, events representing these actions are streamed to Amazon Personalize via the AWS Amplify JavaScript client library where Amazon Personalize automatically adjusts recommendations in real time based on user activity.

With personalization built into the application, you can connect Amazon Personalize with Braze to deliver personalized recommendations through outbound engagement channels such as email, SMS, and push notifications.

Braze allows you to create message templates that use the Liquid templating language to substitute placeholders in your template with values from a customer’s profile or even from an external resource. In the real-time architecture, we use the Recommendations microservice from the sample application as the external resource and Braze Connected Content as the feature to retrieve personalized recommendations to include in your message templates. The following Connected Content Liquid tag, placed at the top of your message, illustrates how to call the Recommendations service from Braze to retrieve recommendations for a user:

{% connected_content http://<RecommendationsServiceHostName>/recommendations?userID={{${user_id}}}&fullyQualifyImageUrls=1&numResults=4 :save result %}

The tag has the following elements:

  • Liquid tags are framed within {% and %} This allows you to embed tags and expressions inside message templates that may also contain text or HTML.
  • The tag type is declared just after the start of the tag. In this case, connected_content is the tag type. For the full list of supported tags, see Personalization Using Liquid Tags.
  • You next define a fully-qualified URL to the HTTP resource that Connected Content calls for each user. You replace <RecommendationsServiceHostName> with the host name for the Elastic Load Balancer for the Recommendations service in your deployment of the sample application.
  • The Recommendations service provides a few resources for different personalization features. The resource for user recommendations is accessed from the /recommendations path.
  • The query string parameters come next. The user is identified by the userID parameter, and the {{${user_id}}} expression instructs Braze to interpolate the user’s ID for each call to the service.
  • The last two query string parameters, fullyQualifyImageUrls=1 and numResults=4, tell the Recommendations service that we want the product image URLs to be fully qualified so they can be displayed in the user’s email client and, in this case, to only return the top four recommendations, respectively.
  • The :save result expression tells Braze to assign the JSON response from the Recommendations service to a template variable named result. With the response saved, you can then access elements of the response using Liquid tags in the rest of the template.

The following code shows the format of a response from the Recommendations service:

[ 
  { 
    "product": { 
      "id": "2", 
      "url": "http://recs.cloudfront.net/#/product/2", 
      "sk": "", 
      "name": "Striped Shirt", 
      "category": "apparel", 
      "style": "shirt", 
      "description": "A classic look for the summer season.", 
      "price": 9.99, 
      "image": "http://recs.cloudfront.net/images/apparel/1.jpg",
      "featured": "true" 
    } 
  }, 
  { 
    "product": { 
      "id": "1", 
      "url": "http://recs.cloudfront.net/#/product/1", 
      "sk": "", 
      "name": "Black Leather Backpack", 
      "category": "accessories", 
      "style": "bag", 
      "description": "Our handmade leather backpack will look great at the office or out on the town.", 
      "price": 109.99, 
      "image": "http://recs.cloudfront.net/images/accessories/1.jpg",
      "featured": "true" 
    } 
  }, 
  ... 
]

For brevity, the preceding code only shows the first two recommended products. Several product attributes are available that you can use in the Braze message template to represent each recommendation. To access a specific element of an array or list as we have here, you can use array subscripting notation in your Liquid tag. For example, the following tag interpolates the product name for the first recommended product in the response. For the preceding sample response, the tag resolves to “Striped Shirt”:

{{result[0].product.name}} 

When you combine the information in the personalized recommendation response from the Recommendations service with Liquid tags, the possibilities for building message designs are endless. The following code is a simplified example of how you could display a product recommendation in an HTML email template:

<table>
  <tr>
    <td>
      <a href="{{result[0].product.url}}" target="_blank">
        <img src="{{result[0].product.image}}" width="200" alt="{{result[0].product.name}}" />
      </a>
    </td>
    <td>
      <h2>{{result[0].product.name}}</h2>
      <p>{{result[0].product.description}}</p>
      <p>Only <strong>$ {{result[0].product.price}}</strong>!</p>
      <a class="button" href="{{result[0].product.url}}">Buy Now</a>
    </td>
  </tr>
</table>

Batch integration

The batch integration architecture replaces the use of the Braze Connected Content feature with an Amazon Personalize batch recommendations job that is used to push attribute updates to Braze. Batch recommendations involve creating a file in an Amazon Simple Storage Service (Amazon S3) bucket that includes the users who you wish to generate recommendations for. A reference to this file is then used to submit a job to Amazon Personalize to generate recommendations for each user in the file and output the results to another Amazon S3 file of your choosing. You can use the output of the batch recommendations job to associate personalized recommendations with user profiles in Braze as custom attributes. The Liquid tags in the message templates we saw earlier are changed to access the recommendations as custom attributes from the user profile rather than the Connected Content response.

As noted earlier, the trade-off you’re making with the batch approach is sacrificing the freshness of real-time recommendations for a more cost-effective solution. Because batch recommendations don’t require an Amazon Personalize campaign, the additional requests from Connected Content to your campaign for each user are eliminated. For Braze campaigns that target extremely large segments, this can result in a significant reduction in requests. Furthermore, if you don’t need an Amazon Personalize campaign for other purposes or you’re creating an Amazon Personalize solution dedicated to email personalization, you can forego creating a campaign entirely.

The following diagram illustrates one of the many possible approaches to designing a batch architecture. The web application components from the real-time architecture still apply; they are excluded from this diagram for brevity.

You use Amazon CloudWatch Events to periodically trigger an AWS Lambda function that builds an input file for an Amazon Personalize batch recommendations job. When the batch recommendations job is complete, another Lambda function processes the output file, decorates the recommended items with rich product information, and enqueues user update events in Amazon Kinesis Data Streams. Finally, another Lambda function consumes the stream’s events and uses the Braze User API to update user profiles.

The use of a Kinesis data stream provides a few key benefits, including decoupling the batch job from the transactional Braze user update process and the ability to pause, restart, and replay user update events.

Real-time integration walkthrough

You implement the real-time integration in the Retail Demo Store sample ecommerce application. In this post, we walk you through the process of deploying this project in your AWS account and describe how to launch the self-guided Braze workshop bundled with the application.

You complete the following steps:

  1. Deploy the Retail Demo Store project to your AWS account using the supplied AWS CloudFormation templates (25–30 minutes).
  2. Build Amazon Personalize solutions and campaigns that provide personalized recommendations (2 hours).
  3. Import users into Braze and build a Braze campaign that uses Connected Content to retrieve personalized recommendations from Amazon Personalize (1 hour).
  4. Clean up resources.

Prerequisites

For this walkthrough, you need the following prerequisites:

  • An AWS account
  • A user in your AWS account with the necessary privileges to deploy the project
  • A Braze account

If you don’t have a Braze account, please contact your Braze representative. We also assume that you have completed at least the Getting Started with Braze LAB course.

Step 1: Deploying the Retail Demo Store to your AWS account

From the following table, choose Launch Stack in the Region of your choice. This list of Regions doesn’t represent all possible Regions where you can deploy the project, just the Regions currently configured for deployment.

Region Launch
US East (N. Virginia)
US West (Oregon)
Europe (Ireland)

Accept all the default template parameter values and launch the template. The deployment of the project’s resources takes 25–30 minutes.

Step 2: Building Amazon Personalize campaigns

Before you can provide personalized product recommendations, you first need to train the ML models and provision the inference endpoints in Amazon Personalize that you need to retrieve recommendations. The CloudFormation template deployed in Step 1 includes an Amazon SageMaker notebook instance that provides a Jupyter notebook with detailed step-by-step instructions. The notebook takes approximately 2 hours to complete.

  1. Sign in to the AWS account where you deployed the CloudFormation template in Step 1.
  2. On the Amazon SageMaker console, choose Notebook instances.
  3. If you don’t see the RetailDemoStore notebook instance, make sure you’re in the same Region where you deployed the project.
  4. To access the notebook instance, choose Open Jupyter or Open JupyterLab.
  5. When the Jupyter web interface is loaded for the notebook instance, choose the workshop/1-Personalization/1.1-Personalize.ipynb.

The notebooks are organized in a directory structure, so you may have to choose the workshop folder to see the notebook subdirectories.

  1. When you have the 1.1-Personalize notebook open, step through the workshop by reading and running each cell.

You can choose Run from the Jupyter toolbar sequentially run the code in the cells.

Step 3: Sending personalized messages from Braze

With the Amazon Personalize solutions and campaigns to produce personalized recommendations in place, you can now import users into your Braze account, build a messaging template that uses Braze Connected Content to retrieve recommendations from Amazon Personalize, and build a Braze campaign to send targeted emails to your users.

Similar to the Personalization workshop in Step 1, the Braze messaging workshop steps you through the process. This notebook takes approximately 1 hour to complete.

  1. If necessary, repeat the instructions in Step 1 to open a Jupyter or JupyterLab browser window from the Amazon SageMaker notebook instance in your Retail Demo Store deployment.
  2. When the Jupyter web interface is loaded for the notebook instance, choose the workshop/4-Messaging/4.2-Braze.ipynb notebook.

As with before, you may have to choose the workshop folder to see the notebook subdirectories.

  1. When you have the 4.2-Braze notebook open, step through the workshop by reading and running each cell.

Step 4: Cleaning up

To avoid incurring future charges, delete the resources the Retail Demo Store project created by deleting the CloudFormation stack you used during deployment. For more information about the source code for this post and the full Retail Demo Store project, see the GitHub repo.

Conclusion

As marketers compete for the attention of customers through outbound messaging, there is increasing pressure to effectively target the right users, at the right time, on the right channel, and with the right messaging. Braze provides the solution to the first three challenges. You can solve the final challenge with Braze Connected Content and Amazon Personalize, and deliver highly personalized product and content recommendations that reflect each customer’s current interests.

How are you using outbound messaging to reach your customers? Is there an opportunity to increase engagement with your customers with more relevant and personalized content?

About Braze

Braze is an AWS Advanced Technology Partner and holder of the AWS Digital Customer Experience and Retail competencies. Top global brands such as ABC News, Urban Outfitters, Rakuten, and Gap are sending tens of billions of messages per month to over 2 billion monthly active users with Braze.


About the Author

James Jory is a Solutions Architect in Applied AI with AWS. He has a special interest in personalization and recommender systems and a background in ecommerce, marketing technology, and customer data analytics. In his spare time, he enjoys camping and auto racing simulation.

 

 

 

Read More

Translating documents, spreadsheets, and presentations in Office Open XML format using Amazon Translate

Translating documents, spreadsheets, and presentations in Office Open XML format using Amazon Translate

Now you can translate .docx, .xlsx, and .pptx documents using Amazon Translate.

Every organization creates documents, spreadsheets, and presentations to communicate and share information with a large group and keep records for posterity. These days, we interact with people who don’t share the same language as ours. The need for translating such documents has become even more critical in a globally interconnected world. Some large organizations hire a team of professional translators to help with document translation, which involves a lot of time and overhead cost. Multiple tools are available online that enable you to copy and paste text to get the translated equivalent in the language of your choice, but there are few secure and easy methods that allow for native support of translating such documents while keeping formatting intact.

Amazon Translate now supports translation of Office Open XML documents in DOCX, PPTX, and XLSX format. Amazon Translate is a fully managed neural machine translation service that delivers high-quality and affordable language translation in 55 languages. For the full list of languages, see Supported Languages and Language Codes. The document translation feature is available wherever batch translation is available. For more information, see Asynchronous Batch Processing.

In this post, we walk you through a step-by-step process to translate documents on the AWS Management Console. You can also access the Amazon Translate BatchTranslation API for document translation via the AWS Command Line Interface (AWS CLI) or the AWS SDK.

Solution overview

This post walks you through the following steps:

  1. Create an AWS Identity and Access Management (IAM) role that can access your Amazon Simple Storage Service (Amazon S3) buckets.
  2. Sort your documents by file type and language.
  3. Perform the batch translation.

Creating an IAM role to access your S3 buckets

In this post, we create a role that has access to all the S3 buckets in your account to translate documents, spreadsheets, and presentations. You provide this role to Amazon Translate to let the service access your input and output S3 locations. For more information, see AWS Identity and Access Management Documentation.

  1. Sign in to your personal AWS account.
  2. On the IAM console, under Access management, choose Roles.
  3. Choose Create role.
  4. Choose Another AWS account.
  5. For Account ID, enter your ID.
  6. Go to the next page.
  7. For Filter policies, search and add the AmazonS3FullAccess policy.
  8. Go to the next page.
  9. Enter a name for the role, for example, TranslateBatchAPI.
  10. Go to the role you just created.
  11. On the Trust relationships tab, choose Edit trust relationship.
  12. Enter the following service principals:
    "Service": [
    "translate.aws.internal",
    "translate.amazonaws.com"
    ],

    For example, see the following screenshot.

Sorting your documents

Amazon Translate batch translation works on documents stored in a folder inside an S3 bucket. Batch translation doesn’t work if the file is saved in the root of the S3 bucket. Batch translation also doesn’t support translation of nested files. So you first need to upload the documents you wish to translate in a folder inside an S3 bucket. Sort the documents such that the folders contain files of the same type (DOCX, PPTX, XLSX) and are in the same language. If you have multiple documents of different file types that you need to translate, sort the files such that each Amazon S3 prefix has only one type of document format written in one language.

  1. On the Amazon S3 console, choose Create bucket.
  2. Walk through the steps to create your buckets.

For this post, we create two buckets: input-translate-bucket and output-translate-bucket.

The buckets contain the following folders for each file type:

  • docx
  • pptx
  • xlsx

Performing batch translation

To implement your batch translation, complete the following steps:

  1. On to the Amazon Translate console, choose Batch Translation.
  2. Choose Create job.

For this post, we walk you through translating documents in DOCX format.

  1. For Name, enter BatchTranslation.
  2. For Source language, choose En.
  3. For Target language, choose Es.
  4. For Input S3 location, enter s3://input-translate-bucket/docx/.
  5. For File format, choose docx.
  6. For Output S3 location, enter s3://output-translate-bucket/.
  7. For Access permissions, select Use an existing IAM role.
  8. For IAM role, enter TranslateBatchAPI.

Because this is an asynchronous translation, the translation begins after the machine resource for the translation is allocated. This can take up to 15 minutes. For more information about performing batch translation jobs, see Starting a Batch Translation Job.

The following screenshot shows the details of your BatchTranslation job.

When the translation is complete, you can find the output in a folder in your S3 bucket. See the following screenshot.

Conclusion

In this post, we discussed implementing asynchronous batch translation to translate documents in DOCX format. You can repeat the same procedure for spreadsheets and presentations. The translation is simple and you pay only for the number of characters (including spaces) you translate in each format. You can start translating office documents today in all Regions where batch translation is supported. If you’re new to Amazon Translate, try out the Free Tier, which offers 2 million characters per month for the first 12 months, starting from your first translation request.


About the Author

Watson G. Srivathsan is the Sr. Product Manager for Amazon Translate, AWS’s natural language processing service. On weekends you will find him exploring the outdoors in the Pacific Northwest.

Read More

Simplifying application onboarding with Amazon CodeGuru Profiler

Simplifying application onboarding with Amazon CodeGuru Profiler

Amazon CodeGuru Profiler provides recommendations to help you continuously fine-tune your application’s performance. It does this by collecting runtime performance data from your live applications. It looks for your most expensive lines of code continuously and provides intelligent recommendations. This helps you more easily understand your applications’ runtime behavior so you can optimize their performance, improve latency, and reduce infrastructure cost.

From a central dashboard, you get different visualizations of your profiling data and details to what CodeGuru Profiler recommends of your application’s performance profile. You also get a report with details on the impact of the issue on your application, what’s causing the issue, and a recommendation on what to change to resolve the issue.

In this post, I introduce you to two enhancements that make it even easier and faster for your applications to start using the machine learning (ML)-based capabilities of CodeGuru Profiler:

  • Resource-based authorizations – You can use resource-based permissions to authorize an identity in your code to upload the profiled data instead of manually configuring AWS Identity and Access Manager (IAM). If the role or user already exists in IAM, you simply select the role or user on the CodeGuru Profiler console.
  • Ability to start the profiler agent on the command line – You can now start the profiler agent on the command line using the -javaagent This means that you no longer need to recompile your application or set build dependencies; you simply download the latest JAR file and add the -javaagent switch to the command line and run. The whole process takes just minutes to complete.

What you can do with CodeGuru Profiler

CodeGuru Profiler is designed to run in your production environment with minimal CPU overhead to help improve your application’s performance and reduce infrastructure costs. With CodeGuru Profiler, you can:

  • Troubleshoot latency and CPU utilization issues in your application
  • Identify application performance issues
  • Identify cost optimization opportunities and where you can reduce the infrastructure costs of running your application

CodeGuru Profiler works with your applications that are hosted on Amazon Elastic Compute Cloud (Amazon EC2), serverless applications running on AWS Fargate and AWS Lambda, and containerized applications running on Amazon Elastic Container Service (Amazon ECS) and Amazon Elastic Kubernetes Service (Amazon EKS). CodeGuru Profiler also works on-premises. It currently supports applications written in all Java virtual machine (JVM) languages, such as Java, Kotlin, and Scala.

Profiling groups

In CodeGuru Profiler, a profiling group is a set of applications that are profiled together as a single unit. The profiling agent sends application data to a single profiling group, where data from all applications in the profiling group are aggregated and analyzed. Your groups are managed on the Profiling groups page on the CodeGuru console. You can see a list of all your profiling groups, their status, or create or delete profiling groups. For more information, see Setting up Amazon CodeGuru Profiler.

A profiling group can profile a single application running on multiple hosts. It can also profile different but related applications. You can profile an application, regardless of what repository the code is in.

Resource-based authorization enabled

The improvements made to application onboarding remove many manual steps, enable resource-based authorization, and remove the need to configure IAM permissions each time you add a new application for profiling. This new setup option helps reduce configuration errors and enables quick onboarding of your application into the profiling group.

In this section, we show you how to grant the required permissions on the CodeGuru Profiler console. The first time you onboard your application, your admin needs to create the IAM role or user that can submit profiling data and configure the agent. That way, the roles and users can appear in the permissions drop-down on the CodeGuru Profiler console. After this setup is complete, you don’t need to go on the IAM console to set permissions.

Creating a profiling group with resource-based authorization

To create a profiling group with resource-based authorization enabled, complete the following steps:

  1. On the CodeGuru Profiler console, choose Create profiling group.

 

  1. For Name, enter test-group.
  2. Choose Create.

We can now show you the resource-based authorization enhancement in action. Prior to the recent enhancement, you needed to set up access credentials to give an application permission to submit data to your account. You can now do this in a few clicks on the Profiling groups page.

  1. In the Set permissions section, choose Give access to users and roles.
  2. For Application permissions, choose the IAM users and roles that you want to give permissions to so they can submit data to CodeGuru Profiler.
  3. Choose Save.

That’s it! You have set permissions for users and roles in your profiling group.

Starting the profiler agent without modifying your application’s code

Next, we walk you through a use case to demonstrate how easy and quick it is to start the profiler agent without modifying the application code.

Prerequisites

To complete this part of the post, you need the following:

  • AWS account
  • Laptop or desktop computer with internet access
  • Terminal emulator of your choice
  • Corretto 8 or JDK version 8 or later installed on your EC2 instance
  • Maven installed on your EC2 instance

You no longer need to modify your application code or add dependencies to run the agent. The benefit to starting the agent with the -javaagent parameter is that you can profile existing applications without recompiling or changing your application’s code. You can still use the previous approach to start the profiling agent. For example, this approach is applicable as the only option to use for Lambda and Spark jobs.

To onboard your application, you complete the following steps:

  1. Create your profiling group and set an environment variable for your Region.
  2. Download the latest Java JAR file from the CodeGuru Profiler console.
  3. Restart your JVM by running the -javaagent parameter

This post has some additional steps because we also set up the demo application to start profiling and onboard two different profiler groups: DemoApplication-WithIssues and DemoApplicationWithoutIssues.

Cloning the demo application

To clone the demo application to your EC2 instance, enter the following code:

git clone  https://github.com/aws-samples/aws-codeguru-profiler-demo-application
cd aws-codeguru-profiler-demo-application 

After you change directories to aws-codeguru-profiler-demo-application, you should see the pom.xml file.

Creating a profiling group

At the command line, enter the following commands, starting with the steps that you always need to perform to onboard the profiler (with aws configure, set up your AWS credentials and default Region, which are the same credentials you created the resource policies for):

aws codeguruprofiler create-profiling-group --profiling-group-name DemoApplication-WithIssues
aws codeguruprofiler create-profiling-group --profiling-group-name DemoApplication-WithoutIssues

Setting up the demo application

To complete the project, you create an Amazon Simple Storage Service (Amazon S3) bucket and an Amazon Simple Queue Service (Amazon SQS) queue. Creating an S3 bucket and SQS queue aren’t part of the onboarding process; you use these services for business logic. The SQS queue is polled for names of images to process and the images get loaded into an S3 bucket, does some image transformation, and then the results back to the S3 bucket.

To create these resources, enter the following code:

aws s3 mb s3://demo-application-test-bucket-YOUR BUCKET NAME
aws sqs create-queue --queue-name DemoApplicationQueue

For this post, replace YOUR BUCKET NAME with a random set of numbers. This helps make sure that your bucket contains a globally unique name.

Next, we set the environment variables by using the following commands. These steps are necessary for this particular application and not related to onboarding the profiler. Also, don’t forget to use your test-bucket name. Replace the referenced Amazon SQS URL with your Amazon SQS URL, which you can find on the Details tab for the queue on the Amazon SQS console.

export DEMO_APP_BUCKET_NAME="demo-application-test-bucket-10245"
export DEMO_APP_SQS_URL="https://sqs.us-east-1.amazonaws.com/12345678/DemoApplicationQueue"
export AWS_CODEGURU_TARGET_REGION=REPLACE WITH YOUR-AWS-REGION

Downloading the Java agent JAR file

First, set the environment variable:

export AWS_CODEGURU_PROFILER_GROUP_NAME=DemoApplication-WithIssues

To create a JAR file in the target directory, enter the following code:

mvn clean install

The process can take up to 1 minute to complete.

For this post, you don’t need to download the CodeGuru Profiler Java agent JAR file. It’s included with the demo application that you downloaded from GitHub. However, when you onboard your application for profiling, you need to download the JAR file from the CodeGuru Profiler console. See the following screenshot.

Running the demo application

Next, you restart your JVM by running the -javaagent switch. Make sure that the application file name is included in the command. When you run this project, the JAR file version may have been updated from the version you see in the following code:

java -javaagent:codeguru-profiler-java-agent-standalone-1.0.0.jar 
-jar target/DemoApplication-1.0-jar-with-dependencies.jar with-issues

The profiling group setup is successful when you see the following message:

INFO: Profiling scheduled, sampling rate is PT1S

You will see messages running on the screen for several minutes. You have the option to enter exit, enter on your keyboard. If you do that, reconnect back into your EC2 instance through SSH to verify that the application is still running in the background, using the following command:

ps -ef |grep javaagent

Checking the CodeGuru console for the profiling group

To verify that your application is profiling, return to the CodeGuru console. You should see the profiling group go from Inactive to Pending status, and then end in Profiling status.

When your profiling group shows the status Profiling, you can visualize the application’s runtime data. It takes about 5 minutes for the CodeGuru Profiling agent to submit code review data. After about 10 more minutes, you can see the flame graph visualizations. Within 1 hour, you have your first recommendation report.

If you want to run this project again, without issues, repeat the previous steps with the following changes:

  • Set the export variables per your setup. As we discussed earlier when running the demo app WithIssues, these environment variables are specific to running the demo app and not part of the profiler onboarding.
    export DEMO_APP_SQS_URL=https://sqs.YOUR-AWS-REGION.queue.amazonaws.com/YOUR-ACCOUNT-ID/DemoApplicationQueue
    export DEMO_APP_BUCKET_NAME=demo-application-test-bucket-1092734-YOUR-BUCKET-NAME
    export AWS_CODEGURU_TARGET_REGION=YOUR-AWS-REGION
    

  • Run the demo application with the following code:
    export AWS_CODEGURU_PROFILER_GROUP_NAME=DemoApplication-WithoutIssues
    mvn clean install ## This command will generate the DemoApplication-1.0-jar-with-dependencies.jar
    java -javaagent:codeguru-profiler-java-agent-standalone-1.0.0.jar 
      -jar target/DemoApplication-1.0-jar-with-dependencies.jar without-issues

For more information about visualizations in CodeGuru Profiler, see the section Understanding CodeGuru Profiler’s visualizations in Optimizing application performance with Amazon CodeGuru Profiler.

Cleaning up

To avoid incurring any future charges, delete the resources you created with this project:

  • Profiling group DemoApplication-WithIssues
  • SQS queue and the S3 bucket
  • EC2 instance

Conclusion

We’re excited to help you make it even quicker and easier to onboard your applications using CodeGuru Profiler. In this post, we reviewed and learned how to use two recent enhancements to CodeGuru Profiler: resource-based permission setting and starting the profiler agent using the -javaagent switch, without needing to modify your application’s code.

CodeGuru Profiler is part of Amazon CodeGuru (Preview), an ML-based service that performs automated code reviews and application performance recommendations. It finds issues in your code and deviations from performance engineering best practices using AWS APIs, SDKs, and a variety of libraries to make specific recommendations to remediate identified issues. CodeGuru is powered by ML capabilities, best practices, and lessons learned from millions of code reviews and thousands of applications profiled on both open-source projects and internally at Amazon.

We hope that you find that using these feature enhancements is straightforward, quick, and easy. Let us know your progress on our GitHub project page!


About the Author

Charles Gibson is an Enterprise Transformation Architect in Professional Services at Amazon Web Services. He helps customers migrate and modernize their businesses on the AWS Cloud. Charles enjoys cooking Northern Italian food for friends and family.

 

 

 

 

Read More

How SNCF Réseau and Olexya migrated a Caffe2 vision pipeline to Managed Spot Training in Amazon SageMaker

How SNCF Réseau and Olexya migrated a Caffe2 vision pipeline to Managed Spot Training in Amazon SageMaker

This blog post is co-written by guest authors from SNCF and Olexya.

Transportation and logistics are fertile ground for machine learning (ML). In this post, we show how the French state-owned railway company Société Nationale des Chemins de fer Français (SNCF) uses ML from AWS with the help of its technology partner Olexya to research, develop, and deploy innovative computer vision solutions.

SNCF was founded in 1938 and employs more than 270,000 people. SNCF Réseau is a subsidiary of SNCF that manages and operates the infrastructure for the rail network. SNCF Réseau and its technology partner Olexya deploy innovative solutions to assist the operations of the infrastructure and keep the bar high for infrastructure safety and quality. The field teams detect anomalies in the infrastructure by using computer vision.

SNCF Réseau researchers have been doing ML for a long time. An SNCF Réseau team developed a computer vision detection model on premises using the Caffe2 deep learning framework. The scientists then reached out to SNCF Réseau technology partner Olexya to assist with the provisioning of GPU to support iteration on the model. To keep operational overhead low and productivity high while retaining full flexibility on the scientific code, Olexya decided to use Amazon SageMaker to orchestrate the training and inference of the Caffe2 model. The process involved the following steps:

  1. Custom Docker creation.
  2. Training configuration ingestion via an Amazon Simple Storage Service (Amazon S3) data channel.
  3. Cost-efficient training via Amazon SageMaker Spot GPU training.
  4. Cost-efficient inference with the Amazon SageMaker training API.

Custom docker creation

The team created a Docker image wrapping the original Caffe2 code that respected the Amazon SageMaker Docker specification. Amazon SageMaker can accommodate multiple data sources, and has advanced integration with Amazon S3. Datasets stored in Amazon S3 can be automatically ingested in training containers running on Amazon SageMaker. To be able to process training data available in Amazon S3, Olexya had to direct the training code to read from the associated local path opt/ml/input/data/<channel name>. Similarly, the model artifact writing location had to be set to opt/ml/model. That way, Amazon SageMaker can automatically compress and ship the trained model artifact to Amazon S3 when training is complete.

Training configuration ingestion via an Amazon S3 data channel

The original Caffe2 training code was parametrized with an exhaustive and flexible YAML configuration file, so that researchers could change model settings without altering the scientific code. This external file was easy to keep external and ingest at training time in the container via the use of data channels. Data channels are Amazon S3 ARNs passed to the Amazon SageMaker SDK at training time and ingested in the Amazon SageMaker container when training starts. Olexya configured the data channels to read via a copy (copy mode), which is the default configuration. It is also possible to stream the data via Unix pipes (Pipe mode).

Cost-efficient training via Amazon SageMaker Spot GPU training

The team configured training infrastructure to be an ml.p3.2xlarge GPU-accelerated compute instance. The Amazon SageMaker ml.p3.2xlarge compute instance is specifically adapted to deep learning computer vision workloads: it’s equipped with an NVIDIA V100 GPU featuring 5,120 cores and 16 GB of High-Bandwidth Memory (HBM), which enables the fast training of large models.

Furthermore, Amazon SageMaker training API calls were set with Managed Spot Instance usage activated, which contributed to a reported savings of 71% compared to the on-demand Amazon SageMaker price. Amazon SageMaker Managed Spot Training is an Amazon SageMaker feature that enables the use of Amazon Elastic Compute Cloud (Amazon EC2) Spot Instance capacity for training. Amazon EC2 Spot Instances allow you to purchase unused Amazon EC2 computer capacity at a highly-reduced rate. In Amazon SageMaker, Spot Instance usage is fully managed by the service, and you can invoke it by setting two training SDK parameters:

  • train_use_spot_instances=True to request usage of Amazon SageMaker Spot Instances
  • train_max_wait set to the maximal acceptable waiting time in seconds

Cost-efficient inference with the Amazon SageMaker training API

In this research initiative, inference interruptions and delayed instantiation were acceptable to the end-user. Consequently, to further optimize costs, the team also used the Amazon SageMaker training API to run inference code, so that managed Amazon SageMaker Spot Instances could be used for inference too. Using the training API came with the additional benefit of a simpler learning curve because the same API is used for both steps of the model life cycle.

Time and cost savings

By applying those four steps, Olexya successfully ported an on-premises Caffe2 deep computer vision detection model to Amazon SageMaker for both training and inference. More impressively, the team completed that onboarding in about 3 weeks, and reported that the training time of the model was reduced from 3 days to 10 hours! The team further estimated that Amazon SageMaker allows a 71% total cost of ownership (TCO) reduction compared the locally available on-premises GPU fleet. A number of extra optimization techniques could reduce the costs even more, such as intelligent hyperparameter search with Amazon SageMaker automatic model tuning and mixed-precision training with the deep learning frameworks that support it.

In addition to SNCF Réseau, numerous AWS customers operating in transportation and logistics have improved their operations and delivered innovation by applying ML to their business. For example:

  • The Dubai-based logistics company Aramex uses ML for address parsing and transit time prediction. The company reports having 150 models in use, doing 450,000 predictions per day.
  • Transport for New South Wales uses the cloud to predict patronage numbers across the entire transport network, which enables the agency to better plan workforce and asset utilization and improve customer satisfaction.
  • Korean Air launched innovative projects with Amazon SageMaker to help predict and preempt maintenance for its aircraft fleet.

Conclusion

Amazon SageMaker supports the whole ML development cycle, from annotation to production deployment and monitoring. As illustrated by the work of Olexya and SNCF Réseau, Amazon SageMaker is framework-agnostic and accommodates a variety of deep learning workloads and frameworks. Although Docker images and SDK objects have been created to closely support Sklearn, TensorFlow, PyTorch, MXNet, XGBoost, and Chainer, you can bring custom Docker containers to onboard virtually any framework, such as PaddlePaddle, Catboost, R, or Caffe2. If you are an ML practitioner, don’t hesitate to test the service, and let us know what you build!


About the Authors

Olivier Cruchant is a Machine Learning Specialist Solution Architect at AWS, based in Lyon, France. Olivier helps French customers – from small startups to large enterprises – develop and deploy production-grade machine learning applications. In his spare time, he enjoys reading research papers and exploring the wilderness with friends and family.

 

 

 

Samuel Descroix is head manager of the Geographic and Analytic Data department at SNCF Réseau. He is in charge of all project teams and infrastructures. To be able to answer to all new use cases, he is constantly looking for most innovative and most relevant solutions to manage growing volumes and needs of complex analysis.

 

 

Alain Rivero is Project Manager in the Technology and Digital Transformation (TTD) department within the General Industrial and Engineering Department of SNCF Réseau. He manages projects implementing in-depth learning solutions to detect defects on rolling stock and tracks to increase traffic safety and guide decision-making within maintenance teams. His research focuses on image processing methods, supervised and unsupervised learning and their applications.

 

 

Pierre-Yves Bonnefoy is data architect at Olexya, currently working for SNCF Réseau IT department. One of his main assignments is to provide environments and sets of datas for Data Scientists and Data Analysts to work on complex analysis, and to help them with software solutions. Thanks to his large range of skills in development and system architecture, he accelerated the deployment of the project on Sagemaker instances, rationalization of costs and optimization of performance.

 

 

Emeric Chaize is certified Solution Architect in Olexya, currently working for SNCF Réseau IT department. He is in charge of Data Migration Project for IT Data Departement, with the responsabilty of covering all needs and usages of the company in data analysis. He defines and plans deployment of all the needed infrastructure for projects and Data Scientists.

Read More

Building a multilingual question and answer bot with Amazon Lex

Building a multilingual question and answer bot with Amazon Lex

You can use Amazon Lex to build a question and answer chatbot. However, if you live in a non-English-speaking country or your business has global reach, you will want a multilingual bot to cater to all your users. This post describes how to achieve that by using the multi-language functionality of your question and answer bot (QnABot).

The QnABot can detect the predominant language in an interaction by using Amazon Comprehend, a natural language processing (NLP) service that uses machine learning to find insights and relationships in text.

The bot then uses Amazon Translate, a neural machine translation service to translate the question to English. Then it can return a preconfigured answer in the end user’s language or translate it back from the default English text.

Although QnABot allows the end-user to interact with an Amazon Lex bot using text or voice, the multilingual feature primarily supports text interactions. Multilingual voice interactions are currently limited to use with Alexa skills.

The solution consists of three easy steps:

  1. Configure the multi-language functionality.
  2. Set up the alternate curated answers in a different language.
  3. Configure your Alexa skill with multiple languages.

For instructions on creating and customizing your bot, see Create a Question and Answer Bot with Amazon Lex and Amazon Alexa or the Q&A Self-Paced Guide. You can also see the following videos on YouTube:

Prerequisites

To implement this solution, you must have an AWS account. If you don’t have a professional account, the AWS Free Tier lets you gain experience with the AWS platform, products, and services.

You also need to deploy QnABot. If you have not already done so, launch the QnABot on an AWS CloudFormation stack in one of the available Regions.

When specifying your stack name, include QnA in the name, for example, MultiLangQnABot.

After you successfully deploy your CloudFormation stack, complete the following steps:

  1. Open the content designer page of your chatbot.

If this is the first time accessing the chatbot design console, you can find the correct URL on the AWS CloudFormation console on the Output tab of your bot stack. Look for the value for the ContentDesignerURL key. You should also receive an email with temporary credentials to access the QnABot designer.

  1. On the designer page, choose the menu icon.
  2. Under Tools, choose Import.
  3. Expand the Examples/Extensions
  4. Next to blog-samples, choose Load.

Configuring the multi-language functionality

You are now ready to configure the multi-language functionality. Complete the following steps:

  1. In the content designer page, under Tools, choose Settings.
  2. For the ENABLE_MULTI_LANGUAGE_SUPPORT parameter, change the value from false to true.
  3. Choose Save.
  4. To test the bot, open the client webpage.
  5. From the designer page, under Tools, choose QnABot Client.
  6. Enter the following questions (in English, Spanish, and French):
    • How do I modify Q and A Bot content ?
    • ¿Cómo modifico el contenido de Q y A Bot ?
    • Comment modifier le contenu Q et A Bot ?

The chatbot answers each time in the language you used, as shown in the following animation.

The QnABot is successfully using Amazon Translate to translate the answer automatically into the user’s native language.

Setting up alternate curated answers in a different language

You might need to provide a more natural experience and want to add a curated answer in the native language of your choice. To further customize the translation for each question, you can use the {{handlebar}} functionality. The QnABot provides the {{handlebar}} function ifLang, which takes the locale as a quoted parameter. You can use any of the languages that Amazon Translate supports. For more information, see What Is Amazon Translate?

For example, to customize the translation in Spanish, the ifLang function uses es as the locale parameter. See the following code:

{{#ifLang 'es'}}
          Su traducción al español
{{/ifLang}}

Additionally, if an unknown language is detected, you can support that with a default response by using the defaultLang function. See the following code:

{{#defaultLang}}
          Your default language answer
{{/defaultLang}}

As an example, modify the question you used earlier. Go back to the content designer and complete the following steps:

  1. Under Tools, choose Edit.
  2. Select 001 and choose the pencil icon on the right.
  3. Replace the text in the answer with the following code:
    {{#ifLang 'es'}}
    Use las herramientas de 'Question and Test' de 'Content Designer' para encontrar sus documentos existentes y editarlos directamente en la consola. También puede exportar documentos existentes como un archivo JSON, realizar cambios en el archivo y volver a importar.
    {{/ifLang}}
    {{#ifLang 'fr'}}
    Utilisez les outils 'Question and Test' de 'Content Designer' pour trouver vos documents existants et les modifier directement dans la console. Vous pouvez également exporter des documents existants sous forme de fichier JSON, apporter des modifications au fichier et réimporter.
    {{/ifLang}}
    {{#defaultLang}} 
    Use the Content Designer Question and Test tools to find your existing documents and edit them directly in the console. You can also export existing documents as a JSON file, make changes to the file, and re-import.
    {{/defaultLang}}
    

    Multi-language and handlebars, in general, also support markdown answers. For example, you could modify the preceding code to highlight the name of the interface that isn’t translated. See the following code:

    {{#ifLang 'es'}}
    Use las herramientas de ***'Question and Test'*** de ***'Content Designer'*** para encontrar sus documentos existentes y editarlos directamente en la consola. También puede exportar documentos existentes como un archivo JSON, realizar cambios en el archivo y volver a importar.
    {{/ifLang}}
    {{#ifLang 'fr'}}
    Utilisez les outils ***'Question and Test'*** de ***'Content Designer'*** pour trouver vos documents existants et les modifier directement dans la console. Vous pouvez également exporter des documents existants sous forme de fichier JSON, apporter des modifications au fichier et réimporter.
    {{/ifLang}}
    {{#defaultLang}} 
    Use the ***Content Designer Question and Test*** tools to find your existing documents and edit them directly in the console. You can also export existing documents as a JSON file, make changes to the file, and re-import.
    {{/defaultLang}}
    

  4. Choose Advanced and enter the new code in the Markdown Answer box.

  5. Choose Update.

If you try to ask your questions again, the answers are different because the chatbot is using your curated version.

You can also import the sample or extension named Language / Multiple Language Support.

This adds two questions to the system: Language.000 and Language.001. The first question allows the end-user to set their preferred language explicitly; the latter resets the preferred language and allow the QnABot to choose the locale based on the automatically detected predominant language.

Debugging the answers in a different language

You can use the ENABLE_DEBUG_RESPONSES setting to see how local language questions are translated to English by QnABot, and to tune the content as needed to ensure QnABot finds the best answer to a non-English question.

Complete the following steps to set up and test:

  1. In the content designer page, under Tools, choose Settings.
  2. For the ENABLE_DEBUG_RESPONSES parameter, change the value from false to true.
  3. Choose Save.
  4. To test the bot, open the client webpage.
  5. From the designer page, under Tools, choose QnABot Client.
  6. Try one of the question we used before, you can read the translation and use this information to tune your answer.

Configuring your Alexa skill with multiple languages

You first need to create your Alexa skill. For instructions, see Create a Question and Answer Bot with Amazon Lex and Amazon Alexa.

When your Alexa skill is working, add the additional languages by completing the following steps:

  1. On the Alexa developer console, open your skill.
  2. From the drop-down menu with your default language, choose Language settings.
  3. Add all the languages you want to support and choose Save.
  4. Under CUSTOM, choose JSON Editor.
  5. Copy the JSON from the editor, switch to the other language you want to support, and enter it in the editor pane (this overwrites the default).
  6. Choose Save Model.
  7. Choose Invocation and change the invocation name.
  8. Choose Save Model.
  9. Repeat these steps for any language you want to support.
  10. Build the model.

Testing your Alexa skill

You can now test your Alexa skill in other languages.

  1. On the Alexa developer console, select your skill.
  2. Choose Test.
  3. Change the language and type the invocation name of your skill for that language.
  4. After Alexa gives her initial greeting, ask the question you used before or any other question you added in the content designer.

Alexa answers you in the selected language.

Now your multilingual chatbot can be published on your website or as an Alexa skill. To integrate the QnABot in your website, you can use lex-web-ui. For instructions, see Deploy a Web UI for your Chatbot.

Conclusion

This post shows you how to configure and use the out-of-the-box feature of your QnABot to localize answers in any language Amazon Translate supports. It is an inexpensive way to deploy a multi-language chatbot, and you don’t need to adapt it to accommodate new Amazon Lex features.

As of this writing, this approach works for text-based interactions only; support for voice is limited to use with Alexa skills only.

For more information about experimenting with the capabilities of QnABot, see the Q&A Self-Paced Guide.


About the Authors

Fabrizio is a Specialist Solutions Architect for Database and Analytics in the AWS Canada Public Sector team. He has worked in the analytics field for the last 20 years, and has recently, and quite by surprise, become a Hockey Dad after moving to Canada.

 

 

 

As a Solutions Architect at AWS supporting our Public Sector customers, Raj excites customers by showing them the art of the possible of what they can build on AWS and helps accelerate their innovation. Raj loves solving puzzles, mentoring, and supporting hackathons and seeing amazing ideas come to life.

Read More

Enhancing your chatbot experience with web browsing

Enhancing your chatbot experience with web browsing

Chatbots are popping up everywhere. They are qualifying leads, assisting with sales, and automating customer service. However, conversational chatbot experiences have been limited to the space available within the chatbot window.

What if these web-based chatbots could provide an interactive experience that expanded beyond the chat window to include relevant web content based on user inputs? In a previous post we showed you how to deploy a web UI for your chatbot. In this post we will show you how to enhance that experience.

Here is an example of how we add an interactive web UI to the Order Flowers chatbot with the lex-web-ui customization.

Installing the chatbot UI

To install your chatbot, complete the following steps:

  1. Deploy the chatbot UI in your AWS account by launching the following AWS CloudFormation stack:
  2. Set EnableCognitoLogin to true in the parameters.
  3. To check if it’s working, on the AWS CloudFormation console, choose Stacks.
  4. Choose the stack you created.
  5. In the Outputs section, choose ParentPageURL.

You have now deployed the bot in the CloudFront distribution you created.

Installing the chatbot UI enhancer

After you install the chatbot UI, launch the following AWS CloudFormation stack:

There are two parameters for this stack:

  • BotName – The chatbot UI bot you deployed. The parameter value of WebUiOrderFlowers is populated by default.
  • lexwebuiStackName – The name of the deployed in the previous step. The parameter value of lex-web-ui is populated by default.

When the stack is complete, find the URL for the new demo site on the Outputs tab on the AWS CloudFormation console.

Enhancing the existing bot with AWS Lambda

To enhance the existing bot with AWS Lambda, complete the following steps:

  1. On the Amazon Lex console, choose Bots.
  2. Choose the bot you created.
  3. In the Lambda initialization and validation section, for Lambda function, choose the function you created as part of the CloudFormation stack (enhanced-orderflowers-<stackname>).

For production workloads, you should publish a new version of the bot. Amazon Lex takes a snapshot copy of the $LATEST version to publish a new version. For more information, see Exercise 3: Publish a Version and Create an Alias.

Enhancing authentication

You have now set up the enhanced chatbot UI. It’s recommended that you authenticate for a production environment. This post uses Amazon Cognito to add a social identity provider (Google) to your user pool. For instructions, see Adding Social Identity Providers to a User Pool.

This step allows your bot to display your Google calendar while you order your flowers. If you skip this step, the bot still functions normally.

Dynamically viewing content on your webpage

Having content appear and disappear on your website based on your interactions with the bot is a powerful feature.

For example, if you ask the bot to order flowers, the bot messaging interface and the webpage change. This example actively builds HTML on the fly with values that the bot sends back to the end-user.

Enhancing pages with external content to help with flower selection

When you ask the bot to buy roses, the result depends on if you’re in unauthenticated or authenticated mode.

In authenticated mode, the iframe changes from the default homepage to a Wikipedia page about roses. The Area chart also changes to a Roses Sold graph that shows the number of roses sold per year.

In authenticated (with Google) mode, the iframe changes to your Google calendar to help you schedule a delivery day. The Area chart still changes to the Roses Sold graph.

This powerful tool allows content from various parts of the website or the internet to appear by interacting with the bot. This also allows the bot to recognize if you’re authenticated or not and tailor your browsing experience.

Parent page, iframes, session attributes, and dynamic HTML tags

Four main components make up the Order Flowers bot page and how the various pieces interact with each other:

  • Parent page – This page houses all the various components, including the chatbot iframe, dynamically created HTML, and the navigation portal (an iframe that displays various HTML pages external and internal to the website).
  • Chatbot iframe – This is the chatbot UI that the end-user interacts with. The chatbot is loaded using a JavaScript snippet that mounts an iframe to the bottom right of the parent page and preloads it with an API to interact with the parent page.
  • Session attributes – These are arbitrary values that get sent back and forth from the chatbot UI backend to the parent page. You can manipulate these values in Lambda. On the parent page, the session attributes event data is made available in a variable called sessionAttributes.
  • Dynamic HTML <Div> tags – This appears on the top right of the page and displays various charts based on the question asked. You can populate it with any data, not just charts. You manipulate the data by returning the values through the session attributes fields. In the parent page, sessionAttributes.appContext houses this data.

The following diagram illustrates the solution architecture.

Chatbot UI user login with Amazon Cognito

When you’re authenticated through the integrated Amazon Cognito feature, the chatbot UI attaches a signed token as a session attribute. The enhanced Order Flowers webpage uses the token to make additional user attributes available, including fields such as given name, family name, and email address. These fields help return personalized information (for example, addressing you by your name).

Limitations

There are certain limitations to displaying outside webpages and content through the chatbot UI parent page.

If cross-origin resource sharing (CORS) is enabled on the external content that is being pulled into the parent page iframe navigation portal, the browser blocks the content. Browsers don’t block different webpages from the same domain or external webpages that don’t have CORS enabled (for example, Wikipedia). For more information, see Cross-Origin Resource Sharing (CORS) on the MDN web docs website.

In most use cases, you should use the navigation portal to pull in content from your own domain, due to the inherent limitations of iframes and CORS.

Additional Resources

The concepts discussed in this blogpost can be used with the QnaBot. The following README goes in detailed instructions on setting up the solution.

Conclusion

This post demonstrates how to enhance the Order Flowers bot with a Lambda function that parses your JWT token and extracts the relevant information. If you are authenticated through Google, the bot extracts information like your name and email address, and displays your Google calendar to help you schedule your delivery date. The function also verifies that the JWT token signature is valid.

The chatbot UI in this post is based on the aws-lex-web-ui open-source project. For more information, see the GitHub repo.


About the Authors

Mohamed Khalil is a Consultant for AWS Professional Services. Bob Strahan is a Principal Consultant for AWS Professional Services. Bob Potterveld is a Senior Consultant for AWS Professional Services. They help our customers and partners on a variety of projects.

Read More

Processing PDF documents with a human loop using Amazon Textract and Amazon Augmented AI

Processing PDF documents with a human loop using Amazon Textract and Amazon Augmented AI

Businesses across many industries, including financial, medical, legal, and real estate, process a large number of documents for different business operations. Healthcare and life science organizations, for example, need to access data within medical records and forms to fulfill medical claims and streamline administrative processes. Amazon Textract is a machine learning (ML) service that makes it easy to process documents at a large scale by automatically extracting text and data from virtually any type of document. For example, it can extract patient information from an insurance claim or values from a table in a scanned medical chart.

Depending on the business use case, you may want to have a human review of ML predictions. For example, extracting information from a scanned mortgage application or medical claim form might require human review of certain fields due to regulatory requirements or potentially low-quality scans. Amazon Augmented AI (Amazon A2I) allows you to build and manage such human review workflows. This allows human review of ML predictions when needed based on a confidence score threshold, and you can audit the predictions on an ongoing basis. For more information, see Using with Amazon Textract with Amazon Augmented AI for processing critical documents.

In this post, we show how you can use Amazon Textract and Amazon A2I to build a workflow that enables multi-page PDF document processing with a human reviewers loop.

Solution overview

The following architecture shows how you can have a serverless architecture to process multi-page PDF documents with a human review. Although Amazon Textract can process images (PNG and JPG) and PDF documents, Amazon A2I human reviewers need to have individual pages as images and process them individually using the AnalyzeDocument API of Amazon Textract.

To implement this architecture, we take advantage of Amazon Step Functions to build the overall workflow. As the workflow starts, it extracts individual pages from the multi-page PDF document. It then uses the Map state to process multiple pages concurrently using the AnalyzeDocument API. When we call Amazon Textract, we also specify the Amazon A2I workflow as part of the request. This workflow is configured to trigger when form fields are detected below a certain confidence threshold. If triggered, Amazon Textract returns the extracted text and data along with the details. When the human review is complete, the callback task token is used to resume the state machine, combine the pages’ results, and store them in an output Amazon Simple Storage Service (Amazon S3) bucket.

For more information about the demo solution, see the GitHub repo.

Prerequisites

Before you get started, you must install the following prerequisites:

  1. Node.js
  2. Python
  3. AWS Command Line Interface (AWS CLI)—for instructions, see Installing the AWS CLI)

Deploying the solution

The following steps deploy the reference implementation in your AWS account. The solution deploys different components, including an S3 bucket, a Step Function, an Amazon Simple Queue Service (Amazon SQS) queue, and AWS Lambda functions using the AWS Cloud Development Kit (AWS CDK), which is an open-source software development framework to model and provision your cloud application resources using familiar programming languages.

  1. Install AWS CDK:
    npm install -g aws-cdk

  2. Download the GitHub repo to your local machine:
    git clone https://github.com/aws-samples/amazon-textract-a2i-pdf

  3. Go to the folder multipagepdfa2i and enter the following:
    pip install -r requirements.txt

  4. Bootstrap AWS CDK:
    cdk bootstrap

  5. Deploy:
    cdk deploy

Creating a private work team

A work team is a group of people that you select to review your documents. You can create a work team from a workforce, which is made up of Amazon Mechanical Turk workers, vendor-managed workers, or your own private workers that you invite to work on your tasks. Whichever workforce type you choose, Amazon A2I takes care of sending tasks to workers. For this post, you create a work team using a private workforce and add yourself to the team to preview the Amazon A2I workflow.

To create and manage your private workforce, you can use the Labeling workforces page on the Amazon SageMaker console. On the console, you can create a private workforce by entering worker emails or importing a pre-existing workforce from an Amazon Cognito user pool.

If you already have a work team for Amazon SageMaker Ground Truth, you can use the same work team with Amazon A2I and skip to the following section.

To create your private work team, complete the following steps:

  1. On the Amazon SageMaker console, choose Labeling workforces.
  2. On the Private tab, choose Create private team.
  3. Choose Invite new workers by email.
  4. In the Email addresses box, enter the email addresses for your work team (for this post, enter your email address).

You can enter a list of up to 50 email addresses, separated by commas.

  1. Enter an organization name and contact email.
  2. Choose Create private team.

After you create the private team, you get an email invitation. The following screenshot shows an example email.

After you click the link and change your password, you are registered as a verified worker for this team. The following screenshot shows the updated information on the Private tab.

Your one-person team is now ready, and you can create a human review workflow.

Creating a human review workflow

You use a human review workflow to do the following:

  • Define the business conditions under which the Amazon Textract predictions of the document content go to a human for review. For example, you can set confidence thresholds for important words in the form that the model must meet. If inference confidence for that word (or form key) falls below your confidence threshold, the form and prediction go for human review.
  • Create instructions to help workers complete your document review task.
  1. On the Amazon SageMaker console, navigate to the Human review workflows page
  2. Choose Create human review workflow.
  3. In the Workflow settings section, for Name, enter a unique workflow name.
  4. For S3 bucket, enter the S3 bucket that was created in CDK deployment step. It should have a name format as multipagepdfa2i-multipagepdf-xxxxxxxxx. This S3 bucket is where A2I will store the human review results.
  5. For IAM role, choose Create a new role from the drop-down menu. Amazon A2I can create a role automatically for you.
  6. For S3 buckets you specify, select Specific S3 buckets.
  7. Enter the S3 bucket you specified earlier in Step 3; for example, multipagepdfa2i-multipagepdf-xxxxxxxxx.
  8. Choose Create.

You see a confirmation when role creation is complete, and your role is now pre-populated in the IAM role drop-down menu.

  1. For Task type, select Amazon Textract – Key-value pair extraction.

Defining the trigger conditions

For this post, you want to trigger a human review if the key Mail Address is identified with a confidence score of less than 99% or not identified by Amazon Textract in the document. For all other keys, a human review starts if a key is identified with a confidence score less than 90%.

  1. Select Trigger a human review for specific form keys based on the form key confidence score or when specific form keys are missing.
  2. For Key name, enter Mail Address.
  3. Set the identification confidence threshold between 0 and 99.
  4. Set the qualification confidence threshold between 0 and 99.
  5. Select Trigger a human review for all form keys identified by Amazon Textract with confidence scores in a specific range.
  6. Set Identification confidence threshold between 0 and 90.
  7. Set Qualification confidence threshold between 0 and 90.

For model-monitoring purposes, you can also randomly send a specific percent of pages for human review. This is the third option on the Conditions for invoking human review page: Randomly send a sample of forms to humans for review. This post doesn’t include this condition.

Creating a UI template

In the next steps, you create a UI template that the worker sees for document review. Amazon A2I provides pre-built templates that workers use to identify key-value pairs in documents.

  1. In the Worker task template creation section, select Create from a default template.
  2. For Template name, enter a name.

When you use the default template, you can provide task-specific instructions to help the worker complete your task. For this post, you can enter instructions similar to the default instructions you see in the console.

  1. Under Task Description, enter something similar to Please review the Key Value Pairs in this document.
  2. Under Instructions, review the default instructions provided and make modifications as needed.
  3. In the Workers section, select Private.
  4. For Private teams, choose the work team you created earlier.
  5. Choose Create.

You’re redirected to the Human review workflows page and see a confirmation message similar to the following screenshot.

Record your new human review workflow ARN, which you use to configure your human loop in the next section.

Updating the solution with the Human Review workflow

You’re now ready to add your human review workflow ARN.

  1. Within the code you downloaded from GitHub repo, open the file multipagepdfa2i/multipagepdfa2i_stack.py.

On line 23, you should see the following code:

SAGEMAKER_WORKFLOW_AUGMENTED_AI_ARN_EV = ""
  1. Within the quotes, enter the human review workflow ARN you copied at the end of the last section.

Line 23 should now look like the following code:

SAGEMAKER_WORKFLOW_AUGMENTED_AI_ARN_EV = "arn:aws:sagemaker: ...."
  1. Save the changes you made.
  2. Deploy by entering the following code:
    cdk deploy

Testing the workflow

To test your workflow, complete the following steps:

  1. Create a folder named uploads in the S3 bucket that was created by CDK deployment (Example: multipagepdfa2i-multipagepdf-xxxxxxxxx)
  2. Upload the sample PDF document to the uploads For example, uploads/Sampledoc.pdf.
  3. On the Amazon SageMaker console, choose Labeling workforces.
  4. On the Private tab, choose the link under Labeling portal sign-in URL.
  5. Sign in with the account you configured with Amazon Cognito.

If the document required a human review, a job appears under Jobs section .

  1. Select the job you want to complete and choose Start working.

In the reviewer UI, you see instructions and the first document to work on. You can use the toolbox to zoom in and out, fit image, and reposition document. See the following screenshot.

This UI is specifically designed for document-processing tasks. On the right side of the preceding screenshot, the key-value pairs are automatically pre-filled with the Amazon Textract response. As a worker, you can quickly refer to this sidebar to make sure the key-values are identified correctly (which is the case for this post).

When you select any field on the right, a corresponding bounding box appears, which highlights its location on the document. See the following screenshot.

In the following screenshot, Amazon Textract didn’t identify Mail Address. The human review workflow identified this as an important field. Even though Amazon Textract didn’t identify it, the worker task UI asks you to enter d details on the right side.

There may be a series of pages you need to submit based on the Amazon Textract confidence score ranges you configured. When you finish reviewing them, continue with steps below.

  1. When you complete the human review, go to the S3 bucket you used earlier (Example: multipagepdfa2i-multipagepdf-xxxxxxxxx)
  2. In the complete folder, choose the folder that has the name of input document (Example: uploads-Sampledoc.pdf-b5d54fdb75b143ee99f7524de56626a3).

That folder contains output.csv, which contains all your key-value pairs.

The following screenshot shows the content of an example output.csv file.

Conclusion

In this post, we showed you how to use Amazon Textract and Amazon A2I to automatically extract data from scanned multi-page PDF documents, and the human review of the pages for given business criteria. For more information about Amazon Textract and Amazon A2I, see Using Amazon Augmented AI with Amazon Textract.

For video presentations, sample Jupyter notebooks, or more information about use cases like document processing, content moderation, sentiment analysis, text translation, and more, see Amazon Augmented AI Resources.


About the Authors

Nicholas Nelson is an AWS Solutions Architect for Strategic Accounts based out of Seattle, Washington. His interests and experience include Computer Vision, Serverless Technology, and Construction Technology. Outside of work, you can find Nicholas out cycling, paddle boarding, or grilling!

 

 

 

Kashif Imran is a Principal Solutions Architect at Amazon Web Services. He works with some of the largest AWS customers who are taking advantage of AI/ML to solve complex business problems. He provides technical guidance and design advice to implement computer vision applications at scale. His expertise spans application architecture, serverless, containers, NoSQL and machine learning.

 

 

 

Anuj Gupta is Senior Product Manager for Amazon Augmented AI. He focuses on delivering products that make it easier for customers to adopt machine learning. In his spare time, he enjoys road trips and watching Formula 1.

Read More

Setting up human review of your NLP-based entity recognition models with Amazon SageMaker Ground Truth, Amazon Comprehend, and Amazon A2I

Setting up human review of your NLP-based entity recognition models with Amazon SageMaker Ground Truth, Amazon Comprehend, and Amazon A2I

Organizations across industries have a lot of unstructured data that you can evaluate to get entity-based insights. You may also want to add your own entity types unique to your business, like proprietary part codes or industry-specific terms. To create a natural language processing (NLP)-based model, you need to label this data based on your specific entities.

Amazon SageMaker Ground Truth makes it easy to build highly accurate training datasets for machine learning (ML), and Amazon Comprehend lets you train a model without worrying about selecting the right algorithms and parameters for model training. Amazon Augmented AI (Amazon A2I) lets you audit, review, and augment these predicted results.

In this post, we cover how to build a labeled dataset of custom entities using the Ground Truth named entity recognition (NER) labeling feature, train a custom entity recognizer using Amazon Comprehend, and review the predictions below a certain confidence threshold from Amazon Comprehend using human reviewers with Amazon A2I.

We walk you through the following steps using this Amazon SageMaker Jupyter notebook:

  1. Preprocess your input documents.
  2. Create a Ground Truth NER labeling Job.
  3. Train an Amazon Comprehend custom entity recognizer model.
  4. Set up a human review loop for low-confidence detection using Amazon A2I.

Prerequisites

Before you get started, complete the following steps to set up the Jupyter notebook:

  1. Create a notebook instance in Amazon SageMaker.

Make sure your Amazon SageMaker notebook has the necessary AWS Identity and Access Management (IAM) roles and permissions mentioned in the prerequisite section of the notebook.

  1. When the notebook is active, choose Open Jupyter.
  2. On the Jupyter dashboard, choose New, and choose Terminal.
  3. In the terminal, enter the following code:
    cd SageMaker
    git clone “https://github.com/aws-samples/augmentedai-comprehendner-groundtruth”

  4. Open the notebook by choosing SageMakerGT-ComprehendNER-A2I-Notebook.ipynb in the root folder.

You’re now ready to run the following steps through the notebook cells.

Preprocessing your input documents

For this use case, you’re reviewing at chat messages or several service tickets. You want to know if they’re related to an AWS offering. We use the NER labeling feature in Ground Truth to label a SERVICE or VERSION entity from the input messages. We then train an Amazon Comprehend custom entity recognizer to recognize the entities from text like tweets or ticket comments.

The sample dataset is provided at data/rawinput/aws-service-offerings.txt in the GitHub repo. The following screenshot shows an example of the content.

You preprocess this file to generate the following:

  • inputs.csv – You use this file to generate input manifest file for Ground Truth NER labeling.
  • Train.csv and test.csv – You use these files as input for training custom entities. You can find these files in the Amazon Simple Storage Service (Amazon S3) bucket.

Refer to Steps 1a and 1b in the notebook for dataset generation.

Creating a Ground Truth NER labeling job

The purpose is to annotate and label sentences within the input document as belonging to a custom entity that we define. In this section, you complete the following steps:

  1. Create the manifest file that Ground Truth needs.
  2. Set up a labeling workforce.
  3. Create your labeling job.
  4. Start your labeling job and verify its output.

Creating a manifest file

We use the inputs.csv file generated during prepossessing to create a manifest file that the NER labeling feature needs. We generate a manifest file named prefix+-text-input.manifest, which you use for data labeling while creating a Ground Truth job. See the following code:

# Create and upload the input manifest by appending a source tag to each of the lines in the input text file. 
# Ground Truth uses the manifest file to determine labeling tasks

manifest_name = prefix + '-text-input.manifest'
# remove existing file with the same name to avoid duplicate entries
!rm *.manifest
s3bucket = s3res.Bucket(BUCKET)

with open(manifest_name, 'w') as f:
    for fn in s3bucket.objects.filter(Prefix=prefix +'/input/'):
        fn_obj = s3res.Object(BUCKET, fn.key)
        for line in fn_obj.get()['Body'].read().splitlines():                
            f.write('{"source":"' + line.decode('utf-8') +'"}n')
f.close()
s3.upload_file(manifest_name, BUCKET, prefix + "/manifest/" + manifest_name)

The NER labeling job requires its input manifest in the {"source": "embedded text"}. The following screenshot shows the generated input.manifest file from inputs.csv.

Creating a private labeling workforce

With Ground Truth, we use a private workforce to create a labeled dataset.

You create your private workforce on the Amazon SageMaker console. For instructions, see the section Creating a private work team in Developing NER models with Amazon SageMaker Ground Truth and Amazon Comprehend.

Alternatively, follow the steps in the notebook.

For this walkthrough, we use the same private workforce to label and augment low-confidence data using Amazon A2I after custom entity training.

Creating a labeling job

The next step is to create the NER labeling job. This post highlights the key steps. For more information, see Adding a data labeling workflow for named entity recognition with Amazon SageMaker Ground Truth.

  1. On the Amazon SageMaker console, under Ground Truth, choose Labeling jobs.
  2. Choose Create labeling job.
  3. For Job name, enter a job name.
  4. For Input dataset location, enter the Amazon S3 location of the input manifest file you created (s3://bucket//path-to-your-manifest.json).
  5. For Output Dataset Location, enter a S3 bucket with an output prefix (for example, s3://bucket-name/output).
  6. For IAM role, choose Create a new Role.
  7. Select Any S3 Bucket.
  8. Choose Create.
  9. For Task category, choose Text.
  10. Select Named entity recognition.
  11. Choose Next.
  12. For Worker type, select Private.
  13. In Private Teams, select the team you created.
  14. In the Named Entity Recognition Labeling Tool section, for Enter a brief description of the task, enter Highlight the word or group of words and select the corresponding most appropriate label from the right.
  15. In the Instructions box, enter Your labeling will be used to train an ML model for predictions. Please think carefully on the most appropriate label for the word selection. Remember to label at least 200 annotations per label type.
  16. Choose Bold Italics.
  17. In the Labels section, enter the label names you want to display to your workforce.
  18. Choose Create.

Starting your labeling job

Your workforce (or you, if you chose yourself as your workforce) received an email with login instructions.

  1. Choose the URL provided and enter your user name and password.

You are directed to the labeling task UI.

  1. Complete the labeling task by choosing labels for groups of words.
  2. Choose Submit.
  3. After you label all the entries, the UI automatically exits.
  4. To check your job’s status, on the Amazon SageMaker console, under Ground Truth, choose Labeling jobs.
  5. Wait until the job status shows as Complete.

Verifying annotation outputs

To verify your annotation outputs, open your S3 bucket and locate <S3 Bucket Name>/output/<labeling-job-name>/manifests/output/output.manifest. You can review the manifest file that Ground Truth created. The following screenshot shows an example of the entries you see.

Training a custom entity model

We now use the annotated dataset or output.manifest Ground Truth created to train a custom entity recognizer. This section walks you through the steps in the notebook.

Processing the annotated dataset

You can provide labels for Amazon Comprehend custom entities through an entity list or annotations. In this post, we use annotations generated using Ground Truth labeling jobs. You need to convert the annotated output.manifest file to the following CSV format:

File, Line, Begin Offset, End Offset, Type
documents.txt, 0, 0, 11, VERSION

Run the following code in the notebook to generate the annotations.csv file:

# Read the output manifest json and convert into a csv format as expected by Amazon Comprehend Custom Entity Recognizer
import json
import csv

# this will be the file that will be written by the format conversion code block below
csvout = 'annotations.csv'

with open(csvout, 'w', encoding="utf-8") as nf:
    csv_writer = csv.writer(nf)
    csv_writer.writerow(["File", "Line", "Begin Offset", "End Offset", "Type"])
    with open("data/groundtruth/output.manifest", "r") as fr:
        for num, line in enumerate(fr.readlines()):
            lj = json.loads(line)
            #print(str(lj))
            if lj and labeling_job_name in lj:
                for ent in lj[labeling_job_name]['annotations']['entities']:
                    csv_writer.writerow([fntrain,num,ent['startOffset'],ent['endOffset'],ent['label'].upper()])
    fr.close()
nf.close()        

s3_annot_key = "output/" + labeling_job_name + "/comprehend/" + csvout

upload_to_s3(s3_annot_key, csvout)

The following screenshot shows the contents of the file.

Setting up a custom entity recognizer

This post uses the API, but you can optionally create the recognizer and batch analysis job on the Amazon Comprehend console. For instructions, see Build a custom entity recognizer using Amazon Comprehend.

  1. Enter the following code. For s3_train_channel, use the train.csv file you generated in preprocessing step for training the recognizer. For s3_annot_channel, use annotations.csv as a label to train your custom entity recognizer.
    custom_entity_request = {
    
          "Documents": { 
             "S3Uri": s3_train_channel
          },
          "Annotations": { 
             "S3Uri": s3_annot_channel
          },
          "EntityTypes": [
                    {
                        "Type": "SERVICE"
                    },
                    {
                        "Type": "VERSION"
                    }
          ]
    }

  2. Create the entity recognizer using CreateEntityRecognizer The entity recognizer is trained with the minimum required number of training samples to generate some low confidence predictions required for our Amazon A2I workflow. See the following code:
    import datetime
    
    id = str(datetime.datetime.now().strftime("%s"))
    create_custom_entity_response = comprehend.create_entity_recognizer(
            RecognizerName = prefix + "-CER", 
            DataAccessRoleArn = role,
            InputDataConfig = custom_entity_request,
            LanguageCode = "en"
    )
    

    When the entity recognizer job is complete, it creates a recognizer with a performance score. As mentioned earlier we trained the entity recognizer with a minimum number of training samples to generate low confidence predictions we need to trigger the Amazon A2I human loop. You can find these metrics on the Amazon Comprehend console. See the following screenshot.

  3. Create a batch entity detection analysis job to detect entities over a large number of documents.

Use the Amazon Comprehend StartEntitiesDetectionJob operation to detect custom entities in your documents. For instructions on creating an endpoint for real-time analysis using your custom entity recognizer, see Announcing the launch of Amazon Comprehend custom entity recognition real-time endpoints.

To use the EntityRecognizerArn for custom entity recognition, you must provide access to the recognizer to detect the custom entity. This ARN is supplied by the response to the CreateEntityRecognizer operation.

  1. Run the custom entity detection job to get predictions on the test dataset you created during the preprocessing step by running the following cell in the notebook:
    s3_test_channel = 's3://{}/{}'.format(BUCKET, s3_test_key) s3_output_test_data = 's3://{}/{}'.format(BUCKET, "output/testresults/") 
    test_response = comprehend.start_entities_detection_job(   InputDataConfig={ 
    'S3Uri': s3_test_channel, 
    'InputFormat': 'ONE_DOC_PER_LINE'
    }, 
    OutputDataConfig={'S3Uri': s3_output_test_data 
    }, 
    DataAccessRoleArn=role, 
    JobName='a2i-comprehend-gt-blog', 
    EntityRecognizerArn=jobArn, 
    LanguageCode='en')
    

    The following screenshot shows the test results.

Setting up a human review loop

In this section, you set up a human review loop for low-confidence detections in Amazon A2I. It includes the following steps:

  1. Choose your workforce.
  2. Create a human task UI.
  3. Create a worker task template creator function.
  4. Create the flow definition.
  5. Check the human loop status and wait for reviewers to complete the task.

Choosing your workforce

For this post, we use the private workforce we created for the Ground Truth labeling jobs. Use the workforce ARN to set up the workforce for Amazon A2I.

Creating a human task UI

Create a human task UI resource with a UI template in liquid HTML. This template is used whenever a human loop is required.

The following example code is compatible with Amazon Comprehend entity detection:

template = """
<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>

<style>
    .highlight {
        background-color: yellow;
    }
</style>

<crowd-entity-annotation
        name="crowd-entity-annotation"
        header="Highlight parts of the text below"
        labels="[{'label': 'service', 'fullDisplayName': 'Service'}, {'label': 'version', 'fullDisplayName': 'Version'}]"
        text="{{ task.input.originalText }}"
>
    <full-instructions header="Named entity recognition instructions">
        <ol>
            <li><strong>Read</strong> the text carefully.</li>
            <li><strong>Highlight</strong> words, phrases, or sections of the text.</li>
            <li><strong>Choose</strong> the label that best matches what you have highlighted.</li>
            <li>To <strong>change</strong> a label, choose highlighted text and select a new label.</li>
            <li>To <strong>remove</strong> a label from highlighted text, choose the X next to the abbreviated label name on the highlighted text.</li>
            <li>You can select all of a previously highlighted text, but not a portion of it.</li>
        </ol>
    </full-instructions>

    <short-instructions>
        Select the word or words in the displayed text corresponding to the entity, label it and click submit
    </short-instructions>

    <div id="recognizedEntities" style="margin-top: 20px">
                <h3>Label the Entity below in the text above</h3>
                <p>{{ task.input.entities }}</p>
    </div>
</crowd-entity-annotation>

<script>

    function highlight(text) {
        var inputText = document.getElementById("inputText");
        var innerHTML = inputText.innerHTML;
        var index = innerHTML.indexOf(text);
        if (index >= 0) {
            innerHTML = innerHTML.substring(0,index) + "<span class='highlight'>" + innerHTML.substring(index,index+text.length) + "</span>" + innerHTML.substring(index + text.length);
            inputText.innerHTML = innerHTML;
        }
    }

    document.addEventListener('all-crowd-elements-ready', () => {
        document
            .querySelector('crowd-entity-annotation')
            .shadowRoot
            .querySelector('crowd-form')
            .form
            .appendChild(recognizedEntities);
    });
</script>
"""

Creating a worker task template creator function

This function is a higher-level abstraction on the Amazon SageMaker package’s method to create the worker task template, which we use to create a human review workflow. See the following code:

def create_task_ui():
    '''
    Creates a Human Task UI resource.

    Returns:
    struct: HumanTaskUiArn
    '''
    response = sagemaker.create_human_task_ui(
        HumanTaskUiName=taskUIName,
        UiTemplate={'Content': template})
    return response
# Task UI name - this value is unique per account and region. You can also provide your own value here.
taskUIName = prefix + '-ui' 

# Create task UI
humanTaskUiResponse = create_task_ui()
humanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']
print(humanTaskUiArn)

Creating the flow definition

Flow definitions allow you to specify the following:

  • The workforce that your tasks are sent to
  • The instructions that your workforce receives

This post uses the API, but you can optionally create this workflow definition on the Amazon A2I console.

For more information, see Create a Flow Definition.

To set up the condition to trigger the human loop review, enter the following code (you can change the value of the CONFIDENCE_SCORE_THRESHOLD based on what confidence level you want to trigger the human review):

human_loops_started = []

import json

CONFIDENCE_SCORE_THRESHOLD = 90
for line in data:
    print("Line is: " + str(line))
    begin_offset=line['BEGIN_OFFSET']
    end_offset=line['END_OFFSET']
    if(line['CONFIDENCE_SCORE'] < CONFIDENCE_SCORE_THRESHOLD):
        humanLoopName = str(uuid.uuid4())
        human_loop_input = {}
        human_loop_input['labels'] = line['ENTITY']
        human_loop_input['entities']= line['ENTITY']
        human_loop_input['originalText'] = line['ORIGINAL_TEXT']
        start_loop_response = a2i_runtime_client.start_human_loop(
        HumanLoopName=humanLoopName,
        FlowDefinitionArn=flowDefinitionArn,
        HumanLoopInput={
                "InputContent": json.dumps(human_loop_input)
            }
        )
        print(human_loop_input)
        human_loops_started.append(humanLoopName)
        print(f'Score is less than the threshold of {CONFIDENCE_SCORE_THRESHOLD}')
        print(f'Starting human loop with name: {humanLoopName}  n')
    else:
         print('No human loop created. n')

Checking the human loop status and waiting for reviewers to complete the task

To define a function that allows you to check the human loop’s status, enter the following code:

completed_human_loops = []
for human_loop_name in human_loops_started:
    resp = a2i_runtime_client.describe_human_loop(HumanLoopName=human_loop_name)
    print(f'HumanLoop Name: {human_loop_name}')
    print(f'HumanLoop Status: {resp["HumanLoopStatus"]}')
    print(f'HumanLoop Output Destination: {resp["HumanLoopOutput"]}')
    print('n')
    
    if resp["HumanLoopStatus"] == "Completed":
        completed_human_loops.append(resp)

Navigate to the private workforce portal that’s provided as the output of cell 2 from the previous step in the notebook. See the following code:

workteamName = WORKTEAM_ARN[WORKTEAM_ARN.rfind('/') + 1:]
print("Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!")
print('https://' + sagemaker.describe_workteam(WorkteamName=workteamName)['Workteam']['SubDomain'])

The UI template is similar to the Ground Truth NER labeling feature. Amazon A2I displays the entity identified from the input text (this is a low-confidence prediction). The human worker can then update or validate the entity labeling as required and choose Submit.

This action generates an updated annotation with offsets and entities as highlighted by the human reviewer.

Cleaning up

To avoid incurring future charges, stop and delete resources such as the Amazon SageMaker notebook instance, Amazon Comprehend custom entity recognizer, and the model artifacts in Amazon S3 when not in use.

Conclusion

This post demonstrated how to create annotations for an Amazon Comprehend custom entity recognition using Ground Truth NER. We used Amazon A2I to augment the low-confidence predictions from Amazon Comprehend.

You can use the annotations that Amazon A2I generated to update the annotations file you created and incrementally train the custom recognizer to improve the model’s accuracy.

For video presentations, sample Jupyter notebooks, or more information about use cases like document processing, content moderation, sentiment analysis, text translation, and more, see Amazon Augmented AI Resources. We’re interested in how you want to extend this solution for your use case and welcome your feedback.


About the Authors

Mona Mona is an AI/ML Specialist Solutions Architect based out of Arlington, VA. She works with World Wide Public Sector team and helps customers adopt machine learning on a large scale. She is passionate about NLP and ML Explainability areas in AI/ML.

 

 

 

 

Prem Ranga is an Enterprise Solutions Architect based out of Houston, Texas. He is part of the Machine Learning Technical Field Community and loves working with customers on their ML and AI journey. Prem is passionate about robotics, is an Autonomous Vehicles researcher, and also built the Alexa-controlled Beer Pours in Houston and other locations.

 

 

 

Read More