Posts in category: Amazon AWS

Amazon Lex provides the power of automatic speech recognition (ASR) for converting speech to text, along with natural language understanding (NLU) for recognizing user intents. This combination allows you to develop sophisticated conversational interfaces using both voice and text for chatbots, IVR bots, and voicebots.

This week, we’re announcing Amazon Lex support for British English and US Spanish. With British English, your conversational bots can be localized to understand the British English accent, while delivering responses in Amazon Polly voices designed to sound like UK English speakers. Brilliant!

With support for US Spanish, you can develop applications for the second-most widely spoken language in the United States. Amazon Lex can now accurately recognize written and spoken Spanish, while providing responses using Amazon Polly’s natural sounding US Spanish voices. Listo!

In this post, we consider a hypothetical appliance manufacturer in the United States. Customers calling into their service center to schedule or change a repair appointment may prefer to speak in English or Spanish. The application for this use case allows callers to select their preferred language by saying “English” or “Español” when prompted. You want to provide them with the best customer service experience, no matter whether it is their washing machine or their lavadora that needs repair.

We show you how to create an Amazon Connect call center experience that supports both US English and US Spanish. Customers can schedule, change, and cancel appointments, using a fully automated solution that converses with them in their preferred language.

Building a multi-language conversational experience

This post uses the following sample conversations:

Agent: Thank you for calling. To continue in English, say “English,” for Spanish, say “Español.”

US English

Agent: I can schedule or change a repair appointment. How can I help?

User: I want to get my dishwasher fixed.

Agent: What city are you in?

User: Philadelphia

Agent: I have technicians available next week. When would you prefer to have them visit?

User: September 24th at noon

Agent: OK, you are all set for your dishwasher repair in Philadelphia on the 24th of September at noon.

US Spanish

Agent: Puede reservar o cambiar una cita de reparación. ¿Cómo puedo ayudar?

User: Me gustaria programar una cita

Agent: Para qué tipo de aparato?

User: Refrigerador

Agent: En que ciudad estas?

User: Brooklyn

Agent: En que fecha te gustaria que vinieran?

User: 24 de Septiembre

Agent: Bien, ya está todo listo para la reparación de su refrigerador en 24 de Septiembre, 2020.

To support these conversation models, you need to create Lex bots with relevant user intents. In this post, we create intents for ScheduleAppointment, ModifyAppointment, and CancelAppointment (and in Spanish, ReservarCita, ModificarCita, CancelarCita).

Deploying the sample Lex bots

To create the sample bots, perform the following steps. For this post, you create two Amazon Lex bots: AppointmentBot_enUS for US English, and AppointmentBot_esUS for US Spanish. To follow along with this post, you can create these Lex bots yourself on the Amazon Lex console, or import them directly.

1. To import the bots, download the US English bot and the US Spanish bot
2. On the Amazon Lex console, choose
3. Select the zip that you downloaded, and choose Import.
4. When the import process is complete, choose AppointmentBot_enUS, and choose
5. When the build it complete, go back to the Amazon Lex console main window, and choose Import.
6. Select the zip that you downloaded, and choose Import.
7. When the import process is complete, choose AppointmentBot_esUS, and choose

At this point, you should have two working Lex bots: one for US English, and one for US Spanish.

Creating your Amazon Connect instance

In this section, we integrate the bots with an Amazon Connect cloud-based call center instance. The first step is to create the Amazon Connect instance:

1. On the AWS Management Console, choose Amazon Connect.
2. If this is your first Amazon Connect instance, choose Get started; otherwise, choose Add an instance.
3. For Identity management, choose Store users within Amazon Connect.
4. Enter a URL prefix, such as appointment-bot-############, where ############ is your current AWS account number.
5. Choose Next step.
6. For Create an administrator, enter a name, password, and email address.
7. Choose Next step.
8. For Telephony Options, leave both call options selected by default.
9. Choose Next step.
10. For Data storage, choose Next step.
11. Review the settings and choose Create instance.

Associating your bots with your Amazon Connect instance

Now that you have an Amazon Connect instance, you can claim a phone number, create a contact flow, and integrate your contact flow with the two Lex bots you created in the prior step. First, associate your bots with your Amazon Connect instance:

1. On the Amazon Connect console, open your instance by choosing the Instance Alias
2. Choose Contact flows.
3. From the drop-down list, choose AppointmentBot_enUS. If you don’t see the bot in the list, make sure you have selected the same Region you used when you created your Lex bot.
4. Choose + Add Lex Bot.
5. From the drop-down list, choose AppointmentBot_esUS and choose + Add Lex Bot.

Configuring Amazon Connect to work with your bot

Now you can use your bots with Amazon Connect. First, claim a phone number for your Amazon Connect instance:

1. On the Amazon Connect console, choose Overview.
2. Choose the Amazon Connect
3. Choose the Login URL link, and enter the user name and password you specified earlier.
4. On the Amazon Connect console, for Step 1, Choose Begin.
5. For your phone number, choose a country, Direct Dial or Toll Free, and a phone number.
6. Choose Next.
7. If you want to test your new phone number, try it on the next screen or choose Skip for now.

For this post, you can skip the hours of operation, creating queues, and creating prompts. For more information on these features, see the Amazon Connect Administrator Guide. Now let’s import the contact flow.

8. Download and unzip the sample Amazon Connect contact flow for this post: manage_repairs.zip.
9. On the Amazon Connect console, go to Step 5, Create contact flows, choose View contact flows.
10. Choose Create contact flow.
11. From the drop-down at the top right side of the page, choose Import flow (beta).
12. Choose Select and select the manage-repairs.json file you downloaded, and select Import.
13. Choose Save, and then Publish.

Your contact flow should look like the following screenshot.

14. Choose the Routing icon from the side menu, and choose Phone numbers.
15. Choose your phone number to edit it, and change the contact flow or IVR to the Manage Repairs contact flow you just created.
16. Choose Save.

Your Amazon Connect instance is now configured to work with your Amazon Lex bots. Try calling the phone number to see how it works!

Conclusion

With the addition of British English and US Spanish language support, along with US English and Australian English, Amazon Lex allows you to create bots that can converse with users in their native language. You can combine Amazon Lex with Amazon Connect to create streamlined, multi-language call center user experiences in minutes. The additional language support in Amazon Lex is available at the same price, and in the same Regions, as US English. You can try these languages via the console, the AWS Command Line Interface (AWS CLI), and the AWS SDKs.

About the Authors

Claire Mitchell is a Design Consultant with the AWS Professional Services Conversational AI team. Occasionally she spends time exploring speculative design practices, textiles, and playing the drums.

Brian Yost is a Senior Consultant with the AWS Professional Services Conversational AI team. In his spare time, he enjoys mountain biking, home brewing, and tinkering with technology.

As a Product Manager on the Amazon Lex team, Harshal Pimpalkhute spends his time trying to get machines to engage (nicely) with humans.

Read More

University of Illinois, Urbana Champaign (UIUC) has partnered with the Amazon Machine Learning Solutions Lab to help UIUC football coaches prepare for games more efficiently and improve their odds of winning.

Previously, coaches prepared for games by creating a game planning sheet that only featured types of plays for a certain down and distance, and where the team was on the field. As a result, the coaching staff might miss important scenarios and opportunities. Additionally, preparing a game planning sheet was a manual process, with new data for each game being entered into a template each week, which is time-consuming and not scalable.

To add more insights to the current call sheet templates and help coaches prepare for games better, the team combined UIUC’s deep expertise in college football and coaching with the machine learning (ML) capabilities of Amazon SageMaker to create a state-of-the-art ML model that predicts the result of UIUC’s football plays. In addition, UIUC coaches now have an auto-generated visual game planning sheet based on key features that the model recommends. This gives them more insights on their strategy for the game and reduces the time it takes to generate the visual game planning sheets from 2.5 hours to less than 30 seconds.

“The UIUC Athletic department collaborated with the Amazon ML Solutions Lab to harness the power of machine learning to derive data-driven insights on what features to include in our planning and preparation for our football games,” says Kingsley Osei-Asibey, Director of Analytics & Football Technology at UIUC. “By selecting AWS as our primary ML/AI platform, we got to work alongside the experts at the ML Solutions Lab to create new and interesting insights using Amazon SageMaker. Now, all the manual analysis of data from past games that took us hours is automated, and our Fighting Illini coaches can generate weekly visual game planning sheets against different opponents at the press of a button.”

This post looks at how the Amazon ML Solutions Lab used features related to the plays during a football game to predict the result of a play, and then used the XGBoost importance score feature and correlation analysis to recommend features for coaches to analyze.

We provide code snippets to show you how we used the Amazon SageMaker XGBoost library to generate feature importance scores.

Data and model

We used UIUC’s game data from the 2018–2019 college football season, covering 24 features including in-game statistics, location of the play, UIUC’s strategies, and their opponent’s play types. We used those features to train an XGBoost model to predict if an offensive play will result in a win or loss. The UIUC coaches decided whether it’s a win or loss for a play based on different situations.

We then used the feature importance scores to select key features. We used the model for feature-selection purposes to recommend important scenarios represented by features. We selected XGBoost because it performs well on features with complex distributions, and it outputs feature importance scores to help us with feature selection and model interpretation.

The main goal was to generate game planning sheets for football coaches to use in games to give them an edge. We used the features from a well performant ML model trained to classify successful and unsuccessful plays to inform coaches and generate game planning sheets.

The following diagram summarizes the modeling steps taken to generate the ML-based features for the game planning sheet.

The rows are shuffled and split into five non-overlapping folds, which are then further split into training and validation sets. The training sets of each fold are balanced using the Synthetic Minority Oversampling Technique (SMOTE) algorithm.

Each fold includes the following steps:

1. Calculate a new feature score:
   1. Train an XGBoost model on the balanced training data set and extract the feature importances feat_i.
   2. Compute the Pearson’s correlation of the features and label in the balanced training dataset corr_i.
   3. Compute a new feature score as the product of absolute correlation and feature importance feature_score_i = feat_i * abs(corr_i).
2. Sort the features based on the feat_score.
3. Train multiple XGBoost models using the top 5 features, top 10 features, and so on, and evaluate validation balanced accuracy for each model.
4. Choose the best-performing model.

After we trained models from each of the five folds, we merged the important features. A feature is selected for the game planning sheet if it appears in the top 10 features (ranked by feature importance score) of at least three folds.

Calculating the new feature score

In the previous section, we described the construction of a new feature score. This new feature score incorporates the feature importance from a non-linear XGBoost model, as well as direct linear correlation. The purpose of this new feature score is to select features that are relevant to winning or losing a play. A feature with a high feature score has high XGBoost feature importance and high linear correlation with the label, making it a relevant feature for game planning sheets.

In this section, we dive deeper into the construction of the new feature score with code snippets. The feature score is a combination of feature importance from a trained XGBoost model and linear correlation of the features and the label.

First, we train an XGBoost model using Amazon SageMaker built-in algorithms. Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy ML models quickly. Amazon SageMaker provides several built-in algorithms (such as XGBoost) for a variety of problem types.

This trained XGBoost model provides a first look into which features are important to the UIUC football team winning a play. See the following code:

from sagemaker.amazon.amazon_estimator import get_image_uri
container = get_image_uri(region, "xgboost", "0.90-1")

hyperparameters = {
    "max_depth":"7",
    "eta":"0.01",
    "gamma":"3",
    "min_child_weight":"6",
    "subsample":"0.6",
    "silent":"0",
    "objective":"binary:logistic",
    "num_round":"330"
}

instance_type = 'ml.m5.2xlarge'
output_path = "s3://{}/{}/{}/output".format(bucket, "model", "xgboost")

job_name = "xgboost-".format(i+1) + time.strftime("%Y-%m-%d-%H-%M-%S", time.gmtime())

estimator = sagemaker.estimator.Estimator(
    container, 
    role, 
    hyperparameters=hyperparameters,
    train_instance_count=1, 
    train_instance_type=instance_type, 
    train_volume_size=5,
    output_path=output_path, 
    sagemaker_session=sagemaker.Session()
)

train_input = sagemaker.s3_input(
    s3_data="s3://{}/{}/{}".format(bucket, "train", "balanced_train_data.csv"), 
    content_type='csv'
)
estimator.fit({"train": train_input}, job_name=job_name)

Amazon SageMaker stores the model object in the specified Amazon Simple Storage Service (Amazon S3) bucket. To calculate the feature score, we need to download model.tar.gz from Amazon S3 to our Amazon SageMaker notebook instance. See the following code:

model_path = "s3://{}/{}/{}/output/{}".format(
    bucket, "model", "xgboost",
    "xgboost-2019-06-16-09-56-39-854/output/model.tar.gz"
)

fs = s3fs.S3FileSystem()

with fs.open(model_path, "rb") as f:
    with tarfile.open(fileobj=f, mode="r") as tar_f:
        with tar_f.extractfile("xgboost-model") as extracted_f:
            xgbooster = pickle.load(extracted_f)

Finally, we calculate the new feature score as feature_score_i = feat_i * abs(corr_i). We use the absolute value of the correlation because our goal is to find features that are relevant to winning or losing a play, and a highly negative correlation indicates a strong linear relationship between the feature and the UIUC football team losing the play. See the following code:

#the xgbooster object replaces the original feature names with 'f0,...f'
#here we create a mapping to obtain the original feature names
feature_name_map = dict(zip([f"f{i}" for i in range(len(feature_names))], feature_names))

features_importance_df = pd.DataFrame([xgbooster.get_fscore()], index=["weight"]).T
features_importance_df["normalized_weight"] = features_importance_df["weight"]/features_importance_df["weight"].sum()
feature_importances_df["feature_name"] = feature_importances_df.index.map(feature_name_map)

correlation_df = pd.DataFrame(balanced_train_data_df[FEATURES].corr()[LABEL])
correlation_df["absolute_corr"] = correlation_df[LABEL].abs()

feature_score_df = pd.merge(
    features_importance_df, correlation_df.reset_index(), 
    left_on="feature_name", right_on="index"
)

feature_score_df["feature_score"] = feature_score_df["absolute_corr"] * feature_score_df["normalized_weight"]

The following graph shows a plot of feature_score vs.rank for each fold. High values on the y-axis indicate that the feature was important for the XGBoost model and has high correlation with winning or losing a play. The key takeaway from this plot is additional features after feature number 105 don’t add any new information, and the optimum number of features to use lies between 0–105.

Evaluating the model

We performed five-fold cross-validation on the XGBoost model, and compared it to three baseline models: a model predicting every sample as lost, a model predicting every sample as win, and a random model assigning win or loss with a 50/50 chance.

Because the dataset is imbalanced with 56% of the plays labeled as lost and 44% as won, we used the weighted accuracy metrics considering the class weights when comparing our model to the naïve baselines. The weighted accuracy for all three naïve baselines is 50%, and average weighted accuracy of the XGBoost is 65.2% across five folds, which shows that our model has 15% improvement compared to the baselines.

The following plot shows validation balanced accuracy vs. the number of top features for each fold. For each data point, an XGBoost model is trained using the top n features, where n is the value on the x-axis, and evaluated on the fold’s validation dataset to obtain the validation balanced accuracy. The top performing model for each is annotated in the plot. For example, Fold 0’s best-performing model uses the top 60 features (as determined in the preceding plot), which has a validation balanced accuracy of 64.8%. Features ranked above 105 aren’t evaluated because the previous plot shows that features ranked above 105 contribute little information.

The following table summarizes the results of the procedure we outlined. For each fold, the balanced accuracy performance improves after performing feature selection, with an average increase of 3.2%.

To further improve the models, we used Amazon SageMaker automated model tuning for hyperparameter optimization. We used the best features identified in the preceding step for each fold, and performed 20 iterations of Bayesian optimization on each fold.

Feature selection and game planning sheet recommendation across five folds

The end goal is to create a new game planning sheet using features derived from the XGBoost models. A high-performing model indicates that the extracted features are relevant to winning a play. The output of the training stage results in an XGBoost model for each fold. A feature is selected for the game planning sheet if it appears in the top 10 features (ranked by feature importance score) of at least three folds.

After reviewing these features with the UIUC coaching staff, the coaches designed new game planning sheets to analyze the best play types based on how their opponent would be playing defense. These additional features will help the coaches prepare more scenarios before the games start, and players can react faster and more accurately against opponents.

Summary

UIUC football coaches partnered with the Amazon ML Solutions Lab and created an ML model to gain more insights on their performance and strategies. This solution also saves the coaches’ time when preparing for a game; instead of manually analyzing the best plays to call under different situations, coaches can automate this process using the features the ML model recommends.

This model is customized for UIUC’s football team and their opponents, and will help UIUC’s coaches prepare for more scenarios in upcoming seasons. Additionally, it will help players react correctly and quickly to game situations.

If you’d like help accelerating the use of ML in your products and services, please contact the Amazon ML Solutions Lab program.

About the Authors

 Ninad Kulkarni is a Data Scientist in the Amazon Machine Learning Solutions Lab. He helps customers adopt ML and AI by building solutions to address their business problems. Most recently, he has built predictive models for sports and automotive customers.

Daliana Zhen Liu is a Data Scientist in the Amazon Machine Learning Solutions Lab. She has built ML models to help customers accelerate their business in sports, media and education. She is passionate about introducing data science to more people.

Tianyu Zhang is a Data Scientist in the Amazon Machine Learning Solutions Lab. He helps customers solve business problems by applying ML and AI techniques. Most recently, he has built NLP model and predictive model for procurement and sports.

Read More

Amazon Comprehend is a natural language processing (NLP) service that uses machine learning (ML) to find insights and relationships like people, places, sentiments, and topics in unstructured text. You can now use Amazon Comprehend ML capabilities to detect and redact personally identifiable information (PII) in customer emails, support tickets, product reviews, social media, and more. No ML experience required. For example, you can analyze support tickets and knowledge articles to detect PII entities and redact the text before you index the documents in the search solution. After that, search solutions are free of PII entities in documents. Redacting PII entities helps you protect privacy and comply with local laws and regulations.

Customer use case: TeraDact Solutions

TeraDact Solutions has already put this new feature to work. TeraDact Solutions’ software offers a robust alternative for secure information sharing in a world of ever-increasing compliance and privacy concerns. With its signature Information Identification & Presentation (IIaP) capabilities, TeraDact’s tools provide the user with a safe information sharing environment. “Using Amazon Comprehend for PII redaction with our tokenization system not only helps us reach a larger set of our customers but also helps us overcome the shortcomings of rules-based PII detection which can result in false alarms or missed details. PII detection is critical for businesses and with the power of context-aware NLP models from Comprehend we can uphold the trust customers place in us with their information. Amazon is innovating in ways to help push our business forward by adding new features which are critical to our business thereby providing enhanced service to 100% of customers able to access Comprehend in AWS.” said Chris Schrichte, CEO, TeraDact Solutions, Inc.

In this post, I cover how to use Amazon Comprehend to detect PII and redact the PII entities via the AWS Management Console and the AWS Command Line Interface (AWS CLI).

Detecting PII in Amazon Comprehend

When you analyze text using Amazon Comprehend real-time analysis, Amazon Comprehend automatically identifies PII, as summarized in the following table.

For each detected PII entity, you get the type of PII, a confidence score, and begin and end offset. These offsets help you locate PII entities in your documents for document processing to redact it at the secure storage or downstream solutions.

Analyzing text on the Amazon Comprehend console

To get started with Amazon Comprehend, all you need is an AWS account. To use the console, complete the following steps:

1. On the Amazon Comprehend console, in the Input text section, select Built-in.
2. For Input text, enter your text.
3. Choose Analyze.

4. On the Insights page, choose the PII

The PII tab shows color-coded text to indicate different PII entity types, such as name, email, address, phone, and others. The Results section shows more information about the text. Each entry shows the PII entity, its type, and the level of confidence Amazon Comprehend has in this analysis.

Analyzing text via the AWS CLI

To perform real-time analysis using the AWS CLI, enter the following code:

aws comprehend detect-pii-entities 
--language-code en 
--text 
" Good morning, everybody. My name is Van Bokhorst Serdar, and today I feel like sharing a whole lot of personal information with you. Let's start with my Email address SerdarvanBokhorst@dayrep.com. My address is 2657 Koontz Lane, Los Angeles, CA. My phone number is 818-828-6231. My Social security number is 548-95-6370. My Bank account number is 940517528812 and routing number 195991012. My credit card number is 5534816011668430, Expiration Date 6/1/2022, my C V V code is 121, and my pin 123456. Well, I think that's it. You know a whole lot about me. And I hope that Amazon comprehend is doing a good job at identifying PII entities so you can redact my personal information away from this document. Let's check."

To view the output, open the JSON response object and look at the detected PII entities. For each entity, the service returns the type of PII, confidence score metric, BeginOffset, and EndOffset. See the following code:

{
    "Entities": [
        {
            "Score": 0.9996334314346313,
            "Type": "NAME",
            "BeginOffset": 36,
            "EndOffset": 55
        },
        {
            "Score": 0.9999902248382568,
            "Type": "EMAIL",
            "BeginOffset": 167,
            "EndOffset": 195
        },
        {
            "Score": 0.9999983310699463,
            "Type": "ADDRESS",
            "BeginOffset": 211,
            "EndOffset": 245
        },
        {
            "Score": 0.9999997615814209,
            "Type": "PHONE",
            "BeginOffset": 265,
            "EndOffset": 277
        },
        {
            "Score": 0.9999996423721313,
            "Type": "SSN",
            "BeginOffset": 308,
            "EndOffset": 319
        },
        {
            "Score": 0.9999984502792358,
            "Type": "BANK_ACCOUNT_NUMBER",
            "BeginOffset": 347,
            "EndOffset": 359
        },
        {
            "Score": 0.9999974966049194,
            "Type": "BANK_ROUTING",
            "BeginOffset": 379,
            "EndOffset": 388
        },
        {
            "Score": 0.9999991655349731,
            "Type": "CREDIT_DEBIT_NUMBER",
            "BeginOffset": 415,
            "EndOffset": 431
        },
        {
            "Score": 0.9923601746559143,
            "Type": "CREDIT_DEBIT_EXPIRY",
            "BeginOffset": 449,
            "EndOffset": 457
        },
        {
            "Score": 0.9999997615814209,
            "Type": "CREDIT_DEBIT_CVV",
            "BeginOffset": 476,
            "EndOffset": 479
        },
        {
            "Score": 0.9998345375061035,
            "Type": "PIN",
            "BeginOffset": 492,
            "EndOffset": 498
        }
    ]
}

Asynchronous PII redaction batch processing on the Amazon Comprehend console

You can redact documents by using Amazon Comprehend asynchronous operations. You can choose redaction mode Replace with PII entity to replace PII entities with PII entity type, or choose to mask PII entity with redaction mode Replace with character and replace the characters in PII entities with a character of your choice (!, #, $, %, &, *, or @).

To analyze and redact large documents and large collections of documents, ensure that the documents are stored in an Amazon Simple Storage Service (Amazon S3) bucket and start an asynchronous operation to detect and redact PII in the documents. The results of the analysis are returned in an S3 bucket.

1. On the Amazon Comprehend console, choose Analysis jobs.
2. Choose Create job.

3. On the Create analysis job page, for Name, enter a name (for this post, we enter comprehend-blog-redact-01).
4. For Analysis type, choose Personally identifiable information (PII).
5. For Language, choose English.

6. In the PII detection settings section, for Output mode, select Redactions.
7. Expand PII entity types and select the entity types to redact.
8. For Redaction mode, choose Replace with PII entity type.

Alternatively, you can choose Replace with character to replace PII entities with a character of your choice (!, #, $, %, &, *, or @).

9. In the Input data section, for Data source, select My documents.
10. For S3 location, enter the S3 path for pii-s3-input.txt.

This text file has the same example content we used earlier for real-time analysis.

11. In the Output data section, for S3 location, enter the path to the output folder in Amazon S3.

Make sure you choose the correct input and output paths based on how you organized the document.

12. In the Access permissions section, for IAM role, select Create an IAM role.

You need an AWS Identity and Access Management (IAM) role with required permissions to access the input and output S3 buckets for the job that is created and propagated.

13. For Permissions to access, choose Input and Output S3 buckets.
14. For Name suffix, enter a suffix for your role (for this post, we enter ComprehendPIIRole).
15. Choose Create job.

You can see the job comprehend-blog-redact-01 with the job status In progress.

When the job status changes to Completed, you can access the output file to view the output. The pii-s3-input.txt file has the same example content we used earlier, and using redaction mode replaces PII with its PII entity type. Your output looks like the following text:

If you have very long entity types, you may prefer to mask PII with a character. If you choose to replace PII with the character *, your output looks like the following text. :

Asynchronous PII redaction batch processing via the AWS CLI

To perform the PII redaction job using the AWS CLI, enter the following code:

aws comprehend start-pii-entities-detection-job 
 --input-data-config S3Uri="s3://ai-ml-services-lab/public/labs/comprehend/pii/input/redact/pii-s3-input.txt"  
 --output-data-config S3Uri="s3://ai-ml-services-lab/public/labs/comprehend/pii/output/redact/"  
 --mode "ONLY_REDACTION" 
 --redaction-config PiiEntityTypes="BANK_ACCOUNT_NUMBER","BANK_ROUTING","CREDIT_DEBIT_NUMBER","CREDIT_DEBIT_CVV","CREDIT_DEBIT_EXPIRY","PIN","EMAIL","ADDRESS","NAME","PHONE","SSN",MaskMode="REPLACE_WITH_PII_ENTITY_TYPE" 
 --data-access-role-arn "arn:aws:iam::<ACCOUNTID>:role/service-role/AmazonComprehendServiceRole-ComprehendPIIRole" 
 --job-name "comprehend-blog-redact-001" 
 --language-code "en"

The request yields the following output:

{
    "JobId": "e41101e2f0919a320bc0583a50f86b5f",
    "JobStatus": "SUBMITTED"
}

To monitor the job request, enter the following code:

aws comprehend describe-pii-entities-detection-job --job-id " e41101e2f0919a320bc0583a50f86b5f "

The following output shows that the job is complete:

{
    "PiiEntitiesDetectionJobProperties": {
        "JobId": "e41101e2f0919a320bc0583a50f86b5f",
        "JobName": "comprehend-blog-redact-001",
        "JobStatus": "COMPLETED",
        "SubmitTime": <SubmitTime>,
        "EndTime": <EndTime>,
        "InputDataConfig": {
            "S3Uri": "s3://ai-ml-services-lab/public/labs/comprehend/pii/input/redact/pii-s3-input.txt",
            "InputFormat": "ONE_DOC_PER_LINE"
        },
        "OutputDataConfig": {
            "S3Uri": "s3://ai-ml-services-lab/public/labs/comprehend/pii/output/redact/<AccountID>-PII-e41101e2f0919a320bc0583a50f86b5f/output/"
        },
        "RedactionConfig": {
            "PiiEntityTypes": [
                "BANK_ACCOUNT_NUMBER",
                "BANK_ROUTING",
                "CREDIT_DEBIT_NUMBER",
                "CREDIT_DEBIT_CVV",
                "CREDIT_DEBIT_EXPIRY",
                "PIN",
                "EMAIL",
                "ADDRESS",
                "NAME",
                "PHONE",
                "SSN"
            ],
            "MaskMode": "REPLACE_WITH_PII_ENTITY_TYPE"
        },
        "LanguageCode": "en",
        "DataAccessRoleArn": "arn:aws:iam::<AccountID>:role/ComprehendBucketAccessRole",
        "Mode": "ONLY_REDACTION"
    }
}

After the job is complete, the output file is plain text (same as the input file). Other Amazon Comprehend asynchronous jobs (start-entities-detection-job) have an output file called output.tar.gz, which is a compressed archive that contains the output of the operation. Start-pii-entities-detection-job retains the folder and file structure as input. Our comprehend-blog-redact-001 job input file pii-s3-input.txt has the respective pii-s3-input.txt.out file with the redacted text in the jobs output folder. You can find the Amazon S3 location in the output from monitoring the job; the JSON element PiiEntitiesDetectionJobProperties.OutputDataConfig.S3uri has the file pii-s3-input.txt.out and the redacted content with PII entity type.

Conclusion

As of this writing, the PII detection feature in Amazon Comprehend is available for US English in the following Regions:

• US East (Ohio)
• US East (N. Virginia)
• US West (Oregon),
• Asia Pacific (Mumbai)
• Asia Pacific (Seoul)
• Asia Pacific (Singapore)
• Asia Pacific (Sydney)
• Asia Pacific (Tokyo)
• EU (Frankfurt)
• EU (Ireland)
• EU (London)
• AWS GovCloud (US-West)

Take a look at the pricing page, give the feature a try, and please send us feedback either via the AWS forum for Amazon Comprehend or through your usual AWS support contacts.

About the Author

Sriharsha M S is an AI/ML specialist solution architect in the Strategic Specialist team at Amazon Web Services. He works with strategic AWS customers who are taking advantage of AI/ML to solve complex business problems. He provides technical guidance and design advice to implement AI/ML applications at scale. His expertise spans application architecture, bigdata, analytics and machine learning.

Read More