Amazon SageMaker Ground Truth is a fully managed data labeling service that makes it easy to build highly accurate training datasets for machine learning. It offers easy access to Amazon Mechanical Turk and private human labelers, and provides them with built-in workflows and interfaces for common labeling tasks.
A labeling team may wish to use the powerful customization features in Ground Truth to modify:
- The look and feel of the workers’ graphical user interface (GUI)
- The backend AWS Lambda functions that perform the preprocessing and postprocessing logic.
Depending on the nature of your labeling job and your use case, your customization requirements may vary.
In this post, via a custom workflow, I show you how to implement a text classification labeling job consisting of a custom GUI, built-in preprocessing and postprocessing logic, and encrypted output. (For our example, workers are tasked to determine whether a sentence references a person, animal, or plant.) I also provide you with an overview of the prerequisites, the code, and estimated costs of implementing the solution.
Understanding task types and processing logic
In this section, I’ll discuss the use cases surrounding built-in vs custom task types and processing logic.
Built-in task types that implement built-in GUIs and built-in processing logic
Ground Truth provides several built-in task types that cover many image, text, video, video frame, and 3D point cloud labeling use cases.
If you want to implement one of these built-in task types, along with a default labeling GUI, creating a labeling job requires no customization steps.
Custom task types that implement custom GUIs and custom processing logic
If the built-in task types don’t satisfy your labeling job requirements, the options for customizing the GUI as well as the preprocessing and postprocessing logic are nearly endless by way of the custom labeling workflow feature.
With this feature, instead of choosing a built-in task type, you define the preprocessing and postprocessing logic via your own Lambda functions. You also have full control over the labeling GUI using HTML elements and the Liquid-based template system. This enables you to do some really cool customization, including Angular framework integration. For more information, see Building a custom Angular application for labeling jobs with Amazon SageMaker Ground Truth.
For more details on custom workflows, see Creating Custom Labeling Workflows and Creating custom labeling jobs with AWS Lambda and Amazon SageMaker Ground Truth.
Built-in task types that implement custom GUIs and built-in processing logic
So far, I’ve discussed the built-in (100% out-of-the-box) option and the custom workflow (100% custom GUI and logic) option for running a job.
What if you wanted to implement a custom GUI, but implement the built-in preprocessing and postprocessing logic that the built-in task types provide? This way, we can adjust the GUI just the way we want, while still relying on the latest AWS-based preprocessing and postprocessing logic (not to mention not having to maintain another codebase).
You can, and I’ll show you how, step-by-step.
Prerequisites
To complete this solution, you need to set up the following prerequisites:
- An AWS account
- The AWS Command Line Interface (AWS CLI)
- AWS Identity and Access Management (IAM) permissions
- An Amazon Simple Storage Service (Amazon S3) bucket
- A Ground Truth work team
- AWS Key Management Service (AWS KMS)
Setting up an AWS account
In this post, you work directly with IAM, SageMaker, AWS KMS, and Amazon S3, so if you haven’t already, create an AWS account. Following along with this post incurs AWS usage charges, so be sure to shut down and delete resources when you’re finished.
Setting up the AWS CLI
Because we use some parameters not available (as of this writing) on the AWS Management Console, you need access to the AWS CLI. For more information, see Installing, updating, and uninstalling the AWS CLI.
All Ground Truth, Amazon S3, and Lambda configurations for this post must be set up within the same Region. This post assumes you’re operating all services out of the us-west-2 region. If you’re operating within another Region, be sure to modify your setup accordingly for a same-Region setup.
Setting up IAM permissions
If you created labeling jobs in the past with Ground Truth, you may already have the permissions needed to implement this solution. Those permissions include the following policies:
- SageMakerFullAccess – To have access to the SageMaker GUI and S3 buckets to perform the steps outlined in this post, you need the SageMakerFullAccess policy applied to the user, group, or role assumed for this post.
- AmazonSageMakerGroundTruthExecution – The Ground Truth labeling jobs you create in this post need to run with an execution role that has the AmazonSageMakerGroundTruthExecution policy attached.
If you have the permissions required to create these roles yourself, the SageMaker GUI walks you through a wizard to set them up. If you don’t have access to create these roles, ask your administrator to create them for you to use during job creation and management.
Setting up an S3 bucket
You need an S3 bucket in the us-west-2 Region to host the SageMaker manifest and categories files for the labeling job. By default, the SageMakerFullAccess and AmazonSageMakerGroundTruthExecution policies only grant access to S3 buckets containing sagemaker or groundtruth in their name (for example, buckets named my-awesome-bucket-sagemaker or marketing-groundtruth-datasets).
Be sure to name your buckets accordingly, or modify the policy accordingly to provide the appropriate access.
For more information on creating a bucket, see Step 1: Create an Amazon S3 Bucket. There is no need for public access to this bucket, so don’t grant it.
As mentioned earlier, all the Ground Truth, Amazon S3, and Lambda configurations for this solution must be in the same Region. For this post, we use us-west-2.
Setting up the Ground Truth work team
When you create a labeling job, you need to assign it to a predefined work team that works on it. If you haven’t created a work team already (or want to create a specific one just for this post), see Create and Manage Workforces.
Setting up AWS KMS
With security as job zero, make sure to encrypt the output manifest file created by the job’s output. To do this, at job creation time, you need to reference a KMS key ID to encrypt the output of the custom Ground Truth job in your S3 bucket.
By default, each account has an AWS managed key (aws/s3) created automatically. For this post, you can use the key ID of the AWS managed key, or you create and use your own customer managed key ID.
For more information about creating and using keys with AWS KMS, see Getting started.
Estimated costs
Running this solution incurs costs for the following:
- Ground Truth labeling – Labeling costs for each job are $0.56 when using your own private workforce (other workforce types, including Mechanical Turk, may have additional costs). For more information, see Amazon SageMaker Ground Truth pricing.
- Amazon S3 storage, retrieval, and data transfer – These costs are less than $0.05 (this assumes you delete all files when you’re finished, and operate the solution for a day or less). For more information, see Amazon S3 pricing.
- Key usage – The cost of an AWS managed KMS key is less than $0.02 for a day’s worth of usage. Storage and usage costs for a customer managed key may be higher. For more information, see AWS Key Management Service pricing.
Setting up the manifest, category, and GUI files
Now that you have met the prerequisites, you can create the manifest, categories, GUI files.
Creating the files
We first create the dataset.manifest file, which we use as the input dataset for the labeling job.
Each object in dataset.manifest contains a line of text describing a person, animal, or plant. One or more of these lines of text is presented as tasks to your workers; they’re responsible for correctly identifying which of the three classifications the line of text best fits.
For this post, dataset.manifest only has seven lines (workers can label up to seven objects), but this input dataset file could have up to 100,000 entries.
Create a file locally named dataset.manifest that contains the following text:
{"source":"His nose could detect over 1 trillion odors!"}
{"source":"Why do fish live in salt water? Because pepper makes them sneeze!"}
{"source":"What did the buffalo say to his son when he went away on a trip? Bison!"}
{"source":"Why do plants go to therapy? To get to the roots of their problems!"}
{"source":"What do you call a nervous tree? A sweaty palm!"}
{"source":"Some kids in my family really like birthday cakes and stars!"}
{"source":"A small portion of the human population carries a fabella bone."}Next, we create the categories.json file. This file is used by Ground Truth to define the categories used to label the data objects.
Create a file locally named categories.json that contains the following code:
{
"document-version": "2018-11-28",
"labels": [{
"label": "person"
},
{
"label": "animal"
},
{
"label": "plant"
}
]
}Finally, we create the worker_gui.html file. This file, when rendered, provides the GUI for the workers’ labeling tasks. The options are endless, but for this post, we create a custom GUI that adds the following custom features:
- An additional Submit button that is styled larger than the default.
- Shortcut keys for submitting and resetting the form.
- JavaScript logic to programatically modify a CSS style (break-all) on the task text output.
Make this custom GUI by creating a file locally named worker_gui.html containing the following code:
<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
<crowd-form>
<crowd-classifier
name="crowd-classifier"
categories="{{ task.input.labels | to_json | escape }}"
header="Please classify"
>
<classification-target>
<strong>{{ task.input.taskObject }}</strong>
</classification-target>
<full-instructions header="Full Instructions">
<div>
<p>Based on the general subject or topic of each sentence presented, please classify it as only one of the following: person, animal, or plant. </p>
</div>
</full-instructions>
<short-instructions>
Complete tasks
</short-instructions>
</crowd-classifier>
</crowd-form>
<script>
document.addEventListener('all-crowd-elements-ready', () => {
// Creating new button to inject in label pane
const button = document.createElement('button');
button.textContent = 'Submit';
button.classList.add('awsui-button', 'awsui-button-variant-primary', 'awsui-hover-child-icons');
// Editing styling to make it larger
button.style.height = '60px';
button.style.width = '100px';
button.style.margin = '15px';
// Adding onclick for submission
const crowdForm = document.querySelector('crowd-form');
button.onclick = () => crowdForm.submit();
// Injecting
const crowdClassifier = document.querySelector('crowd-classifier').shadowRoot;
const labelPane = crowdClassifier.querySelector('.category-picker-wrapper');
labelPane.appendChild(button);
// Adding a Enter hotkey
document.addEventListener('keydown', e => {
if (e.key === 'Enter') {
crowdForm.submit();
}
if (e.key === 'r') {
crowdForm.reset();
}
})
// Implement break-all style in the layout to handle long text tasks
const annotationTarget = crowdClassifier.querySelector('.annotation-area.target');
annotationTarget.style.wordBreak = 'break-all';
});
</script>Previewing the GUI in your web browser
While working on the worker.gui.html file, you may find it useful to preview what you’re building.
At any time you can open the worker_gui.html file from your local file system on your browser for a limited preview of the GUI you’re creating. Some dynamic data, such as that provided by the Lamdba preprocessing functions, may not be visible until you run the job from the job status preview page or worker portal.
To preview with real data, you can create a custom job with Lambda functions. For instructions, see Creating custom labeling jobs with AWS Lambda and Amazon SageMaker Ground Truth. You can preview live from the Ground Truth console’s Create labeling job flow.
For more information about the Liquid-based template system, see Step 2: Creating your custom labeling task template.
Uploading the files to Amazon S3
You can now upload all three files to the root directory of your S3 bucket. When uploading these files to Amazon S3, accept all defaults. For more information, see How Do I Upload Files and Folders to an S3 Bucket?
Creating the custom labeling job
After you upload the files to Amazon S3, you can create your labeling job. For some use cases, the SageMaker console provides the needed interface for creating both built-in and custom workflows. In our use case, we use the AWS CLI because it provides additional options not yet available (as of this writing) on the SageMaker console.
The following scripting instructions assume you’re on MacOS or Linux. If you’re on Windows, you may need to modify the extension and contents of the script for it to work, depending on your environment.
Create a file called createCustom.sh (provide your bucket name, execution role ARN, KMS key ID, and work team ARN):
aws sagemaker create-labeling-job
--labeling-job-name $1
--label-attribute-name "aws-blog-demo"
--label-category-config-s3-uri "s3://YOUR_BUCKET_NAME/categories.json"
--role-arn "YOUR_SAGEMAKER_GROUNDTRUTH_EXECUTION_ROLE_ARN"
--input-config '{
"DataSource": {
"S3DataSource": {
"ManifestS3Uri": "s3://YOUR_BUCKET_NAME/dataset.manifest"
}
}
}'
--output-config '{
"KmsKeyId": "YOUR_KMS_KEY_ID",
"S3OutputPath": "s3://YOUR_BUCKET_NAME/output"
}'
--human-task-config '{
"AnnotationConsolidationConfig": {
"AnnotationConsolidationLambdaArn": "arn:aws:lambda:us-west-2:081040173940:function:ACS-TextMultiClass"
},
"TaskAvailabilityLifetimeInSeconds": 21600,
"TaskTimeLimitInSeconds": 3600,
"NumberOfHumanWorkersPerDataObject": 1,
"PreHumanTaskLambdaArn": "arn:aws:lambda:us-west-2:081040173940:function:PRE-TextMultiClass",
"WorkteamArn": "YOUR_WORKTEAM_ARN",
"TaskDescription": "Select all labels that apply",
"MaxConcurrentTaskCount": 1000,
"TaskTitle": "Text classification task",
"UiConfig": {
"UiTemplateS3Uri": "s3://YOUR_BUCKET_NAME/worker_gui.html"
}
}'Make sure to use your work team ARN, not your workforce ARN. For your KMS key, use the key ID or the AWS managed or customer managed key you want to encrypt the output with. For instructions on retrieving your key, see Finding the key ID and ARN. For more information about types of KMS keys, see Customer master keys (CMKS).
Make the file executable via the command chmod 700 createCustom.sh.
Almost done! But before we run the script, let’s step through what this script is doing in more detail. The script runs the aws sagemaker create-lableing-job CLI command with the following parameters:
- –labeling-job-name – We set this value to $1, which translates to the argument we pass on the command line when we run it.
- –label-attribute-name – The attribute name to use for the label in the output manifest file.
- –label-category-config-s3-url – The path to the
categories.jsonfile we previously uploaded to Amazon S3. - –role-arn – The ARN of the IAM role SageMaker runs the job under. If you aren’t sure what this value is, your administrator should be able to provide it to you.
- –input-config – Points to the location of the input dataset manifest file.
- –output-config – Points to a KMS key ID and the job’s output path.
- –human-task-config – Provides the following parameters:
- PreHumanTaskLambdaArn – The built-in AWS-provided Lambda function that performs the same preprocessing logic as that found in the built-in text classification job type. It handles reading the dataset manifest file in Amazon S3, parsing it, and providing the GUI with the appropriate task data.
- AnnotationConsolidationLambdaArn – The built-in AWS-provided Lambda function that performs the same postprocessing logic as that found in the built-in text classification job type. It handles postprocessing of the data after each labeler submits an answer. As a reminder, all Ground Truth, Amazon S3, and Lambda configurations for this post must be set up within the same Region (for this post,
us-west-2). For nonus-west-2Lambda ARN options, see create-labeling-job. - TaskAvailabilityLifetimeInSeconds – The length of time that a task remains available for labeling by human workers.
- TaskTimeLimitInSeconds – The amount of time that a worker has to complete a task.
- NumberOfHumanWorkersPerDataObject – The number of human workers that label an object.
- WorkteamArn – The ARN of the work team assigned to complete the tasks. Make sure to use your work team ARN and not your workforce ARN in the script.
- TaskDescription – A description of the task for your human workers.
- MaxConcurrentTaskCount – Defines the maximum number of data objects that can be labeled by human workers at the same time.
- TaskTitle – A title for the task for your human workers.
- UiTemplateS3Uri – The S3 bucket location of the GUI template that we uploaded earlier. This is the HTML template used to render the worker GUI for labeling job tasks.
For more information about the options available when creating a labeling job from the AWS CLI, see create-labeling-job.
Running the job
Now that you’ve created the script with all the proper parameters, its time to run it! To run the script, enter ./createCustom.sh JOBNAME from the command line, providing a unique name for the job.
In my example, I named the job gec-custom-template-300, and my command line looked like the following:
gcohen $: ./createCustom.sh gec-custom-template-300
{
"LabelingJobArn": "arn:aws:sagemaker:us-west-2:xxyyzz:labeling-job/gec-custom-template-300"
}Checking the job status and previewing the GUI
Now that we’ve submitted the job, we can easily check its status on the console.
- On the SageMaker console, under Ground Truth, choose Labeling jobs.
You should see the job we just submitted.
- Choose the job to get more details.
- Choose View labeling tool to preview what our labeling workers see when they take the job.
In addition, by using AWS KMS encryption, you can specify authorized users who can decrypt the output manifest file. Who exactly is authorized to decrypt this file varies depending on whether the key is customer managed or AWS managed. For specifics on access permissions for a given key, review the key’s key policy.
Conclusion
In this post, I demonstrated how to implement a custom labeling GUI with built-in preprocessing and postprocessing logic by way of a custom workflow. I also demonstrated how to encrypt the output with AWS KMS. The prerequisites, code, and estimated costs of running it all were also provided.
The code was provided to get you running quickly, but don’t stop there! Try experimenting by adding additional functionality to your workers’ labeling GUIs, either with your own custom libraries or third-party logic. If you get stuck, don’t hesitate to reach out directly, or post an issue on our GitHub repo issues page.
About the Author
Geremy Cohen is a Solutions Architect with AWS where he helps customers build cutting-edge, cloud-based solutions. In his spare time, he enjoys short walks on the beach, exploring the bay area with his family, fixing things around the house, breaking things around the house, and BBQing.
Abhinav Jawadekar is a Senior Partner Solutions Architect at Amazon Web Services. Abhinav works with AWS partners to help them in their cloud journey.
Yunzhi Shi is a data scientist at the Amazon ML Solutions Lab where he helps AWS customers address business problems with AI and cloud capabilities. Recently, he has been building computer vision, search, and forecast solutions for various customers.
Talia Chopra is a Technical Writer in AWS specializing in machine learning and artificial intelligence. She works with multiple teams in AWS to create technical documentation and tutorials for customers using Amazon SageMaker, MxNet, and AutoGluon.
Phil Cunliffe is an engineer turned Software Development Manager for Amazon Human in the Loop services. He is a JavaScript fanboy with an obsession for creating great user experiences.
Azim Siddique serves as Technical Advisor and CoE Architect at Foxconn. He provides architectural direction for the Digital Transformation program, conducts PoCs with emerging technologies, and guides engineering teams to deliver business value by leveraging digital technologies at scale.
Felice Chuang is a Data Architect at Foxconn. She uses her diverse skillset to implement end-to-end architecture and design for big data, data governance, and business intelligence applications. She supports analytic workloads and conducts PoCs for Digital Transformation programs.
Yash Shah is a data scientist in the Amazon ML Solutions Lab, where he works on a range of machine learning use cases from healthcare to manufacturing and retail. He has a formal background in Human Factors and Statistics, and was previously part of the Amazon SCOT team designing products to guide 3P sellers with efficient inventory management.
Dan Volk is a Data Scientist at Amazon ML Solutions Lab, where he helps AWS customers across various industries accelerate their AI and cloud adoption. Dan has worked in several fields including manufacturing, aerospace, and sports and holds a Masters in Data Science from UC Berkeley.
Rodrigo Alarcon is a Sr. Solutions Architect with AWS based out of Santiago, Chile. Rodrigo has over 10 years of experience in IT security and network infrastructure. His interests include machine learning and cybersecurity.
Surya Kari is a Data Scientist who works on AI devices within AWS. His interests lie in computer vision and autonomous systems.