A New Lens on Understanding Generalization in Deep Learning

Hanie Sedghi, Google Research and Preetum Nakkiran, Harvard University

Understanding generalization is one of the fundamental unsolved problems in deep learning. Why does optimizing a model on a finite set of training data lead to good performance on a held-out test set? This problem has been studied extensively in machine learning, with a rich history going back more than 50 years. There are now many mathematical tools that help researchers understand generalization in certain models. Unfortunately, most of these existing theories fail when applied to modern deep networks — they are both vacuous and non-predictive in realistic settings. This gap between theory and practice is largest for overparameterized models, which in theory have the capacity to overfit their train sets, but often do not in practice.

In “The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers”, accepted at ICLR 2021, we present a new framework for approaching this problem by connecting generalization to the field of online optimization. In a typical setting, a model trains on a finite set of samples, which are reused for multiple epochs. But in online optimization, the model has access to an infinite stream of samples, and can be iteratively updated while processing this stream. In this work, we find that models that train quickly on infinite data are the same models that generalize well if they are instead trained on finite data. This connection brings new perspectives on design choices in practice, and lays a roadmap for understanding generalization from a theoretical perspective.

The Deep Bootstrap Framework
The main idea of the Deep Bootstrap framework is to compare the real world, where there is finite training data, to an “ideal world”, where there is infinite data. We define these as:

  • Real World (N, T): Train a model on N train samples from a distribution, for T minibatch stochastic gradient descent (SGD) steps, re-using the same N samples in multiple epochs, as usual. This corresponds to running SGD on the empirical loss (loss on training data), and is the standard training procedure in supervised learning.
  • Ideal World (T): Train the same model for T steps, but use fresh samples from the distribution in each SGD step. That is, we run the exact same training code (same optimizer, learning-rates, batch-size, etc.), but sample a fresh train set in each epoch instead of reusing samples. In this ideal world setting, with an effectively infinite “train set”, there is no difference between train error and test error.
Test soft-error for ideal world and real world during SGD iterations for ResNet-18 architecture. We see that the two errors are similar.

A priori, one might expect the real and ideal worlds may have nothing to do with each other, since in the real world the model sees a finite number of examples from the distribution while in the ideal world the model sees the whole distribution. But in practice, we found that the real and ideal models actually have similar test error.

In order to quantify this observation, we simulated an ideal world setting by creating a new dataset, which we call CIFAR-5m. We trained a generative model on CIFAR-10, which we then used to generate ~6 million images. The scale of the dataset was chosen to ensure that it is “virtually infinite” from the model’s perspective, so that the model never resamples the same data. That is, in the ideal world, the model sees an entirely fresh set of samples.

Samples from CIFAR-5m

The figure below presents the test error of several models, comparing their performance when trained on CIFAR-5m data in the real world setting (i.e., re-used data) and the ideal world (“fresh” data). The solid blue line shows a ResNet model in the real world, trained on 50K samples for 100 epochs with standard CIFAR-10 hyperparameters. The dashed blue line shows the corresponding model in the ideal world, trained on 5 million samples in a single pass. Surprisingly, these worlds have very similar test error — the model in some sense “doesn’t care” whether it sees re-used samples or fresh ones.

The real world model is trained on 50K samples for 100 epochs, and the ideal world model is trained on 5M samples for a single epoch. The lines show the test error vs. the number of SGD steps.

This also holds for other architectures, e.g., a Multi-Layer-Perceptron (red), a Vision Transformer (green), and across many other settings of architecture, optimizer, data distribution, and sample size. These experiments suggest a new perspective on generalization: models that optimize quickly (on infinite data), generalize well (on finite data). For example, the ResNet model generalizes better than the MLP model on finite data, but this is “because” it optimizes faster even on infinite data.

Understanding Generalization from Optimization Behavior
The key observation is that real world and ideal world models remain close, in test error, for all timesteps, until the real world converges (< 1% train error). Thus, one can study models in the real world by studying their corresponding behavior in the ideal world.

This means that the generalization of the model can be understood in terms of its optimization performance under two frameworks:

  1. Online Optimization: How fast the ideal world test error decreases
  2. Offline Optimization: How fast the real world train error converges

Thus, to study generalization, we can equivalently study the two terms above, which can be conceptually simpler, since they only involve optimization concerns. Based on this observation, good models and training procedures are those that (1) optimize quickly in the ideal world and (2) do not optimize too quickly in the real world.

All design choices in deep learning can be viewed through their effect on these two terms. For example, some advances like convolutions, skip-connections, and pre–training help primarily by accelerating ideal world optimization, while other advances like regularization and data-augmentation help primarily by decelerating real world optimization.

Applying the Deep Bootstrap Framework
Researchers can use the Deep Bootstrap framework to study and guide design choices in deep learning. The principle is: whenever one makes a change that affects generalization in the real world (the architecture, learning-rate, etc.), one should consider its effect on (1) the ideal world optimization of test error (faster is better) and (2) the real world optimization of train error (slower is better).

For example, pre-training is often used in practice to help generalization of models in small-data regimes. However, the reason that pre-training helps remains poorly understood. One can study this using the Deep Bootstrap framework by looking at the effect of pre-training on terms (1) and (2) above. We find that the primary effect of pre-training is to improve the ideal world optimization (1) — pre-training turns the network into a “fast learner” for online optimization. The improved generalization of pretrained models is thus almost exactly captured by their improved optimization in the ideal world. The figure below shows this for Vision-Transformers (ViT) trained on CIFAR-10, comparing training from scratch vs. pre-training on ImageNet.

Effect of pre-training — pre-trained ViTs optimize faster in the ideal world.

One can also study data-augmentation using this framework. Data-augmentation in the ideal world corresponds to augmenting each fresh sample once, as opposed to augmenting the same sample multiple times. This framework implies that good data-augmentations are those that (1) do not significantly harm ideal world optimization (i.e., augmented samples don’t look too “out of distribution”) or (2) inhibit real world optimization speed (so the real world takes longer to fit its train set).

The main benefit of data-augmentation is through the second term, prolonging the real world optimization time. As for the first term, some aggressive data augmentations (mixup/cutout) can actually harm the ideal world, but this effect is dwarfed by the second term.

Concluding Thoughts
The Deep Bootstrap framework provides a new lens on generalization and empirical phenomena in deep learning. We are excited to see it applied to understand other aspects of deep learning in the future. It is especially interesting that generalization can be characterized via purely optimization considerations, which is in contrast to many prevailing approaches in theory. Crucially, we consider both online and offline optimization, which are individually insufficient, but that together determine generalization.

The Deep Bootstrap framework can also shed light on why deep learning is fairly robust to many design choices: many kinds of architectures, loss functions, optimizers, normalizations, and activation functions can generalize well. This framework suggests a unifying principle: that essentially any choice that works well in the online optimization setting will also generalize well in the offline setting.

Finally, modern neural networks can be either overparameterized (e.g., large networks trained on small data tasks) or underparmeterized (e.g., OpenAI’s GPT-3, Google’s T5, or Facebook’s ResNeXt WSL). The Deep Bootstrap framework implies that online optimization is a crucial factor to success in both regimes.

Acknowledgements
We are thankful to our co-author, Behnam Neyshabur, for his great contributions to the paper and valuable feedback on the blog. We thank Boaz Barak, Chenyang Yuan, and Chiyuan Zhang for helpful comments on the blog and paper.

Read More

Batch image processing with Amazon Rekognition Custom Labels 

Amazon Rekognition is a computer vision service that makes it easy to add image and video analysis to your applications using proven, highly scalable, deep learning technology that requires no machine learning (ML) expertise to use. With Amazon Rekognition, you can identify objects, people, text, scenes, and activities in images and videos, as well as detect any inappropriate content. Amazon Rekognition also provides highly accurate facial analysis and facial search capabilities that you can use to detect, analyze, and compare faces for a wide variety of use cases.

Amazon Rekognition Custom Labels allows you to identify the objects and scenes in images that are specific to your business needs. For example, you can find your logo in social media posts, identify your products on store shelves, classify machine parts in an assembly line, distinguish healthy and infected plants, and more. The blog post Building your own brand detection shows how to use Amazon Rekognition Custom Labels to build an end-to-end solution to detect brand logos in images and videos.

Amazon Rekognition Custom Labels provides a simple end-to-end experience where you start by labeling a dataset, and Amazon Rekognition Custom Labels builds a custom ML model for you by inspecting the data and selecting the right ML algorithm. After your model is trained, you can start using it immediately for image analysis. If you want to process images in batches (such as once a day or week, or at scheduled times during the day), you can provision your custom model at scheduled times.

In this post, we show how you can build a cost-optimal batch solution with Amazon Rekognition Custom Labels that provisions your custom model at scheduled times, processes all your images, and deprovisions your resources to avoid incurring extra cost.

Overview of solution

The following architecture diagram shows how you can design a cost-effective and highly scalable workflow to process images in batches with Amazon Rekognition Custom Labels. It takes advantage of AWS services such as Amazon EventBridge, AWS Step Functions, Amazon Simple Queue Service (Amazon SQS), AWS Lambda, and Amazon Simple Storage Service (Amazon S3).

This solution uses a serverless architecture and managed services, so it can scale on demand and doesn’t require provisioning and managing any servers. The Amazon SQS queue increases the overall fault tolerance of the solution by decoupling image ingestion from the image processing and enabling reliable delivery of messages for each ingested image. Step Functions makes it easy to build visual workflows to orchestrate a series of individual tasks, such as checking if an image is available for processing and managing the state lifecycle of the Amazon Rekognition Custom Labels project. Although the following architecture shows how you can build a batch processing solution for Amazon Rekognition Custom Labels using AWS Lambda, you can build a similar architecture using services such as AWS Fargate.

The following steps describe the overall workflow:

  1. As an image is stored in Amazon S3 bucket, it triggers a message that gets stored in an Amazon SQS queue.
  2. Amazon EventBridge is configured to trigger an AWS Step Functions workflow at a certain frequency (1 hour by default).
  3. As the workflow runs, it performs the following actions:
    1. It checks the number of items in the Amazon SQS queue. If there are no items to process in the queue, the workflow ends.
    2. If there are items to process in the queue, the workflow starts the Amazon Rekognition Custom Labels model.
    3. The workflow enables Amazon SQS integration with an AWS Lambda function to process those images.
  4. As the integration between the Amazon SQS queue and AWS Lambda is enabled, the following events occur:
    1. AWS Lambda starts processing messages with the image details from Amazon SQS.
    2. The AWS Lambda function uses the Amazon Rekognition Custom Labels project to process the images.
    3. The AWS Lambda function then places the JSON file containing the inferenced labels in the final bucket. The image is also moved from the source bucket to the final bucket.
  5. When all the images are processed, the AWS Step Functions workflow does the following:
    1. It stops the Amazon Rekognition Custom Labels model.
    2. It disables integration between the Amazon SQS queue and the AWS Lambda function by disabling the trigger.

The following diagram illustrates the AWS Step Functions state machine for this solution.

Prerequisites

To deploy this solution, you need the following prerequisites:

  • An AWS account with permission to deploy the solution using AWS CloudFormation, which creates AWS Identity and Access Management (IAM) roles and other resources.
  •  The Amazon Resource Name (ARN) of the Amazon Rekognition Custom Labels project (referred as ProjectArn) and the Amazon Resource Name (ARN) of the model version that was created after training the model (referenced as ProjectVersionArn). These values are required to check the status of the model and also to analyze images using the model.

To learn how to train a model, see Getting Started with Amazon Rekognition Custom Labels.

Deployment

To deploy the solution using AWS CloudFormation in your AWS account, follow the steps in the GitHub repo. It creates the following resources:

  • Amazon S3 bucket
  • Amazon SQS queue
  • AWS Step Functions workflow
  • Amazon EventBridge rules to trigger the workflow
  • IAM roles
  • AWS Lambda Functions

You can see the names of different resources created by the solution in the output section of the CloudFormation stack.

Testing the workflow

To test your workflow, complete the following steps:

  1. Upload sample images to the input S3 bucket that was created by the solution (for example, xxxx-sources3bucket-xxxx).
  2. On the Step Functions console, choose the state machine created by the solution (for example, CustomCVStateMachine-xxxx).

You should see the state machine is triggered by the Amazon EventBridge rule every hour.

  1. You can manually start the workflow by choosing Start execution.
  2. As images are processed, you can go to the output S3 bucket (for example, xxxx-finals3bucket-xxxx) to see the JSON output for each image.

The following screenshot shows the contents of the final S3 bucket with the images, along with their corresponding JSON output from Amazon Rekognition Custom Labels.

Conclusion

In this post, we showed how you can build a cost-optimal batch solution with Amazon Rekognition Custom Labels that can provision your custom model at scheduled times, process all your images, and deprovision your resources to avoid incurring extra cost. Depending on your use case, you can easily adjust the scheduled time window at which the solution should process the batch. For more information about how to create, train, evaluate, and use a model that detects objects, scenes, and concepts in images see getting started with Amazon Rekognition Custom Labels.

While the solution described in this post showed how you can process batch images with Amazon Rekognition Custom Labels, you can easily tweak the solution to process batch images with Amazon Lookout for Vision for defects and anomalies detection. With Amazon Lookout for Vision, manufacturing companies can increase quality and reduce operational costs by quickly identifying differences in images of objects at scale. For example, Amazon Lookout for Vision can be used to identify missing components in products, damage to vehicles or structures, irregularities in production lines, minuscule defects in silicon wafers, and other similar problems. To learn more about Amazon Lookout for Vision, see the developer guide.

About the Authors

Rahul Srivastava is a Senior Solutions Architect at Amazon Web Services and is based in the United Kingdom. He has extensive architecture experience working with large enterprise customers. He is helping our customers with architecture, cloud adoption, developing products with a purpose and take advantage of AI/ ML to solve real world business problems.

 

Kashif Imran is a Principal Solutions Architect at Amazon Web Services. He works with some of the largest AWS customers who are taking advantage of AI/ML to solve complex business problems. He provides technical guidance and design advice to implement computer vision applications at scale. His expertise spans application architecture, serverless, containers, NoSQL and machine learning.

Read More

Translate video captions and subtitles using Amazon Translate

Video is a highly effective a highly effective way to educate, entertain, and engage users. Your company might carry a large collection of videos that include captions or subtitles. To make these videos accessible to a larger audience, you can provide translated captions and subtitles in multiple languages. In this post, we show you how to create an automated and serverless pipeline to translate captions and subtitles using Amazon Translate, without losing their context during translation.

Captions and subtitles help make videos accessible for those hard of hearing, provide flexibility to users in noisy or quiet environments, and assist non-native speakers. Captions or subtitles are normally represented in SRT (.srt) or WebVTT (.vtt) format. SRT stands for SubRip Subtitle, and is the most common file format for subtitles and captions. WebVTT stands for Web Video Text Track, and is becoming a popular format for the same purpose.

Multi-language video subtitling and captioning solution

This solution uses Amazon Translate, a neural machine translation service that delivers fast, high-quality, and affordable language translation. Amazon Translate supports the ability to ignore tags and only translate text content in HTML documents. The following diagram illustrates the workflow of our solution.

The following diagram illustrates the workflow of our solution.

The workflow includes the following steps:

  1. Extract caption text from a WebVTT or SRT file and create a delimited text file using an HTML tag.
  2. Translate this delimited file using the asynchronous batch processing capability in Amazon Translate.
  3. Recreate the WebVTT or SRT files using the translated delimited file.

We provide a more detailed architecture in the next section.

Solution architecture

This solution is based on an event-driven and serverless pipeline architecture, and uses managed services so that it’s scalable and cost-effective. The following diagram illustrates the serverless pipeline architecture.

The following diagram illustrates the serverless pipeline architecture.

The pipeline contains the following steps:

  1. Users upload one or more caption files in the WebVTT (.vtt) or the SRT (.srt) format to an Amazon Simple Storage Service (Amazon S3) bucket.
  2. The upload triggers an AWS Lambda function.
  3. The function extracts text captions from each file, creates a corresponding HTML tag delimited text file, and stores them in Amazon S3.
  4. The function invokes Amazon Translate in batch mode to translate the delimited text files into the target language.
  5. The AWS Step Functions based job poller polls for the translation job to complete.
  6. The Step Functions workflow sends an Amazon Simple Notification Service (Amazon SNS) notification when the translation is complete.
  7. A Lambda function reads the translated delimited files from Amazon S3, creates the caption files in the WebVTT (.vtt) or SRT(.srt) format with the translated text captions, and stores them back in Amazon S3.

We explain Steps 3–7 in more detail in the following sections.

Convert caption files to delimited files

In this architecture, uploading the file with triggerFileName triggers the Lambda function <Stack name>-S3CaptionsFileEventProcessor-<Random string>. The function iterates through the WebVTT and SRT files in the input folder and for each file, it extracts the caption text, converts it into a delimited text file using an HTML (<span>) tag, and places it in the captions-in folder of the Amazon S3 bucket. See the following function code:

try:
        captions = Captions()
        #filter only the VTT and SRT file for processing in the input folder
        objs = S3Helper().getFilteredFileNames(bucketName,"input/",["vtt","srt"])
        for obj in objs:
            try:
                vttObject = {}
                vttObject["Bucket"] = bucketName
                vttObject["Key"] = obj
                captions_list =[]
                #based on the file type call the method that coverts them into python list object
                if(obj.endswith("vtt")):
                    captions_list =  captions.vttToCaptions(vttObject)
                elif(obj.endswith("srt")):
                    captions_list =  captions.srtToCaptions(vttObject)
                #convert the text captions in the list object to a delimited file
                delimitedFile = captions.ConvertToDemilitedFiles(captions_list)
                fileName = obj.split("/")[-1]
                newObjectKey = "captions-in/{}.delimited".format(fileName)
                S3Helper().writeToS3(str(delimitedFile),bucketName,newObjectKey)   
                output = "Output Object: {}/{}".format(bucketName, newObjectKey)

The solution uses a Python library webvtt-py to load, parse, and generate the WebVTT and SRT file formats. All the operations related to the library are abstracted within the Captions module. Also, all Amazon S3 operations are abstracted within the S3Helper module.

Batch translation of delimited files

After the delimited files are stored in the captions-in folder of the Amazon S3 bucket, the Lambda function <Stack name>-S3CaptionsFileEventProcessor-<Random string> invokes the Amazon Translate job startTextTranslationJob with the following parameters:

  • The captions-in folder in the S3 bucket is the input location for files to be translated
  • The captions-out folder in the S3 bucket is the output location for translated files
  • Source language code
  • Destination language code
  • An AWS Identity and Access Management (IAM) role ARN with necessary policy permissions to read and write to the S3 bucket

See the following job code:

translateContext = {}
translateContext["sourceLang"] = sourceLanguageCode
translateContext["targetLangList"] = [targetLanguageCode]
translateContext["roleArn"] = access_role 
translateContext["bucket"] = bucketName
translateContext["inputLocation"] = "captions-in/"
translateContext["outputlocation"] = "captions-out/"
translateContext["jobPrefix"] = "TranslateJob-captions"
#Call Amazon Translate to translate the delimited files in the captions-in folder
jobinfo = captions.TranslateCaptions(translateContext)

Poll the Amazon Translate batch translate job

The solution uses a Step Functions workflow to periodically poll the Amazon Translate service for the status of the submitted job using a Lambda function. When the job is complete, the workflow creates an Amazon SNS notification with details of the Amazon Translate job as the notification payload. For more details on the Step Functions job definition and the Lambda code, see Getting a batch job completion message from Amazon Translate.

Create WebVTT and SRT files from the delimited files

The Amazon SNS notification from the job poller step triggers the Lambda function <Stack name>-TranslateCaptionsJobSNSEventProcessor-<Random string>. The function iterates through the each of the translated delimited files generated in the captions-out folder based on the event details available from the Amazon SNS notification event. See the following code:

output = ""
    logger.info("request: {}".format(request))
    up = urlparse(request["s3uri"], allow_fragments=False)
    accountid = request["accountId"]
    jobid =  request["jobId"]
    bucketName = up.netloc
    objectkey = up.path.lstrip('/')
    basePrefixPath = objectkey  + accountid + "-TranslateText-" + jobid + "/";
    languageCode = request["langCode"]
    logger.debug("Base Prefix Path:{}".format(basePrefixPath))
    captions = Captions()
    #filter only the delimited files with .delimited suffix
    objs = S3Helper().getFilteredFileNames(bucketName,basePrefixPath,["delimited"])
    for obj in objs:
        try:
            #Read the Delimited file contents
            content = S3Helper().readFromS3(bucketName,obj)
            fileName = FileHelper().getFileName(obj)

The solution generates the WebVTT or SRT file using the original WebVTT or SRT file from the input folder for the time markers, but replaces the captions with the translated caption text from the delimited files. See the following code:

logger.debug("SourceFileKey:{}.processed".format(sourceFileName))
            soureFileKey = "input/{}.processed".format(sourceFileName)
            vttObject = {}
            vttObject["Bucket"] = bucketName
            vttObject["Key"] = soureFileKey
            captions_list = []
            #Based on the file format, call the right method to load the file as python object
            if(fileName.endswith("vtt")):
                    captions_list =  captions.vttToCaptions(vttObject)
            elif(fileName.endswith("srt")):
                captions_list =  captions.srtToCaptions(vttObject)
            # Replace the text captions with the translated content
            translatedCaptionsList = captions.DelimitedToWebCaptions(captions_list,content,"<span>",15)
            translatedText = ""
            # Recreate the Caption files in VTT or SRT format
            if(fileName.endswith("vtt")):
                translatedText =  captions.captionsToVTT(translatedCaptionsList)
            elif(fileName.endswith("srt")):
                translatedText =  captions.captionsToSRT(translatedCaptionsList)

The function then writes the new WebVTT or SRT files as S3 objects in the output folder with the following naming convention: TargetLanguageCode-<inputFileName>.vtt or TargetLanguageCode-<inputFileName>.srt. See the following code:

newObjectKey = "output/{}".format(fileName)
# Write the VTT or SRT file into the output S3 folder
S3Helper().writeToS3(str(translatedText),bucketName,newObjectKey)

Solution deployment

You can either deploy the solution using an AWS CloudFormation template or by cloning the GitHub repository.

Deployment using the CloudFormation template

The CloudFormation template provisions the necessary resources needed for the solution, including the IAM roles, IAM policies, and Amazon SNS topics. The template creates the stack the us-east-1 Region.

  1. Launch the CloudFormation template by choosing Launch Stack:

  1. For Stack name, enter a unique stack name for this account; for example, translate-captions-stack.
  2. For SourceLanguageCode, enter the language code for the current language of the caption text; for example, en for English.
  3. For TargetLanguageCode, enter the language code that you want your translated text in; for example, es for Spanish.

For more information about supported languages, see Supported Languages and Language Codes.

  1. For TriggerFileName, enter the name of the file that triggers the translation serverless pipeline (the default is triggerfile).
  2. In the Capabilities and transforms section, and select the check boxes to acknowledge that CloudFormation will create IAM resources and transform the AWS Serverless Application Model (AWS SAM) template.

AWS SAM templates simplify the definition of resources needed for serverless applications. When deploying AWS SAM templates in AWS CloudFormation, AWS CloudFormation performs a transform to convert the AWS SAM template into a CloudFormation template. For more information, see Transform.

  1. Choose Create stack.

Choose Create stack.

The stack creation may take up to 10 minutes, after which the status changes to CREATE_COMPLETE. You can see the name of the newly created S3 bucket along with other AWS resources created on the Outputs tab.

You can see the name of the newly created S3 bucket along with other AWS resources created on the Outputs tab.

Deployment using the GitHub repository

To deploy the solution using GitHub, visit the GitHub repo and follow the instructions in the README.md file. The solution uses AWS SAM to make it easy to deploy in your AWS account.

Test the solution

To test the solution, upload one or more WebVTT (.vtt) or SRT (.srt) files to the input folder. Because this is a batch operation, we recommend uploading multiple files at the same time. The following code shows a sample SRT file:

1
00:00:00,500 --> 00:00:07,000
Hello. My name is John Doe. Welcome to the blog demonstrating the ability to

2
00:00:07,000 --> 00:00:11,890
translate from one language to another using Amazon Translate. Amazon Translate is a neural machine translation service that delivers fast, high-quality, and affordable language translation. 

3
00:00:11,890 --> 00:00:16,320
Neural machine translation is a form of language translation automation that uses deep learning models to deliver more accurate and natural-sounding translation than traditional statistical and rule-based translation algorithms.

4
00:00:16,320 --> 00:00:21,580
The translation service is trained on a wide variety of content across different use cases and domains to perform well on many kinds of content.

5
00:00:21,580 --> 00:00:23,880
Its asynchronous batch processing capability enables you to translate a large collection of text or HTML documents with a single API call.

After you upload all the WebVTT or SRT documents, upload the file that triggers the translation workflow. This file can be a zero-byte file, but the filename should match the TriggerFileName parameter in the CloudFormation stack. The default name for the file is triggerfile.

After you upload all the WebVTT or SRT documents, upload the file that triggers the translation workflow.

After a short time (15–20 minutes), check the output folder to see the WebVTT or SRT files with the following naming convention: TargetLanguageCode-<inputFileName>.vtt or TargetLanguageCode-<inputFileName>.srt.

After a short time (15–20 minutes), check the output folder to see the WebVTT or SRT files

The following snippet shows the SRT file translated into Spanish:

1
00:00:00,500 --> 00:00:07,000
Hola. Mi nombre es John Doe. Bienvenido al blog que demuestra la capacidad de

2
00:00:07,000 --> 00:00:11,890
traducir de un idioma a otro utilizando Amazon Translate. Amazon Translate es un servicio de traducción automática neuronal que ofrece traducción de idiomas rápida, de alta calidad y asequible. 

3
00:00:11,890 --> 00:00:16,320
La traducción automática neuronal es una forma de automatización de la traducción de idiomas que utiliza modelos de aprendizaje profundo para ofrecer una traducción más precisa y natural que los algoritmos de traducción basados en reglas y estadísticas tradicionales. 

4
00:00:16,320 --> 00:00:21,579
El servicio de traducción está capacitado en una amplia variedad de contenido en diferentes casos de uso y dominios para funcionar bien en muchos tipos de contenido. 

5
00:00:21,579 --> 00:00:23,879
Su capacidad de procesamiento por lotes asincrónico le permite traducir una gran colección de documentos de texto o HTML con una sola llamada a la API.

You can monitor the progress of the solution pipeline by checking the Amazon CloudWatch logs generated for each Lambda function that is part of the solution. For more information, see Accessing Amazon CloudWatch logs for AWS Lambda.

To do a translation for a different source-target language combination, you can update the SOURCE_LANG_CODE and TARGET_LANG_CODE environment variable for the <Stack name>-S3CaptionsFileEventProcessor-<Random string> function and trigger the solution pipeline by uploading WebVTT or SRT documents and the TriggerFileName into the input folder.

To do a translation for a different source-target language combination, you can update the SOURCE_LANG_CODE and TARGET_LANG_CODE environment variable

Conclusion

In this post, we demonstrated how to translate video captions and subtitles in WebVTT and SRT file formats using Amazon Translate asynchronous batch processing. This process can be used in several industry verticals, including education, media and entertainment, travel and hospitality, healthcare, finance, law, or any organization with a large collection of subtitled or captioned video assets that wants these translated to their customers in multiple languages.

You can easily integrate the approach into your own pipelines as well as handle large volumes of caption and subtitle text with this scalable architecture. This methodology works for translating captions and subtitles between over 70 languages supported by Amazon Translate (as of this writing). Because this solution uses asynchronous batch processing, you can customize your machine translation output using parallel data. For more information on using parallel data, see Customizing Your Translations with Parallel Data (Active Custom Translation). For a low-latency, low-throughput solution translating smaller caption files, you can perform the translation through the real-time Amazon Translate API. For more information, see Translating documents with Amazon Translate, AWS Lambda, and the new Batch Translate API. If your organization has a large collection of videos that need to be captioned or subtitled, you can use this AWS Subtitling solution.

About the Authors

Siva Rajamani is a Boston-based Enterprise Solutions Architect at AWS. He enjoys working closely with customers and supporting their digital transformation and AWS adoption journey. His core areas of focus are serverless, application integration, and security. Outside of work, he enjoys outdoors activities and watching documentaries.

 

 

Raju Penmatcha is a Senior AI/ML Specialist Solutions Architect at AWS. He works with education, government, and non-profit customers on machine learning and artificial intelligence related projects, helping them build solutions using AWS. Outside of work, he likes exploring new places.

Read More

Is AI Important to Financial Services’ Future? New Survey Says You Can Bank on It

Financial services companies are challenged with defining and executing their AI strategy.

AI solutions contribute to both the top and bottom line for firms by powering nearly every function, including customer service, cybersecurity, new account acquisition and regulatory compliance.

Everyone from executives to data scientists are involved with determining how much to invest, the most profitable use cases to pursue and the biggest challenges that must be overcome in 2021 and beyond.

These are some of the findings of NVIDIA’s recent survey of over 200 financial services professionals from around the world. To fill in a more complete picture of how financial services institutions are using AI, and where it’s headed, our “State of AI in Financial Services” survey consisted of questions covering a range of AI topics, such as deployment models, infrastructure spending, top use cases and biggest challenges. Respondents included C-suite leaders, managers, developers and IT architects from fintechs, investment firms and retail banks.

Getting a Pulse on AI in Financial Services

The survey results showed two consistent themes: AI provides a competitive advantage in financial services, and banks plan to invest significantly in AI infrastructure to unlock its full potential.

Among different roles and subsectors within the industry, the survey data showed finer differences in how AI can best be deployed and the specific challenges for business decision makers and technical implementers.

Three highlights stood out among the survey results:

AI-Enabled Services Grow Revenue and Cut Costs

Our respondents were in widespread agreement on the value of enterprise AI, as 83 percent agreed with the statement that “AI is important to my company’s future success.”

The survey results showed how financial services firms view AI as an enabler of growth opportunities. Over half of those surveyed who had an opinion stated AI will increase their company’s annual revenue by 10 percent or more. In contrast, only 12 percent of respondents — excluding those who marked “Don’t Know” — stated that AI is having no impact on their revenue growth.

AI can also improve the bottom line of financial services institutions through cost savings. For instance, banks, insurers and asset managers are creating significant efficiencies in their daily operations using technologies such as conversational AI, robotic process automation, optical character recognition and other machine learning and deep learning applications.

These AI services save time and reduce expenditures by automating insurance claims processing, augmenting call center agents via automated speech recognition for call transcription and carrying out other manually intensive services.

Passing AI Benefits to Customers

Survey respondents said the top three areas where AI affected their companies were yielding more accurate models (42 percent), creating a competitive advantage (41 percent) and building new products (34 percent).

Utilizing AI to create more accurate models means better outcomes for banks and their customers, particularly in protecting against fraud and maximizing investment returns. These benefits translate into competitive advantage that often leads to increased market share and greater shareholder value. New products from AI enable cross-sell opportunities through enhanced personalization, which generates higher customer retention.

Challenges to Achieving AI Goals

While the benefits of leveraging AI in financial services are unmistakable, the journey from research to enterprise-scale production for AI models within banks, insurers and asset managers is marked with potential pitfalls and challenges.

Our survey identified those barriers, starting with the biggest challenges to achieving a company’s AI goals. The top three cited by respondents were too few data scientists (38 percent), insufficient technology infrastructure (35 percent) and a lack of data (35 percent).

The C-suite is looking to overcome these challenges by building AI expertise across the enterprise. 60 percent of C-level executives responded that their largest focus moving forward is identifying additional AI use cases. One in two respondents from the C-suite noted that their company also plans to hire more AI experts — addressing the gap of too few data scientists.

These findings warrant further exploration, especially in the context of new AI frameworks and platforms for smarter banking.

Popular AI Use Cases for Financial Services

Survey respondents from fintechs and investment firms highlighted portfolio optimization and algorithmic trading as the top AI use cases their companies currently invest in. This data can be understood in the context of maximizing client returns on investment.

Respondents from commercial and retail banks, on the other hand, noted that their companies are mainly investing in AI for fraud detection through payments, transactions and anti-money laundering. These survey results reflect a primary focus on protecting sensitive financial data for their customers.

Powering the Future of Banking with Enterprise AI

With these top use cases for AI in financial services, and dozens if not hundreds more available to banks, insurers and asset managers, the industry is understandably looking to grow its investment in AI. Sixty-two percent of our survey respondents — excluding those who marked “Don’t Know” — agreed that their company should spend more on AI applications.

Financial services professionals not only see the potential in AI, but are willing to invest more to deliver on its promise. That potential is actively being realized by companies who see AI generating competitive advantage, creating new products, adding significant revenues to the top line, and reducing costs to grow the bottom line.

As new use cases are identified and AI becomes more pervasive across organizations, the next challenge for C-suite and IT leadership will be creating enterprise-level AI platforms that deliver the productivity, scalability and return on investment necessary to support the variety of AI teams across their companies.

And, instead of starting from scratch, data scientists building models for a variety of use cases can utilize containers from NGC, NVIDIA’s hub of GPU-optimized software. These include NVIDIA Jarvis for automated speech recognition and speech to text for call center transcription to NVIDIA Merlin for recommendation system application frameworks.

To learn more about AI in the future of finance, download the survey report for more in-depth results.

And join GTC 2021 for free to hear from industry experts at Citibank, Morgan Stanley, Munich Re, Scotiabank, Wells Fargo and other leading financial institutions.

The post Is AI Important to Financial Services’ Future? New Survey Says You Can Bank on It appeared first on The Official NVIDIA Blog.

Read More

Active learning workflow for Amazon Comprehend custom classification models – Part 2

This is the second in a two part series on Amazon Comprehend custom classification models. In Part 1 of this series, we looked at how to build an AWS Step Functions workflow to automatically build, test, and deploy Amazon Comprehend custom classification models and endpoints. In Part 2, we look at real-time classification APIs, feedback loops, and human review workflows that help with continuous model training to keep the model up-to-date with new data and patterns. You can find Part 1 here. 

The Amazon Comprehend custom classification API enables you to easily build custom text classification models using your business-specific labels without learning machine learning (ML). For example, your customer support organization can use custom classification to automatically categorize inbound requests by problem type based on how the customer described the issue. You can use custom classifiers to automatically label support emails with appropriate issue types, thereby routing customer phone calls to the right agents and categorizing social media posts into user segments.

In Part 1 of this series, we looked at how to build an AWS Step Functions workflow to automatically build, test, and deploy Amazon Comprehend custom classification models and endpoints. In this post, we cover the real-time classification APIs, feedback loops, and human review workflows that help with continuous model training to keep it up to date with new data and patterns.

Solution architecture

This post describes a reference architecture for retraining custom classification models. The architecture comprises real-time classification, feedback pipelines, human review workflows using Amazon Augmented AI (Amazon A2I) , preparing new training data from the human review data, and triggering the model building flow that we covered in Part 1 of this series.

The following diagram illustrates this architecture covering the last three components. In the following sections, we walk you through each step in the workflow.

The following diagram illustrates this architecture covering the last three components.

Real-time classification

To use custom classification in Amazon Comprehend in real time, you need to create an API that calls the custom classification model endpoint with the text that needs to be classified. This stage is represented by Steps 1–3 in the preceding architecture:

  1. The end user application calls an Amazon API Gateway endpoint with a text that needs to be classified.
  2. The API Gateway endpoint then calls an AWS Lambda function configured to call an Amazon Comprehend endpoint.
  3. The Lambda function calls the Amazon Comprehend endpoint, which returns the unlabeled text classification and a confidence score.

Feedback collection

When the endpoint returns the classification and the confidence score during the real-time classification, you can send instances with low confidence scores to human review. This type of feedback is called implicit feedback.

  1. The Lambda function sends the implicit feedback to an Amazon Kinesis Data Firehose delivery stream.

The other type of feedback is called explicit feedback, and comes from the application’s end users that use the custom classification feature. This type of feedback comprises the instances of text where the user wasn’t happy with the prediction. You can send explicit feedback either in real time through an API or a batch process.

  1. End users of the application submit explicit real-time feedback through an API Gateway endpoint.
  2. The Lambda function backing the API endpoint transforms the data into a standard feedback format and writes it to the Kinesis Data Firehose delivery stream.
  3. End users of the application can also submit explicit feedback as a batch file by uploading it to an S3 bucket.
  4. A trigger configured on the S3 bucket triggers a Lambda function.
  5. The Lambda function transforms the data into a standard feedback format and writes it to the delivery stream.
  6. Both the implicit and explicit feedback data get sent to a delivery stream in a standard format. All this data is buffered and written to an S3 bucket.

Human classification

The human classification stage includes the following steps:

  1. A trigger configured on the feedback bucket in Step 10 invokes a Lambda function.
  2. The Lambda function creates Amazon A2I human review tasks for all the feedback data received.
  3. Workers assigned to the classification jobs log in to the human review portal and either approve the classification by the model or classify the text with the right labels.
  4. After the human review, all these instances are stored in an S3 bucket and used for retraining the models.

Retraining workflow

The retraining workflow stage includes the following steps:

  1. A trigger configured on the human-reviewed data bucket in Step 14 invokes a Lambda function.
  2. The function transforms the human-reviewed data payload to a comma-separated training data format, required by Amazon Comprehend custom classification models. After transformation, this data is written to a Firehose delivery stream, which acts as an accumulator.
  3. Depending on the time frame set for retraining models, the delivery stream flushes the data into the training bucket that was created in Part 1 of this series. For this post, we set the buffer conditions to 1 MiB or 60 seconds. For your own use case, you might want to adjust these settings so model retraining occurs according to your time or size requirements. This completes the active learning loop, and starts the Step Functions workflow for retraining models.

Solution overview

The next few sections of the post go over how to set up this architecture in your AWS account. We classify news into four categories: World, Sports, Business, and Sci/Tech, using the AG News dataset for custom classification, and set up the implicit and explicit feedback loop. You need to complete two manual steps:

  1. Create an Amazon Comprehend custom classifier and an endpoint.
  2. Create an Amazon SageMaker private workforce, worker task template, and human review workflow.

After this, you run the provided AWS CloudFormation template to set up the rest of the architecture.

Prerequisites

If you’re continuing from Part 1 of this series, you can skip to the step Create a private workforce, worker task template, and human review workflow.

Create a custom classifier and an endpoint

Before you get started, download the dataset and upload it to Amazon S3. This dataset comprises a collection of news articles and their corresponding category labels. We have created a training dataset called train.csv from the original dataset and made it available for download.

The following screenshot shows a sample of the train.csv file.

The following screenshot shows a sample of the train.csv file.

After you download the train.csv file, upload it to an S3 bucket in your account for reference during training. For more information about uploading files, see How do I upload files and folders to an S3 bucket?

To create your classifier for classifying news, complete the following steps:

  1. On the Amazon Comprehend console, choose Custom Classification.
  2. Choose Train classifier.
  3. For Name, enter news-classifier-demo.
  4. Select Using Multi-class mode.
  5. For Training data S3 location, enter the path for train.csv in your S3 bucket, for example, s3://<your-bucketname>/train.csv.
  6. For Output data S3 location, enter the S3 bucket path where you want the output, such as s3://<your-bucketname>/.
  7. For IAM role, select Create an IAM role.
  8. For Permissions to access, choose Input and output (if specified) S3 bucket.
  9. For Name suffix, enter ComprehendCustom.

For Name suffix, enter ComprehendCustom

  1. Scroll down and choose Train Classifier to start the training process.

The training takes some time to complete. You can either wait to create an endpoint or come back to this step later after finishing the steps in the section Create a private workforce, worker task template, and human review workflow.

Create a custom classifier real-time endpoint

To create your endpoint, complete the following steps:

  1. On the Amazon Comprehend console, choose Custom Classification.
  2. From the Classifiers list, choose the name of the custom model for which you want to create the endpoint and select your model news-classifier-demo.
  3. On the Actions drop-down menu, choose Create endpoint.
  4. For Endpoint name, enter classify-news-endpoint and give it one inference unit.
  5. Choose Create endpoint.
  6. Copy the endpoint ARN as shown in the following screenshot. You use it when running the CloudFormation template in a future step.

Copy the endpoint ARN as shown in the following screenshot.

Create a private workforce, worker task template, and human review workflow

This section walks you through creating a private workforce in SageMaker, a worker task template, and your human review workflow.

Create a labeling workforce

For this post, you create a private work team and add only one user (you) to it. For instructions, see Create a Private Workforce (Amazon SageMaker Console).

After the user accepts the invitation, you add them to the workforce. For instructions, see the Add a Worker to a Work Team section the Manage a Workforce (Amazon SageMaker Console).

Create a worker task template

To create a worker task template, complete the following steps:

  1. On the Amazon A2I console, choose Worker task templates.
  2. Choose to Create a template.
  3. For Template name, enter custom-classification-template.
  4. For Template type, choose Custom,
  5. In the Template editor, enter the following GitHub UI template code.
  6. Choose Create.

Choose Create.

Create a human review workflow

To create your human review workflow, complete the following steps:

  1. On the Amazon A2I console, choose Human review workflows.
  2. Choose Create human review workflow.
  3. For Name, enter classify-workflow.
  4. Create a S3 bucket to store the human review output. Make a note of this bucket, because we use this in the later part of the post.
  1. Specify an S3 bucket to store output: s3://<your bucketname>/. Use the bucket created earlier.
  2. For IAM role, select Create a new role.
  3. For Task type, choose Custom.
  4. Under Worker task template creation, select the custom classification template you created.
  5. For Task description, enter Read the instructions and review the document.
  6. Under Workers, select Private.
  7. Use the drop-down list to choose the private team that you created.
  8. Choose Create.
  9. Copy the workflow ARN (see the following screenshot). You will use it when initializing the CloudFormation template parameters in a later step.

Copy the workflow ARN

Deploy the CloudFormation template to set up active learning feedback

Now that you have completed the manual steps, you can run the CloudFormation template to set up this architecture’s building blocks, including the real-time classification, feedback collection, and the human classification.

Before deploying the CloudFormation template, make sure you have the following to pass as parameters:

  • Custom classifier endpoint ARN
  • Amazon A2I workflow ARN
  1. Choose Launch Stack:

  1. You must set this parameter, only if you’re continuing from Part 1 of this series. For BucketFromBlogPart1, enter the bucket name that was created for storing training data in Part 1 of this blog series.
  2. You must set this parameter, only if you’re continuing from Part 1 of this series. For ComprehendEndpointParameterKey, enter /<<StackName of Part1 Blog>>/CURRENT_CLASSIFIER_ENDPOINT. This parameter can be found in the Parameter Store section of the Systems Manager.
  1. You’re not required to set this parameter if you’re continuing from Part 1.For ComprehendEndpointARN, enter the endpoint ARN of your Amazon Comprehend custom classification model.
  1. For HumanReviewWorkflowARN, enter the workflow ARN you copied.
  2. For ComrehendClassificationScoreThreshold, enter 0.5, which means a 50% threshold for low confidence scores.

For ComrehendClassificationScoreThreshold, enter 0.5

  1. Choose Next until the Capabilities
  2. Select the check box to provide acknowledgment to AWS CloudFormation to create AWS Identity and Access Management (IAM) resources and expand the template.

For more information about these resources, see AWS IAM resources.

  1. Choose Create stack.

Choose Create stack.

Wait until the status of the stack changes from CREATE_IN_PROGRESS to CREATE_COMPLETE.

Wait until the status of the stack changes from CREATE_IN_PROGRESS to CREATE_COMPLETE.

  1. On the Outputs tab of the stack (see the following screenshot), copy the value for BatchUploadS3Bucket, FeedbackAPIGatewayID, and TextClassificationAPIGatewayID to interact with the feedback loop.

Both the TextClassificationAPI and FeedbackAPI require an API key to interact with them. The CloudFormation stack output ApiGWKey refers to the name of the API key. As of this writing, this API key is associated with a usage plan that allows 2,000 requests per month.

  1. On the API Gateway console, choose either the TextClassificationAPI or the FeedbackAPI.
  2. In the navigation pane, choose API Keys.
  3. Expand the API key section and copy the value.

Expand the API key section and copy the value.

You can manage the usage plan by following the instructions on Create, configure, and test usage plans with the API Gateway console.

You can also add fine-grained authentication and authorization to your APIs. For more information on securing your APIs, see Controlling and managing access to a REST API in API Gateway.

Enable the trigger to start the retraining workflow

The last step of the process is to add a trigger to the S3 bucket that we created earlier to store the human-reviewed output. The trigger invokes the Lambda function that begins the payload transformation from the Amazon A2I human review output format to a CSV format required for training Amazon Comprehend custom classification models.

  1. Open the Lambda function HumanReviewTrainingDataTransformerFunction, created by running the CloudFormation template.
  2. In the Trigger configuration section, choose S3.
  3. For Bucket, enter the bucket you created earlier in the step 4 of Create a human review workflow section.

For Bucket, enter the bucket you created earlier.

Test the feedback loop

In this section, we walk you through testing your feedback loop, including real-time classification, implicit and explicit feedback, and human review tasks.

Real-time classification

To interact and test these APIs, you need to download Postman.

The API Gateway endpoint receives an unlabeled text document from a client application and internally calls the custom classification endpoint, which returns the predicted label and a confidence score.

  1. Open Postman and enter the TextClassificationAPIGateway URL in POST method.
  2. In the Headers section, configure the API key: x-api-key : << Your API key >>.
  3. In the text field, enter the following JSON code (make sure you have JSON selected and enable raw): 
    {"classifier":"<your custom classifier name>", "sentence":"MS Dhoni retires and a billion people had mixed feelings."}

  1. Choose Send.

You get a response back with a confidence score and class, as seen in the following screenshot.

You get a response back with a confidence score and class, as seen in the following screenshot.

Implicit feedback

When the endpoint returns the classification and the confidence score during the real-time classification, you can route all the instances where the confidence score doesn’t meet the threshold to human review. This type of feedback is called implicit feedback. For this post, we set the threshold as 0.5 as an input to the CloudFormation stack parameter.

You can change this threshold when deploying the CloudFormation template based on your needs.

Explicit feedback

The explicit feedback comes from the end users of the application that uses the custom classification feature. This type of feedback comprises the instances of text where the user wasn’t happy with the prediction. You can send the predicted label by the model’s explicit feedback through the following methods:

  • Real time through an API, which is usually triggered through a like/dislike button on a UI
  • Batch process, where a file with a collection of misclassified utterances is put together based on a user survey conducted by the customer outreach team

Invoke the explicit real-time feedback loop

To test the Feedback API, complete the following steps:

  1. Open Postman and enter the FeedbackAPIGatewayID value from your CloudFormation stack output in POST method.
  2. In the Headers section, configure the API key: x-api-key : << Your API key >>.
  3. In the text field, enter the following JSON code (for classifier, enter the classifier you created, such as news-classifier-demo, and make sure you have JSON selected and enable raw): 
    {"classifier":"<your custom classifier name>","sentence":"Sachin is Indian Cricketer."}

  1. Choose Send.

We recommend that you submit at least four test samples that will result in a confidence score lesser than your set threshold.

We recommend that you submit at least four test samples that will result in a confidence score lesser than your set threshold.

Submit explicit feedback as a batch file

Download the following test feedback JSON file, populate it with your data, and upload it into the BatchUploadS3Bucket created when you deployed your CloudFormation template. We recommend that you submit at least four feedback entries in this file. The following code shows some sample data in the file:

{
   "classifier":"news-classifier-demo",
   "sentences":[
      "US music firms take legal action against 754 computer users alleged to illegally swap music online.",
      "A gamer spends $26,500 on a virtual island that exists only in a PC role-playing game."
   ]
}

Uploading the file triggers the Lambda function that starts your human review loop.

Human review tasks

All the feedback collected through the implicit and explicit methods is sent for human classification. The labeling workforce can include Amazon Mechanical Turk, private teams, or AWS Marketplace vendors. For this post, we create a private workforce. The URL to the labeling portal is located on the SageMaker console, on the Labeling workforces page, on the Private tab.

For this post, we create a private workforce.

After you log in, you can see the human review tasks assigned to you. Select the task to complete and choose Start working.

Select the task to complete and choose Start working.

You see the tasks displayed based on the worker template used when creating the human workflow.

You see the tasks displayed based on the worker template used when creating the human workflow.

After you complete the human classification and submit the tasks, the human-reviewed data is stored in the S3 bucket you configured when creating the human review workflow. This bucket is located under Output location on the workflow details page.

his bucket is located under Output location on the workflow details page.

This human-reviewed data is used to retrain the custom classification model to learn newer patterns and improve its overall accuracy. The following screenshot shows the human-annotated output file output.json in the S3 bucket.

The following screenshot shows the human-annotated output file output.json in the S3 bucket.

This human-reviewed data is then converted to a custom classification model training data format, and transferred to the training bucket that was created in Part 1 of this series, which starts the Step Functions workflow for retraining models. The process of retraining the models with human-reviewed data, selecting the best model, and automatically deploying the new endpoints completes the active learning workflow.

Cleanup

To remove all resources created throughout this process and prevent additional costs, complete the following steps:

  1. On the Amazon S3 console, delete the S3 bucket that contains the training dataset.
  2. On the Amazon Comprehend console, delete the endpoint and the classifier.
  3. On the Amazon A2I console, delete the human review workflow, worker template, and private workforce.
  4. On the AWS CloudFormation console, delete the stack you created. (This removes the resources the CloudFormation template created.)

Conclusion

Amazon Comprehend helps you build scalable and accurate natural language processing capabilities without any ML experience. This post provides a reusable pattern and infrastructure for active learning workflows for custom classification models. The feedback pipelines and human review workflow help the custom classifier learn new data patterns continuously. To learn more about automatic model building, selection, and deployment of custom classification models, you can refer to Active learning workflow for Amazon Comprehend custom classification models – Part 1.

For more information, see Custom Classification. You can discover other Amazon Comprehend features and get inspiration from other AWS blog posts about how to use Amazon Comprehend beyond classification.

About the Authors

Shanthan Kesharaju is a Senior Architect in the AWS ProServe team. He helps our customers with AI/ML strategy, architecture, and developing products with a purpose. Shanthan has an MBA in Marketing from Duke University and an MS in Management Information Systems from Oklahoma State University.

 

 

Mona Mona is an AI/ML Specialist Solutions Architect based out of Arlington, VA. She works with the World Wide Public Sector team and helps customers adopt machine learning on a large scale. She is passionate about NLP and ML explainability areas in AI/ML.

 

 

Joyson Neville Lewis Joyson Neville Lewis obtained his masters in Information Technology from Rutgers University in 2018. He has worked as a Software/Data engineer before diving into the conversational AI domain in 2019, where he works with companies to connect the dots between business and AI using voice and chatbot solutions. Joyson joined Amazon Web Services in February of 2018 as a Big Data Consultant for the AWS Professional Services team in NYC.

Read More

Active learning workflow for Amazon Comprehend custom classification models – Part 1

This is the first in a two part series on Amazon Comprehend custom classification models. In Part 1 of this series, we look at how to build an AWS Step Functions workflow to automatically build, test, and deploy Amazon Comprehend custom classification models and endpoints. In Part 2, we will look at real-time classification APIs, feedback loops, and human review workflows that help with continuous model training to keep the model up-to-date with new data and patterns. You can find Part 2 here. 

The Amazon Comprehend custom classification API enables you to easily build custom text classification models using your business-specific labels without learning ML. For example, your customer support organization can use custom classification to automatically categorize inbound requests by problem type based on how the customer has described the issue. You can use custom classifiers to automatically label support emails with appropriate issue types, route customer phone calls to the right agents, and categorize social media posts into user segments.

For custom classification, you start by creating a training job with a ground truth dataset comprising a collection of text and corresponding category labels. When the job is complete, you have a classifier that can classify any new text into one or more named categories. When the custom classification model classifies a new unlabeled text document, it predicts what it has learned from the training data. Sometimes you may not have a training dataset with various language patterns, or once you deploy the model, you start seeing completely new data patterns. In these cases, the model may not be able to classify these new data patterns accurately. How can we ensure continuous model training to keep it up to date with new data and patterns?

Feedback loops play a pivotal role in keeping the models up to date. This feedback helps the models learn about their misclassifications and learn the right ones. This process of teaching the models continuously through feedback and deploying them is called active learning.

Solution architecture

In this two-part series, we discuss an architecture pattern that allows you to build an active learning workflow for Amazon Comprehend custom classification models. The first post will cover an AWS Step Functions workflow that automates model building, selecting the best model, and deploying an endpoint of the chosen model. The second post describes a workflow comprising real-time classification, feedback pipelines, and human review workflows using Amazon Augmented AI (Amazon A2I).

Step Functions workflow

The following diagram shows the Step Functions workflow for automatic model building, endpoint creation, and deploying Amazon Comprehend custom classification models.

In the following sections, we discuss the six steps in more detail:

  • Model building: Steps 1–2
  • Model selection: Step 3
  • Model deployment: Steps 4–6

Model building

Steps 1–2 in the workflow cover model building, which includes incorporating new data into the ground truth dataset and retraining the model.  If the model is being built for the first time, the new dataset will be marked as ground truth dataset, and the Model selection step would be skipped. This new data can come from different sources, including feedback data that was human reviewed and reclassified, as discussed in Part 2 of this series. The new data is uploaded to an Amazon Simple Storage Service (Amazon S3) bucket, which starts a workflow that includes merging the new data with the ground truth dataset and starting a custom classification model training job that uses the newly merged dataset.

Model selection

Step 3 covers model selection, which includes testing the newly created model with a validation dataset, computing the test results, comparing the results of the new model with the current model in production, and finally selecting the model that performs best with respect to a chosen metric like accuracy, precision, recall, or F1 score. All these steps are orchestrated using the same Step Functions workflow after the model is built.

Model deployment

Steps 4–6 cover model deployment. If the new model outperforms the current model in production, the Step Functions workflow continues to the next step, where a new endpoint is created for the newly created custom classification model, and then updates the AWS Systems Manager Parameter Store values to this new classifier ARN and the new endpoint ARN to be used by the real-time classification API, upgrades the newly merged training dataset as the primary training dataset, and deletes the endpoint of the previous model that was in production. If the newer model doesn’t perform well in production, you can roll back to the previous model and endpoint by manually updating the Parameter Store values to refer to the earlier model and endpoint ARNs.

Deploying the AWS CloudFormation template

You can deploy this architecture using the provided AWS CloudFormation template in us-east-1.

  1. Choose Launch Stack:

  1. For Stack Name, enter the name of your CloudFormation stack.
  2. For StepFunctionName, enter the Step Functions name for automatic model building, endpoint creation, and deploying Amazon Comprehend custom classification models (can be left at the default value of ComprehendModelStepFunction).
  3. For TestThresholdParameterName, choose Accuracy, Precision, Recall, or F1score. (For this post, we leave it at the default value of F1 Score).

We use this metric to check if the newer model is better than the previous model.

  1. Choose Next.
  2. Choose Next again.
  3. In the Capabilities and transforms section, select all three check boxes to provide acknowledgment to AWS CloudFormation to create AWS Identity and Access Management (IAM) resources and expand the template.
  4. Choose Create stack.

This process might take 15 minutes or more to complete, and creates the following resources:

  • Systems Manager parameters to store intermediate parameters, like the current classifier endpoint ARN
  • An S3 bucket for the custom classification model training data
  • An S3 bucket for the custom classification model prediction job on the test data
  • Amazon DynamoDB tables for model predictions on the test data and status of custom classification prediction jobs
  • AWS Lambda functions:
    • StartCustomClassificationModelBuilding – Starts the custom classification model building
    • GetCustomClassificationModelStatus – Gets the status of the model building stage
    • StartCustomClassificationJob – Starts the prediction job using the test dataset
    • GetCustomClassificationJobStatus – Gets the status of the prediction job
    • CustomClassificationModelSelection – Starts the custom classification model selection stage
    • StartCustomClassificationEndpointBuilding – Starts the custom classification model endpoint building stage
    • GetCustomClassificationEndpointStatus – Gets the status of model endpoint building stage
    • DeleteCustomClassificationEndpoint – Deletes the old model endpoint
  • Step Functions to automate the workflow of building, testing, selecting, and deleting the models
  • IAM roles:
    • A Lambda execution role to run Amazon Comprehend custom classification jobs
    • A Lambda execution role to trigger Step Functions
    • A Step Functions role to trigger Lambda functions
    • An Amazon Comprehend data access role to give Amazon Comprehend access to the training data in the S3 bucket

Testing

For testing this blog, you can use your own training dataset or you can download the news dataset and upload it to Amazon S3. The news dataset comprises a collection of news articles and their corresponding category labels.

  1. Find the S3 bucket created by the CloudFormation to store the training data. This can be found by going to the Resources section of the CloudFormation and looking up ComprehendInputDataS3Bucket.
  2. On the Amazon S3 console, inside the input data bucket, create a folder named train.
  3. Upload the training data to the train folder.
  4. On the Step Functions console, choose the new state machine you created.
  5. In the Executions section, choose the latest run.

In the Executions section, choose the latest run.

The following screenshot shows the Graph inspector view. On the Details tab, you can check that Step Functions ran successfully.

The following screenshot shows the Graph inspector view.

It takes approximately 1 hour for the Step Functions state machine to complete.

  1. After Step Functions has successfully ran, on the Systems Manager console, in the navigation pane, under Application Management, choose Parameter Store.

You can check the updated classifier and endpoint from the Parameter Store.

You can check the updated classifier and endpoint from the Parameter Store.

Cleaning up

To avoid incurring any charges in the future, delete the CloudFormation stack. This removes all the resources you created as part of this post.

Conclusion

Active learning in custom classification models ensures that your models are kept up to date with new data and patterns. This two-part series provides you with a reference architecture to build an active learning workflow comprised of real-time classification APIs, feedback loops, a human review workflow, model building, model selection, and model deployment. For more information about the feedback loops and human review workflow, see the second part of this blog series, Active learning workflow for Amazon Comprehend custom classification models – Part 2.

For more information about custom classification in Amazon Comprehend, see Custom Classification. You can discover other Amazon Comprehend features and get inspiration from other AWS blog posts about how to use Amazon Comprehend beyond classification.

About the Authors

Shanthan Kesharaju is a Senior Architect in the AWS ProServe team. He helps our customers with AI/ML strategy, architecture, and developing products with a purpose. Shanthan has an MBA in Marketing from Duke University and an MS in Management Information Systems from Oklahoma State University.

 

 

Marty Jiang is a Conversational AI Consultant with AWS Professional Services. Outside of work, he loves spending time outdoors with his family and exploring new technologies.

Read More

Accelerating Neural Networks on Mobile and Web with Sparse Inference

Posted by Artsiom Ablavatski and Marat Dukhan, Software Engineers, Google Research

On-device inference of neural networks enables a variety of real-time applications, like pose estimation and background blur, in a low-latency and privacy-conscious way. Using ML inference frameworks like TensorFlow Lite with XNNPACK ML acceleration library, engineers optimize their models to run on a variety of devices by finding a sweet spot between model size, inference speed and the quality of the predictions.

One way to optimize a model is through use of sparse neural networks [1, 2, 3], which have a significant fraction of their weights set to zero. In general, this is a desirable quality as it not only reduces the model size via compression, but also makes it possible to skip a significant fraction of multiply-add operations, thereby speeding up inference. Further, it is possible to increase the number of parameters in a model and then sparsify it to match the quality of the original model, while still benefiting from the accelerated inference. However, the use of this technique remains limited in production largely due to a lack of tools to sparsify popular convolutional architectures as well as insufficient support for running these operations on-device.

Today we announce the release of a set of new features for the XNNPACK acceleration library and TensorFlow Lite that enable efficient inference of sparse networks, along with guidelines on how to sparsify neural networks, with the goal of helping researchers develop their own sparse on-device models. Developed in collaboration with DeepMind, these tools power a new generation of live perception experiences, including hand tracking in MediaPipe and background features in Google Meet, accelerating inference speed from 1.2 to 2.4 times, while reducing the model size by half. In this post, we provide a technical overview of sparse neural networks — from inducing sparsity during training to on-device deployment — and offer some ideas on how researchers might create their own sparse models.

Comparison of the processing time for the dense (left) and sparse (right) models of the same quality for Google Meet background features. For readability, the processing time shown is the moving average across 100 frames.

Sparsifying a Neural Network
Many modern deep learning architectures, like MobileNet and EfficientNetLite, are primarily composed of depthwise convolutions with a small spatial kernel and 1×1 convolutions that linearly combine features from the input image. While such architectures have a number of potential targets for sparsification, including the full 2D convolutions that frequently occur at the beginning of many networks or the depthwise convolutions, it is the 1×1 convolutions that are the most expensive operators as measured by inference time. Because they account for over 65% of the total compute, they are an optimal target for sparsification.

Architecture Inference Time
MobileNet 85%
MobileNetV2 71%
MobileNetV3 71%
EfficientNet-Lite   66%
Comparison of inference time dedicated to 1×1 convolutions in % for modern mobile architectures.

In modern on-device inference engines, like XNNPACK, the implementation of 1×1 convolutions as well as other operations in the deep learning models rely on the HWC tensor layout, in which the tensor dimensions correspond to the height, width, and channel (e.g., red, green or blue) of the input image. This tensor configuration allows the inference engine to process the channels corresponding to each spatial location (i.e., each pixel of an image) in parallel. However, this ordering of the tensor is not a good fit for sparse inference because it sets the channel as the innermost dimension of the tensor and makes it more computationally expensive to access.

Our updates to XNNPACK enable it to detect if a model is sparse. If so, it switches from its standard dense inference mode to sparse inference mode, in which it employs a CHW (channel, height, width) tensor layout. This reordering of the tensor allows for an accelerated implementation of the sparse 1×1 convolution kernel for two reasons: 1) entire spatial slices of the tensor can be skipped when the corresponding channel weight is zero following a single condition check, instead of a per-pixel test; and 2) when the channel weight is non-zero, the computation can be made more efficient by loading neighbouring pixels into the same memory unit. This enables us to process multiple pixels simultaneously, while also performing each operation in parallel across several threads. Together these changes result in a speed-up of 1.8x to 2.3x when at least 80% of the weights are zero.

In order to avoid converting back and forth between the CHW tensor layout that is optimal for sparse inference and the standard HWC tensor layout after each operation, XNNPACK provides efficient implementations of several CNN operators in CHW layout.

Guidelines for Training Sparse Neural Networks
To create a sparse neural network, the guidelines included in this release suggest one start with a dense version and then gradually set a fraction of its weights to zero during training. This process is called pruning. Of the many available techniques for pruning, we recommend using magnitude pruning (available in the TF Model Optimization Toolkit) or the recently introduced RigL method. With a modest increase in training time, both of these can successfully sparsify deep learning models without degrading their quality. The resulting sparse models can be stored efficiently in a compressed format that reduces the size by a factor of two compared to their dense equivalent.

The quality of sparse networks is influenced by several hyperparameters, including training time, learning rate and schedules for pruning. The TF Pruning API provides an excellent example of how to select these, as well as some tips for training such models. We recommend running hyperparameter searches to find the sweet spot for your application.

Applications
We demonstrate that it is possible to sparsify classification tasks, dense segmentation (e.g., Meet background blur) and regression problems (MediaPipe Hands), which provides tangible benefits to users. For example, in the case of Google Meet, sparsification lowered the inference time of the model by 30%, which provided access to higher quality models for more users.

Model size comparisons for the dense and sparse models in Mb. The models have been stored in 16- and 32-bit floating-point formats.

The approach to sparsity described here works best with architectures based on inverted residual blocks, such as MobileNetV2, MobileNetV3 and EfficientNetLite. The degree of sparsity in a network influences both inference speed and quality. Starting from a dense network of a fixed capacity, we found modest performance gains even at 30% sparsity. With increased sparsity, the quality of the model remains relatively close to the dense baseline until reaching 70% sparsity, beyond which there is a more pronounced drop in accuracy. However, one can compensate for the reduced accuracy at 70% sparsity by increasing the size of the base network by 20%, which results in faster inference times without degrading the quality of the model. No further changes are required to run the sparsified models, because XNNPACK can recognize and automatically enable sparse inference.

Ablation studies of different sparsity levels with respect to inference time (the smaller the better) and the quality measured by the Intersection over Union (IoU) for predicted segmentation mask.

Sparsity as Automatic Alternative to Distillation
Background blur in Google Meet uses a segmentation model based on a modified MobileNetV3 backbone with attention blocks. We were able to speed up the model by 30% by applying a 70% sparsification, while preserving the quality of the foreground mask. We examined the predictions of the sparse and dense models on images from 17 geographic subregions, finding no significant difference, and released the details in the associated model card.

Similarly, MediaPipe Hands predicts hand landmarks in real-time on mobile and the web using a model based on the EfficientNetLite backbone. This backbone model was manually distilled from the large dense model, which is a computationally expensive, iterative process. Using the sparse version of the dense model instead of distilled one, we were able to maintain the same inference speed but without the labor intensive process of distilling from a dense model. Compared with the dense model the sparse model improved the inference by a factor of two, achieving the identical landmark quality as the distilled model. In a sense, sparsification can be thought of as an automatic approach to unstructured model distillation, which can improve model performance without extensive manual effort. We evaluated the sparse model on the geodiverse dataset and made the model card publicly available.

Comparison of execution time for the dense (left), distilled (middle) and sparse (right) models of the same quality. Processing time of the dense model is 2x larger than sparse or distilled models. The distilled model is taken from the official MediPipe solution. The dense and sparse web demos are publicly available.

Future work
We find sparsification to be a simple yet powerful technique for improving CPU inference of neural networks. Sparse inference allows engineers to run larger models without incurring a significant performance or size overhead and offers a promising new direction for research. We are continuing to extend XNNPACK with wider support for operations in CHW layout and are exploring how it might be combined with other optimization techniques like quantization. We are excited to see what you might build with this technology!

Acknowledgments
Special thanks to all who worked on this project: Karthik Raveendran, Erich Elsen, Tingbo Hou‎, Trevor Gale, Siargey Pisarchyk, Yury Kartynnik, Yunlu Li, Utku Evci, Matsvei Zhdanovich, Sebastian Jansson, Stéphane Hulaud, Michael Hays, Juhyun Lee, Fan Zhang, Chuo-Ling Chang, Gregory Karpiak, Tyler Mullen, Jiuqiang Tang, Ming Guang Yong, Igor Kibalchich, and Matthias Grundmann.

Read More

Introducing a new API to stop in-progress workflows in Amazon Forecast

Amazon Forecast uses machine learning (ML) to generate more accurate demand forecasts, without requiring any prior ML experience. Forecast brings the same technology used at Amazon.com to developers as a fully managed service, removing the need to manage resources or rebuild your systems.

To start generating forecasts through Forecast, you can follow three steps of importing your data, training and evaluating a predictor, and then generating forecasts. Starting today, you can now stop an in-progress Forecast resource workflow if you have mistakenly started a job or misconfigured a workflow before starting, giving you more flexibility to manage your Forecast workflows and to experiment.

Previously, because you couldn’t stop APIs in progress, you had to wait for the job to complete and would incur charges for the job. You can now easily stop the following Forecast resource workflows:

In this post, we walk through the steps to stop workflows on the Forecast console. To review the steps through the APIs, refer to the following notebook in our GitHub repo.

Stop a resource job that is importing your datasets

You have two options to stop importing a dataset. One method is via the dataset details page on the Forecast console. On the Datasets page, choose your dataset and in the Dataset imports section, select the import job that you want to stop and choose Stop.

You can also stop a data import job from the job details page. In the Data imports section of your dataset, choose the import job to go to its details page. Then choose Stop.

Stop a resource job that is training a predictor

You have two options to stop a resource job that is training your predictor. One method is on the Predictors page for your dataset group, where you can select a predictor and choose Stop.

Alternatively, you can select the predictor and choose View details. Here you can stop the resource job that is training the predictor by choosing Stop.

Stop a resource job that is exporting backtest forecasts

Backtest forecasts are the forecasted values from the Forecast internal testing method of splitting the data into training and backtest data groups to compare forecasts versus observed data. When training a model, Forecast automatically splits the historical demand datasets into training and backtesting dataset groups. Forecast trains a model on the training dataset and forecasts at different specified stocking levels for the backtesting period, comparing to the observed values in the backtesting dataset group.

To stop a resource that is exporting these backtest results, select a predictor on the Predictors page of your dataset group. In the Predictor backtest exports section, select an export and choose Stop.

Stop a resource job that is generating forecasts

You have two options to stop a resource job that is generating your forecasts. One method is on the Forecasts page of the dataset group, where you can select a forecast and choose Stop.

Alternatively, you can select a forecast and choose View details. You can then stop the resource job that is generating the forecast by choosing Stop.

Stop a resource job that is exporting forecasts

Lastly, you can stop a resource job that is exporting your forecasts. You have two options to do so. One option is to select the forecast export job listed in the Forecast details section and choose Stop.

The second option is to choose the export job to view its details, and then choose Stop.

Important considerations

Stopping a resource halts the resource job workflow but doesn’t delete the resource. All your resources are still retained, and you can continue to call the Describe operation or access them as part of List APIs. After a resource is marked for stopping, it doesn’t count towards your Max Parallel In Progress limits. If you’re already at the limit, it allows you to submit a new job.

We allow only three resources of a given resource type at any time to be in the Stopping state, and you have to wait for one of the resources to go into the STOPPED state before you can stop more resources. After you initiate a stop, you can’t cancel it. You also can’t resume a stopped job. When you stop a predictor training or forecast generation job, you’re billed for the resources used up to the point when the job stopped.

Conclusion

You now have more flexibility in managing your Forecast workflows with the ability to stop in-progress resource workflows that may have been started unintentionally. To get started with this capability, see Stopping Resources and go through the notebook in our GitHub repo that walks you through how to use the Forecast Stop Resource APIs. You can use this capability in all Regions where Forecast is publicly available. For more information about Region availability, see AWS Regional Services.

About the Authors

Namita Das is a Sr. Product Manager for Amazon Forecast. Her current focus is to democratize machine learning by building no-code and low-code ML services. Outside of AWS, she frequently advises startups and is raising a puppy named Imli.

 

 

Gunjan Garg is a Sr. Software Development Engineer in the AWS Vertical AI team. In her current role at Amazon Forecast, she focuses on engineering problems and enjoys building scalable systems that provide the most value to end users. In her free time, she enjoys playing Sudoku and Minesweeper.

 

 

Punit Jain works as SDE on the Amazon Forecast team. His current work includes building large-scale distributed systems to solve complex machine learning problems with high availability and low latency as a major focus. In his spare time, he enjoys hiking and cycling.

 

 

Shannon Killingsworth is a UX Designer for Amazon Forecast and Amazon Personalize. His current work is creating console experiences that are usable by anyone, and integrating new features into the console experience. In his spare time, he is a fitness and automobile enthusiast.

 

 

 

 

Read More