Learning to Fix Programs from Error Messages

Learning to Fix Programs from Error Messages

Machine Learning for Program Repair

When writing programs, a lot of time is spent debugging or fixing source code errors, both for beginners (imagine the intro programming classes you took) as well as for professional developers (for example, this case study from Google 1). Automating program repair could dramatically enhance the productivity of both programming and learning programming. In our recent work published at ICML 2020, we study how to use machine learning to repair programs automatically.

Problem Setting

Programmers write programs incrementally: write code, compile or execute it, and if there are any errors, repair the program based on the received feedback. Can we model and solve this problem with machine learning?

Let’s say we have a broken C++ program (figure left), where the char in line 5 should actually be string. When we compile it, we get an error (figure top right), which says “line 9 is requesting for size in a which is of type char”. From this message, a programmer can notice that the error is related to the type of the variable a, track how a has been used or declared in the source code, reaching line 5, and then edit the line to fix the error. Thus, the concrete task we want our machine learning model to solve is, given broken code (figure left) and an error message (figure top right), localize the error line (line 5) and generate a repaired version of it (“string tmp, a, b;”) (figure bottom right).

Challenges:
This task poses two main challenges. First, on the modeling side, we need to connect and jointly reason over two modalities, the program and the error message: for instance, tracking variables that caused the error as we saw in the example above. Second, on the training data side, we need an efficient source of data that provides supervision for correcting broken programs; unfortunately, existing labeled datasets with <broken code, fixed code> pairs are small and hard to come by, and don’t scale up. In this work, we introduce promising solutions to those two challenges by: 1) modeling program repair with program-feedback graph, and 2) introducing a self-supervised training scheme that uses unlabeled programs.

Modeling Approach: Program-Feedback Graph

How can we effectively connect the two modalities (programs and error messages) and perform the reasoning needed for repair? To achieve this, we introduce a program-feedback graph, a joint graph representation that connects symbols across the program and error message. For instance, the compiler message in the example mentions a, size, and char, so we connect these symbols to their occurrences in the source code, to capture semantic correspondence. This way, we treat the two modalities in a shared semantic space rather than separately. We then perform reasoning over the symbols in this space using graph attention 2.

Specifically, for the model architecture, we build on the encoder-decoder framework commonly used in NLP, which encodes input sequences (in our case, the program and error message; next figure bottom) and then decodes outputs (in our case, the localized line index, and the repaired version of the line; figure top), and we incorporate a graph attention module applied to the program-feedback graph in the intermediate layer of the architecture (figure middle).

Training Approach: Self-Supervised Learning

Our second technique is self-supervised learning. Labeled datasets of program repair are small, but there are vast amounts of unlabeled programs available online. For example, GitHub has more than 30M public repositories. Using this large amount of freely available code to improve learning program repair would significantly enhance the scalability and reliability of our system.
Our idea is as follows: we first collect unlabeled, working programs from online resources such as GitHub and codeforce.com (figure left). We then design randomized program corruption procedures (e.g. delete/insert/replace tokens) and corrupt the unlabeled programs (figure middle). As a result, the corrupted programs give us errors (figure right). This way, we can create a lot of new examples of program repair, <broken code, error message, fixed code>. We can use this extra data to pre-train the program repair model, and then fine-tune on the labeled target dataset.

Let’s use our program repair model!

We apply and evaluate our repair model (we call DrRepair) on two benchmark tasks:

Application to DeepFix (Correcting Student Programs)

In DeepFix, the task is to correct C programs written by students in an intro programming class so that they will compile. The input programs may have multiple lines with errors, so we apply the repair model iteratively, addressing one error at a time. For instance, the following figure shows an example program in DeepFix, which has a compiler error saying that “i is undeclared”. By applying the repair model, DrRepair, it repairs this error by inserting a declaration of i in line 5. After this fix, we notice that there is another error, which says “expected semicolon before brace”. We can apply the repair model again – this time, the model inserts a semicolon in line 12, and now the repaired program compiles successfully! This approach is conducive to the idea of iterative refinement: we can keep running the repair model and progressively fixing errors.

What is the effect of using error messages, program-feedback graphs, and self-supervised pre-training? Existing repair systems studied on DeepFix did not use compiler error messages – they aimed to directly translate from broken code to fixed code. To see the effect of using error messages in the first place, we tried removing all our techniques from the system: the use of compiler messages, program-feedback graphs, and pre-training. This version of our model (“ours: no compiler” in the figure below) achieves 34% repair accuracy on DeepFix, which is comparable to the existing systems. Now we add compiler messages to our input. We find that this model achieves much better performance and generalization (62.5% accuracy; “ours: base” in the figure). This suggests that with an access to error messages, the model learns the right inductive bias to repair the code based on the feedback. Next, we add program-feedback graphs and self-supervised pre-training. We find that both provide further improvements (“ours: base+graph” and “ours: base+graph+pretrain”), and our final system can fix 68.2% of the broken programs in DeepFix!

Application to SPoC (Natural Language to Code)

Program synthesis, in particular systems that can translate natural language descriptions (e.g. English) into code (e.g. Python, C++), are useful because they can help a wider range of people use programming languages. In SPoC (Pseudocode-to-Code), the task is to synthesize C++ implementation from pseudocode, a natural language description of a program. However, one challenge experienced by existing synthesizers (machine translation models applied to SPoC) is that they tend to output inconsistent code that does not compile – for instance, in the figure below, the variable i is declared twice in the synthesized code. We find that we can apply our program repair model to this invalid code and fix it into a correct one, helping the program synthesis task. In the evaluation on SPoC, the use of our repair model improves the final synthesis success rate from the existing system’s 34% to 37.6%.

Conclusion

In this work, we studied how to use machine learning to repair programs from error messages, and developed three key insights:

  1. Error messages provide a crucial signal for learning program repair.
  2. Program-feedback graphs (joint representations of code & error messages) help model the reasoning of repair (e.g. tracking variables that caused the error).
  3. Self-supervised learning allows us to turn freely-available, unlabeled programs (e.g. GitHub code) into useful training examples of program repair.

This work also provides a general framework of “learning from feedback”, which has various applications: editing documents based on comments, learning from users in interactive dialog, etc.

You can check out our full paper (ICML 2020) here and our source code/data on GitHub. You can also find the presentation slides on this work here. If you have questions, please feel free to email us!

Acknowledgments

Many thanks to Percy Liang, as well as members of the P-Lambda lab and the Stanford NLP group for their valuable feedback, and to Sidd Karamcheti and Andrey Kurenkov for edits on this blog post!

  1. Programmers’ Build Errors: A Case Study (at Google). Hyunmin Seo, Caitlin Sadowski, Sebastian Elbaum, Edward Aftandilian, Robert Bowdidge. 2014 ↩

  2. Graph Attention Networks. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio. 2018. ↩

  3. DeepFix: Fixing common C language errors by deep learning. Rahul Gupta, Soham Pal, Aditya Kanade, Shirish Shevade. 2017. ↩

  4. SPoC: Search-based Pseudocode to Code. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken and Percy Liang. 2019. ↩

Read More

Bringing your own custom container image to Amazon SageMaker Studio notebooks

Bringing your own custom container image to Amazon SageMaker Studio notebooks

Amazon SageMaker Studio is the first fully integrated development environment (IDE) for machine learning (ML). SageMaker Studio lets data scientists spin up Studio notebooks to explore data, build models, launch Amazon SageMaker training jobs, and deploy hosted endpoints. Studio notebooks come with a set of pre-built images, which consist of the Amazon SageMaker Python SDK and the latest version of the IPython runtime or kernel. With this new feature, you can bring your own custom images to Amazon SageMaker notebooks. These images are then available to all users authenticated into the domain. In this post, we share how to bring a custom container image to SageMaker Studio notebooks.

Developers and data scientists may require custom images for several different use cases:

  • Access to specific or latest versions of popular ML frameworks such as TensorFlow, MxNet, PyTorch, or others.
  • Bring custom code or algorithms developed locally to Studio notebooks for rapid iteration and model training.
  • Access to data lakes or on-premises data stores via APIs, and admins need to include the corresponding drivers within the image.
  • Access to a backend runtime, also called kernel, other than IPython such as R, Julia, or others. You can also use the approach outlined in this post to install a custom kernel.

In large enterprises, ML platform administrators often need to ensure that any third-party packages and code is pre-approved by security teams for use, and not downloaded directly from the internet. A common workflow might be that the ML Platform team approves a set of packages and frameworks for use, builds a custom container using these packages, tests the container for vulnerabilities, and pushes the approved image in a private container registry such as Amazon Elastic Container Registry (Amazon ECR). Now, ML platform teams can directly attach approved images to the Studio domain (see the following workflow diagram). You can simply select the approved custom image of your choice in Studio. You can then work with the custom image locally in your Studio notebook. With this release, a single Studio domain can contain up to 30 custom images, with the option to add a new version or delete images as needed.

We now walk through how you can bring a custom container image to SageMaker Studio notebooks using this feature. Although we demonstrate the default approach over the internet, we include details on how you can modify this to work in a private Amazon Virtual Private Cloud (Amazon VPC).

Prerequisites

Before getting started, you need to make sure you meet the following prerequisites:

  • Have an AWS account.
  • Ensure that the execution role you use to access Amazon SageMaker has the following AWS Identity and Access Management (IAM) permissions, which allow SageMaker Studio to create a repository in Amazon ECR with the prefix smstudio, and grant permissions to push and pull images from this repo. To use an existing repository, replace the Resource with the ARN of your repository. To build the container image, you can either use a local Docker client or create the image from SageMaker Studio directly, which we demonstrate here. To create a repository in Amazon ECR, SageMaker Studio uses AWS CodeBuild, and you also need to include the CodeBuild permissions shown below. 
    {
            "Effect": "Allow",
            "Action": [
                "ecr:CreateRepository",
                "ecr:BatchGetImage",
                "ecr:CompleteLayerUpload",
                "ecr:DescribeImages",
                "ecr:DescribeRepositories",
                "ecr:UploadLayerPart",
                "ecr:ListImages",
                "ecr:InitiateLayerUpload",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:PutImage"
            ],
            "Resource": "arn:aws:ecr:*:*:repository/smstudio*"
        },
        {
            "Effect": "Allow",
            "Action": "ecr:GetAuthorizationToken",
            "Resource": "*"
           }
{
            "Effect": "Allow",
            "Action": [
                "codebuild:DeleteProject",
                "codebuild:CreateProject",
                "codebuild:BatchGetBuilds",
                "codebuild:StartBuild"
            ],
            "Resource": "arn:aws:codebuild:*:*:project/sagemaker-studio*"
}

  • Your SageMaker role should also have a trust policy with AWS CodeBuild as shown below. For more information, see Using the Amazon SageMaker Studio Image Build CLI to build container images from your Studio notebooks 
    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "codebuild.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

  • Install the AWS Command Line Interface (AWS CLI) on your local machine. For instructions, see Installing the AWS.

If you wish to use your private VPC to securely bring your custom container, you also need the following:

To set up these resources, see Securing Amazon SageMaker Studio connectivity using a private VPC and the associated GitHub repo.

Creating your Dockerfile

To demonstrate the common need from data scientists to experiment with the newest frameworks, we use the following Dockerfile, which uses the latest TensorFlow 2.3 version as the base image. You can replace this Dockerfile with a Dockerfile of your choice. Currently, SageMaker Studio supports a number of base images, such as Ubuntu, Amazon Linux 2, and others. The Dockerfile installs the IPython runtime required to run Jupyter notebooks, and installs the Amazon SageMaker Python SDK and boto3.

In addition to notebooks, data scientists and ML engineers often iterate and experiment on their local laptops using various popular IDEs such as Visual Studio Code or PyCharm. You may wish to bring these scripts to the cloud for scalable training or data processing. You can include these scripts as part of your Docker container so they’re visible in your local storage in SageMaker Studio. In the following code, we copy the train.py script, which is a base script for training a simple deep learning model on the MNIST dataset. You may replace this script with your own scripts or packages containing your code.

FROM tensorflow/tensorflow:2.3.0
RUN apt-get update 
RUN apt-get install -y git
RUN pip install --upgrade pip
RUN pip install ipykernel && 
    python -m ipykernel install --sys-prefix && 
    pip install --quiet --no-cache-dir 
    'boto3>1.0<2.0' 
    'sagemaker>2.0<3.0'
COPY train.py /root/train.py #Replace with your own custom scripts or packages

import tensorflow as tf
import os 
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])
model.fit(x_train, y_train, epochs=1)
model.evaluate(x_test, y_test)

Instead of a custom script, you can also include other files, such as Python files that access client secrets and environment variables via AWS Secrets Manager or AWS Systems Manager Parameter Store, config files to enable connections with private PyPi repositories, or other package management tools. Although you can copy the script using the custom image, any ENTRYPOINT or CMD commands in your Dockerfile don’t run.

Setting up your installation folder

You need to create a folder on your local machine, and add the following files in that folder:

  • The Dockerfile that you created in the previous step
  • A file named app-image-config-input.json with the following content: 
    "AppImageConfigName": "custom-tf2",
    "KernelGatewayImageConfig": {
        "KernelSpecs": [
            {
                "Name": "python3",
                "DisplayName": "Python 3"
            }
        ],
        "FileSystemConfig": {
            "MountPath": "/root/data",
            "DefaultUid": 0,
            "DefaultGid": 0
        }
    }
}

We set the backend kernel for this Dockerfile as an IPython kernel, and provide a mount path to the Amazon Elastic File System (Amazon EFS). Amazon SageMaker recognizes kernels as defined by Jupyter. For example, for an R kernel, set Name in the preceding code to ir.

  • Create a file named default-user-settings.json with the following content. If you’re adding multiple custom images, just add to the list of CustomImages. 
    {
  "DefaultUserSettings": {
    "KernelGatewayAppSettings": {
      "CustomImages": [
          {
                   "ImageName": "tf2kernel",
                   "AppImageConfigName": "custom-tf2"
                }
            ]
        }
    }
}

Creating and attaching the image to your Studio domain

If you have an existing domain, you simply need to update the domain with the new image. In this section, we demonstrate how existing Studio users can attach images. For instructions on onboarding a new user, see Onboard to Amazon SageMaker Studio Using IAM.

First, we use the SageMaker Studio Docker build CLI to build and push the Dockerfile to Amazon ECR. Note that you can use other methods to push containers to ECR such as your local docker client, and the AWS CLI.

    1. Log in to Studio using your user profile.
    2. Upload your Dockerfile and any other code or dependencies you wish to copy into your container to your Studio domain.
    3. Navigate to the folder containing the Dockerfile.
    4. In a terminal window or in a notebook —>
!pip install sagemaker-studio-image-build
  1. Export a variable called IMAGE_NAME, and set it to the value you specified in the default-user-settings.json 
    sm-docker build . --repository smstudio-custom:IMAGE_NAME

  2. If you wish to use a different repository, replace smstudio-custom in the preceding code with your repo name.

SageMaker Studio builds the Docker image for you and pushes the image to Amazon ECR in a repository named smstudio-custom, tagged with the appropriate image name. To customize this further, such as providing a detailed file path or other options, see Using the Amazon SageMaker Studio Image Build CLI to build container images from your Studio notebooks. For the pip command above to work in a private VPC environment, you need a route to the internet or access to this package in your private repository.

  1. In the installation folder from earlier, create a new file called create-and-update-image.sh: 
    ACCOUNT_ID=AWS ACCT ID # Replace with your AWS account ID
REGION=us-east-2 #Replace with your region
DOMAINID=d-####### #Replace with your SageMaker Studio domain name.
IMAGE_NAME=tf2kernel #Replace with your Image name

# Using with SageMaker Studio
## Create SageMaker Image with the image in ECR (modify image name as required)
ROLE_ARN='The Execution Role ARN for the execution role you want to use'

aws --region ${REGION} sagemaker create-image 
    --image-name ${IMAGE_NAME} 
    --role-arn ${ROLE_ARN}

aws --region ${REGION} sagemaker create-image-version 
    --image-name ${IMAGE_NAME} 
    --base-image "${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom:${IMAGE_NAME}"
    
## Create AppImageConfig for this image (modify AppImageConfigName and KernelSpecs in app-image-config-input.json as needed)
aws --region ${REGION} sagemaker create-app-image-config --cli-input-json file://app-image-config-input.json

## Update the Domain, providing the Image and AppImageConfig
aws --region ${REGION} sagemaker update-domain --domain-id ${DOMAINID} --cli-input-json file://default-user-settings.json

    Refer to the AWS CLI to read more about the arguments you can pass to the create-image API. To check the status, navigate to your Amazon SageMaker console and choose Amazon SageMaker Studio from the navigation pane.

Attaching images using the Studio UI

You can perform the final step of attaching the image to the Studio domain via the UI. In this case, the UI will handle the setting up of the

  1. On the Amazon SageMaker console, choose Amazon SageMaker Studio.

On the Control Panel page, you can see that the Studio domain was provisioned, along with any user profiles that you created.

  1. Choose Attach image.

  1. Select whether you wish to attach a new or pre-existing image.
    1. If you select Existing image, choose an image from the Amazon SageMaker image store.
    2. If you select New image, provide the Amazon ECR registry path for your Docker image. The path needs to be in the same Region as the studio domain. The ECR repo also needs to be in the same account as your Studio domain or cross-account permissions for Studio need to be enabled.
  2. Choose Next.
  1. For Image name, enter a name.
  2. For Image display name, enter a descriptive name.
  3. For Description, enter a label definition.
  4. For IAM role, choose the IAM role required by Amazon SageMaker to attach Amazon ECR images to Amazon SageMaker images on your behalf.
  5. Additionally, you can tag your image.
  6. Choose Next.

  1. For Kernel name, enter Python 3.
  2. Choose Submit.

The green check box indicates that the image has been successfully attached to the domain.

The Amazon SageMaker image store automatically versions your images. You can select a pre-attached image and choose Detach to detach the image and all versions, or choose Attach image to attach a new version. There is no limit to the number of versions per image or the ability to detach images.

User experience with a custom image

Let’s now jump into the user experience for a Studio user.

  1. Log in to Studio using your user profile.
  2. To launch a new activity, choose Launcher.
  3. For Select a SageMaker image to launch your activity, choose tf2kernel.

  1. Choose the Notebook icon to open a new notebook with the custom kernel.

The notebook kernel takes a couple minutes to spin up and you’re ready to go!

Testing your custom container in the notebook

When the kernel is up and running, you can run code in the notebook. First, let’s test that the correct version of TensorFlow that was specified in the Dockerfile is available for use. In the following screenshot, we can see that the notebook is using the tf2kernel we just launched.

Amazon SageMaker notebooks also display the local CPU and memory usage.

Next, let’s try out the custom training script directly in the notebook. Copy the training script into a notebook cell and run it. The script downloads the mnist dataset from the tf.keras.datasets utility, splits the data into training and test sets, defines a custom deep neural network algorithm, trains the algorithm on the training data, and tests the algorithm on the test dataset.

To experiment with the TensorFlow 2.3 framework, you may wish to test out newly released APIs, such as the newer feature preprocessing utilities in Keras. In the following screenshot, we import the keras.layers.experimental library released with TensorFlow 2.3, which contains newer APIs for data preprocessing. We load one of these APIs and re-run the script in the notebook.

Amazon SageMaker also dynamically modifies the CPU and memory usage as the code runs. By bringing your custom container and training scripts, this feature allows you to experiment with custom training scripts and algorithms directly in the Amazon SageMaker notebook. When you’re satisfied with the experimentation in the Studio notebook, you can start a training job.

What about the Python files or custom files you included with the Dockerfile using the COPY command? SageMaker Studio mounts the elastic file system in the file path provided in the app-image-config-input.json, which we set to root/data. To avoid Studio from overwriting any custom files you want to include, the COPY command loads the train.py file in the path /root. To access this file, open a terminal or notebook and run the code:

! cat /root/train.py

You should see an output as shown in the screenshot below.

The train.py file is in the specified location.

Logging in to CloudWatch

SageMaker Studio also publishes kernel metrics to Amazon CloudWatch, which you can use for troubleshooting. The metrics are captured under the /aws/sagemaker/studio namespace.

To access the logs, on the CloudWatch console, choose CloudWatch Logs. On the Log groups page, enter the namespace to see logs associated with the Jupyter server and the kernel gateway.

Detaching an image or version

You can detach an image or an image version from the domain if it’s no longer supported.

To detach an image and all versions, select the image from the Custom images attached to domain table and choose Detach.

You have the option to also delete the image and all versions, which doesn’t affect the image in Amazon ECR.

To detach an image version, choose the image. On the Image details page, select the image version (or multiple versions) from the Image versions attached to domain table and choose Detach. You see a similar warning and options as in the preceding flow.

Conclusion

SageMaker Studio enables you to collaborate, experiment, train, and deploy ML models in a streamlined manner. To do so, data scientists often require access to the newest ML frameworks, custom scripts, and packages from public and private code repositories and package management tools. You can now create custom images containing all the relevant code, and launch these using Studio notebooks. These images will be available to all users in the Studio domain. You can also use this feature to experiment with other popular languages and runtimes besides Python, such as R, Julia, and Scala. The sample files are available on the GitHub repo. For more information about this feature, see Bring your own SageMaker image.

About the Authors

Stefan Natu is a Sr. Machine Learning Specialist at AWS. He is focused on helping financial services customers build end-to-end machine learning solutions on AWS. In his spare time, he enjoys reading machine learning blogs, playing the guitar, and exploring the food scene in New York City.

 

 

Jaipreet Singh is a Senior Software Engineer on the Amazon SageMaker Studio team. He has been working on Amazon SageMaker since its inception in 2017 and has contributed to various Project Jupyter open-source projects. In his spare time, he enjoys hiking and skiing in the Pacific Northwest.

 

 

Huong Nguyen is a Sr. Product Manager at AWS. She is leading the user experience for SageMaker Studio. She has 13 years’ experience creating customer-obsessed and data-driven products for both enterprise and consumer spaces. In her spare time, she enjoys reading, being in nature, and spending time with her family.

Read More

New Coral APIs and tools for AI at the edge

New Coral APIs and tools for AI at the edge

Posted by Carlos Mendonça, Coral

Coral Fall 2020 image

Fall has finally arrived and with it a new release of Coral’s C++ and Python APIs and tools, along with new models optimized for the Edge TPU and further support for TensorFlow 2.0-based workflows.

Coral is a complete toolkit to build products with local AI. Our on-device inferencing capabilities allow you to build products that are efficient, private, fast and offline with the help of TensorFlow Lite and the Edge TPU.

From the beginning, we’ve provided APIs in Python and C++ that enable developers to take advantage of the Edge TPU’s local inference speed. Offline processing for machine learning models allows for considerable savings on bandwidth and cloud compute costs, it keeps data local, and it preserves user privacy. More recently, we’ve been hard at work to refactor our APIs and make them more modular, reusable and performant, while at the same time eliminating unnecessary API abstractions and surfacing more of the native TensorFlow Lite APIs that developers are familiar with.

So in our latest release, we’re now offering two separate reusable libraries, each built upon the powerful TensorFlow Lite APIs and each isolated in their own repositories: libcoral for C++ and PyCoral for Python.

libcoral (C++)

Unlike some of our previous APIs, libcoral doesn’t hide tflite::Interpreter. Instead, we’re making this native TensorFlow Lite class a first-class component and offering some additional helper APIs that simplify some of your code when working with common models such as classification and detection.

With our new libcoral library, developers should typically follow the pattern below to perform an inference in C++:

  1. Create tflite::Interpreter instance with the Edge TPU context and allocate memory.

    To simplify this step, libcoral provides the MakeEdgeTpuInterpreter() function: 

     
    // Load the model
    auto model = coral::LoadModelOrDie(absl::GetFlag(FLAGS_model_path));

    // Get the Edge TPU context
    auto tpu_context = coral::ContainsEdgeTpuCustomOp(*model) ?
     	coral::GetEdgeTpuContextOrDie() :
     	nullptr;

    // Get the interpreter
    auto interpreter = coral::MakeEdgeTpuInterpreterOrDie(
     	*model,
     	tpu_context.get());
  2. Configure the interpreter’s input.
  3. Invoke the interpreter: 

    4. interpreter->Invoke();

    As an alternative to Invoke(), you can achieve higher performance with the InvokeWithMemBuffer() and InvokeWithDmaBuffer() functions, which enable processing the input data without copying from another region of memory or from a DMA file descriptor, respectively.

  4. Process the interpreter’s output.

To simplify this step, libcoral provides some adapters, requiring less code from you: 


auto result = coral::GetClassificationResults(
 	*interpreter,
 	/* threshold= */0.0f,
 	/*top_k=*/3);

The above is an example of the classification adapter, where developers can specify the minimum confidence threshold, as well as the maximum number of results to return. The API also features a detection adapter with its own result filtering parameters.

For a full view of the example application source code, see classify_image.cc on GitHub and for instructions on how to integrate libcoral into your application, refer to README.md on GitHub.

This new release also brings updates to on-device retraining with the decoupling of imprinting functions from inference on the updated ImprintingEngine. The new design makes the imprinting engine work with the tflite::Interpreter directly.

To easily address the Edge TPUs available on the host, libcoral supports labels such as "usb:0" or "pci:1“. This should make it easier to manage resources on multi-Edge TPU systems.

Finally, we’ve made a number of performance improvements such as more efficient memory usage and memory-based instead of file-based abstractions. Also, the design of the API is more consistent by leveraging the Abseil library for error propagation, generic interfaces and other common patterns, which should provide a more consistent and stable developer experience.

PyCoral (Python)

The new PyCoral library (provided in a new pycoral Python module) follows some of the design patterns introduced with libcoral, and brings parity across our C++ and Python APIs. PyCoral implements the same imprinting decoupling design, model adapters for classification and detection, and the same label-based TPU addressing semantics.

On PyCoral, the “run inference” functionality is now entirely delegated to the native TensorFlow Lite library, as we’ve done-away with the model “engines” that abstracted the TensorFlow interpreter. This change allowed us to eliminate the code duplication introduced by the Coral-specific BasicEngine, ClassificationEngine and DetectionEngine classes (those APIs—from the “Edge TPU Python library”—are now deprecated).

To perform an inference with PyCoral, we follow a similar pattern to that of libcoral:

  1. Create an interpreter: 

    2. interpreter = edgetpu.make_interpreter(model_file)
    interpreter.allocate_tensors()
  2. Configure the interpreter’s input: 

    3. common.set_input(interpreter, image)
  3. Invoke the interpreter: 

    4. interpreter.invoke()
  4. Process the interpreter’s output: 

    5. classes = classify.get_classes(interpreter, top_k=3)

    For fully detailed example code, check out our documentation for Python.

    Updates to the Coral model garden

    With this release, we’re further expanding the Coral model garden with MobileDet. MobileDets refer to a family of lightweight, single-shot detectors using the TensorFlow Object Detection API that achieve state-of-the-art accuracy-latency tradeoff on Edge TPUs. It is a lower-latency detection model that offers better accuracy, compared to the MobileNet family of models.

    Check out the full collection of models available from Coral for the Edge TPU, including Classification, Detection, Segmentation and models specially prepared for on-device training.

    Migrating our entire workflow and model collection to TensorFlow 2 is an ongoing effort. This release of the Coral machine learning API starts introducing support for TensorFlow 2-based workflows. For now, MobileNet v1 (ImageNet), MobileNet v2 (ImageNet), MobileNet v3 (ImageNet), ResNet50 v1 (ImageNet), and UNet MobileNet v2 (Oxford pets) all support training and conversion with TensorFlow 2.

    Model Pipelining

    Both libcoral and PyCoral have graduated the model pipelining functionality from Beta to General Availability. Model pipelining makes it possible for large models to be partitioned and distributed across multiple Edge TPUs to run them considerably faster.

    Refer to the documentation for examples of the API in C++ and Python.

    The partitioning of models is done with the Edge TPU Compiler, which employs a parameter count algorithm, partitioning the model into segments with similar parameter sizes. For cases where this algorithm doesn’t provide the throughput you need, this release is introducing a new tool that supports a profiling-based algorithm, which divides the segments based on latency observed by actually running the model multiple times, possibly resulting in a more balanced output.

    The new profiling_partition tool can be used as such: 


    ./profiling_partition 
      --edgetpu_compiler_binary $PATH_TO_COMPILER 
      --model_path $PATH_TO_MODEL 
      --output_dir $OUT_DIR 
      --num_segments $NUM_SEGMENTS

    Learn more

    For more information about the Coral APIs mentioned above, see the following documentation:

Read More

Amazon Translate now enables you to mark content to not get translated

Amazon Translate now enables you to mark content to not get translated

While performing machine translations, you may have situations where you wish to preserve specific sections of text from being translated, such as names, unique identifiers, or codes. We at the Amazon Translate team are excited to announce a tag modifications that allows you to specify what text should not be translated. This feature is available in both the real-time TranslateText API and asynchronous batch TextTranslation API. You can tag segments of text that you don’t want to translate in an HTML element. In this post, we walk through the step-by-step method to use this feature.

Using the translate-text operation in Command Line Interface

The following example shows you how to use the translate-text operation from the command line. This example is formatted for Unix, Linux, and macOS. For Windows, replace the backslash () Unix continuation character at the end of each line with a caret (^). At the command line, enter the following code:

aws translate translate-text 
--source-language-code "en" 
--target-language-code "es" 
--region us-west-2 
--text “This can be translated to any language. <p translate=no>But do not translate this!</p>”

You can specify any type of HTML element to do so, for example, paragraph <p>, text section <span>, or block section <div>. When you run the command, you get the following output:

{
    "TranslatedText": "Esto se puede traducir a cualquier idioma. <p translate=no>But do not translate this!</p>",
    "SourceLanguageCode": "en",
    "TargetLanguageCode": "es"
}

Using the span tag in Amazon Translate Console

In this example, we translate the following text from French to English:

Musée du Louvre, c’est ainsi que vous dites Musée du Louvre en français.

You don’t want to translate the first instance of “Musée du Louvre,” but you do want to translate the second instance to “Louvre Museum.” You can tag the first instance using a simple span tag:

Musée du Louvre, c'est ainsi que vous dites Musée du Louvre en français.

The following screenshot shows the output on the Amazon Translate console.

The following screenshot shows the output translated to Arabic.

Conclusion

In this post, we showed you how to tag and specify text that should not be translated. For more information, see the Amazon Translate Developer Guide and Amazon Translate resources. If you’re new to Amazon Translate, try it out using our Free Tier, which offers 2 million characters per month for free for the first 12 months, starting from your first translation request.

 

About the Author

Watson G. Srivathsan is the Sr. Product Manager for Amazon Translate, AWS’s natural language processing service. On weekends you will find him exploring the outdoors in the Pacific Northwest.

Read More

Intelligently connect to customers using machine learning in the COVID-19 pandemic

Intelligently connect to customers using machine learning in the COVID-19 pandemic

The pandemic has changed how people interact, how we receive information, and how we get help. It has shifted much of what used to happen in-person to online. Many of our customers are using machine learning (ML) technology to facilitate that transition, from new remote cloud contact centers, to chatbots, to more personalized engagements online. Scale and speed are important in the pandemic—whether it’s processing grant applications or limiting call wait times for customers. ML tools like Amazon Lex and Amazon Connect are just a few of the solutions helping to power this change with speed, scale, and accuracy. In this post, we explore companies who have quickly pivoted to take advantage of AI capabilities to engage more effectively online and deliver immediate impact.

Chatbots connect governments and their citizens

GovChat is South Africa’s largest citizen engagement platform, connecting over 50 million citizens to 10,000 public representatives in the government. Information flowing to and from the government has gained a new level of urgency, and this connection between citizens and the government is critical in how we adjust and respond to the pandemic. GovChat exists to meet that demand—working directly with the South African government to facilitate the digitization of their COVID-19 social relief grants, help citizens find their closest COVID-19 testing facility, and enable educational institutions to reopen safely.

GovChat uses a chatbot powered by Amazon Lex, a managed AI service for building conversational interfaces into any application using voice and text. The chatbot, available on popular social media platforms such as WhatsApp and Facebook, provides seamless communication between the government and its citizens.

At the beginning of the pandemic, GovChat worked with the South African Social Security Agency to digitize, facilitate, and track applications for a COVID-19 social relief grant. The plan was to create a chatbot that could help citizens easily file and track their grant applications. GovChat needed to act quickly and provide an infrastructure that could rapidly scale to support the unprecedented demand for government aid. To provide speed of delivery and scalability while keeping costs down, GovChat turned to Amazon Lex for voice and text conversational interfaces and AWS Lambda, a serverless compute service. Within days, the chatbot was handling up to 14.2 million messages a day across social media platforms in South Africa regarding the social relief grant.

More recently, the South African Human Rights Commission (SAHRC) turned to GovChat to help gauge schools’ readiness to reopen safety. Parents, students, teachers, and community members can use their mobile devices to provide first-hand, real-time details of their school’s COVID-19 safety checks and readiness as contact learning is resumed, with special attention paid to children with disabilities. In GovChat’s engagements during the COVID-19 pandemic, they found that 28% of service requests at schools have been in relation to a disruption in access to water, which is critical for effective handwashing—a preventative component to fight the spread of the virus. With the real-time data provided by citizens via the chatbot, the government was able to better understand the challenges schools faced and identify areas of improvement. GovChat has processed over 250 million messages through their platform, playing an important role in enabling more effective and timely communications between citizens and their government.

ML helps power remote call centers

Organizations of all kinds have also experienced a rapid increase in call volume to their call centers—from local government, to retail, to telecommunications, to healthcare providers. Organizations have also had to quickly shift to a remote work environment in response to the pandemic. Origin Energy, one of Australia’s largest integrated energy companies serving over 4 million customer accounts, launched an Amazon Connect contact center in March as part of their customer experience transformation. Amazon Connect is an omnichannel cloud contact center with AI/ML capabilities that understands context and can transcribe conversations.

This transition to Amazon Connect accelerated Origin’s move to remote working during the COVID-19 pandemic. This allowed their agents to continue to serve their customers, while also providing increased self-service and automation options such as bill payments, account maintenance, and plan renewals to customers. They deployed new AI/ML capabilities, including neural text-to-speech through Amazon Polly. Since the March 2020 launch, they’ve observed an increase in call quality scores, improved customer satisfaction, and agent productivity—all while managing up to 1,200 calls at a time. They’re now looking to further leverage natural language understanding with Amazon Lex and automated quality management with built-in speech-to-text and sentiment analysis from Contact Lens for Amazon Connect. Amazon Connect has supported Origin in their efforts to respond rapidly to opportunities and customer feedback as they focus on continually improving their customer experience with affordable, reliable, and sustainable energy.

Conclusion

Organizations are employing creative strategies to engage their customers and provide a more seamless experience. This is a two-way street; not only can organizations more effectively distribute key information, but—more importantly—they can listen. They can hear the evolving needs of their customers and adjust in real time to meet them.

To learn about another way AWS is working toward solutions from the COVID-19 pandemic, check out the blog article Introducing the COVID-19 Simulator and Machine Learning Toolkit for Predicting COVID-19 Spread.

 

 

About the Author

Taha A. Kass-Hout, MD, MS, is director of machine learning and chief medical officer at Amazon Web Services (AWS). Taha received his medical training at Beth Israel Deaconess Medical Center, Harvard Medical School, and during his time there, was part of the BOAT clinical trial. He holds a doctor of medicine and master’s of science (bioinformatics) from the University of Texas Health Science Center at Houston.

Read More

Announcing the AWS DeepComposer Chartbusters challenge, Keep Calm and Model On

Announcing the AWS DeepComposer Chartbusters challenge, Keep Calm and Model On

We are back with another AWS DeepComposer Chartbusters challenge, Keep Calm and Model On! This challenge is open for submissions throughout AWS re:Invent until January 31, 2021. In this challenge, you can experiment with our newly launched Transformers algorithm and generate an original piece of music. Chartbusters is a global monthly challenge where you can use AWS DeepComposer to create compositions on the console using generative AI techniques, compete to top the charts, and win prizes. This challenge launches today and you can submit your compositions until January 31, 2021.

Participation is easy — in 5 simple steps you can generate your original piece of music on the AWS DeepComposer console using the Transformers technique. You can then use Edit melody and add or remove notes.

How to Compete

To participate in Keep Calm and Model On, just do the following:

  1. Go to AWS DeepComposer Music Studio and choose a sample melody recommended for Transformers.
  2. Under Generative AI technique choose Transformers. Then choose TransformerXIClassical for Model. You then have six advanced parameters that you can choose to adjust including Sampling technique that defines how the next note for your input melody is chosen, and Track extension duration that attempts to add additional seconds to your melody. After adjusting the values, choose Extend Input Melody.

  1. Use Edit melody to add or remove notes. You can also change the note duration and pitch. When finished, choose Apply changes. Repeat these steps until you’re satisfied with the generated music.

  1. When you’re happy with your composition, choose Download composition. You can choose to post-process your composition; however, one of the judging criteria is how close your final submission is to the track generated using AWS DeepComposer.

  1. In the navigation panel, choose Chartbusters; and on the chartbusters page, choose Submit a composition. Then, choose Import a post-processed audio track, upload your composition, provide a track name for your composition, and choose Submit.

 

AWS DeepComposer then submits your composition to the Keep Calm and Model On playlist on SoundCloud.

Congratulations!

You’ve successfully submitted your composition to the AWS DeepComposer Chartbusters challenge Keep Calm and Model On. Now you can invite your friends and family to listen to your creation on SoundCloud and join the fun by participating in the competition.

About the Author

Maryam Rezapoor is a Senior Product Manager with AWS AI Devices team. As a former biomedical researcher and entrepreneur, she finds her passion in working backward from customers’ needs to create new impactful solutions. Outside of work, she enjoys hiking, photography, and gardening.

Read More

Adapting on the Fly to Test Time Distribution Shift

Adapting on the Fly to Test Time Distribution Shift

Imagine that you are building the next generation machine learning model for handwriting transcription. Based on previous iterations of your product, you have identified a key challenge for this rollout: after deployment, new end users often have different and unseen handwriting styles, leading to distribution shift. One solution for this challenge is to learn an adaptive model that can specialize and adjust to each user’s handwriting style over time. This solution seems promising, but it must be balanced against concerns about ease of use: requiring users to provide feedback to the model may be cumbersome and hinder adoption. Is it possible instead to learn a model that can adapt to new users without labels?