Incremental training with Amazon SageMaker JumpStart

by Amazon AWS

In December 2020, AWS announced the general availability of Amazon SageMaker JumpStart, a capability of Amazon SageMaker that helps you quickly and easily get started with machine learning (ML). SageMaker JumpStart provides one-click fine-tuning and deployment of a wide variety of pre-trained models across popular ML tasks, as well as a selection of end-to-end solutions that solve common business problems. These features remove the heavy lifting from each step of the ML process, making it easier to develop high-quality models and reducing time to deployment.

All JumpStart content was previously available only through Amazon SageMaker Studio, which provides a user-friendly graphical interface to interact with the feature. Recently, we also announced the launch of easy-to-use JumpStart APIs as an extension of the SageMaker Python SDK, allowing you to programmatically deploy and fine-tune a vast selection of JumpStart-supported pre-trained models on your own datasets. This launch unlocks the usage of JumpStart capabilities in your code workflows, MLOps pipelines, and anywhere else you’re interacting with SageMaker via SDK.

In this post, we’re excited to announce that all trainable JumpStart models now support incremental training. Incremental training allows you to train a model you have already fine-tuned using an expanded dataset that contains an underlying pattern not accounted for in previous fine-tuning runs, which resulted in poor model performance. Incremental training saves both time and resources because you don’t need to retrain the model from scratch. If you want to jump straight into the JumpStart API code we explain in this post, you can refer to the sample notebook.

JumpStart overview

JumpStart is a multi-faceted product that includes different capabilities to help get you quickly started with ML on SageMaker. At the time of writing, JumpStart enables you to do the following:

  • Deploy pre-trained models for common ML tasks – JumpStart enables you to address common ML tasks with no development effort by providing easy deployment of models pre-trained on large, publicly available datasets. The ML research community has put a large amount of effort into making a majority of recently developed models publicly available for use; JumpStart hosts a collection of over 300 models, spanning the 15 most popular ML tasks such as object detection, text classification, and text generation, making it easy for beginners to use them. These models are drawn from popular model hubs, such as TensorFlow, PyTorch, Hugging Face, and MXNet Hub.
  • Fine-tune pre-trained models – JumpStart allows you to fine-tune pre-trained models with no need to write your own training algorithm. In ML, the ability to transfer the knowledge learned in one domain to another is called transfer learning. You can use transfer learning to produce accurate models on your smaller datasets, with much lower training costs than the ones involved in training the original model. JumpStart also includes popular training algorithms based on LightGBM, CatBoost, XGBoost, and Scikit-learn that you can train from scratch for tabular regression and classification.
  • Use pre-built solutions – JumpStart provides a set of 17 solutions for common ML use cases such as demand forecasting and industrial and financial applications, which you can deploy with just a few clicks. Solutions are end-to-end ML applications that string together various AWS services to solve a particular business use case. They use AWS CloudFormation templates and reference architectures for quick deployment, which means they are fully customizable.
  • Use notebook examples for SageMaker algorithms – SageMaker provides a suite of built-in algorithms to help data scientists and ML practitioners get started with training and deploying ML models quickly. JumpStart provides sample notebooks that you can use to quickly apply these algorithms.
  • Review training videos and blogs – JumpStart also provides numerous blog posts and videos that teach you how to use different functionalities within SageMaker.

JumpStart accepts custom VPC settings and AWS Key Management Service (AWS KMS) encryption keys, so you can use the available models and solutions securely within your enterprise environment. You can pass your security settings to JumpStart within Studio or through the SageMaker Python SDK.

Image classification

Image classification refers to classifying an image into one of the class labels in the training dataset. You can fine-tune the model to any given dataset comprising images belonging to any number of classes. The model available for fine-tuning on JumpStart attaches a classification layer to the corresponding feature extractor model and initializes the layer parameters to random values. The output dimension of the classification layer is determined based on the number of classes in the input data. The fine-tuning step tunes the classification layer parameters, while keeping the parameters of the feature extractor model frozen, and returns the fine-tuned model. The objective is to minimize prediction error on the input data.

For our dataset, the input is a directory with as many sub-directories as the number of classes. Each sub-directory should have images belonging to that class in .jpg format. The input directory should look like the following hierarchy if the training data contains images from two classes: roses and dandelion:

input_directory 
     |--roses 
           |--abc.jpg 
           |--def.jpg 
     |--dandelion 
           |--ghi.jpg 
           |--jkl.jpg

The names of the folders, classes, and .jpg file names can be anything.

We provide the tf_flowers1 dataset as a default dataset for fine-tuning the model. This dataset comprises images of five types of flowers. The dataset has been downloaded from TensorFlow.

Walkthrough overview

The following sections provide a step-by-step demo to perform image classification with JumpStart, both via the Studio UI and JumpStart APIs.

We walk through the following steps:

  1. Access JumpStart through the Studio UI:
    1. Fine-tune the pre-trained model.
    2. Deploy the fine-tuned model.
    3. Incrementally train the fine-tuned model and redeploy.
  2. Use JumpStart programmatically with the SageMaker Python SDK:
    1. Fine-tune the pre-trained model.
    2. Deploy the fine-tuned model.
    3. Incrementally train the fine-tuned model and redeploy.

Access JumpStart through the Studio UI

In this section, we demonstrate how to fine-tune and deploy JumpStart models through the Studio UI. Additionally, we show how to incrementally train a model that you have previously fine-tuned.

Fine-tune the pre-trained model

The following video shows you how to find a pre-trained image classification model on JumpStart and fine-tune it. The model page contains valuable information about the model, how to use it, expected data format, and some fine-tuning details.

For demonstration purposes, we fine-tune the model using the dataset provided by default, which is the tf_flowers dataset, composed of different varieties of flowers. Fine-tuning on your own dataset involves taking the correct formatting of data (as explained on the model page), uploading it to Amazon Simple Storage Service (Amazon S3), and specifying its location in the data source configuration.

We use the same hyperparameter values set by default (number of epochs, learning rate, and batch size). We also use a GPU-backed ml.p3.2xlarge instance as our SageMaker training instance.

You can monitor your training job directly on the Studio console, and are notified upon its completion.

Deploy the fine-tuned model

After training is complete, you can deploy the fine-tuned model from the same page that holds the training job details. To deploy our model, we pick a different instance type, ml.p2.xlarge. It still provides the GPU acceleration needed for low inference latency, but at a lower price point. After you configure the SageMaker hosting instance, choose Deploy. It may take 5–10 minutes until your persistent endpoint is up and running.

Then your endpoint is operational and ready to respond to inference requests!

To accelerate your time to inference, JumpStart provides a sample notebook that shows you how to run inference on your freshly deployed endpoint. Choose Open Notebook under Use Endpoint from Studio.

Incrementally train the fine-tuned model and deploy

When fine-tuning is complete, you can further train the model to boost performance. This step is very similar to the initial fine-tuning process, except that we use the already fine-tuned model as the starting point. You may use new data, but the dataset format must be the same (same set of classes).

Use JumpStart programmatically with the SageMaker SDK

In the preceding sections, we showed how you can use the JumpStart UI to fine-tune, deploy, and incrementally train a model interactively in a matter of a few clicks. You can also use JumpStart’s models and easy fine-tuning programmatically by using APIs that are integrated into the SageMaker SDK. We now go over a quick example of how you can replicate the preceding process. All the steps in this demo are available in the accompanying notebooks Introduction to JumpStart – Image Classification.

Fine-tune the pre-trained model

To fine-tune a selected model, we need to get that model’s URI, as well as that of the training script and the container image used for training. Thankfully, these three inputs depend solely on the model name, version (for a list of available models, see JumpStart Available Model Table), and type of instance you want to train on. This is demonstrated in the following code snippet:

from sagemaker import image_uris, model_uris, script_uris

model_id, model_version = "pytorch-ic-mobilenet-v2", "1.0.0"
training_instance_type = "ml.p3.2xlarge"

# Retrieve the docker image
train_image_uri = image_uris.retrieve(
    region=None,
    framework=None,
    model_id=model_id,
    model_version=model_version,
    image_scope="training",
    instance_type=training_instance_type,
)

# Retrieve the training script
train_source_uri = script_uris.retrieve(model_id=model_id, model_version=model_version, script_scope="training")

# Retrieve the pre-trained model tarball to further fine-tune
train_model_uri = model_uris.retrieve(model_id=model_id, model_version=model_version, model_scope="training")

We retrieve the model_id corresponding to the same model we used previously. The ic in the identifier corresponds to image classification.

You can now fine-tune this JumpStart model on your own custom dataset using the SageMaker SDK. We use the same tf_flowers dataset that is publicly hosted on Amazon S3, conveniently focused on sentiment analysis. Your dataset should be structured for fine-tuning, as explained in the previous section. See the following example code:

# URI of your training dataset
training_dataset_s3_path = "s3://jumpstart-cache-prod-us-west-2/training-datasets/tf_flowers/"
training_job_name = name_from_base(f"jumpstart-example-{model_id}-transfer-learning")

# Create SageMaker Estimator instance
ic_estimator = Estimator(
    role=aws_role,
    image_uri=train_image_uri,
    source_dir=train_source_uri,
    model_uri=train_model_uri,
    entry_point="transfer_learning.py",
    instance_count=1,
    instance_type=training_instance_type,
    max_run=360000,
    hyperparameters=hyperparameters,
    output_path=s3_output_location,
)

# Launch a SageMaker Training job by passing s3 path of the training data
ic_estimator.fit({"training": training_dataset_s3_path}, logs=True)

We obtain the same default hyperparameters for our selected model as the ones we saw in the previous section, using sagemaker.hyperparameters.retrieve_default(). We then instantiate a SageMaker estimator and call the .fit method to start fine-tuning our model, passing it the Amazon S3 URI for our training data. As you can see, the entry_point script provided is named transfer_learning.py (the same for other tasks and models), and the input data channel passed to .fit must be named training.

Deploying the fine-tuned model

When training is complete, you can deploy your fine-tuned model. To do so, all we need to obtain is the inference script URI (the code that determines how the model is used for inference once deployed) and the inference container image URI, which includes an appropriate model server to host the model we chose. See the following code:

# Retrieve the inference docker container uri
deploy_image_uri = image_uris.retrieve(
    region=None,
    framework=None,
    image_scope="inference",
    model_id=model_id,
    model_version=model_version,
    instance_type=inference_instance_type,
)
# Retrieve the inference script uri
deploy_source_uri = script_uris.retrieve(
    model_id=model_id, model_version=model_version, script_scope="inference"
)

endpoint_name = name_from_base(f"jumpstart-example-FT-{model_id}-")

# Use the estimator from the previous step to deploy to a SageMaker endpoint
finetuned_predictor = ic_estimator.deploy(
    initial_instance_count=1,
    instance_type=inference_instance_type,
    entry_point="inference.py",
    image_uri=deploy_image_uri,
    source_dir=deploy_source_uri,
    endpoint_name=endpoint_name,
)

After a few minutes, our model is deployed and we can get predictions from it in real time!

Next, we invoke the endpoint to predict what type of flowers exist in the example image. We use the query_endpoint and parse_response helper functions, which are defined in the accompanying notebook.

query_response = finetuned_predictor.predict(
        img, {"ContentType": "application/x-image", "Accept": "application/json;verbose"}
    )
model_predictions = json.loads(query_response)
predicted_label = model_predictions["predicted_label"]
display(
    HTML(
        f'<img src={image_filename} alt={image_filename} align="left" style="width: 250px;"/>'
        f"<figcaption>Predicted Label: {predicted_label}</figcaption>"
    )
)

Incrementally train the fine-tuned model and redeploy

We can increase the performance of a fine-tuned model by further training it on new images. You may use any number of new or old images for this, however the dataset format must remain the same (same set of classes). The incremental training step is similar to the fine-tuning process, with an important difference: in the initial fine-tuning we start with a pre-trained model, whereas in incremental training we start with an existing fine-tuned model. See the following code:

last_trained_model_path =  f"{s3_output_location}/{last_training_job_name}/output/model.tar.gz"
incremental_s3_output_location = f"s3://{output_bucket}/{incremental_output_prefix}/output"incremental_train_estimator = Estimator(
    role=aws_role,
    image_uri=train_image_uri,
    source_dir=train_source_uri,
    model_uri=last_trained_model_path,
    entry_point="transfer_learning.py",
    instance_count=1,
    instance_type=training_instance_type,
    max_run=360000,
    hyperparameters=hyperparameters,
    output_path=incremental_s3_output_location,
    base_job_name=incremental_training_job_name,
)

incremental_train_estimator.fit({"training": training_dataset_s3_path}, logs=True)

When training is complete, we can use the same steps as the ones described in the preceding section to deploy the model.

Conclusion

JumpStart is a capability in SageMaker that allows you to quickly get started with ML. JumpStart uses open-source pre-trained models to solve common ML problems like image classification, object detection, text classification, sentence pair classification, and question answering.

In this post, we showed you how to fine-tune and deploy a pre-trained image classification model. We also showed how to incrementally train a fine-tuned model for image classification. With JumpStart, you can easily perform this process with no need to code. Try out the solution on your own and let us know how it goes in the comments. To learn more about JumpStart, check out the AWS re:Invent 2020 video Get started with ML in minutes with Amazon SageMaker JumpStart.

References

  1. The TensorFlow Team, 2019

About the Authors

Dr. Vivek Madan is an Applied Scientist with the Amazon SageMaker JumpStart team. He got his PhD. from University of Illinois at Urbana-Champaign and was a Post Doctoral Researcher at Georgia Tech. He is an active researcher in machine learning and algorithm design and has published papers in EMNLP, ICLR, COLT, FOCS, and SODA conferences.

João Moura is an AI/ML Specialist Solutions Architect at Amazon Web Services. He is mostly focused on NLP use cases and helping customers optimize deep learning model training and deployment. He is also an active proponent of low-code ML solutions and ML-specialized hardware.

Dr. Ashish Khetan is a Senior Applied Scientist with Amazon SageMaker JumpStart and Amazon SageMaker built-in algorithms and helps develop machine learning algorithms. He is an active researcher in machine learning and statistical inference and has published many papers in NeurIPS, ICML, ICLR, JMLR, ACL, and EMNLP conferences.

Read More

How eMagazines utilizes Amazon Polly to voice articles for school-aged kids

by Amazon AWS

This is a guest post by Andrew Degenholtz, CEO and Founder of eMagazines, the parent company of ReadAlong.ai. eMagazines’ technology seamlessly transforms print products into premium digital and audio experiences. Leveraging Amazon technology, ReadAlong.ai offers a simple, turn-key way for publishers to add audio to their websites with a single line of code.

eMagazines supports publishers in bringing high-quality journalism content to readers across digital platforms. Our ReadAlong.ai brand allows our customers to deepen their connection to readers by adding audio to traditional text-first publishing formats. In March 2020, we helped TIME for Kids launch a digital version of its popular magazine for school-aged kids. This premium subscription product helped their users transition to digital when the pandemic forced schools to close and families needed high-quality educational tools to supplement classroom learning materials.

In this post, we share how we created an automated way for TIME for Kids to seamlessly add audio for early readers and pre-readers through ReadAlong.ai, which uses Amazon Polly technology.

Why did TIME for Kids decide to start creating audio narration of their articles?

The addition of audio with auto scrolling and highlighting of text supports pre-readers and those students still learning to read. Listening while reading supports vocabulary development and reading comprehension, and new words are more likely to be learned when both their oral and written forms are provided. A report from the National Center on Early Childhood Development, Teaching, and Learning states that developing brains need to hear language even before learning to talk, and that even infants’ brains are preparing to speak months before they say their first words. Not only that, but the report also revealed that listening to stories read aloud helps expand both the volume and variety of words entering young vocabularies and fields of comprehension. Experts at Scholastic report that being read to also helps early readers “focus on the sounds of words read without interruption and provides a model of fluent reading,” and also noted that resources like audio help children learn how to listen, a prerequisite to learning to read.

What was the business challenge we addressed?

TIME for Kids originally addressed pre-reader accessibility by hiring voice actors to record their stories. The earlier iteration of their audio play button used an HTML audio player without speed variation or the option to scroll the page or highlight the text. The experience was expensive and time-consuming, and the user experience wasn’t as engaging as it could be. TIME for Kids was also unable to see even basic data around play or completion rates.

Why Amazon Polly?

We chose Amazon Polly because its APIs and web services support our goal of automating processes and making things easy for our clients.

Amazon Polly’s neural text-to-speech synthesis does the best job of voicing words within the context of a sentence, and the consistency in speech quality allows for the automation of article rendering.

Additionally, Amazon Polly offers a responsive API and powerful SSML support. This offers support for those cases where more control is needed to change inflection, and in the event that text contains challenging names (people, brands, companies) or word and phrase replacements (reading out abbreviations or acronyms in a particular way).

Amazon Polly also supports speech marks, which are crucial for highlighting the text that is currently being read out.

For TIME for Kids, the Kevin voice was a clear winner. TIME for Kids loved the approachable sound of the Kevin voice—they wanted wanting a voice that sounded like a child’s in order to help establish a sense of connection with young readers. Hear an example of a TIME for Kids article using the Kevin voice.

The technical challenge

TIME for Kids needed an educational audio solution for their website. It needed to be a one-time setup that was highly automated and very low friction. The solution also needed to process new articles as they were added dynamically, on a daily basis. And when a user listens to the audio, the page needed to scroll along with the text and highlight the sentence currently being read out loud.

Part of our challenge was to reliably and programmatically identify which content should be read aloud. In a typical publishing context, the audio player needs to read the article title and content, but avoid reading the header and footer text, navigation bars, and certain kinds of ads or captions. Our page analysis solution combines positive and negative query selectors. For each configuration, defined by a set of articles that share the same structure and layout, the http://readalong.ai solution supports a set of allow list selectors and a set of deny list selectors that together capture the appropriate content for synthesizing speech.

Furthermore, the TIME for Kids website posed many technical challenges because some pages are available only for paying subscribers, whereas some are open to the public. TIME for Kids offers four grade-specific editions, teaching materials, curriculum guides, and weekly virtual learning plans for each issue, as well as worksheets and quizzes. Therefore, each article has multiple versions for different reading levels in both English and Spanish—some with as many as seven different reading levels in both languages.

Our solution

We created a simple drop-in script that allowed TIME for Kids to only add one line of code to the header of any page where they wanted to offer audio. The script automated everything from page content delivery to audio-synthesis to webpage integration. Since the start of the school year, we’ve added the Kevin and Lupe voices (for English and Spanish content, respectively) to thousands of articles on timeforkids.com.

Our solution allowed for automated content delivery and audio synthesizing, which meant no need to sign into a dashboard, FTP, Dropbox, or otherwise send new article content to ReadAlong.ai each time a new page was added. The user-friendly backend of the solution also allows TIME for Kids to easily make word replacements, including global rules, to give the audio synthesizer engine lexicon hints for context-based pronunciations and difficult names, brands, or acronyms.

In addition to positioning and styling the launcher and player to match the TIME for Kids site design, as part of the customization, we added functionality to highlight and scroll the text as the article is read aloud, which is another helpful tool to support children in learning to recognize words and connect them to sounds. We customized this feature to be visible but not distracting, so the audio and visual elements could work in tandem to aid young readers. To support this enhanced feature, we implemented the detailed word- and sentence-level metadata available in Amazon Polly to provide a fluid highlighting experience that helps readers follow along as they encounter new words and concepts. This allows the listener to identify what they’re hearing as they view the content as it’s highlighted on the browser.

We also created a default for the Amazon Polly Kevin and Lupe voices to start at a slower speed, so the default pacing is at .9x, rather than at 1x, as another way to help early readers and pre-readers better access the content. Listeners have the ability to lower the default voice speed to .75x or increase to 1.5x, in order to accommodate more reading levels.

Business benefits for the customer

With our product in place on their site, TIME for Kids was able to voice their content in a scalable way. They deliver content on an article-by-article basis in two different languages (English and Spanish) and in seven different reading levels.

They’re also now able to easily collect and analyze data in real time, including both play and completion rates, and view most popular articles as well as articles with the most audio engagement.

We now know that 55% of kids that click to listen to an article complete 100% of the article, and 66% of kids that listen to an article complete more than half of the article. These significant completion rates reinforce the benefit and confirm that listeners are comfortable with the technology and the voice is relatable. The ReadAlong.ai audio also helped TIME for Kids promote its advanced accessibility features, including key articles with Spanish translation and read-aloud functionality, because the presence of the audio is featured prominently on the preview of each article along with other benefits (such as Spanish translation).

Stacy Bien, Director of Curriculum for TIME for Kids, was impressed with both the solution and the engagement data, saying,

“This is really a thing of beauty. This solution will help so many early readers develop their reading skills and easily consume more content. For us, we’ve seen a huge lift in engagement. That, coupled with the ease of use and cost-effectiveness, makes this a slam dunk.”

Conclusion

ReadAlong.ai used Amazon Polly to help TIME for Kids streamline the process of adding high-quality audio voiceover content to its premium subscription product. Our solution enabled the customer to significantly improve product time, precision, and cost. For example, a voiceover artist typically spends 1 hour or more to record an article, edit the audio, and master the final audio output. Now, once the ReadAlong.ai script has been added to the site, when new articles are created, the content is automatically processed without any time spent by a voiceover artist, audio editor, or administrator. The audio reads articles precisely and rarely requires adjustments, creating a valuable and immeasurable savings of both time and cost.

Collected KPIs tell us that not only did this become an easy way for the TIME for Kids team to manage audio functionality, but that the end-users—children early in the development of their reading abilities—take to the functionality as another tool on their reading path.

About the Author

Andrew Degenholtz is CEO and Founder of eMagazines and ReadAlong.ai, and is President of ValueMags, which he founded in 1999. Degenholtz holds a master’s in marketing from Northwestern University and a B.A. from Muhlenberg College. Previously, he was a member of the Alliance for Audited Media digital edition task force, created to develop best practices for acquisition of digital magazine subscribers.

Read More

Weekly forecasts can now start on Sunday with Amazon Forecast

by Amazon AWS

We are excited to announce that in Amazon Forecast, you can now start your forecast horizon at custom starting points, including on Sundays for weekly forecasts. This allows you to more closely align demand planning forecasts to local business practices and operational requirements.

Forecast is a fully managed service that uses statistical and machine learning (ML) algorithms to deliver highly accurate time series forecasts. It uses state-of-the-art algorithms to predict future time series data based on historical data, and requires no ML experience. Typical Forecast applications include resource planning for inventory, workforce staffing, and web traffic. In this post, we review a new option that allows you to align forecasts with business and demand cycles, while reducing operational cost by offloading aggregation workflows.

To optimize demand planning, forecasts need to align with business operations. Previously, starting points for forecasts were fixed: daily forecasts assumed demand starting at midnight each day, weekly predictions assumed Monday as the first day of the week, and monthly predictions started on the first day of each month. These predefined starting points presented two challenges. First, if your business cycle began at a different point than the fixed value, you had to manually aggregate forecasts to your required starting point. For example, if your business week began on a Sunday and you wanted to produce weekly forecasts, you had to manually aggregate daily forecasts to a Sunday–Saturday week. This additional work added cost and compute time, and presented opportunities for errors. Second, the training data and forecast periods weren’t consistent; if your data reflects a demand cycle that begins on Sundays, the predictor and forecast should also use Sunday as the starting point.

Custom forecast horizon starting points now align business operations and forecasts, eliminating the need for manual aggregation work and saving cost and compute. If you have a business week starting on Sundays, you can automatically aggregate daily data to generate weekly forecasts that begin on Sundays. Or you can begin daily forecasts starting at 9:00 AM. Predictors can now be aligned with your ground truth data, providing consistency between inputs and forecasts. Forecast horizon starting points are easily defined when training new predictors via the Forecast console or using Forecast APIs.

Define custom forecast horizon starting periods

The forecast horizon, also called frequency, is the length of time for which a forecast is made, and is bounded by a starting and ending point. In Forecast, you can now select specific starting points for daily, weekly, monthly, and yearly forecast horizons when training new predictors. These starting points—also called boundary values—are selected at one frequency unit finer than the forecast horizon, as shown in the following table.

Forecast frequency unit Boundary unit Boundary values
Daily Hour 0–23
Weekly Day of week Monday through Sunday
Monthly Day of month 1 through 28
Yearly Month January through December

With custom starting points, you can align forecasts to start at specific points in time that match your business processes and ground truth data, for example, the month of May, the 15th of the month, Sundays, or 15:00 hours. For forecast horizons coarser than the provided time series frequency, Forecast aggregates the time series data based on the custom starting point. For example:

  • When generating daily forecasts from hourly data with a 9:00 AM starting period, forecasts are aggregated with hourly data each day between 9:00 AM to the following day at 8:00 AM
  • When generating weekly forecasts from daily data with a Sunday starting period, forecasts are aggregated with daily data each week from Sunday to the following Saturday
  • When generating monthly forecasts from daily data with a starting day of the 15th of the month, forecasts are aggregated with daily data from the 15th of the current month to the 14th of the next month
  • When generating yearly forecasts from monthly data with a starting month of May, forecasts are aggregated with monthly data from May of the current year to April of next year

Available forecast frequencies

The following screenshots show examples of custom daily, weekly, monthly, and yearly forecast frequencies and starting points (the Time alignment boundary field on the Forecast console).




Specify custom forecast horizon starting points

You can define custom forecast horizon starting points when creating a new predictor. The following steps demonstrate how to do this using the Forecast console. We also offer a sample notebook that provides an example of how to integrate this new setting into your workflows.

  1. On the Forecast console, choose View dataset groups, and then Create dataset group.
  2. Create your dataset group, a target time series dataset, and load your data.
    You’re redirected to the Forecast console as your data is loaded.
  3. After your target time series dataset is loaded into your dataset group and active, choose Start under Train a predictor.
  4. In the Train predictor section, provide values for the Name, Forecast frequency, and Forecast horizon fields.
  5. In the optional Time alignment boundary field, specify the starting point the predictor uses for the forecast.
    The values in this list depend on the Forecast frequency value you choose. In this example, we create weekly forecasts with a 1-week horizon, with Sunday as the starting day of the week and of the forecast.
  6. Provide other optional configurations as needed and choose Create.

    After you create the predictor, you can create your forecast.
  7. In the navigation pane, under your dataset group choose Predictors.
  8. Select your new predictor.
  9. Choose Create forecast.
  10. Provide the necessary details and choose Start to create your forecast.
  11. When the forecast is complete, choose Create forecast export to export the results.

The following screenshots are samples of the original input file (left) and the exported forecast results (right). The input file is at an hourly frequency, whereas the forecast is produced at a weekly frequency, beginning with Sunday as the first day of the week. This is an example of Forecast automatically aggregating over two levels of forecast frequencies (from hours to days).

Conclusion

Custom forecast horizon starting points in Forecast allow you to produce forecasts that align with your specific operational requirements. Work weeks start on different days in different regions, requiring forecasts that begin on days other than Mondays, and that are aligned with ground truth training and ongoing data. Or you may want to generate hourly forecasts that reflect a demand cycle beginning at 7:00 AM each day, for example.

Forecast also automatically aggregates fine-grained forecasts to higher-level frequencies (such as days into weeks). This allows you to produce forecasts aligned with your operations, and saves you costs by removing the need to stand up and manage aggregation workflows.

Custom starting points are optional. If you don’t provide specific starting points, forecasts start at default times. Specific forecast horizon starting points are only available with AutoPredictor. For more information, refer to New Amazon Forecast API that creates up to 40% more accurate forecasts and provides explainability and CreateAutoPredictor.

To learn more about forecast frequencies, refer to Data aggregation for different forecast frequencies. All these new capabilities are available in all Regions where Forecast is publicly available. For more information about Region availability, see AWS Regional Services.

About the Authors

Dan Sinnreich is a Sr. Product Manager for Amazon Forecast. He is focused on democratizing low-code/no-code machine learning and applying it to improve business outcomes. Outside of work, he can be found playing hockey, trying to improve his tennis serve, scuba diving, and reading science fiction.

Paras Arora is a Software Development Engineer in the Amazon Forecast Team. He is passionate about building cutting edge AI/ML solutions in the cloud. In his spare time, he enjoys hiking and traveling.

Chetan Surana is a Software Development Engineer in the Amazon Forecast team. His interests lie at the intersection of machine learning and software development, applying thoughtful design and engineering skills to solve problems. Outside of work, he enjoys photography, hiking, and cooking.

Read More

Continuously monitor predictor accuracy with Amazon Forecast

by Amazon AWS

We’re excited to announce that you can now automatically monitor the accuracy of your Amazon Forecast predictors over time. As new data is provided, Forecast automatically computes predictor accuracy metrics, providing you with more information to decide whether to keep using, retrain, or create new predictors.

Monitoring predictor quality and identifying deterioration in accuracy over time is important to achieving business goals. However, the processes required to continuously monitor predictor accuracy metrics can be time-consuming to set up and challenging to manage: forecasts have to be evaluated, and updated accuracy metrics have to be computed. In addition, metrics have to be stored and charted to understand trends and make decisions about keeping, retraining, or recreating predictors. These processes can result in costly development and maintenance burdens, and place meaningful operational stress on data science and analyst teams. And for customers not willing to take on this time-consuming process (they would prefer to retrain new predictors even when not needed), this wastes time and compute.

With today’s launch, Forecast now automatically tracks predictor accuracy over time as new data is imported. You can now quantify your predictor’s deviation from initial quality metrics and systematically evaluate model quality by visualizing trends, and make more informed decisions about keeping, retraining, or rebuilding your models as new data comes in. Predictor monitoring can be enabled for new predictors at inception, or turned on for existing models. You can enable this feature with one click on the AWS Management Console or using Forecast APIs.

Predictor accuracy over time

A predictor is a machine learning model created at a point in time, using an original set of training data. After a predictor is created, it’s used on an ongoing basis over days, weeks, or months into the future to generate time series forecasts with new ground truth data generated through actual transactions. As new data is imported, the predictor generates new forecasted data points based on the latest data provided to it.

When a predictor is first created, Forecast produces accuracy metrics such as weighted quantile loss (wQL), mean absolute percentage error (MAPE), or root mean squared error (RMSE) to quantify the accuracy of the predictor. These accuracy metrics are used to determine whether a predictor will be put into production. However, the performance of a predictor will fluctuate over time. External factors such as changes in the economic environment or in consumer behavior can change the fundamental factors underlying a predictor. Other factors include new products, items, and services that may be created; changes in the financial or economic environment; or changes in the distribution of data.

For example, consider a predictor trained when a certain color of a product was popular. Months later, new colors may appear or become more popular and the distribution of values change. Or a shift occurs in the business environment that modifies long-standing purchasing patterns (such as from high-margin to low-margin products). All things considered, the predictor may need to be retrained, or a new predictor may need to be created to ensure highly accurate predictions continue to be made.

Automated predictor monitoring

Predictor monitoring is designed to automatically analyze your predictor’s performance as new ground truth time series data becomes available and is used to create new forecasts. This monitoring provides you with continuous model performance information, and saves you time so you don’t have to set up the process yourself.

If predictor monitoring is enabled in Forecast, each time you import new data and produce a new forecast, performance statistics are updated automatically. Until now, these performance statistics were only available when the predictor was initially trained; now these statistics are produced on a continuous basis using new ground truth data, and can be actively monitored to gauge predictor performance.

This allows you to use predictor performance statistics to decide when to train or retrain a new predictor. For example, as the average wQL metric deviates from the initial baseline values, you can determine whether to retrain a new predictor. If you decide to retrain a predictor or create a new one, you can begin generating new forecasted data points using the more accurate predictor.

The following graphs provide two examples of predictor monitoring. In the first chart, the average wQL metric is decreasing from the baseline (the initial value when the predictor was trained), indicating that forecast accuracy is increasing over time. The chart shows average wQL dropping from 0.3 to 0.15 over the course of a few days, meaning that forecast accuracy is increasing. In this case, there is no need to retrain the predictor because it’s producing more accurate forecasts than when it was first trained.

In the next figure, the opposite is true: the average wQL is increasing, indicating that accuracy is decreasing over time. In this case, you should consider retraining or rebuilding the predictor with new data.

In Forecast, you have the choice of retraining the current predictor or rebuilding it from scratch. Retraining is done with one click and incorporates more up-to-date data and any updates and improvements in the Forecast algorithms. Rebuilding the predictor allows you to provide new inputs (such as forecast frequency, horizon, or new dimension) to create a new predictor.

Enable predictor monitoring

You can enable predictor monitoring when creating a new predictor, or turn it on for existing predictors. The steps in this section demonstrate how to perform these steps using the Forecast console. There is also a Jupyter notebook that walks through a sequence of steps to enable predictor monitoring using APIs and generate predictor monitor results.

This example uses the time-sliced sample dataset available from the predictor monitoring notebook. In our example, we start with a 100,000-row dataset of New York City taxi pickups containing a timestamp, location ID, and target value (the number of pickups requested during the timestamp at the location ID).

Complete the following steps:

  1. On the Forecast console, choose View dataset groups in the navigation pane.
  2. Choose Create dataset group and provide your dataset group details.
    After you create the dataset group, you’re prompted to create a target time series dataset. You use this dataset to train the predictor and create forecasts.
  3. On the Create target time series dataset page, provide your data’s schema, frequency, and location.
  4. Choose Start to import your target dataset.
    Next, you build your predictor and train it using your initial dataset.
  5. In the navigation pane, choose Predictors.
  6. Choose Train new predictor.
  7. In the Predictor settings section, enter a name for your predictor, how long in the future you want to forecast and at what frequency, and the number of quantiles you want to forecast for.
  8. For Optimization metric, you can choose an optimization metric to optimize AutoPredictor to tune a model for a specific accuracy metric of your choice. We leave this as default for our walkthrough.
  9. To get the predictor explainability report, select Enable predictor explainability.
  10. To enable predictor monitoring, select Enable predictor monitoring.
  11. Under the input data configuration, you can add local weather information and national holidays for more accurate demand forecasts.
  12. Choose Start to start training your predictor.

    Forecast now trains the predictor with this initial dataset. With predictor monitoring enabled, every time new data is provided in this dataset group, Forecast is able to compute updated predictor accuracy metrics.
  13. After the predictor has been trained, choose it to evaluate the initial accuracy metrics.

    The Metrics tab shows initial predictor quality metrics. Because you haven’t generated any forecasts from your predictor or imported any new ground truth data, there is nothing to show on the Monitoring tab.
    The next step is to generate a forecast using the new predictor.
  14. Choose Forecasts in the navigation pane.
  15. Choose Create forecast to create a new forecast based on the time series data you just imported and the predictor settings.
  16. Provide the forecast name, predictor name, and any additional quantile metrics you wish to compute.

After you create the forecast, you can view and export its details and results on the Forecast details page.

Predictor monitoring: Evaluating accuracy over time

Through the passage of time, new ground truth data is created by your business processes, for example, updated sales figures, staffing levels, or manufacturing output. To create new forecasts based on that new data, you can import your data to the dataset you created.

  1. On the Amazon Forecast console, on the Dataset groups page, choose your dataset group.
  2. Choose your dataset.
  3. In the Dataset imports section, choose Create dataset import.
  4. Provide additional details about your updated data, including its location.
  5. Choose Start.

With predictor monitoring, Forecast compares this new data to the previous forecast generated, and computes accuracy metrics for the predictor. Updated predictor quality metrics are computed on an ongoing basis as new data is added to the dataset.

You can follow these steps to import additional data, representing additional transactions that have occurred through time.

Evaluate predictor monitoring results

To see predictor monitoring results, you must add new ground truth data after generated the initial forecasts. Forecast compares this new ground truth data to the previous forecast, and produces updated model accuracy values for monitoring.

  1. On the Dataset groups page, choose the relevant dataset groups and select the Target Time Series to update it with new ground truth data.
  2. Choose Create Dataset Import and add your new ground truth data.

    After you provide the additional ground truth data, you can open your predictor and view initial predictor monitoring statistics.
  3. Choose your predictor and navigate to the Monitoring tab.

You can follow these steps to run additional forecasts using this predictor and add further iterations of ground truth data. The progression of model accuracy statistics for your predictor are available on the Monitoring tab.

This example shows model accuracy statistics for a predictor that has been evaluated with four additional data updates. The predictor had an initial baseline MAPE of 0.55 when it was initially trained. As additional data was loaded, the MAPE dropped to .42 with the first additional dataset, indicating a more accurate predictor, and fluctuated within a tight range from .42 to .48 with subsequent datasets.

You can toggle the chart to view additional metrics. In the following examples, MASE and average wQL show similar fluctuations from the baseline over time.

The Monitoring History section at the bottom of the page provides full details on all predictor accuracy metrics tracked over time.

Set up prediction monitoring on an existing predictor

You can easily enable monitoring for existing predictors. To do so, complete the following steps:

  1. In the navigation pane, under your dataset, choose Predictors.
  2. From here there are two ways to enable monitoring:
    1. Choose Start monitoring under the Monitoring column.
    2. Choose your predictor and on the Monitoring tab, under Monitor details, choose Start monitor.
  3. In the pop-up dialog, choose Start to start monitoring for the selected predictor.

The Monitoring tab now shows that predictor monitoring has started, and results are generated as you import more data.

Stop and restart predictor monitoring

You can also stop and restart predictor monitoring. Consider the following:

  • Cost – Predictor monitoring consumes additional resources. With typical small datasets, the cost is minimal, but may increase with large datasets (number of items in the input dataset, and forecast horizon).
  • Privacy – A copy of your forecast is stored during monitoring. If you don’t want to store this copy, you can stop monitoring.
  • Noise – If you’re experimenting with a predictor and don’t want to see noise in your predictor monitor results, you can temporarily stop predictor monitoring and start it again when your predictor is stable again.

To stop predictor monitoring, complete the following steps:

  1. Navigate to the Monitoring tab for a predictor where monitoring is enabled.
  2. Choose Stop Monitor to stop the monitoring of the predictor.
  3. Verify your choice when prompted.

A message shows on the next page to indicate that predictor monitoring is stopped.

You can restart predictor monitoring by choosing Resume monitor.

Conclusion

Monitoring the quality of your predictors over time is important to achieve your demand planning and forecasting objectives, and ultimately your business goals. However, predictor monitoring can be a time-consuming exercise, and the processes required to stand up and maintain the necessary workflows can lead to higher operational costs.

Forecast can now automatically track the quality of your predictors, allowing you to reduce operational efforts, while helping you make more informed decisions about keeping, retraining, or rebuilding your predictors. To enable predictor monitoring, you can follow the steps outlined in this post, or follow our GitHub notebook.

Please note that predictor monitoring is only available with AutoPredictor. For more information, refer to New Amazon Forecast API that creates up to 40% more accurate forecasts and provides explainability and CreateAutoPredictor.

To learn more, refer to Predictor Monitoring. We also recommend reviewing the pricing for using these new features. All these new capabilities are available in all Regions where Forecast is publicly available. For more information about Region availability, see AWS Regional Services.

About the Authors

Dan Sinnreich is a Sr. Product Manager for Amazon Forecast. He is focused on democratizing low code/no code machine learning and applying it to improve business outcomes. Outside of work he can be found playing hockey, trying to improve his tennis serve, and reading science fiction.

 Adarsh Singh works as a Software Development Engineer in the Amazon Forecast team. In his current role, he focuses on engineering problems and building scalable distributed systems that provide the most value to end users. In his spare time, he enjoys watching anime and playing video games.

Shannon Killingsworth is a UX Designer for Amazon Forecast. His current work is creating console experiences that are usable by anyone, and integrating new features into the console experience. In his spare time, he is a fitness and automobile enthusiast.

Read More

Unified data preparation and model training with Amazon SageMaker Data Wrangler and Amazon SageMaker Autopilot

by Amazon AWS

Data fuels machine learning (ML); the quality of data has a direct impact on the quality of ML models. Therefore, improving data quality and employing the right feature engineering techniques are critical to creating accurate ML models. ML practitioners often tediously iterate on feature engineering, choice of algorithms, and other aspects of ML in search of optimal models that generalize well on real-world data and deliver the desired results. Because speed in doing business disproportionately matters, this extremely tedious and iterative process may lead to project delays and lost business opportunities.

Amazon SageMaker Data Wrangler reduces the time to aggregate and prepare data for ML from weeks to minutes, and Amazon SageMaker Autopilot automatically builds, trains, and tunes the best ML models based on your data. With Autopilot, you still maintain full control and visibility of your data and model. Both services are purpose-built to make ML practitioners more productive and accelerate time to value.

Data Wrangler now provides a unified experience enabling you to prepare data and seamlessly train a ML model in Autopilot. With this newly launched feature, you can now prepare your data in Data Wrangler and easily launch Autopilot experiments directly from the Data Wrangler user interface (UI). With just a few clicks, you can automatically build, train, and tune ML models, making it easier to employ state-of-the-art feature engineering techniques, train high-quality ML models, and gain insights from your data faster.

In this post, we discuss how you can use this new integrated experience in Data Wrangler to analyze datasets and easily build high-quality ML models in Autopilot.

Dataset overview

Pima Indians are an Indigenous group that live in Mexico and Arizona, US. Studies show Pima Indians as a high-risk population group for diabetes mellitus. Predicting the probability of an individual’s risk and susceptibility to a chronic illness like diabetes is an important task in improving the health and well-being of this often underrepresented minority group.

We use the Pima Indian Diabetes public dataset to predict the susceptibility of an individual to diabetes. We focus on the new integration between Data Wrangler and Autopilot to prepare data and automatically create an ML model without writing a single line of code.

The dataset contains information about Pima Indian females 21 years or older and includes several medical predictor (independent) variables and one target (dependent) variable, Outcome. The following chart describes the columns in our dataset.

Column Name Description
Pregnancies The number of times pregnant
Glucose Plasma glucose concentration in an oral glucose tolerance test within 2 hours
BloodPressure Diastolic blood pressure (mm Hg)
SkinThickness Triceps skin fold thickness (mm)
Insulin 2-hour serum insulin (mu U/ml)
BMI Body mass index (weight in kg/(height in m)^2)
DiabetesPedigree Diabetes pedigree function
Age Age in years
Outcome The target variable

The dataset contains 768 records, with 9 total features. We store this dataset in Amazon Simple Storage Bucket (Amazon S3) as a CSV file and then import the CSV directly into a Data Wrangler flow from Amazon S3.

Solution overview

The following diagram summarizes what we accomplish in this post.[KT1]

Data scientists, doctors, and other medical domain experts provide patient data with information on glucose levels, blood pressure, body mass index, and other features used to predict the likelihood of having diabetes. With the dataset in Amazon S3, we import the dataset into Data Wrangler to perform exploratory data analysis (EDA), data profiling, feature engineering, and splitting the dataset into train and test for model building and evaluation.

We then use Autopilot’s new feature integration to quickly build a model directly from the Data Wrangler interface. We choose Autopilot’s best model based on the model with the highest F-beta score. After Autopilot finds the best model, we run a SageMaker Batch Transform job on the test (holdout) set with the model artifacts of the best model for evaluation.

Medical experts can provide new data to the validated model to obtain a prediction to see if a patient will likely have diabetes. With these insights, medical experts can start treatment early to improve the health and well-being of vulnerable populations. Medical experts can also explain a model’s prediction by referencing the model’s detail in Autopilot because they have full visibility into the model’s explainability, performance, and artifacts. This visibility in addition to validation of the model from the test set gives medical experts greater confidence in the model’s predictive ability.

We walk you through the following high-level steps.

  1. Import the dataset from Amazon S3.
  2. Perform EDA and data profiling with Data Wrangler.
  3. Perform feature engineering to handle outliers and missing values.
  4. Split data into train and test sets.
  5. Train and build a model with Autopilot.
  6. Test the model on a holdout sample with a SageMaker notebook.
  7. Analyze validation and test set performance.

Prerequisites

Complete the following prerequisite steps:

  1. Upload the dataset to an S3 bucket of your choice.
  2. Make sure you have the necessary permissions. For more information, refer to Get Started with Data Wrangler.
  3. Set up a SageMaker domain configured to use Data Wrangler. For instructions, refer to Onboard to Amazon SageMaker Domain.

Import your dataset with Data Wrangler

You can integrate a Data Wrangler data flow into your ML workflows to simplify and streamline data preprocessing and feature engineering using little to no coding. Complete the following steps:

  1. Create a new Data Wrangler flow.

If this is your first time opening Data Wrangler, you may have to wait a few minutes for it to be ready.

  1. Choose the dataset stored in Amazon S3 and import it into Data Wrangler.

After you import the dataset, you should see the beginnings of a data flow within the Data Wrangler UI. You now have a flow diagram.

  1. Choose the plus sign next to Data types and choose Edit to confirm that Data Wrangler automatically inferred the correct data types for your data columns.

If the data types aren’t correct, you can easily modify them through the UI. If multiple data sources are present, you can join or concatenate them.

We can now create an analysis and add transformations.

Perform exploratory data analysis with the data insights report

Exploratory data analysis is a critical part of the ML workflow. We can use the new data insights report from Data Wrangler to gain a better understanding of the profile and distribution of our data. The report includes summary statistics, data quality warnings, target column insights, a quick model, and information about anomalous and duplicate rows.

  1. Choose the plus sign next to Data types and choose Get data insights.

  1. For Target column, choose Outcome.
  2. For Problem type, and (optionally) select Classification.
  3. Choose Create.

The results show a summary data with the dataset statistics.

We can also view the distribution of the labeled rows with a histogram, an estimate of the expected predicted quality of the model with the quick model feature, and a feature summary table.

We don’t go into the details of analyzing the data insights report; refer to Accelerate data preparation with data quality and insights in Amazon SageMaker Data Wrangler for additional details about how you can use the data insights report to accelerate your data preparation steps.

Perform feature engineering

Now that we’ve profiled and analyzed the distribution of our input columns at a high level, the first consideration for improving the quality of our data could be to handle missing values.

For example, we know that zeros (0) for the Insulin column represent missing values. We could follow the recommendation to replace the zeros with NaN. But on closer examination, we find that the minimum value is 0 for others columns such as Glucose, BloodPressure, SkinThickness, and BMI. We need a way to handle missing values, but need to be sensitive to columns with zeros as valid data. Let’s see how we can fix this.

In the Feature Details section, the report raises a Disguised missing value warning for the feature Insulin.

Because zeros in the Insulin column are in fact missing data, we use the Convert regex to missing transform to transform zero values to empty (missing values).

  1. Choose the plus sign next to Data types and choose Add transform.
  2.  Choose Search and edit.
  3. For Transform, choose Convert regex to missing.
  4. For Input columns, choose the columns Insulin, Glucose, BloodPressure, SkinThickness, and BMI.
  5. For Pattern, enter 0.
  6. Choose Preview and Add to save this step.

The 0 entries under Insulin, Glucose, BloodPressure, SkinThickness, and BMI are now missing values.

Data Wrangler gives you a few other options to fix missing values.

  1. We handle missing values by imputing the approximate median for the Glucose column.

We also want to ensure that our features are on the same scale. We don’t want to accidentally give more weight to a certain feature just because they contain a larger numeric range. We normalize our features to do this.

  1. Add a new Process numeric transform and choose Scale values.
  2. For Scaler, choose Min-max scaler.
  3. For Input columns, choose the columns Pregnancies, BloodPressure, Glucose, SkinThickness, Insulin, BMI, and Age.
  4. Set Min to 0 and Max to 1.

This makes sure that our features are between the values 0 and 1.

Now that’s we’ve created some features, we split our dataset into training and testing before we build a model.

Split data into training and testing

In the model building phase of your ML workflow, you test the efficacy of your model by running batch predictions. You can set aside a testing or holdout dataset for evaluation to see how your model performs by comparing the predictions to the ground truth. Generally, if more of the model’s predictions match the true labels, we can determine the model is performing well.

We use Data Wrangler to split our dataset for testing. We retain 90% of our dataset for training because we have a relatively small dataset. The remaining 10% of our dataset serves as the test dataset. We use this dataset to validate the Autopilot model later in this post.

We split our data by choosing the Split data transform and choosing Randomized split as the method. We designate 0.9 as the split percentage for training and 0.1 for testing.

With the data transformation and featuring engineering steps complete, we’re now ready to train a model.

Train and validate the model

We can use the new Data Wrangler integration with Autopilot to directly train a model from the Data Wrangler data flow UI.

  1. Choose the plus sign next to Dataset and choose Train model.

  1. For Amazon S3 location, specify the Amazon S3 location where SageMaker exports your data.

Autopilot uses this location to automatically train a model, saving you time from having to define the output location of the Data Wrangler flow, then having to define the input location of the Autopilot training data. This makes for a more seamless experience.

  1. Choose Export and train to initiate model building with Autopilot.

Autopilot automatically selects the training data input and output locations. You only need to specify the target column and click Create Experiment to train your model.

Test the model on a holdout sample

When Autopilot completes the experiment, we can view the training results and explore the best model.

  1. Choose View model details for your desired model, then choose the Performance tab on the model details page.

The Performance tab displays several model measurement tests, including a confusion matrix, the area under the precision/recall curve (AUCPR), and the area under the receiver operating characteristic curve (ROC). These illustrate the overall validation performance of the model, but they don’t tell us if the model will generalize well. We still need to run evaluations on unseen test data to see how accurately the model predicts if an individual will have diabetes.

To ensure the model generalizes well enough, we set aside the test sample for independent sampling. We can do so in the Data Wrangler flow UI.

  1.  Choose the plus sign next to Dataset, choose Export to, and choose Amazon S3.

  1. Specify an Amazon S3 path.

We refer to this path when we run batch inference for validation in the next section.

  1. Create a new SageMaker notebook to perform batch inferencing on the holdout sample and assess the test performance. Refer to the following GitHub repo for a sample notebook to run batch inference for validation.

Analyze validation and test set performance

When the batch transform is complete, we create a confusion matrix to compare the actual and predicted outcomes of the holdout dataset.

We see 23 true positives and 33 true negatives from our results. In our case, true positives refer to the model correctly predicting an individual as having diabetes. In contrast, true negatives refer to the model correctly predicting an individual as not having diabetes.

In our case, precision and recall are important metrics. Precision essentially measures all individuals predicted to have diabetes, how many really have diabetes? In contrast, recall helps measure all individual who indeed have diabetes, how many were predicted to have diabetes? For example, you may want to use a model with high precision because you want to treat as many individuals as you can, especially if the first stage of treatment has no effect on individuals without diabetes (these are false positives—those labeled as having it when in fact they do not).

We also plot the area under the ROC curve (AUC) graph to evaluate the results. The higher the AUC, the better the model is at distinguishing between classes, which in our case is how well the model performs at distinguishing patients with and without diabetes.

Conclusion

In this post, we demonstrated how to integrate your data processing, featuring engineering, and model building using Data Wrangler and Autopilot. We highlighted how you can easily train and tune a model with Autopilot directly from the Data Wrangler user interface. With this integration feature, we can quickly build a model after completing feature engineering, without writing any code. Then we referenced Autopilot’s best model to run batch predictions using the AutoML class with the SageMaker Python SDK.

Low-code and AutoML solutions like Data Wrangler and Autopilot remove the need to have deep coding knowledge to build robust ML models. Get started using Data Wrangler today to experience how easy it is to build ML models using SageMaker Autopilot.

About the Authors

Peter Chung is a Solutions Architect for AWS, and is passionate about helping customers uncover insights from their data. He has been building solutions to help organizations make data-driven decisions in both the public and private sectors. He holds all AWS certifications as well as two GCP certifications. He enjoys coffee, cooking, staying active, and spending time with his family.

Pradeep Reddy is a Senior Product Manager in the SageMaker Low/No Code ML team, which includes SageMaker Autopilot, SageMaker Automatic Model Tuner. Outside of work, Pradeep enjoys reading, running and geeking out with palm sized computers like raspberry pi, and other home automation tech.

Arunprasath Shankar is an Artificial Intelligence and Machine Learning (AI/ML) Specialist Solutions Architect with AWS, helping global customers scale their AI solutions effectively and efficiently in the cloud. In his spare time, Arun enjoys watching sci-fi movies and listening to classical music.

Srujan Gopu is a Senior Frontend Engineer in SageMaker Low Code/No Code ML helping customers of Autopilot and Canvas products. When not coding, Srujan enjoys going for a run with his dog Max, listening to audio books and VR game development.

Read More

Integrate Amazon Lex and Uneeq’s digital human platform

by Amazon AWS

In today’s digital landscape, customers are expecting a high-quality experience that is responsive and delightful. Chatbots and virtual assistants have transformed the customer experience from a point-and-click or a drag-and-drop experience to one that is driven by voice or text. You can create a more engaging experience by further augmenting the interaction with a visual modality.

Uneeq is an AWS Partner that specializes in developing animated visualizations of these voice bots and virtual agents, called. Uneeq’s digital humans can help provide a next-generation customer experience that is visual, animated, and emotional. Having worked with brands across numerous verticals such as UBS (financial services), Vodafone (telecommunications ), and Mentemia (healthcare), Uneeq helps customers enable innovative customer experiences powered by Amazon Lex.

Amazon Lex is a service for building conversational interfaces into any application using voice and text. Amazon Lex provides natural language understanding (NLU) and automatic speech recognition (ASR), enabling customer experiences that are highly engaging through conversational interactions.

In this post, we guide you through the steps required to configure an Amazon Lex V2 chatbot, connect it to Uneeq’s digital human, and manage a conversation.

Overview of solution

This solution uses the following services:

The following diagram illustrates the architecture of our solution.

The architecture utilizes AWS serverless resources for ease of deployment and to minimize any associated run costs with the deploying the solution.

The Uneeq digital human interfaces with a simple REST API, configured with Lambda proxy integration that in turn interacts with a deployed Amazon Lex bot.

After you deploy the bot, you need to configure it with a basic Welcome intent. In the first interaction with Uneeq’s digital human, the Welcome intent determines the initial phrase the Uneeq digital human gives. For example, “Hi, my name is Crissy and I am your digital assistant today. How can I help you?”

You deploy the solution with three high-level steps:

  1. Deploy an Amazon Lex bot.
  2. Deploy the integration, which is a simple API Gateway REST API and Lambda function using AWS Serverless Application Model (AWS SAM) .
  3. Create a Uneeq 14-day free trial account and connect Uneeq’s digital human to the Amazon Lex bot.

Prerequisites

To implement this solution, you need the following prerequisites:

These instructions assume a general working knowledge of the listed Amazon services, particularly AWS SAM and AWS CloudFormation.

Deploy an Amazon Lex Bot

For this solution, we use the BookTrip sample bot that is provided in Amazon Lex.

  1. On the Amazon Lex v2 console, choose Bots in the navigation pane.
  2. Choose Create bot.
  3. Select Start with an example.
  4. For Example bot, choose BookTrip.
  5. In the Bot configuration section, enter a bot name and optional description.
  6. Under IAM permissions, select Create a role with basic Amazon Lex permissions.
  7. Because this is a bot for demo purposes, it’s not subject to COPPA, so in the Children’s Online Privacy Protection Act (COPPA) section, select No.
  8. Leave the remainder of the settings as default and choose Next.
  9. Choose your preferred language and voice, which is provided by Amazon Polly.
  10. Choose Done to create your bot.

Edit the BookTrip bot welcome intent

When first initiated, Uneeq’s digital human utters dialog to introduce itself based on a welcome intent defined in the Amazon Lex bot.

  1. To add the welcome intent, browse to the intents for the BookTrip bot just created and create a new intent called Welcome by choosing Add intent.
  2. To configure the welcome intent, in the Closing Response section, enter the initial phrase that you want Uneeq’s digital human to utter. For this post, we use “Hi, my name is Crissy and I am your digital assistant today. How can I help you?”

This is the only configuration required for this intent.

  1. Choose Save intent.
  2. Choose Build to build the bot with the Welcome intent.
  3. Record the bot ID, alias ID, locale ID, and Welcome intent name to use in the next step to deploy the integration.

Deploy the integration using AWS SAM

Browse to the GitHub repo and clone the lexV2 branch. The template.yaml file is the AWS SAM configuration for the application; the swagger.yaml is the OpenAPI configuration for the API.

  1. Deploy this application by following the instructions in the README file.
  2. Make sure your AWS Command Line Interface (AWS CLI) configuration can access an AWS account.
  3. Browse to the root of the cloned repository and install the required dependencies by running the following command: 
    cd function && npm install && cd ..

  4. Prior to running the deploy command, upload the swagger.yaml file to an S3 bucket.
  5. Deploy the serverless application by running the following command from the root of the repository, and assign values to the listed parameters:
      1. pLexBotID
      2. pLexBotAliasID
      3. pWelcomeIntentName
      4. pLocaleID
      5. pS3BucketName
    sam deploy --template-file template.yml --s3-bucket %S3BUCKETNAME% --stack-name %STACKNAME% --parameter-overrides pLexBotID=%LexV2BotID% pLexBotAliasID=%AliasID% pWelcomeIntentName=Welcome pLocaleID=en_AU pS3BucketName=%S3BucketName% --capabilities CAPABILITY_NAMED_IAM

  6. Confirm the deployment has been successful by reviewing the output of the AWS SAM deployment.
  7. Take note of the API endpoint URL; you use this for configuring Uneeq’s digital human.

Create a Uneeq trial account and configure Uneeq’s digital human

Let’s start by creating a 14-day free trial account on the Uneeq website.

  1. On the Uneeq website, choose Free Trial.
  2. Enter the required details and verify your email address via a unique code that is sent to the provided email address.
  3. Choose a Uneeq digital human from the three provided to you as part of the free trial.

Uneeq has multiple personas available, but some require a paid subscription.

  1. Choose a background for Uneeq’s digital human.
  2. Enter a name for Uneeq’s digital human.
  3. Choose your preferred language and voice for Uneeq’s digital human.

You can choose Test Voice to hear an example of the voice.

  1. After you create Uneeq’s digital human, browse to the Uneeq dashboard and choose Personas.
  2. Choose the edit icon for Uneeq’s digital human you just created.
  3. In the Conversation settings section, choose Bring Your Own Conversation Platform.
  4. For API URL, enter the URL of our deployed API.
  5. Return to the Personas page and choose Try to start Uneeq’s digital human.

Uneeq’s digital human begins the interaction by uttering the dialog configured in your welcome intent.

For a demonstration of Uneeq’s digital human and Amazon Lex integration, watch Integrating Digital Humans with AWS Lambda – Devs in the Shed Episode 16.

Conclusion

In this post, I implemented a solution that integrates Amazon Lex with Uneeq’s digital human by enhancing the visual modality of the user experience. You can use this solution for multiple use cases by simply configuring it to a different Amazon Lex bot.

It’s easy to get started. Sign up for a free trial account with Uneeq’s digital human, and clone the GitHub repo to get started enhancing your customers’ interactions with your business. For more information about Amazon Lex, see Getting started with Amazon Lex and the V2 Developer Guide.

About the Author

Barry Conway is an Enterprise Solutions Architect with years of experience in the technology industry bridging the gap between business and technology. Barry has helped banking, manufacturing, logistics, and retail organizations realize their business goals.

Read More

Easily create and store features in Amazon SageMaker without code

by Amazon AWS

Data scientists and machine learning (ML) engineers often prepare their data before building ML models. Data preparation typically includes data preprocessing and feature engineering. You preprocess data by transforming data into the right shape and quality for training, and you engineer features by selecting, transforming, and creating variables when building a predictive model.

Amazon SageMaker helps you perform these tasks by simplifying feature preparation with Amazon SageMaker Data Wrangler and storage and feature serving with Amazon SageMaker Feature Store. You can prepare your data and engineer features using over 300 built-in transformations with Data Wrangler. Then you can persist those features to a purpose-built feature store for ML with Feature Store. These services help you build automatic and repeatable processes to streamline your data preparation tasks, all without writing code.

We’re excited to announce a new capability that seamlessly integrates Data Wrangler with Feature Store. You can now easily create features with Data Wrangler and store those features in Feature Store with just a few clicks in Amazon SageMaker Studio.

In this post, we demonstrate creating features with Data Wrangler and persisting them in Feature Store using the hotel booking demand dataset. We focus on the data preparation and feature engineering tasks to show how easily you can create and stores features in SageMaker without code using Data Wrangler. After the features are stored, they can be used for training and inference by multiple models and teams.

Solution overview

To demonstrate feature engineering and feature storage, we use a hotel booking demand dataset. You can download the dataset and view the full description of each variable. The dataset contains information such as when a hotel booking was made, the booking location, the length of stay, the number of parking spaces, and other features.

Our goal is to engineer features to predict if a user will cancel a booking.

We host the dataset in an Amazon Simple Storage Service (Amazon S3) bucket. We also open a Studio domain to utilize the native Data Wrangler and Feature Store capabilities. We import the dataset into a Data Wrangler flow and define the data transformation steps we want to apply using the Data Wrangler user interface (UI). We then have SageMaker run our feature engineering steps and store the features in Feature Store.

The following diagram illustrates the solution workflow.

To demonstrate Data Wrangler’s feature engineering steps, we assume we’ve already conducted exploratory data analysis (EDA). EDA helps you understand your data by identifying patterns in your data. For example, we might find that customers who book resort hotels tend to stay longer than city hotels. Or customers that stay over the weekend purchase more meals. Because these patterns aren’t evident with data in tables, data scientists use visualization tools to help identify patterns. EDA is often a necessary step to determine which features to create, delete, and transform.

If you already have features ready to export to Feature Store, you can navigate to the Save features to Feature Store section to learn how you can easily save your prepared features to Feature Store.

Prerequisites

If you want to follow along with this post, you should have the following prerequisites:

Create features with Data Wrangler

To create features with Data Wrangler, complete the following steps:

  1. Enter your Studio domain.
  2. Choose Data Wrangler as your resource to view.
  3. Choose New flow.
  4. Choose Import and import your data.

You can see a preview of the data in the Data Wrangler UI when selecting your dataset. You can also choose a sampling method. Because our dataset is relatively small, we choose not to sample our data. The flow editor now shows two steps in the UI, representing the step you took to import the data and a data validation step Data Wrangler automatically completes for you.

  1. Choose the plus sign next to Data types and choose Add transform.

Assuming we’ve spent time in EDA, we can remove redundant columns that contribute to target leakage. Target leakage occurs when some data in a training dataset is strongly correlated with the target label, but isn’t available in real-world data. After we conduct a target leakage analysis, we determine we should drop redundant columns. Data Wrangler helped identify 10 columns to drop.

  1. Add a step and choose the Drop column transform step.

Additionally, we determine we can remove columns like agent and adults after a multicollinearity analysis. Multicollinearity is the presence of high correlations between two or more independent variables. We usually want to avoid variables to be correlated to each other because they can lead to misleading and inaccurate models.

We also want to drop duplicate rows. In our case, nearly 28% of all rows in our dataset are duplicates. Because duplicates may have undesirable effects on our model, we use the transform set to remove them.

  1. Add a new transform and choose Manage rows from the list of available transforms.
  2. Choose Drop duplicates on the Transform drop-down menu.

Next, we want to handle missing values. We find that many hotel guests didn’t travel with children, and have a blank value for the children column. We can replace this blank value with 0.

  1. Choose Handle missing as the transform step and Fill missing as the transform type.
  2. Add a transform to fill blank values with the 0 value by choosing children as the input column.

From our EDA, we see that there are many missing values for the country column. However, the data reveals most of the hotel guests are from Europe. We determine that missing country column values can be replaced with the most commonly occurring country—Portugal (PRT).

  1. Choose the Handle missing transform step and choose Fill missing as the transform type.
  2. Choose country as the input column, and enter PRT as the Fill value.

ML algorithms like linear regression, logistic regression, neural networks, and others that use gradient descent as an optimization technique require data to be scaled. Normalization (also known as min-max scaling) is a scaling technique that transforms values to be in the range of 0–1. Standardization is another scaling technique where the values are centered around the mean with a standard deviation unit. In our case, we normalize the numeric feature columns to a standard scale between [0, 1].

  1. Choose the Process numeric transform step and Scale values as the transform type.
  2. Choose Min-max scaler as the scaler and lead_time, booking_changes, adr, and others as the input columns.
  3. Leave 0 as Min and 1 as Max default values.

We also want to handle categorical data by representing them as numeric values. For example, if your categories are Dog and Cat, you may encode this information into two vectors, [1,0] to represent Dog, and [0,1] to represent Cat. For our dataset, we use one-hot encoding to encode categories into an integer between 0 and the total number of categories within the column.

  1. Choose the One-hot encode transform type from the Encode categorical transform.

ML models are sensitive to the distribution and range of your feature values. Outliers can negatively impact model accuracy and lead to longer training times. For our dataset, we apply the standard deviation numeric outliers transform with a set of configuration values as shown in the following screenshot. We apply this transform on the numeric columns.

  1. Choose the Standard Deviation Numeric Outliers transform type from the Handle outliers transform.

Lastly, we want to balance the target variable for class imbalance. In Data Wrangler, we can handle class imbalance using three different techniques:

  • Random undersample
  • Random oversample
  • SMOTE
  1. In the Data Wrangler transform pane, choose Balance data as the group and choose Random oversample for the Transform field.

The ratio of positive to negative cases is around 0.38 before balancing.

After oversampling and balancing the dataset, the ratio equates to 1.

Now that we’ve completed our feature engineering tasks, we’re ready to export our features to Feature Store with one click.

Save features to Feature Store

You can easily export your generated features to SageMaker Feature Store by selecting it as the destination.

You can save the features into an existing feature group or create new one. For this post, we create a new feature group. Studio directs you to a new tab where you can create a new feature group.

  1. Choose the plus sign, choose Export to, and choose SageMaker Feature Store.

  1. Choose Create Feature Group.

  1. Optionally, select Create “EventTime” column.
  2. Choose Next.

  1. Copy the JSON schema, then choose Create.

  1. Provide a feature group name and an optional description for your feature group.
  2. Select a feature group storage configuration that is either online or offline, or both.

Online stores serve features with low millisecond latency for real-time inference, whereas offline stores are ideal for retrieving your features for training models or for batch scoring. Additionally, you can run queries on your offline feature stores by registering your features in an AWS Glue Data Catalog. For more information, see Query Feature Store with Athena and AWS Glue.

  1. Choose Continue.

Next, you specify the feature definitions. You specify the data type (string, integral, fractional) for each feature definition.

  1. Enter the JSON schema from the previous step to define your feature definitions.
  2. Choose Continue.

  1. Next, you specify a record identifier name and a timestamp to uniquely identify a record within a feature group.

The record identifier name must refer to one of the names of a feature defined in the feature group’s feature definition. In our case, we use the existing identifier, distribution-channel, which was in our source dataset, and EventTime.

  1. Choose Continue.

  1. Lastly, apply any relevant tags and review your feature group details.
  2. Choose Create feature group to finalize the process.

  1. After we create our feature group, we can return to the Data Wrangler flow UI.
  2. Choose the plus sign, choose Add destination, and choose SageMaker Feature Store.

  1. We choose the desired destination feature group to ensure that the features we’re storing match the feature group schema.

If the newly created feature group doesn’t show up in the UI, refresh the list to reload the groups.

  1. Chose the message under the Validation column to have Data Wrangler validate the schema of the dataset with the schema of the feature group.

If you missed specifying the event time column, Data Wrangler will notify you of an error and request that you add one to your dataset.

Once validated, Data Wrangler informs you that the data frame matches the feature group schema.

  1. If you enabled both the online and offline stores for the feature group, you can optionally select Write to offline store only to only ingest data to the offline store.

This is helpful for historical data backfilling scenarios.

  1. Choose Add to add another step to our Data Wrangler flow.
  2. With all our steps defined, choose Create job to run our ML workflow from feature engineering to ingesting features into our feature group.

  1. Give the job a name, then provide the job specifications like the type and number of instances.
  2. Choose Run.

Congratulations! You’ve successfully engineered features using Data Wrangler and stored them in a persistent feature store without writing any code. You can easily explore features, see details of your feature group, and update the feature group schema when necessary.

Conclusion

In this post, we created features with Data Wrangler, and easily stored those features in Feature Store. We showed an example workflow for feature engineering in the Data Wrangler UI. Then we saved those features into Feature Store directly from Data Wrangler by creating a new feature group. Finally, we ran a processing job to ingest those features into Feature Store. These services helped us build automatic and repeatable processes to streamline our data preparation tasks, all without writing code.

With this new integration, you can accelerate your ML tasks with a more streamlined experience between feature engineering and feature ingestion. For more information, refer to Get Started with Data Wrangler and Get started with Amazon SageMaker Feature Store.

About the Authors

Peter Chung is a Solutions Architect for AWS, and is passionate about helping customers uncover insights from their data. He has been building solutions to help organizations make data-driven decisions in both the public and private sectors. He holds all AWS certifications as well as two GCP certifications. He enjoys coffee, cooking, staying active, and spending time with his family.

Patrick Lin is a Software Development Engineer with Amazon SageMaker Data Wrangler. He is committed to making Amazon SageMaker Data Wrangler the number one data preparation tool for productionized ML workflows. Outside of work, you can find him reading, listening to music, having conversations with friends, and serving at his church.

Ziyao Huang is a Software Development Engineer with Amazon SageMaker Data Wrangler. He is passionate about building great product that makes ML easy for the customers. Outside of work, Ziyao likes to read, and hang out with his friends

Read More

Create train, test, and validation splits on your data for machine learning with Amazon SageMaker Data Wrangler

by Amazon AWS

In this post, we talk about how to split a machine learning (ML) dataset into train, test, and validation datasets with Amazon SageMaker Data Wrangler so you can easily split your datasets with minimal to no code.

Data used for ML is typically split into the following datasets:

  • Training – Used to train an algorithm or ML model. The model iteratively uses the data and learns to provide the desired result.
  • Validation – Introduces new data to the trained model. You can use a validation set to periodically measure model performance as training is happening, and also tune any hyperparameters of the model. However, validation datasets are optional.
  • Test – Used on the final trained model to assess its performance on unseen data. This helps determine how well the model generalizes.

Data Wrangler is a capability of Amazon SageMaker that helps data scientists and data engineers quickly and easily prepare data for ML applications using a visual interface. It contains over 300 built-in data transformations so you can quickly normalize, transform, and combine features without having to write any code.

Today, we’re excited to announce a new data transformation to split datasets for ML use cases within Data Wrangler. This transformation splits your dataset into training, test, and optionally validation datasets without having to write any code.

Overview of the split data transformation

The split data transformation includes four commonly used techniques to split the data for training the model, validating the model, and testing the model:

  • Random split – Splits data randomly into train, test, and, optionally validation datasets using the percentage specified for each dataset. It ensures that the distribution of the data is similar in all datasets. Choose this option when you don’t need to preserve the order of your input data. For example, consider a movie dataset where the dataset is sorted by genre and you’re predicting the genre of the movie. A random split on this dataset ensures that the distribution of the data includes all genres in all three datasets.
  • Ordered split – Splits data in order, using the percentage specified for each dataset. An ordered split ensures that the data in each split is non-overlapping while preserving the order of the data. When training, we want to avoid past or future information leaking across datasets. The ordered split option prevents data leakage. For example, consider a scenario where you have customer engagement data for the first few months and you want to use this historical data to predict customer engagement in the next month. You can perform this split by providing an optional input column (numeric column). This operation uses the values of a numeric column to ensure that the data in each split doesn’t overlap while preserving the order. This helps avoid data leakage across splits. If no input column is provided, the order of the rows is used, so the data in each split still comes before the data in the next split. This is useful where the rows of the dataset are already ordered (for example, by date) and the model may need to be fit to earlier data and tested on later data.
  • Stratified split – Splits the dataset so that each split is similar with respect to a column specifying different categories for your data, for example, size or country. This split ensures that the train, test, and validation datasets have the same proportions for each category as the input dataset. This is useful with classification problems where we’re trying to ensure that the train and test sets have approximately the same percentage of samples of each target class. Choose this option if you have imbalanced data across different categories and you need to have it balanced across split datasets.
  • Split by key – Takes one or more columns as input (the key) and ensures that no combination of values across the input columns occurs in more than one of the splits (split by key). This is useful to avoid data leakage for unordered data. Choose this option if your data for key columns needs to be in the same split. For example, consider customer transactions split by customer ID; the split ensures that customer IDs don’t overlap across split datasets.

Solution overview

For this post, we demonstrate how to split data into train, test, and validation datasets using the four new split options in Data Wrangler. We use a hotel booking dataset available publicly on Kaggle, which has the year, month, and date that bookings were made, along with reservation statuses, cancellations, repeat customers, and other features.

Prerequisites

Before getting started, upload the dataset to an Amazon Simple Storage Service (S3) bucket, then import it into Data Wrangler. For instructions, refer to Import data from Amazon S3.

Random split

After we import the data into Data Wrangler, we start the transformation. We first demonstrate a random split.

  1. On the Data Wrangler console, choose the plus sign and choose Add transform.
  2. To add the split data transformation, choose Add step.

    You’re redirected to the page where all transformations are displayed.
  3. Scroll down the list and choose Split data.

    The split data transformation has a drop-down menu that lists the available transformations to split your data, which include random, ordered, stratified, and split by key. By default, Randomized split is displayed.
  4. Choose the default value Randomized split.
  5. In the Splits section, enter the name Train with an 0.8 split percentage, and Test with a 0.2 percentage.
  6. Choose the plus sign to add an additional split.
  7. Add the Validation split with 0.2, and adjust Train to 0.7 and Test to 0.1.
    The split percentage can be any value you want, provided all three splits sum to 1 (100%).We can also specify optional fields like Error threshold and Random seed. We can achieve an exact split by setting the error threshold to 0. A smaller error threshold can lead to more processing time for splitting the data. This allows you to control the trade-off between time and accuracy on the operation. The Random seed option is for reproducibility. If not specified, Data Wrangler uses a default random seed value. We leave it blank for the purpose of this post.
  8. To preview your data split, choose Preview.

    The preview page displays the data split. You can choose Train, Test, or Validation on the drop-down menu to review the details of each split.
  9. When you’re satisfied with your data split, choose Add to add the transformation to your Data Wrangler flow.

To analyze the train dataset, choose Add analysis.

You can perform a similar analysis on the validation and test datasets.

Ordered split

We now use the hotel bookings dataset to demonstrate an ordered split transformation. The hotel dataset contains rows ordered by date.

  1. Repeat the steps to add a split, and choose Ordered split on the drop-down menu.
  2. Specify your three splits and desired percentages.
  3. Preview your data and choose Add to add the transformation to the Data Wrangler flow.
  4. Use the Add analysis option to verify the splits.

Stratified split

In the hotel booking dataset, we have an is_cancelled column, which indicates whether the booking was cancelled or not. We want to use this column to split the data. A stratified split ensures that the train, test, and validation datasets have same percentage of samples of is_cancelled.

  1. Repeat the steps to add a transformation, and choose Stratified split.
  2. Specify your three splits and desired percentages.
  3. For Input column, choose is_canceled.
  4. Preview your data and choose Add to add the transformation to the Data Wrangler flow.
  5. Use the Add analysis option to verify the splits.

Split by key

The split by key transformation splits the data by the key or multiple keys we specify. This split is useful to avoid having the same data in the split datasets created during transformation and to avoid data leakage.

  1. Repeat the steps to add a transformation, and choose Split by key.
  2. Specify your three splits and desired percentages.
  3. For Key column, we can specify the columns to form the key. For this post, choose the following columns:
    1. is_cancelled
    2. arrival_date_year
    3. arrival_date_month
    4. arrival_date_week_number
    5. reservation_status
  4. Preview your data and choose Add to add the transformation to the Data Wrangler flow.
  5. Use the Add analysis option to verify the splits.

Considerations

The node labeled as Data types cannot be deleted. Deleting a split node deletes all its datasets and downstream datasets and its nodes.

Conclusion

In this post, we demonstrated how to split an input dataset into train, test, and validation datasets with Data Wrangler using the split techniques random, ordered, stratified, and split by key.

To learn more about using data flows with Data Wrangler, refer to Create and Use a Data Wrangler Flow. To get started with Data Wrangler, see Prepare ML Data with Amazon SageMaker Data Wrangler.

About the Authors

Gopi Mudiyala is a Senior Technical Account Manager at AWS. He helps customers in the Financial Services Industry with their operations in AWS. As a machine learning specialist, Gopi works to support customers succeed in their ML journey.

Patrick Lin is a Software Development Engineer with Amazon SageMaker Data Wrangler. He is committed to making Amazon SageMaker Data Wrangler the number one data preparation tool for productionized ML workflows. Outside of work, you can find him reading, listening to music, having conversations with friends, and serving at his church.

Xiyi Li is a Front End Engineer at Amazon SageMaker Data Wrangler. She helps support Amazon SageMaker Data Wrangler and is passionate about building products that provide a great user experience. Outside of work, she enjoys hiking and listening to classical music.

Vishaal Kapoor is a Senior Applied Scientist with AWS AI. He is passionate about helping customers understand their data in Data Wrangler. In his spare time, he mountain bikes, snowboards, and spends time with his family.

Read More