Posts in category: Amazon AWS

Feature engineering is a process of applying transformations on raw data that a machine learning (ML) model can use. As an organization scales, this process is typically repeated by multiple teams that use the same features for different ML solutions. Because of this, organizations are forced to develop their own feature management system.

Additionally, you can also have a non-negotiable Java compatibility requirement due to existing data pipelines developed in Java, supporting services that can only be integrated with Java, or in-house applications that only expose Java APIs. Creating and maintaining such a feature management system can be expensive and time-consuming.

In this post, we address this challenge by adopting Amazon SageMaker Feature Store, a fully managed, purpose-built repository to securely store, update, retrieve, and share ML. We use Java to create a feature group; describe and list the feature group; ingest, read, and delete records from the feature group; and lastly delete the feature group. We also demonstrate how to create custom utility functions such as multi-threaded ingest to meet performance requirements. You can find the code used to build and deploy this solution into your own AWS account in the GitHub repo.

Credit card fraud use case

In our example, Organization X is a finance-tech company and has been combating credit card fraudulence for decades. They use Apache frameworks to develop their data pipelines and other services in Java. These data pipelines collect, process, and transform raw streaming data into feature sets. These feature sets are then stored in separated databases for model training and inference. Over the years, these data pipelines have created a massive number of feature sets and the organization doesn’t have a system to properly manage them. They’re looking to use Feature Store as their solution.

To help with this, we first configure a Java environment in an Amazon SageMaker notebook instance. With the use of a synthetic dataset, we walk through a complete end-to-end Java example with a few extra utility functions to show how to use Feature Store. The synthetic dataset contains two tables: identity and transactions. The transaction table contains the transaction amount and credit or debit card, and the identity table contains user and device information. You can find the datasets on GitHub.

Differences between Boto3 SDK and Java SDK

Each AWS service, including SageMaker, exposes an endpoint. An endpoint is the URL of the entry point for an AWS offering. The AWS SDKs and AWS Command Line Interface (AWS CLI) automatically use the default endpoint for each service in an AWS Region, but you can specify an alternate endpoint for your API requests.

You can connect directly to the SageMaker API or to the SageMaker Runtime through HTTPs implementations. But in that case, you have to handle low-level implementation details such as credentials management, pagination, retry algorithms (adaptive retry, disable retries), logging, debugging, error handling, and authentication. Usually these are handled by the AWS SDK for Python (Boto3), a Python-specific SDK provided by SageMaker and other AWS services.

The SDK for Python implements, provides, and abstracts away the low-level implementational details of querying an endpoint URL. While doing this, it exposes important tunable parameters via configuration parameters. The SDK implements elemental operations such as read, write, delete, and more. It also implements compound operations such as bulk read and bulk write. The definition of these compound operations is driven by your specific need; in the case of Feature Store, they’re provided in the form of bulk ingest.

Sometimes organizations can also have a non-negotiable Java compatibility requirement for consumption of AWS services. In such a situation, the AWS service needs to be consumed via the AWS SDK for Java and not via SDK for Python. This further drives the need to compose these compound functions using Java and reimplement the SDK for Python functionality using the SDK for Java.

Set up a Java environment

In this section, we describe the setup pattern for the Java SDK.

To help set up this particular example without too much hassle, we prefer using SageMaker notebooks instance as a base from which to run these examples. SageMaker notebooks are available in various instance types; for this post, an inexpensive instance such as ml.t3.medium suffices.

Perform the following commands in the terminal of the SageMaker notebook instance to confirm the Java version:

sh-4.2$ java --version
openjdk 11.0.9.1-internal 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1-internal+0-adhoc..src)
OpenJDK 64-Bit Server VM (build 11.0.9.1-internal+0-adhoc..src, mixed mode)

Next, install Maven:

cd /opt
sudo wget https://apache.osuosl.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
sudo tar xzvf apache-maven-3.6.3-bin.tar.gz
export PATH=/opt/apache-maven-3.6.3/bin:$PATH

Finally, clone our repository from GitHub, which includes code and the pom.xml to set up this example.

After the repository is cloned successfully, run this example from within the Java directory:

mvn compile; mvn exec:java -Dexec.mainClass="com.example.customername.FeatureStoreAPIExample"

Java end-to-end solution overview

After the environment is fully configured, we can start calling the Feature Store API from Java. The following diagram workflow outlines the end-to-end solution for Organization X for their fraud detection datasets.

Configure the feature store

To create the feature groups that contain the feature definitions, we first need to define the configurations for the online and offline feature store into which we ingest the features from the dataset. We also need to set up a new Amazon Simple Storage Service (Amazon S3) bucket to use as our offline feature store. Then, we initialize a SageMaker client using an ARN role with the proper permissions to access both Amazon S3 and the SageMaker APIs.

The ARN role that you use must have the following managed policies attached to it: AmazonSageMakerFullAccess and AmazonSageMakerFeatureStoreAccess.

The following code snippet shows the configuration variables that need to be user-specified in order to establish access and connect to the Amazon S3 client, SageMaker client, and Feature Store runtime client:

public static void main(String[] args) throws IOException {

  // Specify the region for your env
  static final Region REGION = Region.US_EAST_1;
  
  //S3 bucket where the Offline store data is stored
  // Replace with your value
  static final String BUCKET_NAME = "YOUR_BUCKET_NAME";
  
  // Replace with your value
  static final String FEATURE_GROUP_DESCRIPTION = "YOUR_DESCRIPTION";
  
  // Replace with your value
  static final String SAGEMAKER_ROLE_ARN = "YOUR_SAGEMAKER_ARN_ROLE";
  
  // CSV file path
  static final String FILE_PATH = "../data/Transaction_data.csv";
  
  // Feature groups to create
  static final String[] FEATURE_GROUP_NAMES = {
    "Transactions"
  };
  
  // Unique record identifier name for feature group records
  static final String RECORD_IDENTIFIER_FEATURE_NAME = "TransactionID";
  
  // Timestamp feature name for Feature Store to track
  static final String EVENT_TIME_FEATURE_NAME = "EventTime";
  
  // Number of threads to create per feature group ingestion, or can be    
  // determined dynamically through custom functions
  static final int NUM_OF_THREADS = 4;
  
  // Utility function which contains the feature group API operations
featureGroupAPIs(
    BUCKET_NAME, FILE_PATH, FEATURE_GROUP_NAMES,     
    RECORD_IDENTIFIER_FEATURE_NAME, EVENT_TIME_FEATURE_NAME, 
    FEATURE_GROUP_DESCRIPTION, SAGEMAKER_ROLE_ARN, NUM_OF_THREADS, REGION);
  
  System.exit(0);
};

We have now set up the configuration and can start invoking SageMaker and Feature Store operations.

Create a feature group

To create a feature group, we must first identify the types of data that exist in our dataset. The identification of feature definitions and the data types of the features occur during this phase, as well as creating the list of records in feature group ingestible format. The following steps don’t have to occur right after the initial data identification, but in our example, we do this at the same time to avoid parsing the data twice and to improve overall performance efficiency.

The following code snippet loads the dataset into memory and runs the identification utility functions:

// Read csv data into list
List < String[] > csvList = CsvIO.readCSVIntoList(filepath);

// Get the feature names from the first row of the CSV file
String[] featureNames = csvList.get(0);

// Get the second row of data for data type inferencing
String[] rowOfData = csvList.get(1);

// Initialize the below variable depending on whether the csv has an idx 
// column or not
boolean isIgnoreIdxColumn = featureNames[0].length() == 0 ? true : false;

// Get column definitions
List < FeatureDefinition > columnDefinitions = 
    FeatureGroupRecordOperations.makeColumnDefinitions(
        featureNames, rowOfData, EVENT_TIME_FEATURE_NAME, isIgnoreIdxColumn);

// Build and create a list of records
List < List < FeatureValue >> featureRecordsList = 
    FeatureGroupRecordOperations.makeRecordsList(featureNames, csvList, 
        isIgnoreIdxColumn, true);

After we create the list of feature definitions and feature values, we can utilize these variables for creation of our desired feature group and ingest the data by establishing connections to the controlling SageMaker client, Feature Store runtime client, and by building the online and offline Feature Store configurations.

The following code snippet demonstrates the setup for our creation use case:

S3StorageConfig s3StorageConfig = 
    S3StorageConfig.builder()
        .s3Uri(
            String.format("s3://%1$s/sagemaker-featurestore-demo/",
            BUCKET_NAME))
        .build();

OfflineStoreConfig offlineStoreConfig = 
    OfflineStoreConfig.builder()
        .s3StorageConfig(s3StorageConfig)
        .build();

OnlineStoreConfig onlineStoreConfig = 
    OnlineStoreConfig.builder()
        .enableOnlineStore(Boolean.TRUE)
        .build();

SageMakerClient sageMakerClient = 
    SageMakerClient.builder()
        .region(REGION)
        .build();

S3Client s3Client = 
    S3Client.builder()
        .region(REGION)
        .build();

SageMakerFeatureStoreRuntimeClient sageMakerFeatureStoreRuntimeClient = 
    SageMakerFeatureStoreRuntimeClient.builder()
        .region(REGION)
        .build();

We develop and integrate a custom utility function with the SageMaker client to create our feature group:

// Create feature group 
FeatureGroupOperations.createFeatureGroups(
    sageMakerClient, FEATURE_GROUP_NAMES, FEATURE_GROUP_DESCRIPTION, 
    onlineStoreConfig, EVENT_TIME_FEATURE_NAME, offlineStoreConfig, 
    columnDefinitions, RECORD_IDENTIFIER_FEATURE_NAME, 
    SAGEMAKER_ROLE_ARN);

For a deeper dive into the code, refer to the GitHub repo.

After we create a feature group and its state is set to ACTIVE, we can invoke API calls to the feature group and add data as records. We can think of records as a row in a table. Each record has a unique RecordIdentifier and other feature values for all FeatureDefinitions that exist in the FeatureGroup.

Ingest data into the feature group

In this section, we demonstrate the code incorporated into the utility function we use to multi-thread a batch ingest to our created feature group using the list of records (FeatureRecordsList) that we created earlier in the feature definition identification step:

// Total number of threads to create for batch ingestion
static final int NUM_OF_THREADS_TO_CREATE = 
    NUM_OF_THREADS * FEATURE_GROUP_NAMES.length;

// Ingest data from csv data
Ingest.batchIngest(
    NUM_OF_THREADS_TO_CREATE, sageMakerFeatureStoreRuntimeClient, 
    featureRecordsList, FEATURE_GROUP_NAMES, EVENT_TIME_FEATURE_NAME);

During the ingestion process, we should see the ingestion progress on the console as status outputs. The following code is the output after the ingestion is complete:

Starting batch ingestion
Ingest_0 is running
Ingest_1 is running
Ingest_2 is running
Number of created threads: 4
Ingest_3 is running
Thread: Ingest_2 => ingested: 500 out of 500
Thread: Ingest_2, State: TERMINATED
Thread: Ingest_1 => ingested: 500 out of 500   
Thread: Ingest_1, State: TERMINATED
Thread: Ingest_0 => ingested: 500 out of 500   
Thread: Ingest_0, State: TERMINATED
Thread: Ingest_3 => ingested: 500 out of 500   
Thread: Ingest_3, State: TERMINATED
Ingestion finished 
Ingested 2000 of 2000

Now that we have created the desired feature group and ingested the records, we can run several operations on the feature groups.

List and describe a feature group

We can list and describe the structure and definition of the desired feature group or of all the feature groups in our feature store by invoking getAllFeatureGroups() on the SageMaker client then calling describeFeatureGroup() on the list of feature group summaries that is returned in the response:

// Invoke the list feature Group API 
List < FeatureGroupSummary > featureGroups =    
    FeatureGroupOperations.getAllFeatureGroups(sageMakerClient);

// Describe each feature Group
FeatureGroupOperations.describeFeatureGroups(sageMakerClient, featureGroups);

The preceding utility function iterates through each of the feature groups and outputs the following details:

Feature group name is: Transactions

Feature group creation time is: 2021-04-28T23:24:54.744Z

Feature group feature Definitions is: 
[
    FeatureDefinition(FeatureName=TransactionID, FeatureType=Integral),
    FeatureDefinition(FeatureName=isFraud, FeatureType=Integral), 
    FeatureDefinition(FeatureName=TransactionDT, FeatureType=Integral),
    FeatureDefinition(FeatureName=TransactionAmt, FeatureType=Fractional), 
    FeatureDefinition(FeatureName=card1, FeatureType=Integral),
    FeatureDefinition(FeatureName=card2, FeatureType=Fractional),
…
    FeatureDefinition(FeatureName=card_type_0, FeatureType=Integral),
    FeatureDefinition(FeatureName=card_type_credit, FeatureType=Integral), 
    FeatureDefinition(FeatureName=card_type_debit, FeatureType=Integral),
    FeatureDefinition(FeatureName=card_bank_0, FeatureType=Integral), 
    FeatureDefinition(FeatureName=card_bank_american_express, FeatureType=Integral),
    FeatureDefinition(FeatureName=card_bank_discover, FeatureType=Integral), 
    FeatureDefinition(FeatureName=card_bank_mastercard, FeatureType=Integral),
    FeatureDefinition(FeatureName=card_bank_visa, FeatureType=Integral), 
    FeatureDefinition(FeatureName=EventTime, FeatureType=Fractional)
]

Feature group description is: someDescription

Retrieve a record from the feature group

Now that we know that our features groups are populated, we can fetch records from the feature groups for our fraud detection example. We first define the record that we want to retrieve from the feature group. The utility function used in this section also measures the performance metrics to show the real-time performance of the feature store when it’s used to train models for inference deployments. See the following code:

// Loop getRecord for FeatureGroups in our feature store
static final int AMOUNT_TO_REPEAT = 1;
static final String RECORD_IDENTIFIER_VALUE = "2997887";
FeatureGroupOperations.runFeatureGroupGetTests(
    sageMakerClient, sageMakerFeatureStoreRuntimeClient, featureGroups, 
    AMOUNT_TO_REPEAT, RECORD_IDENTIFIER_VALUE);

You should see the following output from the record retrieval:

Getting records from feature group: Transactions
Records retrieved: 1 out of: 1
Retrieved record feature values: 
[
    FeatureValue(FeatureName=TransactionID, ValueAsString=2997887), 
    FeatureValue(FeatureName=isFraud, ValueAsString=1), 
    FeatureValue(FeatureName=TransactionDT, ValueAsString=328678), 
    FeatureValue(FeatureName=TransactionAmt, ValueAsString=13.051), 
    FeatureValue(FeatureName=card1, ValueAsString=2801), 
    FeatureValue(FeatureName=card2, ValueAsString=130.0), 
    FeatureValue(FeatureName=card3, ValueAsString=185.0), 
…
    FeatureValue(FeatureName=card_type_0, ValueAsString=0), 
    FeatureValue(FeatureName=card_type_credit, ValueAsString=1), 
    FeatureValue(FeatureName=card_type_debit, ValueAsString=0), 
    FeatureValue(FeatureName=card_bank_0, ValueAsString=0), 
    FeatureValue(FeatureName=card_bank_american_express, ValueAsString=0), 
    FeatureValue(FeatureName=card_bank_discover, ValueAsString=0), 
    FeatureValue(FeatureName=card_bank_mastercard, ValueAsString=0), 
    FeatureValue(FeatureName=card_bank_visa, ValueAsString=1), 
    FeatureValue(FeatureName=EventTime, ValueAsString=1619655284.177000)
]

Delete a record from the feature group

The deleteRecord API call deletes a specific record from the list of existing records in a specific feature group:

// Delete record with id 2997887
FeatureGroupOperations.deleteRecord(
    sageMakerFeatureStoreRuntimeClient, FEATURE_GROUP_NAMES[0], 
    RECORD_IDENTIFIER_VALUE);

The preceding operation should log the following output with status 200, showing that the delete operation was successful:

Deleting record with identifier: 2997887 from feature group: Transactions
Record with identifier deletion HTTP response status code: 200

Although feature store is used for ongoing ingestion and update of the features, you can still remove the feature group after the fraud detection example use case is over in order to save costs, because you only pay for what you provision and use with AWS.

Delete the feature group

Delete the feature group and any data that was written to the OnlineStore of the feature group. Data can no longer be accessed from the OnlineStore immediately after DeleteFeatureGroup is called. Data written into the OfflineStore is not deleted. The AWS Glue database and tables that are automatically created for your OfflineStore are not deleted. See the following code:

// Delete featureGroups
FeatureGroupOperations.deleteExistingFeatureGroups(
    sageMakerClient, FEATURE_GROUP_NAMES);

The preceding operation should output the following to confirm that deletion has properly completed:

Deleting feature group: Transactions
...
Feature Group: Transactions cannot be found. Might have been deleted.

Feature group deleted is: Transactions

Close connections

Now that we have completed all the operations on the feature store, we need to close our client connections and stop the provisioned AWS services:

sageMakerFeatureStoreRuntimeClient.close();
sageMakerClient.close();
s3Client.close();

Conclusion

In this post, we showed how to configure a Java environment in a SageMaker instance. We walked through an end-to-end example to demonstrate not only what Feature Store is capable of, but also how to develop custom utility functions in Java, such as multi-threaded ingestion to improve the efficiency and performance of the workflow.

To learn more about Amazon SageMaker Feature Store, check out this overview of its key features. It is our hope that this post, together with our code examples, can help organizations and Java developers integrate Feature Store into their services and application. You can access the entire example on GitHub. Try it out, and let us know what you think in the comments.

About the Authors

Ivan Cui is a Data Scientist with AWS Professional Services, where he helps customers build and deploy solutions using machine learning on AWS. He has worked with customers across diverse industries, including software, finance, pharmaceutical, and healthcare. In his free time, he enjoys reading, spending time with his family, and maximizing his stock portfolio.

Chaitanya Hazarey is a Senior ML Architect with the Amazon SageMaker team. He focuses on helping customers design, deploy, and scale end-to-end ML pipelines in production on AWS. He is also passionate about improving explainability, interpretability, and accessibility of AI solutions.

Daniel Choi is an Associate Cloud Developer with AWS Professional Services, who helps customers build solutions using the big data analytics and machine learning platforms and services on AWS. He has created solutions for clients in the AI automated manufacturing, AR/MR, broadcasting, and finance industries utilizing his data analytics specialty whilst incorporating his SDE background. In his free time, he likes to invent and build new IoT home automation devices and woodwork.

Raghu Ramesha is a Software Development Engineer (AI/ML) with the Amazon SageMaker Services SA team. He focuses on helping customers migrate ML production workloads to SageMaker at scale. He specializes in machine learning, AI, and computer vision domains, and holds a master’s degree in Computer Science from UT Dallas. In his free time, he enjoys traveling and photography.

Read More

You might use traditional methods to forecast future business outcomes, but these traditional methods are often not flexible enough to account for varying factors, such as weather or promotions, outside of the traditional time series data considered. With the advancement of machine learning (ML) and the elasticity that the AWS Cloud brings, you can now enjoy more accurate forecasts that influence business decisions. You will learn how to interpret and format your data according to what Amazon Forecast needs based on your business questions.

This post shows you how to prepare your data to optimally use with Amazon Forecast. Amazon Forecast is a fully managed service that allows you to forecast your time series data with high accuracy. It uses ML to analyze complex relationships in historical data and doesn’t require any prior ML experience. With its deep integration capabilities with the AWS Cloud, your forecasting process can be fully automated and highly flexible.

We will begin by understanding the different types of input data that Forecast accepts. With a retail use case, we will discuss how to structure your data to match the use case and forecasting granularity of the business metric that you are interested in forecasting. Then, we will discuss how to clean your data and handle challenging scenarios, such as missing values, to generate the most accurate forecasts.

Factors affecting forecast accuracy

Amazon Forecast uses your data to train a private, custom model tailored to your use case. ML models are only as good as the data put into them, and it’s important to understand what the model needs. Amazon Forecast can accept three types of datasets: target time series, related time series, and item metadata. Amongst those, target time series is the only mandatory dataset. This historical data provides the majority of the model’s accuracy.

Amazon Forecast provides predefined dataset domains that specify a schema of what data to include in which input datasets for common use cases, such as forecasting for retail, web traffic, and more. The domains are convenient column names only. The underlying models aren’t affected by these column names because they’re dropped prior to training. For the remainder of this post, we use the retail domain as an example.

Target time series data

Target time series data defines the historical demand for the resources you’re predicting. As mentioned earlier, the target time series dataset is mandatory. It contains three required fields:

• item_id – Describes a unique identifier for the item or category you want to predict. This field may be named differently depending on your dataset domain (for example, in the workforce domain this is workforce_type, which helps distinguish different groups of your labor force).
• timestamp – Describes the date and time at which the observation was recorded.
• demand – Describes the amount of the item, specified by item_id, that was consumed at the timestamp specified. For example, this could be the number of pink shoes sold on a certain day.

You can also add additional fields in your input data. For example, in the retail dataset domain, you can optionally add an additional field titled location. This can help to add context about where the consumption occurred for that record and forecast the demand for items on a per-store basis where multiple stores are selling the same item. The best practice is to create a concatenated item_id identifier that includes product and location identifiers. One exception to this rule is if you know more than string names of locations, such as “Store 1”. If you know actual geolocations, such as postal codes or latitude/longitude points, then geolocation data such as weather can be pulled in automatically. This geolocation field needs to be separate from the item_id.

The frequency of your observations in the historical data you provide is also important, because it dictates the frequency of your forecasts that can be generated. You can provide target time series data with fine granularity such as a per-minute frequency, where historical demand is recorded every minute, up to as wide of a granularity as a yearly frequency. The data granularity must be smaller than or equal to your desired forecast granularity. If you want predictions on a monthly basis for each item, you should input data with monthly or finer granularity. The granularity shouldn’t be larger than your desired forecast frequency (for example, giving yearly observations in historical data when you want forecasts on a monthly basis).

High-quality datasets consist of dense data where there is almost a data point for every item and timestamp. Sparse data doesn’t give Amazon Forecast enough information to determine historical patterns to forecast with. To achieve accurate forecasts, ensure that you can supply dense data or fill in missing data points with null filling, as described later in this post.

Related time series data

In addition to historical sales data, other data may be known per item at exactly the same time as every sale. This data is called related time series data. Related data can give more clues to what future predictions could look like. The best related data is also known in the future. Examples of related data include prices, promotions, economic indicators, holidays, and weather. Although related time series data is optional, including additional information can help increase accuracy by providing context of various conditions that may have affected demand.

The related time series dataset must include the same dimensions as the target time series, such as the timestamp and item_id. Additionally, you can include up to a maximum of 13 related features. For more information about useful features you may want to include for different use cases, see Predefined Dataset Domains and Dataset Types.

Amazon Forecast trains a model using all input data. If the related time series doesn’t improve accuracy, it’s not used. When training with related data, it’s best to train using the CNN-QR algorithm, if possible, then check the model parameters to see if your related time series data was useful for improving accuracy.

Item metadata

Providing item metadata to Amazon Forecast is optional, but can help refine forecasts by adding contextual information about items that appear in your target time series data. Item metadata is static information that doesn’t change with time, describing features about items such as the color and size of a product being sold. Amazon Forecast uses this data to create predictions based on similarities between products.

To use item metadata, you upload a separate file to Amazon Forecast. Each row in the CSV file you upload must contain the item ID, followed by the metadata features for that item. Each row can have a maximum of 10 fields, including the field that contained the item ID.

Item metadata is required when forecasting demand for an item that has no historical demand, known as the cold start problem. This could be a new product that you want to launch, for example. Because item metadata is required, demand for new products can’t be forecasted except if your data qualifies to train a deep learning algorithm. By understanding the demand of items with similar features, Amazon Forecast predicts demand for your new product. For more information about forecasting for cold start scenarios, see the following best practices on GitHub.

Now that you understand the different types of input data and their formats, we explore how to manipulate your data to achieve your business objectives.

Structure your input data based on your business questions

When preparing your input data for Amazon Forecast, consider the business questions you want to ask. As mentioned earlier, Amazon Forecast requires three mandatory input columns (timestamp, item_id, and value) as part of your time series data. You need to prepare your input data by applying aggregations to your input data while keeping the eventual structure in line to the input format. The following scenarios explain how you can manipulate and prepare your input data depending on your business questions.

Imagine we have the following dataset showing your daily sales per product. In this example, your company is selling two different products (Product A and Product B) in different stores (Store 1 and Store 2) across two different countries (Canada and the US).

The granularity of the provided sales data is on a per-store, country, item ID, and per-day basis. This initial assessment is useful when we prepare the data for the input.

Now imagine you need to ask the following forecasting question: “How many sales should I anticipate for Product A on January 4?”

The question is looking for an answer for a particular day, so you need to tell Amazon Forecast to predict at a daily frequency. Amazon Forecast can produce the forecasts at the desired daily frequency because the raw data is reported at the same granularity level or less.

The question also asks for a specific product, Product A. Because the raw data reports sales on a per-product granularity already, no further data preparation action is required for product aggregation.

The source data shows that sales are reported per store. Because we’re not interested in forecasting on a per-store basis, you need to aggregate all the sales data of each product across all the stores.

Taking these into account, your Amazon Forecast input structure looks like the following table.

Another business question you might ask could be: “How many sales should I anticipate from Canada on January 4?”

In this question, the granularity is still daily, so Amazon Forecast can produce daily forecasts. The question doesn’t ask for a specific product or store. However, it asks for a prediction on a country level. The source data shows that the data is broken down on a per-store basis, and each store has one-to-one mapping to a country. That means you need to sum up all sales across all the different stores within the same country.

Your Amazon Forecast input structure looks like the following table.

Lastly, we ask the following question: “How much overall sales should I anticipate for February?”

This question doesn’t mention any dimensions other than time. That means that all the sales data should be aggregated across all products, stores, and countries per month. Because Amazon Forecast requires a specific date to use as the timestamp, you can use the first of each month to indicate a month’s aggregated demand. Your Amazon Forecast input structure looks like the following table.

This example data is just for demonstration purposes. Real-life datasets should be much larger, because a larger historical dataset yields more accurate predictions. For more information, see the data size best practices on GitHub. Remember that while you’re doing aggregations across dimensions, you’re reducing the total number of input data points. If there is little historical data, aggregation leads to fewer input data points, which may not be enough for Amazon Forecast to accurately train your predictor. You can experiment with different aggregation levels within your data and explore how they affect the accuracy of your predictions through iteration.

Data cleaning

Cleaning your data for Amazon Forecast is important because it can affect the accuracy of the forecasts that are created. To demonstrate some best practices, we use the Department store sales and stocks dataset provided by the Government of Canada. The data is already prepared for Amazon Forecast to predict on a monthly basis for each unique department using historical data from January 1991 to December 1997. The following table shows an excerpt of the cleaned data.

In the following sections, we describe some of the steps that were taken to understand and cleanse our data.

Visualize the data

Previously, we discussed how granularity of data dictates forecast frequency and how you can manipulate data granularity to suit your business questions. With visualization, you can see at what levels of time and product granularity your data exhibits smoother patterns, which give the ML model better inputs for learning. If your data appears to be intermittent or sparse, try to aggregate data into a higher granularity (for example, aggregating all sales for a given day as a single data point) with equally spaced time intervals. If your data has too few observations to determine a trend over time, your data has been aggregated at too high a level and you should reduce the granularity to a finer level. For sample Python code, see our Data Prep notebook.

In the following chart of yearly demand for bedding and household items, we visualize the data from earlier at the yearly aggregation level. The chart shows a one-time bump in the year 1994 that isn’t repeated. This is a bad aggregation level to use because there is no repeatable pattern to the historical sales. In addition, yearly aggregation results in too little historical data, which isn’t enough for Amazon Forecast to use.

Next, we can visualize our sample dataset at a monthly granularity level to identify patterns in our data. In the following figure, we plotted data for the bedding and household items department and added a trendline. We can observe a seasonal trend that is predictable, which Amazon Forecast can learn and predict with.

Handle missing and zero values

You must also be careful of gaps and zero values within your target time series data. If the target field value (such as demand) is zero for a timestamp and item ID combination, this could mean that data was simply missing, the item wasn’t in stock, and so on. Having zeroes in your data that aren’t actual zeroes, such as values representing new or end-of-life products, can bias a model toward zero. When preparing your data, one best practice is to convert all zero values to null and let Amazon Forecast do the heavy lifting by automatically detecting new products and end-of-life products. In addition, adding an out-of-stock related variable per item_id and timestamp can improve accuracy. When you replace zeroes with null values, they’re replaced according to the filling logic you specify, which you can change based on your null filling strategy.

In our sample dataset, the data for the plumbing, heating, and building materials department is blank or contains a 0 after June 1993.

Upon further inspection, only blank and zero values are observed until the end of the dataset. This is known as the end-of-life problem. We have two options: simply remove these items from training data because we know that their forecast should be zero, or replace all zeroes with nulls and let Amazon Forecast automatically detect end-of-life products with null filling logic.

Conclusion

This post outlined how to prepare your data to predict according to your business outcomes with Amazon Forecast. When you follow these best practices, Amazon Forecast can create highly accurate probabilistic forecasts. To learn more about data preparation for Amazon Forecast and best practices, refer to the Amazon Forecast Cheat Sheet and the sample data preparation Jupyter notebook. You can also take a self-learning workshop and browse our other sample Jupyter notebooks that show how to productionize with Amazon Forecast.

About the Authors

Murat Balkan is an AWS Solutions Architect based in Toronto. He helps customers across Canada to transform their businesses and build industry leading solutions on AWS.

Christy Bergman is working as an AI/ML Specialist Solutions Architect at AWS. Her work involves helping AWS customers be successful using AI/ML services to solve real-world business problems. Prior to joining AWS, Christy worked as a data scientist in banking and software industries. In her spare time, she enjoys hiking and bird watching.

Brandon How is an AWS Solutions Architect who works with enterprise customers to help design scalable, well-architected solutions on the AWS Cloud. He is passionate about solving complex business problems with the ever-growing capabilities of technology.

Read More

Have you noticed that your shopping preferences are influenced by the weather? For example, on hot days would you rather drink a lemonade vs. a hot coffee?

Customers from consumer-packaged goods (CPG) and retail industries wanted to better understand how weather conditions like temperature and rain can be used to provide better purchase suggestions to consumers. Data was taken over a year long period. Based on observations like demand for refreshing beverages increasing on hot days, customers want to enrich their Amazon Personalize models with weather information.

In December 2019, AWS announced support for contextual recommendations in Amazon Personalize. This feature allows you to include contextual information with interactions when training a model, as well as include context when generating recommendations for users. For more information about how to implement contextual recommendations in real time, see Increasing the relevance of your Amazon Personalize recommendations by leveraging contextual information.

In this post, we share a hands-on approach to include context when generating recommendations, by using data from our partners in the AWS Data Exchange, a service that makes it easy to find, subscribe to, and use third-party data in the cloud.

Solution overview

For this example, we use a subset of the temperatures and precipitation dataset provided by Weather Trends International, combined with a fictitious dataset representing a bottler company beverage catalog and a user demo dataset from the Retail Demo Store. Retail Demo Store is an application and workshop platform intended as an educational tool for demonstrating how to use AWS services to build compelling customer experiences for ecommerce.

With all three datasets, we create sampled interactions to mimic a direct-to-customer ecommerce experience and an Amazon Personalize model to show how weather affects the recommendations for different users.

Preparing the data

First, we prepare the data to be used by Amazon Personalize. Let’s look at each dataset.

interactions.csv

An interactions dataset stores historical and real-time data from interactions between users and items. At minimum, you must provide the following for each interaction: User ID, Item ID, and Timestamp (in Unix epoch time format). You can also include metadata fields to further describe the interaction. For example, you can include the event type or other relevant contextual information like device type used or daily temperature.

To create the interactions dataset, we use a slightly modified upsampling process from Retail Demo Store. Basically, we create fictitious interactions events between our users and items, and add the average temperature for that day as a categorical attribute in our interaction’s dataset. Our ranges are as follows:

• Less than 5˚ Celsius = very cold
• Between 5–10˚ Celsius = cold
• Between 10–15˚ Celsius = slightly cold
• Between 15–21˚ Celsius = lukewarm
• Between 21–28˚ Celsius = hot
• More than 28˚ Celsius = very hot

The upsampling process takes into account the user persona field to generate interactions. A user with the persona sparkling_waters_other can view, add, and purchase products from those categories.

The interactions dataset enriched with temperature data looks like the following screenshot.

Context is environmental information that can shift rapidly over time, influencing your customer’s perception and behavior. We use DAILY_TEMPERATURE as an optional categorical attribute in our interactions dataset as context information at inference time.

Now let’s look at the distribution of temperatures from the interactions generated during the period of 3 months starting on March 25, 2020.

The preceding graphic shows that our interactions dataset doesn’t have many hot or cold days. This could be a good signal to the model to learn about the behavior customers display during those days and use that information at inference time. We can provide Amazon Personalize the weather context when asking for recommendations, and explore how the recommendations are influenced by this contextual information.

items.csv

This fictitious dataset simulates a product catalog for a CPG company.

The items dataset stores metadata about your items. This dataset is optional when using Amazon Personalize; however, in this use case we need to include attributes category and size to test the recommendations behavior.

users.csv

This dataset is a subset of a fictitious dataset taken from the Retail Demo Store project.

A users dataset stores metadata about your users. This can include information such as age, gender, name, and more. It’s an optional dataset when using Amazon Personalize.

For this post, we need the information from columns id and persona to be included in our model. persona is a categorical metadata field representing the categories of products that a customer likes and is used to test the recommendations behavior.

weather_data.csv

We use a real weather dataset provided by Weather Trends International to enrich the interactions dataset with temperature data.

For simplicity, we keep only the rows representing Santiago de Chile, location_id = ‘ST650’.

Create Amazon Personalize components

When the datasets are ready to be imported, we’re ready to start creating an Amazon Personalize deployment. If you want to reproduce this process in your own account, you can use our sample notebook to complete the following steps:

1. Create a dataset group with interactions, users, and items metadata datasets.
2. Import historical data into each dataset.
3. Configure a solution with a context-aware algorithm (aws-user-personalization) and train a solution version.
4. Explore the solution version metrics.
5. Create a campaign (managed recommendations endpoint).
6. Query for personalized recommendations.

Explore Amazon Personalize recommendations

Now we can explore the recommendations provided by the campaign we created earlier. To do so, we pick a random user with ID 170 and get the top five beverages recommended for them with and without daily temperature as context.

The following screenshot shows the top five without context information.

The following screenshot shows the top five with DAILY_TEMPERATURE = “cold”.

This user’s persona is other_waters_sparkling, which indicates a preference for those categories of beverages. The Amazon Personalize recommendations are consistent with their preferences with and without context. However, when we ask for recommendations on a cold day, the ranking of recommendations change:

• The category of the top three items in the ranking doesn’t change
• Tea item 33 went down from first to third in the ranking
• Coconut water, a refreshing item 52, was replaced by the same beverage in a bigger size format: item 15

These changes were driven by the behavior of customers in our dataset. For example, maybe user 170 likes different brands to prepare hot and iced tea, and the recommendations ranking is reflecting that preference on cold days.

Context is especially useful when dealing with new users without previous interactions. The following screenshot shows the top recommendations for a nonexistent user, ID 7000, without context.

The following shows the recommendations for the same user with DAILY_TEMPERATURE = “hot”.

In the case of generating recommendations for a cold user, recommended items are the same, but the position in the ranking varies when considering a hot day vs. not providing context. In ecommerce with anonymous shopping functionality, this is particularly useful to prioritize which products to visualize, or suggest popular items to an unknown user considering their current context.

Conclusion

As stated in study How Context Affects Choice, context influences customer outcomes by altering the process by which decisions are made. Data like temperature, precipitation, device type, and channel of interaction can enrich models and improve the relevance of recommendations generated by Amazon Personalize. More relevant recommendations lead to better user experience and better business outcomes.

If you found this post useful and want to test how it works in real life, you can gather your own context data, or purchase datasets on AWS Data Exchange and use the approach of this post to put your use case into production. Feel free to explore any other creative ideas you can find.

If you want to recreate this use case with your own data, Daily Historical Weather Dataset – Demo is free for a month and code from this post is available on GitHub.

Additional resources about Amazon Personalize can be found on the Amazon Personalize GitHub repository.

About the Author

Angel Goñi is an Enterprise Solutions Architect at AWS. He helps enterprise customers drive their digital transformation process leveraging AWS services. Usually works with Consumer Packaged Goods customers with emphasis on SAP migrations to AWS.

Read More