Posts in category: Amazon AWS

This post demonstrates how to create a serverless Machine Learning Operations (MLOps) pipeline to develop and visualize a forecasting model built with Amazon Forecast. Because Machine Learning (ML) workloads need to scale, it’s important to break down the silos among different stakeholders to capture business value. The MLOps model makes sure that the data science, production, and operations teams work together seamlessly across workflows that are as automated as possible, ensuring smooth deployments and effective ongoing monitoring.

Similar to the DevOps model in software development, the MLOps model in ML helps build code and integration across ML tools and frameworks. You can automate, operationalize, and monitor data pipelines without having to rewrite custom code or rethink existing infrastructures. MLOps helps scale existing distributed storage and processing infrastructures to deploy and manage ML models at scale. It can also be implemented to track and visualize drift over time for all models across the organization in one central location and implement automatic data validation policies.

MLOps combines best practices from DevOps and the ML world by applying continuous integration, continuous deployment, and continuous training. MLOps helps streamline the lifecycle of ML solutions in production. For more information, see the whitepaper Machine Learning Lens: AWS Well-Architected Framework.

In the following sections you will build, train, and deploy a time-series forecasting model leveraging an MLOps pipeline encompassing Amazon Forecast, AWS Lambda, and AWS Step Functions. To visualize the generated forecast, you will use a combination of AWS serverless analytics services such as Amazon Athena and Amazon QuickSight.

Solution architecture

In this section, you deploy an MLOps architecture that you can use as a blueprint to automate your Amazon Forecast usage and deployments. The provided architecture and sample code help you build an MLOps pipeline for your time series data, enabling you to generate forecasts to define future business strategies and fulfil customer needs.

You can build this serverless architecture using AWS-managed services, which means you don’t need to worry about infrastructure management while creating your ML pipeline. This helps iterate through a new dataset and adjust your model by tuning features and hyperparameters to optimize performance.

The following diagram illustrates the components you will build throughout this post.

In the preceding diagram, the serverless MLOps pipeline is deployed using a Step Functions workflow, in which Lambda functions are stitched together to orchestrate the steps required to set up Amazon Forecast and export the results to Amazon Simple Storage Service (Amazon S3).

The architecture contains the following components:

• The time series dataset is uploaded to the Amazon S3 cloud storage under the /train directory (prefix).
• The uploaded file triggers Lambda, which initiates the MLOps pipeline built using a Step Functions state machine.
• The state machine stitches together a series of Lambda functions to build, train, and deploy a ML model in Amazon Forecast. You will learn more details about the state machine’s Lambda components in the next section.
• For log analysis, the state machine uses Amazon CloudWatch, which captures Forecast metrics. You use Amazon Simple Notification Service (Amazon SNS) to send email notifications when the final forecasts become available in the source Amazon S3 bucket in the /forecast directory. The ML pipeline saves any old forecasts in the /history directory.
• Finally, you will use Athena and QuickSight to provide a visual presentation of the current forecast.

In this post, you use the Individual household electric power consumption dataset available in the UCI Machine Learning Repository. The time series dataset aggregates hourly energy consumption for various customers households and shows spikes in energy utilization over weekdays. You can replace the sample data as needed for later use cases.

Now that you are familiar with the architecture, you’re ready to explore the details of each Lambda component in the state machine.

Building an MLOps pipeline using Step Functions

In the previous section, you learned that the Step Functions state machine is the core of the architecture automating the entire MLOps pipeline. The following diagram illustrates the workflow deployed using the state machine.

As shown in preceding diagram, the Lambda functions from the Step Functions workflow are as follows (the steps also highlight the mapping between Lambda functions and the parameters used from the params.json file stored in Amazon S3):

• Create-Dataset – Creates a Forecast dataset. The information about the dataset helps Forecast understand how to consume the data for model training.
• Create-DatasetGroup – Creates a dataset group.
• Import-Data – Imports your data into a dataset that resides inside a dataset group.
• Create-Predictor – Creates a predictor with a forecast horizon that the parameters file specifies.
• Create-Forecast – Creates the forecast and starts an export job to Amazon S3, including quantiles specified in the parameters file.
• Update-Resources – Creates the necessary Athena resources and transforms the exported forecasts to the same format as the input dataset.
• Notify Success – Sends an email alerting when the job is finished by posting a message to Amazon SNS.
• Strategy-Choice – Checks whether to delete the Forecast resources, according to the parameters file.
• Delete-Forecast – Deletes the forecast and keeps the exported data.
• Delete-Predictor – Deletes the predictor.
• Delete-ImportJob – Deletes the Import-Data job in Forecast.

In Amazon Forecast, a dataset group is an abstraction that contains all the datasets for a particular collection of forecasts. There is no information sharing between dataset groups; to try out various alternatives to the schemas, you create a new dataset group and make changes inside its corresponding datasets. For more information, see Datasets and Dataset Groups. For this use case, the workflow imports a target time series dataset into a dataset group.

After completing these steps, the workflow triggers the predictor training job. A predictor is a Forecast-trained model used for making forecasts based on time series data. For more information, see Predictors.

When your predictor is trained, the workflow triggers the creation of a forecast using that predictor. During forecast creation, Amazon Forecast trains a model on the entire dataset before hosting the model and making inferences. For more information, see Forecasts.

The state machine sends a notification email to the address specified at the deployment of a successful forecast export. After exporting your forecast, the Update-Resources step reformats the exported data so Athena and QuickSight can easily consume it.

You can reuse this MLOps pipeline to build, train, and deploy other ML models by replacing the algorithms and datasets in the Lambda function for each step.

Prerequisites

Before you deploy the architecture, complete the following prerequisite steps:

1. Install Git.
2. Install the AWS Serverless Application Model (AWS SAM) CLI on your system. For instructions, see Installing the AWS SAM CLI. Make sure the latest version is installed with the following code:

   sam --version 

Deploying the sample architecture to your AWS account

To simplify deployment, this post provides the entire architecture as infrastructure as code using AWS CloudFormation and is available in the Forecast Visualization Automation Blogpost GitHub repo. You use AWS SAM to deploy this solution.

1. Clone the Git repo. See the following code:

   git clone https://github.com/aws-samples/amazon-forecast-samples.git

   The code is also available on the Forecast Visualization Automation Blogpost GitHub repo.

2. Navigate to the newly created amazon-forecast-samples/ml_ops/visualization_blog directory and enter the following code to start solution deployment:

   cd amazon-forecast-samples/ml_ops/visualization_blog
   
   sam build && sam deploy --guided
   

   At this stage, AWS SAM builds a CloudFormation template change set. After a few seconds, AWS SAM prompts you to deploy the CloudFormation stack.

3. Provide parameters for the stack deployment. This post uses the following parameters; you can keep the default parameters:

   Setting default arguments for 'sam deploy'
   	=========================================
   	Stack Name [ForecastSteps]:  <Enter Stack Name e.g.  - forecast-blog-stack>
   	AWS Region [us-east-1]: <Enter region e.g. us-east-1>
   	Parameter Email [youremail@yourprovider.com]: <Enter valid e-mail id>
   	Parameter ParameterFile [params.json]: <Leave Default>
   	#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
   	Confirm changes before deploy [Y/n]: y
   	#SAM needs permission to be able to create roles to connect to the resources in   
           your template
   	Allow SAM CLI IAM role creation [Y/n]: y
           Save arguments to samconfig.toml [Y/n]: n 
   

   AWS SAM creates an AWS CloudFormation change set and asks for confirmation.

4. Enter Y.

For more information about change sets, see Updating Stacks Using Change Sets.

After a successful deployment, you see the following output:

CloudFormation outputs from the deployed stack
------------------------------------------------------------
Outputs                                                                                                                                
-------------------------------------------------------------
Key                 AthenaBucketName                                                                                                   
Description         Athena bucket name to drop your files                                                                              
Value               forecast-blog-stack-athenabucket-1v6qnz7n5f13w                                                                     

Key                 StepFunctionsName                                                                                                  
Description         Step Functions Name                                                                                                
Value               arn:aws:states:us-east-1:789211807855:stateMachine:DeployStateMachine-5qfVJ1kycEOj                                 

Key                 ForecastBucketName                                                                                                 
Description         Forecast bucket name to drop your files                                                                            
Value               forecast-blog-stack-forecastbucket-v61qpov2cy8c                                                                    
-------------------------------------------------------------
Successfully created/updated stack - forecast-blog-stack in us-east-1

4. On the AWS CloudFormation console, on the Outputs tab, record the value of ForecastBucketName, which you use in the testing step.
   

Testing the sample architecture

The following steps outline how to test the sample architecture. To trigger the Step Functions workflow, you need to upload two files to the newly created S3 bucket: a parameter file and the time series training dataset.

1. Under the same directory in which you cloned the GitHub repo, enter the following code, replacing YOURBUCKETNAME with the value from the AWS CloudFormation Outputs tab that you copied earlier:

   aws s3 cp ./testing-data/params.json s3://{YOURBUCKETNAME}

   The preceding command copied the parameters file that the Lambda functions use to configure your Forecast API calls.

2. Upload the time series dataset by entering the following code:

   aws s3 sync ./testing-data/ s3://{YOURBUCKETNAME}

3. On the Step Functions dashboard, locate the state machine named DeployStateMachine-<random string>.
4. Choose the state machine to explore the workflow in execution.

In the preceding screenshot, all successfully executed steps (Lambda functions) are in a green box. The blue box indicates that a step is still in progress. All boxes without colors are steps that are pending execution. It can take up to 2 hours to complete all the steps of this workflow.

After the successful completion of the workflow, you can go to the Amazon S3 console and find an Amazon S3 bucket with the following directories:

/params.json    # Your parameters file.
/train/         # Where the training CSV files are stored
/history/       # Where the previous forecasts are stored
/history/raw/   # Contains the raw Amazon Forecast exported files
/history/clean/ # Contains the previous processed Amazon Forecast exported files
/quicksight/    # Contains the most updated forecasts according to the train dataset
/tmp/           # Where the Amazon Forecast files are temporarily stored before processing

The parameter file params.json stores attributes to call Forecast APIs from the Lambda functions. These parameter configurations contain information such as forecast type, predictor setting, and dataset setting, in addition to forecast domain, frequency, and dimension. For more information about API actions, see Amazon Forecast Service.

Now that your data is in Amazon S3, you can visualize your results.

Analyzing forecasted data with Athena and QuickSight

To complete your forecast pipeline, you need to query and visualize your data. Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. QuickSight is a fast, cloud-powered business intelligence service that makes it easy to uncover insights through data visualization. To start analyzing your data, you first ingest your data into QuickSight using Athena as a data source.

If you’re new to AWS, set up QuickSight to create a QuickSight account. If you have an AWS account, subscribe to QuickSight to create an account.

If this is your first time using Athena on QuickSight, you have to give permissions to QuickSight to query Amazon S3 using Athena. For more information, see Insufficient Permissions When Using Athena with Amazon QuickSight.

1. On the QuickSight console, choose New Analysis.
2. Choose New Data Set.
3. Choose Athena.
4. In the New Athena data source window, for Data source name, enter a name; for example, Utility Prediction.
5. Choose Validate connection.
6. Choose Create data source.

The Choose your table window appears.

7. Choose Use custom SQL.
   
8. In the Enter custom SQL query window, enter a name for your query; for example, Query to merge Forecast result with training data.
9. Enter the following code into the query text box:

   SELECT LOWER(forecast.item_id) as item_id,
            forecast.target_value,
            date_parse(forecast.timestamp, '%Y-%m-%d %H:%i:%s') as timestamp,
            forecast.type
   FROM default.forecast
   UNION ALL
   SELECT LOWER(train.item_id) as item_id,
            train.target_value,
            date_parse(train.timestamp, '%Y-%m-%d %H:%i:%s') as timestamp,
            'history' as type
   FROM default.train
   

10. Choose Confirm query.

You now have the option to import your data to SPICE or query your data directly.

11. Choose either option, then choose Visualize.

You see the following fields under Fields list:

• item_id
• target_value
• timestamp
• type

The exported forecast contains the following fields:

• item_id
• date
• The requested quantiles (P10, P50, P90)

The type field contains the quantile type (P10, P50, P90) for your forecasted window and history as its value for your training data. This was done through the custom query to have a consistent historical line between your historical data and the exported forecast.

You can customize the quantiles by using the CreateForecast API optional parameter called ForecastType. For this post, you can configure this in the params.json file in Amazon S3.

12. For X axis, choose timestamp.
13. For Value, choose target_value.
14. For Color, choose type.

In your parameters, you specified a 72-hour horizon. To visualize results, you need to aggregate the timestamp field on an hourly frequency.

15. From the timestamp drop-down menu, choose Aggregate and Hour.

The following screenshot is your final forecast prediction. The graph shows a future projection in the quantiles p10, p50m, and p90, at which probabilistic forecasts are generated.

Conclusion

Every organization can benefit from more accurate forecasting to better predict product demand, optimize planning and supply chains, and more. Forecasting demand is a challenging task, and ML can narrow the gap between predictions and reality.

This post showed you how to create a repeatable, AI-powered, automated forecast generation process. You also learned how to implement an ML operation pipeline using serverless technologies and used a managed analytics service to get data insights by querying the data and creating a visualization.

There’s even more that you can do with Forecast. For more information about energy consumption predictions, see Making accurate energy consumption predictions with Amazon Forecast. For more information about quantiles, see Amazon Forecast now supports the generation of forecasts at a quantile of your choice.

If this post helps you or inspires you to solve a problem, share your thoughts and questions in the comments. You can use and extend the code on the GitHub repo.

About the Author

Luis Lopez Soria is an AI/ML specialist solutions architect working with the AWS machine learning team. He works with AWS customers to help them adopt machine learning on a large scale. He enjoys playing sports, traveling around the world, and exploring new foods and cultures.

Saurabh Shrivastava is a solutions architect leader and AI/ML specialist working with global systems integrators. He works with AWS partners and customers to provide them with architectural guidance for building scalable architecture in hybrid and AWS environments. He enjoys spending time with his family outdoors and traveling to new destinations to discover new cultures.

Pedro Sola Pimentel is an R&D solutions architect working with the AWS Brazil commercial team. He works with AWS to innovate and develop solutions using new technologies and services. He’s interested in recent computer science research topics and enjoys traveling and watching movies.

Read More

Amazon Kendra is an easy-to-use enterprise search service that allows you to add search capabilities to your applications so end-users can easily find information stored in different data sources within your company. This could include invoices, business documents, technical manuals, sales reports, corporate glossaries, internal websites, and more. You can harvest this information from storage solutions like Amazon Simple Storage Service (Amazon S3) and OneDrive; applications such as SalesForce, SharePoint and Service Now; or relational databases like Amazon Relational Database Service (Amazon RDS)

When you type a question, the service uses machine learning (ML) algorithms to understand the context and return the most relevant results, whether that’s a precise answer or an entire document. Most importantly, you don’t need to have any ML experience to do this—Amazon Kendra also provides you with the code that you need to easily integrate with your new or existing applications.

This post shows you how to create your internal enterprise search by using the capabilities of Amazon Kendra. This enables you to build a solution to create and query your own search index. For this post, you use Amazon.com help documents in HTML format as the data source, but Amazon Kendra also supports MS Office (.doc, .ppt), PDF, and text formats.

Overview of solution

This post provides the steps to help you create an enterprise search engine on AWS using Amazon Kendra. You can provision a new Amazon Kendra index in under an hour without much technical depth or ML experience.

The post also demonstrates how to configure Amazon Kendra for a customized experience by adding FAQs, deploying Amazon Kendra in custom applications, and synchronizing data sources. This post addresses and answers these questions in the subsequent sections.

Prerequisites

For this walkthrough, you should have the following prerequisites:

• An AWS account
• Basic knowledge of AWS
• An S3 bucket for your documents. For more information, see Create a Bucket and What is Amazon S3?

Creating and configuring your document repository

Before you can create an index in Amazon Kendra, you need to load documents into an S3 bucket. This section contains instructions to create an S3 bucket, get the files, and load them into the bucket. After completing all the steps in this section, you have a data source that Amazon Kendra can use.

1. On the AWS Management Console, in the Region list, choose US East (N. Virginia) or any Region of your choice that Amazon Kendra is available in.
2. Choose Services.
3. Under Storage, choose S3.
4. On the Amazon S3 console, choose Create bucket.
5. Under General configuration, provide the following information:
   • Bucket name: kendrapost-{your account id}.
   • Region: Choose the same Region that you use to deploy your Amazon Kendra index (this post uses US East (N. Virginia) us-east-1).
6. Under Bucket settings for Block Public Access, leave everything with the default values.
7. Under Advanced settings, leave everything with the default values.
8. Choose Create bucket.
9. Download amazon_help_docs.zip and unzip the files.
10. On the Amazon S3 console, select the bucket that you just created and choose Upload.
11. Upload the unzipped files.

Inside your bucket, you should now see two folders: amazon_help_docs (with 3,100 objects) and faqs (with one object).

The following screenshot shows the contents of amazon_help_docs.

The following screenshot shows the contents of faqs.

Creating an index

An index is the Amazon Kendra component that provides search results for documents and frequently asked questions. After completing all the steps in this section, you have an index ready to consume documents from different data sources. For more information about indexes, see Index.

To create your first Amazon Kendra index, complete the following steps:

1. On the console, choose Services.
2. Under Machine Learning, choose Amazon Kendra.
   
3. On the Amazon Kendra main page, choose Create an Index.
   
4. In the Index details section, for Index name, enter kendra-blog-index.
5. For Description, enter My first Kendra index.
6. For IAM role, choose Create a new role.
7. For Role name, enter -index-role (your role name has the prefix AmazonKendra-YourRegion-).
8. For Encryption, don’t select Use an AWS KMW managed encryption key.

(Your data is encrypted with an Amazon Kendra-owned key by default.)

9. Choose Next.
   

For more information about the IAM roles Amazon Kendra creates, see Prerequisites.

Amazon Kendra offers two editions. Kendra Enterprise Edition provides a high-availability service for production workloads. Kendra Developer Edition is suited for building a proof-of-concept and experimentation. For this post, you use the Developer edition.

10. In the Provisioning editions section, select Developer edition.
11. Choose Create.
    

For more information on the free tier, document size limits, and total storage for each Amazon Kendra edition, see Amazon Kendra pricing.

The index creation process can take up to 30 minutes. When the creation process is complete, you see a message at the top of the page that you successfully created your index.

Adding a data source

A data source is a location that stores the documents for indexing. You can synchronize data sources automatically with an Amazon Kendra index to make sure that searches correctly reflect new, updated, or deleted documents in the source repositories.

After completing all the steps in this section, you have a data source linked to Amazon Kendra. For more information, see Adding documents from a data source.

Before continuing, make sure that the index creation is complete and the index shows as Active.

1. On the kendra-blog-index page, choose Add data sources.

Amazon Kendra supports six types of data sources: Amazon S3, SharePoint Online, ServiceNow, OneDrive, Salesforce online, and Amazon RDS. For this post, you use Amazon S3.

2. Under Amazon S3, choose Add connector.
   

For more information about the different data sources that Amazon Kendra supports, see Adding documents from a data source.

3. In the Define attributes section, for Data source name, enter amazon_help_docs.
4. For Description, enter AWS services documentation.
5. Choose Next.
   
6. In the Configure settings section, for Enter the data source location, enter the S3 bucket you created: kendrapost-{your account id}.
7. Leave Metadata files prefix folder location

By default, metadata files are stored in the same directory as the documents. If you want to place these files in a different folder, you can add a prefix. For more information, see S3 document metadata.

8. For Select decryption key, leave it deselected.
9. For Role name, enter source-role (your role name is prefixed with AmazonKendra-).
10. For Additional configuration, you can add a pattern to include or exclude certain folders or files. For this post, keep the default values.
    
11. For Frequency, choose Run on demand.

This step defines the frequency with which the data source is synchronized with the Amazon Kendra index. For this walkthrough, you do this manually (one time only).

12. Choose Next.
    
13. On the Review and create page, choose Create.
    
14. After you create the data source, choose Sync now to synchronize the documents with the Amazon Kendra index.
    

The duration of this process depends on the number of documents that you index. For this use case, it may take 15 minutes, after which you should see a message that the sync was successful.

In the Sync run history section, you can see that 3,099 documents were synchronized.

Exploring the search index using the search console

The goal of this section is to let you explore possible search queries via the built-in Amazon Kendra console.

To search the index you created above, complete the following steps:

1. Under Indexes, choose kendra-blog-index.
   
2. Choose Search console.
   

Kendra can answer three types of questions: factoid, descriptive, and keyword. For more information, see Amazon Kendra FAQs. You can ask some questions using the Amazon.com help documents that you uploaded earlier.

In the search field, enter What is Amazon music unlimited?

With a factoid question (who, what, when, where), Amazon Kendra can answer and also offer a link to the source document.

As a keyword search, enter shipping rates to Canada. The following screenshot shows the answer Amazon Kendra gives.

Adding FAQs

You can also upload a list of FAQs to provide direct answers to common questions your end-users ask. To do this, you need to load a .csv file with the information related to the questions. This section contains instructions to create and configure that file and load it into Amazon Kendra.

1. On the Amazon Kendra console, navigate to your index.
2. Under Data management, choose FAQs.
   
3. Choose Add FAQ.
   
4. In the Define FAQ project section, for FAQ name, enter kendra-post-faq.
5. For Description, enter My first FAQ list.
   

Amazon Kendra accepts .csv files formatted with each row beginning with a question followed by its answer. For example, see the following table.

This is how the .CSV file included for this use case looks like:

"How do I sign up for the Amazon Prime free Trial?"," To sign up for the Amazon Prime free trial, your account must have a current, valid credit card. Payment options such as an Amazon.com Corporate Line of Credit, checking accounts, pre-paid credit cards, or gift cards cannot be used. "," https://www.amazon.com/gp/help/customer/display.html/ref=hp_left_v4_sib?ie=UTF8&nodeId=201910190”

6. Under FAQ settings, for S3, enter s3://kendrapost-{your account id}/faqs/kendrapost.csv.
7. For IAM role, choose Create a new role.
8. For Role name, enter faqs-role (your role name is prefixed with AmazonKendra-).
   
9. Choose Add.
10. Wait until you see the status show as Active.
    

You can now see how the FAQ works on the search console.

11. Under Indexes, choose your index.
12. Under Data management, choose Search console.
    
13. In the search field, enter How do I sign up for the Amazon Prime free Trial?
14. The following screenshot shows that Amazon Kendra added the FAQ that you uploaded previously to the results list, and provides an answer and a link to the related documentation.
    

Using Amazon Kendra in your own applications

You can add the following components from the search console in your application:

• Main search page – The main page that contains all the components. This is where you integrate your application with the Amazon Kendra API.
• Search bar – The component where you enter a search term and that calls the search function.
• Results – The component that displays the results from Amazon Kendra. It has three components: suggested answers, FAQ results, and recommended documents.
• Pagination – The component that paginates the response from Amazon Kendra.

Amazon Kendra provides source code that you can deploy in your website. This is offered free of charge under a modified MIT license so you can use it as is or change it for your own needs.

This section contains instructions to deploy Amazon Kendra search to your website. You use a Node.js demo application that runs locally in your machine. This use case is based on a MacOS environment.

To run this demo, you need the following components:

• Npm (Node.js)
• IAM credentials from a user with the appropriate permissions to use Amazon Kendra

1. Download amazon_aws-kendra-sample-app-master.zip and unzip the file.
2. Open a terminal window and go to the aws-kendra-sample-app-master folder:

   cd /{folder path}/aws-kendra-sample-app-master

3. Create a copy of the .env.development.local.example file as .env.development.local:

   cp .env.development.local.example .env.development.local

4. Edit the .env.development.local file and add the following connection parameters:
   • REACT_APP_INDEX – Your Amazon Kendra index ID (you can find this number on the Index home page)
   • REACT_APP_AWS_ACCESS_KEY_ID – Your account access key
   • REACT_APP_AWS_SECRET_ACCESS_KEY – Your account secret access key
   • REACT_APP_AWS_SESSION_TOKEN – Leave it blank for this use case
   • REACT_APP_AWS_DEFAULT_REGION – The Region that you used to deploy the Kendra index (for example, us-east-1)
5. Save the changes.
6. Install the Node.js dependencies:

   npm install

7. Launch the local development server:

   npm start

8. View the demo app at http://localhost:3000/. You should see the following screenshot.
   
9. Enter the same question you used to test the FAQs: How do I sign up for the Amazon Prime free Trial?

The following screenshot shows that the result is the same as the one you got from the Amazon Kendra console, even though the demo webpage is running locally in your machine.

Cleaning up

To avoid incurring future charges and to clean out unused roles and policies, delete the resources you created: the Amazon Kendra index, S3 bucket, and corresponding IAM roles.

1. To delete the Amazon Kendra index, under Indexes, choose kendra-blog-index.
   
2. In the index settings section, from the Actions drop-down menu, choose Delete.
   
3. To confirm deletion, enter Delete in the field and choose Delete.
   

Wait until you get the confirmation message; the process can take up to 15 minutes.

For instructions on deleting your S3 bucket, see How do I delete an S3 Bucket?

Conclusion

In this post, you learned how to use Amazon Kendra to deploy an enterprise search service. You can use Amazon Kendra to improve the search experience in your company, powered by ML. You can enable rapid look for your documents using natural language, without any previous ML/AI experience. For more information about Amazon Kendra, see AWS re:Invent 2019 – Keynote with Andy Jassy on YouTube, Amazon Kendra FAQs, and What is Amazon Kendra?

About the Author

Leonardo Gómez is a Big Data Specialist Solutions Architect at AWS. Based in Toronto, Canada, He works with customers across Canada to design and build big data architectures.

Read More

It’s 5 PM on a Friday. You’ve spent all afternoon coding out a complex, sophisticated feature engineering strategy. It just started working on your Amazon SageMaker Studio t3.medium notebook, and all you want to do is plug this onto a massive instance, scale it out over the rest of your dataset, and go home. You could upgrade your notebook instance, but the job would stop as soon as you close your laptop. Why not schedule the job from your notebook directly?

Amazon SageMaker provides a fully-managed solution for building, training, and deploying machine learning (ML) models. In this post, we demonstrate using Amazon SageMaker Processing Jobs to execute Jupyter notebooks with the open-source project Papermill. The combination of Amazon SageMaker with Amazon CloudWatch, AWS Lambda, and the entire AWS stack have always provided the modular backbone you need to scale up jobs, like feature engineering, both on the fly and on a schedule. We’re happy to provide a do-it-yourself toolkit to simplify this process, using AWS CloudFormation to set up permissions, Lambda to launch the job, and Amazon Elastic Container Registry (Amazon ECR) to create a customized execution environment. It includes a library and CLI to initiate notebook execution from any AWS client and a Jupyter plugin for a seamless user experience.

As of this writing, you can write code in a Jupyter notebook and run it on an Amazon SageMaker ephemeral instance with the click of a button, either immediately or on a schedule. With the tools provided here, you can do this from anywhere: at a shell prompt, in JupyterLab on Amazon SageMaker, in another JupyterLab environment you have, or automated in a program you’ve written. We’ve written sample code that simplifies setup by using AWS CloudFormation to handle the heavy lifting and provides convenience tools to run and monitor executions.

For more information about executing notebooks, see the GitHub repo. All the source code is available in aws-samples on GitHub. Read on to learn all about how to use scheduled notebook execution.

When to use this solution

This toolkit is especially useful for running nightly reports. For example, you may want to analyze all the training jobs your data science team ran that day, run a cost/benefit analysis, and generate a report about the business value your models are going to bring after you deploy them into production. That would be a perfect fit for a scheduled notebook—all the graphs, tables, and charts are generated by your code, the same as if you stepped through the notebook yourself, except now they are handled automatically, in addition to persisting in Amazon Simple Storage Service (Amazon S3). You can start your day with the latest notebook, executed overnight, to move your analysis forward.

Or, imagine that you want to scale up a feature engineering step. You’ve already perfected the for-loop to knock out all your Pandas transformations, and all you need is time and compute to run this on the full 20 GB of data. No problem—just drop your notebook into the toolkit, run a job, close your laptop, and you’re done. Your code continues to run on the scheduled instance, regardless of whether or not you’re actively using Jupyter at the moment.

Perhaps you’re on a data science team that still trains models on local laptops or Amazon SageMaker notebooks, and haven’t yet adopted the Amazon SageMaker ephemeral instances for training jobs. With this toolkit, you can easily use the advanced compute options only for the time you’re training a model. You can spin up a p3.xlarge only for the hour your model trains but use your Studio environment all day on the affordable t3.medium. You can easily connect these resources to the Experiments SDK with a few lines of code. Although it’s still fully supported to run Amazon SageMaker notebooks and Amazon SageMaker Studio on p3 instances, developing a habit of using the largest instances only for short periods is a net cost-savings exercise.

You may have an S3 bucket full of objects and need to run a full notebook on each object. These could be dates of phone call records in your call center or Tweet-streams from particular users in your social network. You can easily write a for-loop over those objects by using this toolkit, which schedules a job for each file, runs it on its dedicated instance, and stores the completed notebook in Amazon S3. These could even be model artifacts loaded in from your preferred training environment—package up your inference code in a notebook and use the toolkit to easily deploy them!

Finally, customers tell us that reporting on the performance of their models is a key asset for their stakeholders. With this toolkit, you can implement a human-in-the-loop solution that analyzes feature importance, produces ROC curves, and estimates how your model will perform on the tricky edge cases that are crucial to your final product. You can build a model profiler that all the data scientists on your team can easily access. You can trigger this model profiler to run after every training job is complete, closing the loop on the value of your analysis to your stakeholders.

Three Ways to Execute Notebooks on a Schedule in SageMaker

To execute a notebook in Amazon SageMaker, you use a Lambda function that sets up and runs an Amazon SageMaker Processing job. The function can be invoked directly by the user or added as a target of an Amazon EventBridge rule to run on a schedule or in response to an event. The notebook to run is stored as an Amazon S3 object so it’s available to run even if you’re not online when the execution happens. The following diagram illustrates this architecture.

We outline three different ways to install and use this capability that let you work with notebooks and schedules just the way you want.

Using the AWS APIs or CLI directly

You can use the AWS APIs directly to execute and schedule notebooks. To make the process easier, we have provided a CloudFormation template to set up the Lambda function you need and some AWS Identity and Access Management (IAM) roles and policies that you use when running notebooks. We also provided scripts for building and customizing the Docker container images that Amazon SageMaker Processing Jobs uses when running the notebooks.

After you instantiate the CloudFormation template and create a container image, you can run a notebook like this with the following code:

$ aws lambda invoke --function-name RunNotebook 
             --payload '{"input_path": "s3://mybucket/mynotebook.ipynb", 
                         "parameters": {"p": 0.75}}' result.json

To create a schedule, enter the following code, replacing the Region and account number in the arn, as well as the input_path to your S3 bucket.

$ aws events put-rule --name "RunNotebook-test" --schedule "cron(15 1 * * ? *)"
$ aws lambda add-permission --statement-id EB-RunNotebook-test 
                            --action lambda:InvokeFunction 
                            --function-name RunNotebook 
                            --principal events.amazonaws.com 
                            --source-arn arn:aws:events:us-east-1:123456789:rule/RunNotebook-test
$ aws events put-targets --rule RunNotebook-test 
                         --targets '[{"Id": "Default", 
                                      "Arn": "arn:aws:lambda:us-east-1:123456789:function:RunNotebook", 
                                      "Input": "{ "input_path": "s3://mybucket/mynotebook.ipynb", 
                                                  "parameters": {"p": 0.75}}"}]‘

With this approach, you manage moving the notebook to Amazon S3, monitoring Amazon SageMaker Processing Jobs, and retrieving the output notebook from Amazon S3.

This is a great solution when you’re a knowledgeable AWS user who wants to craft a solution without taking on extra dependencies. You can even modify the Lambda function we’ve written or the Papermill execution container to meet your exact needs.

For more information about scheduling notebooks with the AWS APIs, see the full setup instructions on the GitHub repo.

Making things easier with a convenience package

To make it easier to schedule notebooks (especially if you aren’t an AWS expert), we’ve created a convenience package that wraps the AWS tools in a CLI and Python library that give you a more natural interface to running and scheduling notebooks. This package lets you build customized execution environments without Docker, via AWS CodeBuild instead, and manages the Amazon S3 interactions and job monitoring for you.

After you run the setup, execute the notebook with the following code:

$ run-notebook run mynotebook.ipynb -p p=0.5 -p n=200

Schedule a notebook with the following code:

$ run-notebook schedule --at "cron(15 1 * * ? *)" --name nightly weather.ipynb -p "name=Boston, MA"

The convenience package also contains tools to monitor jobs and view schedules. See the following code:

$ run-notebook list-runs
Date                 Rule                 Notebook              Parameters           Status     Job
2020-06-15 15:31:40                       fraud-analysis.ipynb  name=Tom             Completed  papermill-fraud-analysis-2020-06-15-22-31-39
2020-06-15 01:00:08  DailyForecastSeattle DailyForecast.ipynb   place=Seattle, WA    Completed  papermill-DailyForecast-2020-06-15-08-00-08
2020-06-15 01:00:03  DailyForecastNewYork DailyForecast.ipynb   place=New York, NY   Completed  papermill-DailyForecast-2020-06-15-08-00-02
2020-06-12 22:34:06                       powers.ipynb          p=0.5                Completed  papermill-powers-2020-06-13-05-34-05
                                                                n=20
$ 

For more information about the convenience package, see the GitHub repo.

Executing notebooks directly from JupyterLab with a GUI

For those who prefer an interactive experience, the convenience package includes a JupyterLab extension that you can enable for JupyterLab running locally, in Amazon SageMaker Studio, or on an Amazon SageMaker notebook instance.

After you set up the Jupyter extension for Amazon SageMaker Studio users, you see the new notebook execution sidebar (the rocket ship icon). The sidebar lets you execute or schedule the notebook you’re viewing . You can use the notebook-runner container that was created by the default setup or any other container you built. Enter the ARN for the execution role these jobs utilize and your instance preference, and you’re ready to go!

After you choose Run Now, the Lambda function picks up your notebook and runs it on an Amazon SageMaker Processing job. You can view the status of that job by choosing Runs. See the following screenshot.

When the job is complete, the finished notebook is stored in Amazon S3. Remember, this means your previous runs will persist, so you can easily revert back to them.

Finally, import the output notebook by choosing View Output and Import Notebook. If you don’t import the notebook, it’s never copied to your local directory. This is great when you want to see what happened, but don’t want to clutter things up with lots of extra notebooks.

For instructions on setting up the JupyterLab extension and using the GUI to run and monitor your notebooks, see the GitHub repo.

Summary

This post discussed how you can combine the modular capabilities of Amazon SageMaker and the AWS cloud to give data scientists and ML engineers the seamless experience of running notebooks on ephemeral instances. We are releasing an open-source toolkit to simplify this process, including a CLI, convenience package, and Jupyter widget. We discussed a variety of use cases for this, from running nightly reports to scaling up feature engineering to profiling models on the latest datasets. We shared examples from the various ways of running the toolkit. Feel free to walk through the Quick Start on GitHub and step through even more examples on the GitHub repo.

Author Bios

Emily Webber is a machine learning specialist SA at AWS, who alternates between data scientist, machine learning architect, and research scientist based on the day of the week. She lives in Chicago, and you can find her on YouTube, LinkedIn, GitHub, or Twitch. When not helping customers and attempting to invent the next generation of machine learning experiences, she enjoys running along beautiful Lake Shore Drive, escaping into her Kindle, and exploring the road less traveled.

Tom Faulhaber is a Principal Engineer on the Amazon SageMaker team. Lately, he has been focusing on unlocking all the potential uses of the richness of Jupyter notebooks and how they can add to the data scientist’s toolbox in non-traditional ways. In his spare time, Tom is usually found biking and hiking to discover all the wild spaces around Seattle with his kids.

Read More

We are excited to announce the launch of AWS DeepComposer Chartbusters, a monthly challenge where developers can use AWS DeepComposer to create original compositions and compete to top the charts and win prizes. AWS DeepComposer gives developers a creative way to get started with machine learning (ML) and generative AI techniques. With AWS DeepComposer, developers, regardless of their background in ML, can get started with generative AI techniques to learn how to train and optimize their models to create original music. The first AWS DeepComposer Chartbusters challenge, Bach to the Future, requires developers to use a new generative AI algorithm provided in the AWS DeepComposer console to create compositions in the style of Bach.

Every month through October, 2020, AWS will release a new Chartbusters challenge that has a different monthly theme to introduce you to a variety of generative AI techniques. You don’t need any musical knowledge to participate in the challenge. Before participating in a challenge, you can use learning capsules available in the AWS DeepComposer console to learn the generative AI concepts required for each month’s challenge. Learning capsules provide easy-to-consume, bite-size content to help you learn the concepts of generative AI algorithms.

How to participate in the challenge

The challenge is open worldwide for developers to participate. To get started, you will need to use one of the generative AI algorithms available in the AWS DeepComposer console to create compositions. Once you are ready to submit your composition for the challenge, select Submit a composition in the console to submit your creations to SoundCloud. AWS DeepComposer will add your submission to the Chartbuster challenge playlist on SoundCloud.

You can invite your family and friends to listen and like your composition by using the social sharing buttons available on SoundCloud. At the end of each challenge period, AWS will shortlist the top 20 compositions using a sum of customer likes and count of plays on SoundCloud. Our human AWS experts and DeepComposer AI judge will evaluate the shortlist based on musical quality and creativity to select the top 10 ranked compositions. The DeepComposer AI judge is trained on original Bach compositions and scores how similar your composition is to Bach’s style.

At the end of each challenge period, we will announce the top 10 compositions in an AWS ML blog post and feature them in an exclusive AWS top 10 playlist on SoundCloud and in the AWS DeepComposer console. The winner for each month’s challenge will receive an AWS DeepComposer Chartbusters gold record mailed to their physical address. Additionally, we will interview the winner to share their experience and feature them in an AWS ML blog post. You can continue to participate in multiple challenges so that if you don’t make it to the top 10 in one challenge, you can participate in the next challenge for another chance to top the charts.

Bach to the Future challenge

The first AWS DeepComposer Chartbuster challenge titled Bach to the Future launches today and is open until July 25^th. To participate in the challenge, you will need to use the autoregressive CNN (AR-CNN) algorithm available in the AWS DeepComposer console to create compositions in the style of Bach. AWS will announce the top 10 compositions for the Bach to the Future challenge and the theme for the next challenge on July 31st, 2020, in an AWS ML blog post.

The AR-CNN algorithm enhances the original input melody by adding or removing musical notes from the input melody. If the algorithm detects off-key or extraneous notes, it may choose to remove them. If it identifies additional specific notes that are highly probable in a Bach composition, it may decide to add them. Listen to the following example of a composition that is generated by applying the AR-CNN algorithm. You might recognize this tune from Jonathan Coulton, as it is available as a sample input melody in the AWS DeepComposer console.

Input: me-and-my-jar-with-extra-missing-notes.midi

Enhanced composition: me-and-my-jar-enhanced.midi

You can use the Introduction to autoregressive convolutional neural network learning capsule available in the AWS DeepComposer console to learn the concepts. To access the learning capsule in the console, navigate to learning capsules using the left navigation menu. Choose Introduction to autoregressive convolutional neural network to begin learning.

Creating and submitting a composition

To get started, log in to the AWS DeepComposer console and navigate to the music studio using the left navigation menu. You can use either the sample melodies provided in the console, record a custom melody using the keyboard, or import your own input track. You can adjust the tempo and pitch for your melody in the music studio.

Choose Autoregressive generative AI technique, and then choose Autoregressive CNN Bach model. You have four parameters that you can choose to adjust: Maximum notes to add, Maximum notes to remove, Sampling iterations, and Creative risk. For this example, let’s choose the defaults and select Enhance input melody.

The AR-CNN algorithm allows you to collaborate iteratively with the machine learning algorithm by experimenting with the parameters; you can use the output from one iteration of the AR-CNN algorithm as input to the next iteration.

To submit your composition for the challenge, choose Chartbusters using the left navigation menu, and select Submit a composition. Choose your composition from the drop-down menu, provide a track name for your composition, and select Submit. AWS DeepComposer will submit your composition to the Bach to the Future playlist on SoundCloud. You can select the Vote on SoundCloud button in the console to review and listen to other submissions for the challenge.

Congratulations! You have submitted your first entry for the AWS DeepComposer Chartbusters challenge. Invite your friends and family to listen to and like your composition.

To celebrate the launch of the first Chartbuster challenge, we are offering the AWS DeepComposer keyboard at a special price of $79 for a limited period from 6/23/2020 to 7/15/2020 on amazon.com. The pricing includes the keyboard and a 3-month free trial of AWS DeepComposer services.

Learn more about AWS DeepComposer Chartbusters at https://aws.amazon.com/deepcomposer/chartbusters.

About the Author

Jyothi Nookula is a Principal Product Manager for AWS AI devices. She loves to build products that delight her customers. In her spare time, she loves to paint and host charity fund raisers for her art exhibitions.

Read More

Fraudulent users and malicious accounts can result in billions of dollars in lost revenue annually for businesses. Although many businesses use rule-based filters to prevent malicious activity in their systems, these filters are often brittle and may not capture the full range of malicious behavior.

However, some solutions, such as graph techniques, are especially suited for detecting fraudsters and malicious users. Fraudsters can evolve their behavior to fool rule-based systems or simple feature-based models, but it’s difficult to fake the graph structure and relationships between users and other entities captured in transaction or interaction logs. Graph neural networks (GNNs) combine information from the graph structure with attributes of users or transactions to learn meaningful representations that can distinguish malicious users and events from legitimate ones.

This post shows how to use Amazon SageMaker and Deep Graph Library (DGL) to train GNN models and detect malicious users or fraudulent transactions. Businesses looking for a fully-managed AWS AI service for fraud detection can also use Amazon Fraud Detector, which makes it easy to identify potentially fraudulent online activities, such as the creation of fake accounts or online payment fraud.

In this blog post, we focus on the data preprocessing and model training with Amazon SageMaker.  To train the GNN model, you must first construct a heterogeneous graph using information from transaction tables or access logs. A heterogeneous graph is one that contains different types of nodes and edges. In the case where nodes represent users or transactions, the nodes can have several kinds of distinct relationships with other users and possibly other entities, such as device identifiers, institutions, applications, IP addresses and so on.

Some examples of use cases that fit under this include:

• A financial network where users transact with other users and specific financial institutions or applications
• A gaming network where users interact with other users but also with distinct games or devices
• A social network where users can have different types of links to other users

The following diagram illustrates a heterogeneous financial transaction network.

GNNs can incorporate user features like demographic information or transaction features like activity frequency. In other words, you can enrich the heterogeneous graph representation with features for nodes and edges as metadata. After the node and relations in the heterogeneous graph are established, with their associated features, you can train a GNN model to learn to classify different nodes as malicious or legitimate, using both the node or edge features as well as the graph structure. The model training is set up in a semi-supervised manner—you have a subset of nodes in the graph already labeled as fraudulent or legitimate. You use this labeled subset as a training signal to learn the parameters of the GNN. The trained GNN model can then predict the labels for the remaining unlabeled nodes in the graph.

Architecture

To get started, you can use the full solution architecture that uses Amazon SageMaker to run the processing jobs and training jobs. You can trigger the Amazon SageMaker jobs automatically with AWS Lambda functions that respond to Amazon Simple Storage Service (Amazon S3) put events, or manually by running cells in an example Amazon SageMaker notebook. The following diagram is a visual depiction of the architecture.

The full implementation is available on the GitHub repo with an AWS CloudFormation template that launches the architecture in your AWS account.

Data preprocessing for fraud detection with GNNs

In this section, we show how to preprocess an example dataset and identify the relations that will make up the heterogeneous graph.

Dataset

For this use case, we use the IEEE-CIS fraud dataset to benchmark the modeling approach. This is an anonymized dataset that contains 500 thousand transactions between users. The dataset has two main tables:

• Transactions table – Contains information about transactions or interactions between users
• Identity table – Contains information about access logs, device, and network information for users performing transactions

You use a subset of these transactions with their labels as a supervision signal for the model training. For the transactions in the test dataset, their labels are masked during training. The task is to predict which masked transactions are fraudulent and which are not.

The following code example gets the data and uploads it to an S3 bucket that Amazon SageMaker uses to access the dataset during preprocessing and training (run this in a Jupyter notebook cell):

# Replace with an S3 location or local path to point to your own dataset
raw_data_location = 's3://sagemaker-solutions-us-west-2/Fraud-detection-in-financial-networks/data'

bucket = 'SAGEMAKER_S3_BUCKET'
prefix = 'dgl'
input_data = 's3://{}/{}/raw-data'.format(bucket, prefix)

!aws s3 cp --recursive $raw_data_location $input_data

# Set S3 locations to store processed data for training and post-training results and artifacts respectively
train_data = 's3://{}/{}/processed-data'.format(bucket, prefix)
train_output = 's3://{}/{}/output'.format(bucket, prefix)

Despite the efforts of fraudsters to mask their behavior, fraudulent or malicious activities often have telltale signs like high out-degree or activity aggregation in the graph structure. The following sections show how to perform feature extraction and graph construction to allow the GNN models to take advantage of these patterns to predict fraud.

Feature extraction

Feature extraction consists of performing numerical encoding on categorical features and some transformation of numerical columns. For example, the transaction amounts are logarithmically transformed to indicate the relative magnitude of the amounts, and categorical attributes can be converted to numerical form by performing one hot encoding. For each transaction, the feature vector contains attributes from the transaction tables with information about the time delta between previous transactions, name and addresses matches, and match counts.

Constructing the graph

To construct the full interaction graph, split the relational information in the data into edge lists for each relation type. Each edge list is a bipartite graph between transaction nodes and other entity types. These entity types each constitute an identifying attribute about the transaction. For example, you can have an entity type for the kind of card (debit or credit) used in the transaction, the IP address of the device the transaction was completed with, and the device ID or operating system of the device used. The entity types used for graph construction consist of all the attributes in the identity table and a subset of attributes in the transactions table, like credit card information or email domain. The heterogeneous graph is constructed with the set of per relation type edge lists and the feature matrix for the nodes.

Using Amazon SageMaker Processing

You can execute the data preprocessing and feature extraction step using Amazon SageMaker Processing. Amazon SageMaker Processing is a feature of Amazon SageMaker that lets you run preprocessing and postprocessing workloads on fully managed infrastructure. For more information, see Process Data and Evaluate Models.

First define a container for the Amazon SageMaker Processing job to use. This container should contain all the dependencies that the data preprocessing script requires. Because the data preprocessing here only depends on the pandas library, you can have a minimal Dockerfile to define the container. See the following code:

FROM python:3.7-slim-buster

RUN pip3 install pandas==0.24.2
ENV PYTHONUNBUFFERED=TRUE

ENTRYPOINT ["python3"]

You can build the container and push the built container to an Amazon Elastic Container Registry (Amazon ECR) repository by entering the following of code:

import boto3

region = boto3.session.Session().region_name
account_id = boto3.client('sts').get_caller_identity().get('Account')
ecr_repository = 'sagemaker-preprocessing-container'
ecr_repository_uri = '{}.dkr.ecr.{}.amazonaws.com/{}:latest'.format(account_id, region, ecr_repository)

!bash data-preprocessing/container/build_and_push.sh $ecr_repository docker

When the data preprocessing container is ready, you can create an Amazon SageMaker ScriptProcessor that sets up a Processing job environment using the preprocessing container. You can then use the ScriptProcessor to run a Python script, which has the data preprocessing implementation, in the environment defined by the container. The Processing job terminates when the Python script execution is complete and the preprocessed data has been saved back to Amazon S3. This process is completely managed by Amazon SageMaker. When running the ScriptProcessor, you have the option of passing in arguments to the data preprocessing script. Specify what columns in the transaction table should be considered as identity columns and what columns are categorical features. All other columns are assumed to be numerical features. See the following code:

from sagemaker.processing import ScriptProcessor, ProcessingInput, ProcessingOutput

script_processor = ScriptProcessor(command=['python3'],
                                   image_uri=ecr_repository_uri,
                                   role=role,
                                   instance_count=1,
                                   instance_type='ml.r5.24xlarge')

script_processor.run(code='data-preprocessing/graph_data_preprocessor.py',
                     inputs=[ProcessingInput(source=input_data,
                                             destination='/opt/ml/processing/input')],
                     outputs=[ProcessingOutput(destination=train_data,
                                               source='/opt/ml/processing/output')],
                     arguments=['--id-cols', 'card1,card2,card3,card4,card5,card6,ProductCD,addr1,addr2,P_emaildomain,R_emaildomain',
                                '--cat-cols',' M1,M2,M3,M4,M5,M6,M7,M8,M9'])

The following code example shows the outputs of the Amazon SageMaker Processing job stored in Amazon S3:

from os import path
from sagemaker.s3 import S3Downloader
processed_files = S3Downloader.list(train_data)
print("===== Processed Files =====")
print('n'.join(processed_files))Output:

===== Processed Files =====
s3://graph-fraud-detection/dgl/processed-data/features.csv
s3://graph-fraud-detection/dgl/processed-data/relation_DeviceInfo_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_DeviceType_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_P_emaildomain_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_ProductCD_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_R_emaildomain_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_TransactionID_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_addr1_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_addr2_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_card1_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_card2_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_card3_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_card4_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_card5_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_card6_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_01_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_02_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_03_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_04_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_05_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_06_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_07_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_08_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_09_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_10_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_11_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_12_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_13_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_14_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_15_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_16_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_17_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_18_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_19_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_20_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_21_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_22_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_23_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_24_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_25_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_26_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_27_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_28_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_29_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_30_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_31_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_32_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_33_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_34_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_35_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_36_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_37_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/relation_id_38_edgelist.csv
s3://graph-fraud-detection/dgl/processed-data/tags.csv
s3://graph-fraud-detection/dgl/processed-data/test.csv

All the relation edgelist files represent the different kinds of edges used to construct the heterogenous graph during training. Features.csv contains the final transformed features of the transaction nodes, and tags.csv contains the labels of the nodes used as the training supervision signal. Test.csv contains the TransactionID data to use as a test dataset to evaluate the performance of the model. The labels for these nodes are masked during training.

GNN model training

Now you can use Deep Graph Library (DGL) to create the graph and define a GNN model, and use Amazon SageMaker to launch the infrastructure to train the GNN. Specifically,  a relational graph convolutional neural network model can be used to learn embeddings for the nodes in the heterogeneous graph, and a fully connected layer for the final node classification.

Hyperparameters

To train the GNN, you need to define a few hyperparameters that are fixed before the training process, such as the kind of graph you’re constructing, the class of GNN models you’re using, the network architecture, and the optimizer and optimization parameters. See the following code:

edges = ",".join(map(lambda x: x.split("/")[-1], [file for file in processed_files if "relation" in file]))
params = {'nodes' : 'features.csv',
          'edges': 'relation*.csv',
          'labels': 'tags.csv',
          'model': 'rgcn',
          'num-gpus': 1,
          'batch-size': 10000,
          'embedding-size': 64,
          'n-neighbors': 1000,
          'n-layers': 2,
          'n-epochs': 10,
          'optimizer': 'adam',
          'lr': 1e-2
        }

The preceding code shows a few of the hyperparameters. For more information about all the hyperparameters and their default values, see estimator_fns.py in the GitHub repo.

Model training with Amazon SageMaker

With the hyperparameters defined, you can now kick off the training job. The training job uses DGL, with MXNet as the backend deep learning framework, to define and train the GNN. Amazon SageMaker makes it easy to train GNN models with the framework estimators, which have the deep learning framework environments already set up. For more information about training GNNs with DGL on Amazon SageMaker, see Train a Deep Graph Network.

You can now create an Amazon SageMaker MXNet estimator and pass in the model training script, hyperparameters, and the number and type of training instances you want. You can then call fit on the estimator and pass in the training data location in Amazon S3. See the following code:

from sagemaker.mxnet import MXNet

estimator = MXNet(entry_point='train_dgl_mxnet_entry_point.py',
                  source_dir='dgl-fraud-detection',
                  role=role, 
                  train_instance_count=1, 
                  train_instance_type='ml.p2.xlarge',
                  framework_version="1.6.0",
                  py_version='py3',
                  hyperparameters=params,
                  output_path=train_output,
                  code_location=train_output,
                  sagemaker_session=sess)

estimator.fit({'train': train_data})

Results

After training the GNN, the model learns to distinguish legitimate transactions from fraudulent ones. The training job produces a pred.csv file, which contains the model’s predictions for the transactions in test.csv. The ROC curve depicts the relationship between the true positive rate and the false positive rate at various thresholds, and the Area Under the Curve (AUC) can be used as an evaluation metric. The following graph shows that the GNN model we trained outperforms both fully connected feed forward networks and gradient boosted trees that use the features but don’t fully take advantage of the graph structure.

Conclusion

In this post, we showed how to construct a heterogeneous graph from user transactions and activity and use that graph and other collected features to train a GNN model to predict which transactions are fraudulent. This post also showed how to use DGL and Amazon SageMaker to define and train a GNN that achieves high performance on this task. For more information about the full implementation of the project and other GNN models for the task, see the GitHub repo.

Additionally, we showed how to perform data processing to extract useful features and relations from raw transaction data logs using Amazon SageMaker Processing. You can get started with the project by deploying the provided CloudFormation template and passing in your own dataset to detect malicious users and fraudulent transactions in your data.

About the Author

Soji Adeshina is a Machine Learning Developer who works on developing deep learning based solutions for AWS customers. Currently, he’s working on graph learning with applications in financial services and advertising but he also has a background in computer vision and recommender systems. In his spare time, he likes to cook and read philosophical texts.

Read More