Posts in category: Amazon AWS

For many organizations, consolidating information assets and making them available to employees when needed remains a challenge. Commonly used technology like spreadsheets, relational databases, and NoSQL databases exacerbate this issue by creating more and more unconnected, unstructured data.

Knowledge graphs can provide easier access and understanding to this data by organizing this data and capturing dataset semantics, properties, and relationships. While some organizations use knowledge graphs like Amazon Neptune to add structure to their data, they still lack a targeted search engine that users can leverage to search this information.

Amazon Kendra is an intelligent search service powered by machine learning. Ken­dra reimagines enterprise search for your websites and applications so your employees and customers can easily find the content they are looking for, even when it’s scattered across multiple locations and content repositories within your organization.

This solution illustrates how to create an intelligent search engine on AWS using Amazon Kendra to search a knowledge graph stored in Amazon Neptune. We illustrate how you can provision a new Amazon Kendra index in just a few clicks – no prior Machine Learning (ML) experience required! We then show how an existing knowledge graph stored in Amazon Neptune can be connected to Amazon Kendra’s pipeline as metadata as well as into the knowledge panels to build a targeted search engine. Knowledge panels are information boxes that appear on search engines when you search for entities (people, places, organizations, things) that are contained in the knowledge graph. Finally, this solution provides you with a ranked list of the top notable entities that match certain criteria and finds more relevant search results extracted from the knowledge graph.

The following are several common use cases for integrating an enterprise search engine with a knowledge graph:

• Add a knowledge graph as metadata to Amazon Kendra to more relevant results
• Derive a ranked list of the top notable entities that match certain criteria
• Predictively complete entities in a search box
• Annotate or organize content using the knowledge graph entities by querying in Neptune

Required Services

In order to complete this solution, you will require the following:

• An AWS account
• Basic knowledge of AWS
• An S3 bucket for your documents (for more information, see Creating a bucket and What is Amazon S3?)
• Amazon Kendra
• Neptune
• AWS Cloud9
• AWS Amplify
• Node.js

Sample dataset

For this solution, we use a subset of the Arbitration Awards Online database, which is publicly available. Arbitration is an alternative to litigation or mediation when resolving a dispute. Arbitration panels are composed of one or three arbitrators who are selected by the parties. They read the pleadings filed by the parties, listen to the arguments, study the documentary and testimonial evidence, and render a decision.

The panel’s decision, called an award, is final and binding on all the parties. All parties must abide by the award, unless it’s successfully challenged in court within the statutory time period. Arbitration is generally confidential, and documents submitted in arbitration are not publicly available, unlike court-related filings.

However, if an award is issued at the conclusion of the case, the Financial Industry Regulatory Authority (FINRA) posts it in its Arbitration Awards Online database, which is publicly available. We use a subset of this dataset for our use case under FINRA licensing (©2020 FINRA. All rights reserved. FINRA is a registered trademark of the Financial Industry Regulatory Authority, Inc. Reprinted with permission from FINRA) to create a knowledge graph for awards.

Configuring your document repository

Before you can create an index in Amazon Kendra, you need to load documents into an S3 bucket. This section contains instructions to create an S3 bucket, get the files, and load them into the bucket. After completing all the steps in this section, you have a data source that Amazon Kendra can use.

1. On the AWS Management Console, in the Region list, choose US East (N. Virginia) or any Region of your choice that Amazon Kendra is available in.
2. Choose Services.
3. Under Storage, choose S3.
4. On the Amazon S3 console, choose Create bucket.
5. Under General configuration, provide the following information:
   1. Bucket name – enterprise-search-poc-ds-UNIQUE-SUFFIX
   2. Region – Choose the same Region that you use to deploy your Amazon Kendra index (this post uses US East (N. Virginia) us-east-1)
6. Under Bucket settings for Block Public Access, leave everything with the default values.
7. Under Advanced settings, leave everything with the default values.
8. Choose Create bucket.
9. Download kendra-graph-blog-data and unzip the files.
10. Upload the index_data folder from the unzipped files.

Inside your bucket, you should now see two folders: index_data (with 20 objects) and graph_data (with two objects).

The following screenshot shows the contents of enterprise-search-poc-ds-UNIQUE-SUFFIX.

The following screenshot shows the contents of index_data.

The index_data folder contains two files: an arbitration PDF file and arbitration metadata file.

The following code is an example for arbitration metadata. DocumentId is the arbitration case number, and we use this identifier to create a correlation between the Amazon Kendra index and a graph dataset that we load into Neptune.

{
  "DocumentId": "17-00486",
  "ContentType": "PDF",
  "Title": "17-00486",
  "Attributes": {
    "_source_uri": "https://www.finra.org/sites/default/files/aao_documents/17-00486.pdf"
  }
}

The following screenshot shows the contents of graph_data.

Setting up an Amazon Kendra index

In this section, we set up an Amazon Kendra index and configure an S3 bucket as the data source.

1. Sign in to the console and confirm that you have set the Region to us-east-1.
2. Navigate to the Amazon Kendra service and choose Launch Amazon Kendra.
3. For Index name, enter enterprise-search-poc.
4. For IAM role, choose Create a new role.
5. For Role name, enter poc-role.
6. Leave Use an AWS KMS managed encryption key
7. Choose Create.

The index creation may take some time. For more information about AWS Identity and Access Management (IAM) access roles, see IAM access roles for Amazon Kendra.

8. When index creation is complete, on the Amazon Kendra console, choose your new index.
9. In the Index settings section, locate the index ID.

You use the index ID in a later step.

Adding a data source

To add your data source, complete the following steps:

1. On the Amazon Kendra console, choose your index.
2. Choose Add data source.
3. For Select connector type for your data source, choose Add connector under Amazon S3.
4. For Data source name, enter a name for your data source (for example, ent-search-poc-ds-001).
5. For Enter the data source location, choose Browse S3 and choose the bucket you created earlier.
6. For IAM role, choose Create new role.
7. For Role name, enter poc-ds-role.
8. In the Additional configuration section, on the Include pattern tab, add index_data.
9. In the Set sync run schedule section, choose Run on demand.
10. Choose Next.

11. Review your details and choose Create.
12. In the details page of your data source, choose Sync now.

Kendra starts crawling and indexing the data source from Amazon S3 and prepares the index.

You can also monitor the process on Amazon CloudWatch.

Searching the ingested documents

To test the index, complete the following steps:

1. On the Amazon Kendra console, navigate to your index.
2. Choose Search console.

3. Enter a question (for example, how much was initial claim fees for case 17-00486?).

The following screenshot shows your search results.

Knowledge graph

This section describes a knowledge graph of the entities and relationships that participate in arbitration panels. We use Apache TinkerPop Gremlin format to load the data to Neptune. For more information, see Gremlin Load Data Format.

To load Apache TinkerPop Gremlin data using the CSV format, you must specify the vertices and the edges in separate files. The loader can load from multiple vertex files and multiple edge files in a single load job.

The following diagram shows the graph ontology. Each award has properties, such as Problem, Customer, Representative, Firm, and Subtype. The is_related edge shows relationships between awards.

You can access the CSV files for both vertex and nodes in the enterprise-search-poc.zip file. The following screenshot shows a tabular view of the vertex file.

The following screenshot shows a tabular view of the edge file.

Launching the Neptune-SageMaker stack

You can launch the Neptune-SageMaker stack from the AWS CloudFormation console by choosing Launch Stack:

Region	View	Launch
US East 1 (N. Virginia)	View

Acknowledge that AWS CloudFormation will create IAM resources, and choose Create.

The Neptune and Amazon SageMaker resources described here incur costs. With Amazon SageMaker hosted notebooks, you pay simply for the Amazon Elastic Compute Cloud (Amazon EC2) instance that hosts the notebook. For this post, we use an ml.t2.medium instance, which is eligible for the AWS free tier.

The solution creates five stacks, as shown in the following screenshot.

Browsing and running the content

After the stacks are created, you can browse your notebook instance and run the content.

1. On the Amazon SageMaker console, choose Notebook instances on the navigation pane.
2. Select your instance and from the Actions menu, choose Open Jupyter.

3. In the Jupyter window, in the Neptune directory, open the Getting-Starteddirectory.

The Getting-Starteddirectory contains three notebooks:

• 01-Introduction.ipynb
• 02-Labelled-Property-Graph.ipynb
• 03-Graph-Recommendations.ipynb

The first two introduce Neptune and the property graph data model. The third contains an runnable example of an arbitration knowledge graph recommendation engine. When you run the content, the notebook populates Neptune with a sample award dataset and issues several queries to generate related cases recommendations.

4. To see this in action, open 03-Graph-Recommendations.ipynb.
5. Change the bulkLoad Amazon S3 location to your created Amazon S3 location (enterprise-search-poc-ds-UNIQUE-SUFFIX).

6. Run each cell in turn, or choose Run All from the Cell drop-down menu.

You should see the results of each query printed below each query cell (as in the following screenshot).

Architecture overview

DocumentId in Amazon Kendra is the key for an Amazon Kendra index and knowledge graph data. DocumentId in an Amazon Kendra index should be the same as ~id in the graph node. This creates the association between the Amazon Kendra results and graph nodes. The following diagram shows the architecture for integrating Amazon Kendra and Neptune.

The architecture workflow includes the following steps:

1. The search user interface (UI) sends the query to Amazon Kendra.
2. Amazon Kendra returns results based on its best match.
3. The UI component calls Neptune via Amazon API Gateway and AWS Lambda with the docId as the request parameter.
4. Neptune runs the query and returns all related cases with the requested docId.
5. The UI component renders the knowledge panel with the graph responses.

Testing Neptune via API Gateway

To test Neptune via API Gateway, complete the following steps:

1. On the API Gateway console, choose APIs.
2. Choose KendraGraphAPI to open the API page.
   
3. Select the POST method and choose Test.
4. Enter the following sample event data:

   {
     "docId": "14-02936",
     "repCrd": "5048331",
     "firm": ["23131","29604"]
   }

5. Choose Test.

This sends an HTTP POST request to the endpoint, using the sample event data in the request body. In the following screenshot, the response shows the related cases for award 14-02936.

6. On the navigation name, choose Stages.
7. Choose dev.
8. Copy the Invoke URL value and save it to use in the next step.

9. To test the HTTP, enter the following CURL command. Replace the endpoint with your API Gateway invoke URL.

   curl --location --request POST 'REPLACE_WITH_API_GATEWAY_ENDPOINT' 
   --header 'Content-Type: application/json' 
   --data-raw '{
   "docId": "14-02936",
   "repCrd": "5048331",
   "firm": ["23131","29604"]
   }'
   

Testing Neptune via Lambda

Another way to test Neptune is with a Lambda function. This section outlines how to invoke the Lambda function using the sample event data provided.

1. On the Lambda console, choose Kendra-Neptune-Graph-AddLamb-NeptuneLambdaFunction.
2. Choose Test.
3. In the Configure test event page, choose Create new test event.
4. For Event template, choose the default Hello World
5. For Event name, enter a name and note the following sample event template:

   {
     "docId": "14-02936",
     "repCrd": "5048331",
     "firm": ["23131","29604"]
   }

6. Choose Create.

7. Choose Test.

Each user can create up to 10 test events per function. Those test events aren’t available to other users.

Lambda runs your function on your behalf. The handler in your Lambda function receives and processes the sample event.

8. After the function runs successfully, view the results on the Lambda console.

The results have the following sections:

• Execution result – Shows the run status as succeeded and also shows the function run results, returned by the return statement.
• Summary – Shows the key information reported in the Log output section (the REPORT line in the run log).
• Log output – Shows the log Lambda generates for each run. These are the logs written to CloudWatch by the Lambda function. The Lambda console shows these logs for your convenience. The Click here link shows the logs on the CloudWatch console. The function then adds logs to CloudWatch in the log group that corresponds to the Lambda function.

Developing the web app

In this section, we develop a web app with a search interface to search the documents. We use AWS Cloud9 as our integrated development environment (IDE) and Amplify to build and deploy the web app.

AWS Cloud9 is a cloud-based IDE that lets you write, run, and debug your code with just a browser. It includes a code editor, debugger, and a terminal. AWS Cloud9 comes prepackaged with essential tools for popular programming languages, including JavaScript, Python, PHP, and more, so you don’t need to install files or configure your development machine to start new projects.

The AWS Cloud9 workspace should be built by an IAM user with administrator privileges, not the root account user. Please ensure you’re logged in as an IAM user, not the root account user.

Ad blockers, JavaScript disablers, and tracking blockers should be disabled for the AWS Cloud9 domain, otherwise connecting to the workspace might be impacted.

Creating a new environment

To create your environment, complete the following steps:

1. On the AWS Cloud9 console, make sure you’re using one of the following Regions:
   • US East (N. Virginia)
   • US West (Oregon)
   • Asia Pacific (Singapore)
   • Europe (Ireland)
2. Choose Create environment.
3. Name the environment kendrapoc.
4. Choose Next step.
5. Choose Create a new instance for environment (EC2) and choose small.
6. Leave all the environment settings as their defaults and choose Next step.
7. Choose Create environment.

Preparing the environment

To prepare your environment, make sure you’re in the default directory in an AWS Cloud9 terminal window (~/environment) before completing the following steps:

1. Enter the following code to download the code for the Amazon Kendra sample app and extract it in a temporary directory:

   mkdir tmp
   cd tmp
   aws s3 cp s3://aws-ml-blog/artifacts/Incorporating-your-enterprise-knowledge-graph-into-Amazon-Kendra/kendra-graph-blog-data/kendrasgraphsampleui.zip .
   unzip kendrasgraphsampleui.zip
   rm kendrasgraphsampleui.zip
   cd ..

2. We build our app in ReactJS with the following code:

   echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p 

   npx create-react-app kendra-poc

3. Change the working directory to kendra-poc and ensure that you’re in the /home/ec2-
user/environment/kendra-poc directory:

   cd kendra-poc

4. Install a few prerequisites with the following code:

   npm install --save node-sass typescript bootstrap react-bootstrap @types/lodash aws-sdk
   npm install --save semantic-ui-react
   npm install aws-amplify @aws-amplify/ui-react
   npm install -g @aws-amplify/cli

5. Copy the source code to the src directory:

   cp -r ../tmp/kendrasgraphsampleui/* src/

Initializing Amplify

To initialize Amplify, complete the following steps:

1. On the command line, in the kendra-poc directory, enter the following code:

   amplify init

2. Choose Enter.
3. Accept the default project name kendrapoc.
4. Enter dev for the environment name.
5. Choose None for the default editor (we use AWS Cloud9).
6. Choose JavaScript and React when prompted.
7. Accept the default values for paths and build commands.
8. Choose the default profile when prompted.

Your run should look like the following screenshot.

Adding authentication

To add authentication to the app, complete the following steps:

1. Enter the following code:

   amplify add auth

2. Choose Default Configuration when asked if you want to use the default authentication and security configuration.
3. Choose Username when asked how you want users to sign in.
4. Choose No, I am done. when asked about advanced settings.

This session should look like the following screenshot.

5. To create these changes in the cloud, enter:

   amplify push

6. Confirm you want Amplify to make changes in the cloud for you.

Provisioning takes a few minutes to complete. The Amplify CLI takes care of provisioning the appropriate cloud resources and updates src/aws-exports.js with all the configuration data we need to use the cloud resources in our app.

Amazon Cognito lets you add user sign-up, sign-in, and access control to your web and mobile apps quickly and easily. We made a user pool, which is a secure user directory that lets our users sign in with the user name and password pair they create during registration. Amazon Cognito (and the Amplify CLI) also supports configuring sign-in with social identity providers, such as Facebook, Google, and Amazon, and enterprise identity providers via SAML 2.0. For more information, see Amazon Cognito Developer and Amplify Authentication documentations

Configuring an IAM role for authenticated users

To configure your IAM role for authenticated users, complete the following steps:

1. In the AWS Cloud9 IDE, in the left panel, browse to the file kendra-poc/amplify/team-provider-info.json and open it (double-click).
2. Note the value of AuthRoleName.

3. On the IAM console, choose Roles.
4. Search for the AuthRole using the value from the previous step and open that role.

5. Choose Add inline policy.
6. Choose JSON and replace the contents with the following policy. Replace enterprise-search-poc-ds-UNIQUE-SUFFIX with the name of the S3 bucket that is configured as the data source (leave the * after the bucket name, which allows the policy to access any object in the bucket). Replace ACCOUNT-NUMBER with the AWS account number and KENDRA-INDEX-ID with the index ID of your index.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": "kendra:ListIndices",
               "Resource": "*"
           },
           {
               "Sid": "VisualEditor1",
               "Effect": "Allow",
               "Action": [
                   "kendra:Query",
                   "s3:ListBucket",
                   "s3:GetObject",
                   "kendra:ListFaqs",
                   "kendra:ListDataSources",
                   "kendra:DescribeIndex",
                   "kendra:DescribeFaq",
                   "kendra:DescribeDataSource"
               ],
               "Resource": [
                   "arn:aws:s3:::enterprise-search-poc-ds-UNIQUE-SUFFIX*",
                   "arn:aws:kendra:us-east-1:ACCOUNT-NUMBER:index/KENDRA-INDEX-ID"
               ]
           }
       ]
   }

7. Choose Review policy.
8. Enter a policy name, such as my-kendra-poc-policy.
9. Choose Create policy.
10. Browse back to the role and confirm that my-kendra-poc-policy is present.

11. Create a user (poctester - customer) and add to the corresponding groups by entering the following at the command prompt:

USER_POOL_ID=`grep user_pools_id src/aws-exports.js| awk 'BEGIN {FS =""" } {print $4}'`
aws cognito-idp create-group --group-name customer --user-pool-id $USER_POOL_ID
aws cognito-idp admin-create-user --user-pool-id $USER_POOL_ID --username poctester --temporary-password AmazonKendra
aws cognito-idp admin-add-user-to-group --user-pool-id $USER_POOL_ID --username poctester --group-name customer

Configuring the application

You’re now ready to configure the application.

1. In the AWS Cloud9 environment, browse to the file kendra-poc/src/search/Search.tsx and open it for editing.

A new window opens.

2. Replace REPLACE_WITH_KENDRA-INDEX-ID with your index ID.

3. In the AWS Cloud9 environment, browse to the file kendra-poc/src/properties.js and open it for editing.
4. In the new window that opens, replace REPLACE_WITH_API_GATEWAY_ENDPOINT with the PI Gateway invoke URL value from earlier.

5. Start the application in the AWS Cloud9 environment by entering the following code in the command window in the ~/environment/kendra-poc directory:

   npm start

Compiling the code and starting takes a few minutes.

6. Preview the running application by choosing Preview on the AWS Cloud9 menu bar.
7. From the drop-down menu, choose Preview Running Application.

A new browser window opens.

8. Log in with any of the users we configured earlier (poctester) with the temporary password AmazonKendra.

Amazon Cognito forces a password reset upon first login.

Using the application

Now we can try out the app we developed by making a few search queries, such as “how much was initial claim fees for case 17-00486?”

The following screenshot shows the Amazon Kendra results and the knowledge panel, which shows the related cases for each result. The knowledge panel details on the left is populated from the graph database and shows all the related cases for each search item.

Conclusion

This post demonstrated how to build a targeted and flexible cognitive search engine with a knowledge graph stored in Neptune and integrated with Amazon Kendra. You can enable rapid search for your documents and graph data using natural language, without any previous AI or ML experience. Finally, you can create an ensemble of other content types, including any combination of structured and unstructured documents, to make your archives indexable and searchable for harvesting knowledge and gaining insight. For more information about Amazon Kendra, see AWS re:Invent 2019 – Keynote with Andy Jassy on YouTube, Amazon Kendra FAQs, and What is Amazon Kendra?

About the Authors

Dr. Yazdan Shirvany is all 12 AWS-certified Senior Solution Architect with deep experience in AI/ML, IOT and big data technologies including NLP, Knowledge Graph, applications reengineering, and optimizing software to leverage the cloud. Dr. Shirvany has 20+ scientific publications, and several issued patents in AI/ML field. Dr. Shirvany holds a M.S and Ph.D. in Computer Science from Chalmers University of Technology.

Dipto Chakravarty is a leader in Amazon’s Alexa engineering group and heads up the Personal Mobility team in HQ2 utilizing AI, ML and IoT to solve local search analytics challenges. He has 12 patents issued to date and has authored two best-selling books on computer architecture and operating systems published by McGraw-Hill and Wiley. Dipto holds a B.S and M.S in Computer Science and Electrical Engineering from U. of Maryland, an EMBA from Wharton School, U. Penn, and a GMP from Harvard Business School.

Mohit Mehta is a leader in the AWS Professional Services Organization with expertise in AI/ML and Big Data technologies. Mohit holds a M.S in Computer Science, all 12 AWS certifications, MBA from College of William and Mary and GMP from Michigan Ross School of Business.

Read More

If you’re a Formula 1 (F1) fan, have you ever wondered why F1 teams have very different performances between qualifying and practice sessions? Why do they have multiple practice sessions in the first place? Can practice session results actually tell something about the upcoming qualifying race? In this post, we answer these questions and more. We show you how we can predict qualifying results based on practice session performances by harnessing the power of data and machine learning (ML). These predictions are being integrated into the new “Qualifying Pace” insight for each F1 Grand Prix (GP). This work is part of the continuous collaboration between F1 and the Amazon ML Solutions Lab to generate new F1 Insights powered by AWS.

Each F1 GP consists of several stages. The event starts with three practice sessions (P1, P2, and P3), followed by a qualifying (Q) session, and then the final race. Teams approach practice and qualifying sessions differently because these sessions serve different purposes. The practice sessions are the teams’ opportunities to test out strategies and tire compounds to gather critical data in preparation for the final race. They observe the car’s performance with different strategies and tire compounds, and use this to determine their overall race strategy.

In contrast, qualifying sessions determine the starting position of each driver on race day. Teams focus solely on obtaining the fastest lap time. Because of this shift in tactics, Friday and Saturday practice session results often fail to accurately predict the qualifying order.

In this post, we introduce deterministic and probabilistic methods to model the time difference between the fastest lap time in practice sessions and the qualifying session (∆t = t_q-t_p). The goal is to more accurately predict the upcoming qualifying standings based on the practice sessions.

Error sources of ∆t

The delta of the fastest lap time between practice and qualifying sessions (∆t) comes primarily from variations in fuel level and tire grip.

A higher fuel level adds weight to the car and reduces the speed of the car. For practice sessions, teams vary the fuel level as they please. For the second practice session (P2), it’s common to begin with a low fuel level and run with more fuel in the latter part of the session. During qualifying, teams use minimal fuel levels in order to record the fastest lap time. The impact of fuel on lap time varies from circuit to circuit, depending on how many straights the circuit has and how long these straights are.

Tires also play a significant role in an F1 car’s performance. During each GP event, the tire supplier brings various tire types with varying compounds suitable for different racing conditions. Two of these are for wet circuit conditions: intermediate tires for light standing water and wet tires for heavy standing water. The remaining dry running tires can be categorized into three compound types: hard, medium, and soft. These tire compounds provide different grips to the circuit surface. The more grip the tire provides, the faster the car can run.

Past racing results showed that car performance dropped significantly when wet tires were used. For example, in the 2018 Italy GP, because the P1 session was wet and the qualifying session was dry, the fastest lap time in P1 was more than 10 seconds slower than the qualifying session.

Among the dry running types, the hard tire provides the least grip but is the most durable, whereas the soft tire has the most grip but is the least durable. Tires degrade over the course of a race, which reduces the tire grip and slows down the car. Track temperature and moisture affects the progression of degradation, which in turn changes the tire grip. As in the case with fuel level, tire impact on lap time changes from circuit to circuit.

Data and attempted approaches

Given this understanding of factors that can impact lap time, we can use fuel level and tire grip data to estimate the final qualifying lap time based on known practice session performance. However, as of this writing, data records to directly infer fuel level and tire grip during the race are not available. Therefore, we take an alternative approach with data we can currently obtain.

The data we used in the modeling were records of fastest lap times for each GP since 1950 and partial years of weather data for the corresponding sessions. The lap times data included the fastest lap time for each session (P1, P2, P3, and Q) of each GP with the driver, car and team, and circuit name (publicly available on F1’s website). Track wetness and temperature for each corresponding session was available in the weather data.

We explored two implicit methods with the following model inputs: the team and driver name, and the circuit name. Method one was a rule-based empirical model that attributed observed  to circuits and teams. We estimated the latent parameter values (fuel level and tire grip differences specific to each team and circuit) based on their known lap time sensitivities. These sensitivities were provided by F1 and calculated through simulation runs on each circuit track. Method two was a regression model with driver and circuit indicators. The regression model learned the sensitivity of ∆t for each driver on each circuit without explicitly knowing the fuel level and tire grip exerted. We developed and compared deterministic models using XGBoost and AutoGluon, and probabilistic models using PyMC3.

We built models using race data from 2014 to 2019, and tested against race data from 2020. We excluded data from before 2014 because there were significant car development and regulation changes over the years. We removed races in which either the practice or qualifying session was wet because ∆t for those sessions were considered outliers.

Managed model training with Amazon SageMaker

We trained our regression models on Amazon SageMaker.

Amazon SageMaker is a fully managed service that provides every developer and data scientist with the ability to build, train, and deploy ML models quickly. Specifically for model training, it provides many features to assist with the process.

For our use case, we explored multiple iterations on the choices of model feature sets and hyperparameters. Recording and comparing the model metrics of interest was critical to choosing the most suitable model. The Amazon SageMaker API allowed customized metrics definition prior to launching a model training job, and easy retrieval after the training job was complete. Using the automatic model tuning feature reduced the mean squared error (MSE) metric on the test data by 45% compared to the default hyperparameter choice.

We trained an XGBoost model using the Amazon SageMaker’s built-in implementation. Its built-in implementation allowed us to run model training through a general estimator interface. This approach provided better logging, superior hyperparameter validation, and a larger set of metrics than the original implementation.

Rule-based model

In the rule-based approach, we reason that the differences of lap times ∆t primarily come from systematic variations of tire grip for each circuit and fuel level for each team between practice and qualifying sessions. After accounting for these known variations, we assume residuals are random small numbers with a mean of zero. ∆t can be modeled with the following equation:

∆t_f(c) and ∆t_g(c) are known sensitivities of fuel mass and tire grip, and  is the residual. A hierarchy exists among the factors contained in the equation. We assume grip variations for each circuit (g(c)) are at the top level. Under each circuit, there are variations of fuel level across teams (f(t,c)).

To further simplify the model, we neglect  because we assume it is small. We further assume fuel variation for each team across all circuits is the same (i.e., f(t,c) = f(t)). We can simplify the model to the following:

Because ∆t_f(c) and ∆t_g(c) are known, f(t) and g(c), we can estimate team fuel variations and tire grip variations from the data.

The differences in the sensitivities depend on the characteristics of circuits. From the following track maps, we can observe that the Italian GP circuit has fewer corner turns and the straight sections are longer compared to the Singapore GP circuit. Additional tire grip gives a larger advantage in the Singapore GP circuit.

ML regression model

For the ML regression method, we don’t directly model the relation between  and fuel level and grip variations. Instead, we fit the following regression model with just the circuit, team, and driver indicator variables:

I_c, I_t, and I_d represent the indicator variables for circuits, teams, and drivers.

Hierarchical Bayesian model

Another challenge with modeling the race pace was due to noisy measurements in lap times. The magnitude of random effect (ϵ) of ∆t could be non-negligible. Such randomness might come from drivers’ accidental drift from their normal practice at the turns or random variations of drivers’ efforts during practice sessions. With deterministic approaches, such random effect wasn’t appropriately captured. Ideally, we wanted a model that could quantify uncertainty about the predictions. Therefore, we explored Bayesian sampling methods.

With a hierarchical Bayesian model, we account for the hierarchical structure of the error sources. As with the rule-based model, we assume grip variations for each circuit (g(c))) are at the top level. The additional benefit of a hierarchical Bayesian model is that it incorporates individual-level variations when estimating group-level coefficients. It’s a middle ground between two extreme views of data. One extreme is to pool data for every group (circuit and driver) without considering the intrinsic variations among groups. The other extreme is to train a regression model for each circuit or driver. With 21 circuits, this amounts to 21 regression models. With a hierarchical model, we have a single model that considers the variations simultaneously at the group and individual level.

We can mathematically describe the underlying statistical model for the hierarchical Bayesian approach as the following varying intercepts model:

Here, i represents the index of each data observation, j represents the index of each driver, and k represents the index of each circuit. μ_jk represents the varying intercept for each driver under each circuit, and θ_k represents the varying intercept for each circuit. w_p and w_q represent the wetness level of the track during practice and qualifying sessions, and ∆T represents the track temperature difference.

Test models in the 2020 races

After predicting ∆t, we added it into the practice lap times to generate predictions of qualifying lap times. We determined the final ranking based on the predicted qualifying lap times. Finally, we compared predicted lap times and rankings with the actual results.

The following figure compares the predicted rankings and the actual rankings for all three practice sessions for the Austria, Hungary, and Great Britain GPs in 2020 (we exclude P2 for the Hungary GP because the session was wet).

For the Bayesian model, we generated predictions with an uncertainty range based on the posterior samples. This enabled us to predict the ranking of the drivers relatively with the median while accounting for unexpected outcomes in the drivers’ performances.

The following figure shows an example of predicted qualifying lap times (in seconds) with an uncertainty range for selected drivers at the Austria GP. If two drivers’ prediction profiles are very close (such as MAG and GIO), it’s not surprising that either driver might be the faster one in the upcoming qualifying session.

Metrics on model performance

To compare the models, we used mean squared error (MSE) and mean absolute error (MAE) for lap time errors. For ranking errors, we used rank discounted cumulative gain (RDCG). Because only the top 10 drivers gain points during a race, we used RDCG to apply more weight to errors in the higher rankings. For the Bayesian model output, we used median posterior value to generate the metrics.

The following table shows the resulting metrics of each modeling approach for the test P2 and P3 sessions. The best model by each metric for each session is highlighted.

All models reduced the qualifying lap time prediction errors significantly compared to directly using the practice session results. Using practice lap times directly without considering pace correction, the MSE on the predicted qualifying lap time was up to 2.8 seconds. With machine learning methods which automatically learned pace variation patterns for teams and drivers on different circuits, we brought the MSE down to smaller than half a second. The resulting prediction was a more accurate representation of the pace in the qualifying session. In addition, the models improved the prediction of rankings by a small margin. However, there was no one single approach that outperformed all others. This observation highlighted the effect of random errors on the underlying data.

Summary

In this post, we described a new Insight developed by the Amazon ML Solutions Lab in collaboration with Formula 1 (F1).

This work is part of the six new F1 Insights powered by AWS that are being released in 2020, as F1 continues to use AWS for advanced data processing and ML modeling. Fans can expect to see this new Insight unveiled at the 2020 Turkish GP to provide predictions for the upcoming qualifying races at practice sessions.

If you’d like help accelerating the use of ML in your products and services, please contact the Amazon ML Solutions Lab .

About the Author

Guang Yang is a data scientist at the Amazon ML Solutions Lab where he works with customers across various verticals and applies creative problem solving to generate value for customers with state-of-the-art ML/AI solutions.

Read More

Documents are a primary tool for communication, collaboration, record keeping, and transactions across industries, including financial, medical, legal, and real estate. The format of data can pose an extra challenge in data extraction, especially if the content is typed, handwritten, or embedded in a form or table. Furthermore, extracting data from your documents is manual, error-prone, time-consuming, expensive, and does not scale. Amazon Textract is a machine learning (ML) service that extracts printed text and other data from documents as well as tables and forms.

We’re pleased to announce two new features for Amazon Textract: support for handwriting in English documents, and expanding language support for extracting printed text from documents typed in Spanish, Portuguese, French, German, and Italian.

Handwriting recognition with Amazon Textract

Many documents, such as medical intake forms or employment applications, contain both handwritten and printed text. The ability to extract text and handwriting has been a need our customers have asked us for. Amazon Textract can now extract printed text and handwriting from documents written in English with high confidence scores, whether it’s free-form text or text embedded in tables and forms. Documents can also contain a mix of typed text or handwritten text.

The following image shows an example input document containing a mix of typed and handwritten text, and its converted output document.

You can log in to the Amazon Textract console to test out the handwriting feature, or check out the new demo by Amazon Machine Learning Hero Mike Chambers.

Not only can you upload documents with both printed text and handwriting, you can also use Amazon Augmented AI (Amazon A2I), which makes it easy to build workflows for a human review of the ML predictions. Adding in Amazon A2I can help you get to market faster by having your employees or AWS Marketplace contractors review the Amazon Textract output for sensitive workloads. For more information about implementing a human review, see Using Amazon Textract with Amazon Augmented AI for processing critical documents. If you want to use one of our AWS Partners, take a look at how Quantiphi is using handwriting recognition for their customers.

Additionally, we’re pleased to announce our language expansion. Customers can now extract and process documents in more languages.

New supported languages in Amazon Textract

Amazon Textract now supports processing printed documents in Spanish, German, Italian, French, and Portuguese. You can send documents in these languages, including forms and tables, for data and text extraction, and Amazon Textract automatically detects and extracts the information for you. You can simply upload the documents on the Amazon Textract console or send them using either the AWS Command Line Interface (AWS CLI) or AWS SDKs.

AWS customer success stories

AWS customers like yourself are always looking for ways to overcome document processing. In this section, we share what our customers are saying about Amazon Textract.

Intuit

Intuit is a provider of innovative financial management solutions, including TurboTax and QuickBooks, to approximately 50 million customers worldwide.

“Intuit’s document understanding technology uses AI to eliminate manual data entry for our consumer, small business, and self-employed customers. For millions of Americans who rely on TurboTax every year, this technology simplifies tax filing by saving them from the tedious, time-consuming task of entering data from financial documents. Textract is an important element of Intuit’s document understanding capability, improving data extraction accuracy by analyzing text in the context of complex financial forms.”

– Krithika Swaminathan, VP of AI, Intuit

Veeva

Veeva helps cosmetics, consumer goods and chemical companies bring innovative, high quality products to market faster without compromising compliance.

“Our customers are processing millions of documents per year and have a critical need to extract the information stored within the documents to make meaningful business decisions. Many of our customers are multinational organizations which means the documents are submitted in various languages like Spanish or Portuguese. Our recent partnership with AWS allowed us early access to Amazon Textract’s new feature that supports additional languages like Spanish, and Portuguese. This partnership with Textract has been key to work closely, iterate and deliver exceptional solutions to our customers.”

– Ali Alemdar, Sr Product Manager, Veeva Industries

Baker Tilly

Baker Tilly is a leading advisory, tax and assurance firm dedicated to building long-lasting relationships and helping customers with their most pressing problems — and enabling them to create new opportunities.

“Across all industries, forms are one of the most popular ways of collecting data. Manual efforts can take hours or days to “read” through digital forms. Leveraging Amazon Textract’s Optical Character Recognition (OCR) technology we can now read through these digital forms quicker and effortlessly. We now leverage handwriting as part of Textract to parse out handwritten entities. This allows our customers to upload forms with both typed and handwritten text and improve their ability to make key decisions through data quickly and in a streamlined process. Additional, Textract easily integrates with Amazon S3 and RDS for instantaneous access to processed forms and near real-time analytics.”

-Ollie East – Director of Advanced Analytics and Data Engineering

ARG Group

ARG Group is the leading end-to-end provider of digital solutions for the corporate and government market.

“At ARG Group, we work with different transportation companies and their physical asset maintenance teams. Their processes have been refined over many years. Previous attempts to digitize the process caused too much disruption and consequently failed to be adopted. Textract allowed us to provide a hybrid solution to gain the benefits of predictive insights coming from digitizing maintenance data, whilst still allowing our customer workforce to continue following their preferred handwritten process. This is expected to result in a 22% reduction in downtime and 18% reduction in maintenance cost, as we can now predict when parts are likely to fail and schedule for maintenance to happen outside of production hours. We are also expecting the lifespan of our customer assets to increase, now that we are preventing failure scenarios.”

– Daniel Johnson, Business Segment Director, ARG Group

Belle Fleur

Belle Fleur believes the ML revolution is altering the way we live, work, and relate to one another, and will transform the way every business in every industry operates.

“We use Amazon Textract to detect text for our clients that have the three Vs when it pertains to data: Variety, Velocity, and Volume, and particularly our clients that have different document formats to process information and data properly and efficiently. The feature designed to recognize the various different formats, whether it’s tables or forms and now with handwriting recognition, is an AI dream come true for our medical, legal, and commercial real estate clients. We are so excited to roll out this new handwriting feature to all of our customers to further enhance their current solution, especially those with lean teams. We are able to allow the machine learning to handle the heavy lifting via automation to read thousands of documents in a fraction of the time and allow their teams to focus on higher-order assignments.”

– Tia Dubuisson, President, Belle Fleur

Lumiq

Lumiq is a data analytics company, holding the deep domain and technical expertise to build and implement AI- and ML-driven products and solutions. Their data products are built like building blocks and run on AWS, which helps their customers scale the value of their data and drive tangible business outcomes.

“With thousands of documents being generated and received across different stages of the consumer engagement lifecycle every day, one of our customers (a leading insurance service provider in India) had to invest several manual hours for data entry, data QC, and validation. The document sets consisted of proposal forms, supporting documents for identity, financials, and medical reports, among others. These documents were in different, non-standardized formats and some of them were handwritten, resulting in an increased average lag in lead to policy issuance and impacted customer experience.

“We leveraged Amazon’s machine learning-powered Textract to extract information and insights from various types of documents, including handwritten text. Our custom solution built on top of Amazon Textract and other AWS services helped in achieving a 97% reduction in human labor for PII redaction and a projected 70% reduction in work hours for data entry. We are excited to further deep-dive into Textract to enable our customers with an E2E paperless workflow and enhance their end-consumer experience with significant time savings.”

– Mohammad Shoaib, Founder and CEO, Lumiq (Crisp Analytics)

QL Resources

QL is among Asean’s largest egg producers and surimi manufacturers, and is building a presence in the sustainable palm oil sector with activities including milling, plantations, and biomass clean energy.

“We have a large amount of handwritten documents that are generated daily in our factories, where it is challenging to ubiquitously install digital capturing devices. With the custom solution developed by our AWS partner Axrail using Amazon Textract and various AWS services, we are able to digitize documents for both printed and handwritten hard copy forms that we generated on the production floor daily, especially in production areas where digital capturing tools are not available or economical. This is a sensible solution and completes the missing link for full digitization of our production data.”

– Chia Lik Khai, Director, QL Resources

Summary

We continually make improvements to our products based on your feedback, and we encourage you to log in to the Amazon Textract console and upload a sample document and use the APIs available. You can also talk with your account manager about how best to incorporate these new features. Amazon Textract has many resources to help you get started, like blog posts, videos, partners, and getting started guides. Check out the Textract resources page for more information.

You have millions of documents, which means you have a ton of meaningful and critical data within those documents. You can extract and process your data in seconds rather than days, and keep it secure by using Amazon Textract. Get started today.

About the Author

Andrea Morton-Youmans is a Product Marketing Manager on the AI Services team at AWS. Over the past 10 years she has worked in the technology and telecommunications industries, focused on developer storytelling and marketing campaigns. In her spare time, she enjoys heading to the lake with her husband and Aussie dog Oakley, tasting wine and enjoying a movie from time to time.

Read More

Over the past few years, businesses have experienced a steep rise in the volume of documents they have to deal with. These include a wide range of structured and unstructured text spread across different document formats. Processing these documents and extracting data is labor-intensive and costly. It involves complex operations that can easily go wrong, leading to regulatory breaches and hefty fines. Many digitally mature organizations, therefore, have started using intelligent document processing solutions.

Quantiphi has been a part of this transformation and witnessed higher adoption of QDox, our document processing solution built on top of Amazon Textract. It extracts information to gain business insights and automate downstream business processes. We have helped customers across insurance, healthcare, financial services, manufacturing automate loans processing, patient onboarding, and compliance management, to name a few.

Although these solutions have solved some crucial problems for businesses in reducing their manual efforts, extracting information from handwritten text has been a challenge. This is primarily because handwritten texts come with their own set of complexities, such as:

• Differences in handwriting styles
• Poor quality or illegible handwriting
• Joined or cursive handwritten text
• Compression or expansion of text

These challenges make it difficult to capture data correctly and gain meaningful insights for companies.

Use case: Insurance provider

Recently, one of our customers, a large supplemental insurance provider based in the US, was facing a similar challenge in extracting vital information from a doctor’s handwritten notes that accounted for 20% of the total documents. Initially, they manually sifted through the documents to decide on the claims payout and asked to automate the process, because it took 5–6 days to process the claim. As part of the process, we built a solution to extract printed and handwritten text from several supporting documents to verify the claim. To ease the process for policyholders, we built a user interface that could interact with users using a conversational agent, and the agent could request the necessary supporting documents to process the claim. The solution extracted information from the supporting documents, such as claim application, doctor notes, and invoices to validate the claim.

The following diagram illustrates the process flow.

The solution reduced manual intervention by over 70%, but extracting and validating information from a doctor’s handwritten note was still a task. The accuracy was low and required human intervention to validate the information, which impacted process efficiency.

Solution: Amazon Textract

As an AWS partner, we reached out to the Amazon Textract product team with a need to support handwriting recognition. They assured us they were developing a solution to address such challenges. When Amazon Textract came out with a beta version of the product for handwritten text, we were among the first to get private access. The Textract team worked closely with us and iterated quickly to improve the accuracy for a wide variety of documents. Below is an example of one of our documents that Textract recognized. In fact, our customers are also happy that it does even better than other handwriting recognition services we tested for them.

We used the Amazon Textract handwriting beta version with a few sample customer documents, and we saw it improved the accuracy of the entire process by over 90%, while reducing manual efforts significantly. This enabled us to expand the scope of our platform to additional offices of our client.

Armed with the success of our customers, we’re planning to implement the Amazon Textract handwriting solution into different processes across industries. As the product is set to launch, we believe that the implementation will become much easier and the results will improve considerably.

Summary

Overall, our partnership with AWS has helped us solve some challenging business problems to bring value to our customers. We plan to continue working with AWS to solve more challenging problems to bring real value to our customers.

There are many ways to get started with Amazon Textract: reach out to our AWS Partner Quantiphi, reach out to your account manager or solutions architect, or visit our Amazon Textract product page and learn more about the resources available.

About the Author

Vibhav Sangam Gupta is a Client Solutions Partner at Quantiphi, Inc, an Applied AI and Machine Learning software and services company focused on helping organizations translate the big promise of Big Data & Machine Learning technologies into quantifiable business impact.

Read More

We’re excited to announce dynamic filters in Amazon Personalize, which allow you to apply business rules to your recommendations on the fly, without any extra cost. Dynamic filters create better user experiences by allowing you to tailor your business rules for each user when you generate recommendations. They save you time by removing the need to define all possible permutations of your business rules in advance and enable you to use your most recent information to filter recommendations. You control the recommendations for your users in real time while responding to their individual needs, preferences, and changing behavior to improve engagement and conversion.

For online retail use cases, you can use dynamic filters in Amazon Personalize to generate recommendations within the criteria specified by the shopper, such as brand choices, shipping speed, or ratings. For video-on-demand users, you can make sure you include movies or television shows from their favorite directors or actors based on each individual user’s preferences. When users subscribe to premium services, or their subscription expires, you can ensure that their content recommendations are based on their subscription status in real time.

Based on over 20 years of personalization experience, Amazon Personalize enables you to improve customer engagement by powering personalized product and content recommendations and targeted marketing promotions. Amazon Personalize uses machine learning (ML) to create higher-quality recommendations for your websites and applications. You can get started without any prior ML experience, using simple APIs to easily build sophisticated personalization capabilities in just a few clicks. All your data is encrypted to be private and secure, and is only used to create recommendations for your users.

Setting up and applying filters to your recommendations is simple; it only takes only a few minutes to define and deploy your business rules. You can use the Amazon Personalize console or API to create a filter with your logic using the Amazon Personalize DSL (Domain Specific Language).

To create a filter that is customizable in real-time, you define your filter expression criteria using a placeholder parameter instead of a fixed value. This allows you to choose the values to filter by applying a filter to a recommendation request, rather than when you create your expression. You provide a filter when you call the GetRecommendations or GetPersonalizedRanking API operations, or as a part of your input data when generating recommendations in batch mode through a batch inference job.

This post walks you through the process of defining and applying item and user metadata-based recommendation filters with statically or dynamically defined filter values in Amazon Personalize.

Prerequisites

To define and apply filters, you first need to set up some Amazon Personalize resources on the Amazon Personalize console. For full instructions, see Getting Started (Console).

1. Create a dataset group.
2. Create an Interactions dataset using the following schema :

   {
       "type": "record",
       "name": "Interactions",
       "namespace": "com.amazonaws.personalize.schema",
       "fields": [
           {
               "name": "USER_ID",
               "type": "string"
           },
           {
               "name": "ITEM_ID",
               "type": "string"
           },
           {
               "name": "EVENT_TYPE",
               "type": "string"
           },
           {
               "name": "EVENT_VALUE",
               "type": ["null","float"]
           },
           {
               "name": "IMPRESSION",
               "type": [“null”,"string"]
           },
           {
               "name": "TIMESTAMP",
               "type": "long"
           }
       ],
       "version": "1.0"
   }

3. Import the data using the following data file.
4. Create an Items dataset using the following schema:

   {
   	"type": "record",
   	"name": "Items",
   	"namespace": "com.amazonaws.personalize.schema",
   	"fields": [
   		{
   			"name": "ITEM_ID",
   			"type": "string"
   		},
   		{
   			"name": “TITLE”,
   			"type": "string"
   		},
   		{
   			"name": "GENRE",
   			"type": [
   				"null",
   				"string"
   			],
   			"categorical": true
   		},
   		{
               		"name": "CREATION_TIMESTAMP",
               		"type": "long"
           		}
   	],
   	"version": "1.0"
   }

5. Import data using the following data file.
6. Create a Users dataset using the following schema:

   {
       "type": "record",
       "name": "Users",
       "namespace": "com.amazonaws.personalize.schema",
       "fields": [
           {
               "name": "USER_ID",
               "type": "string"
           },
           {
               "name": "AGE",
               "type": ["null", "int"]
           },
           {
               "name": “SUBSCRIPTION”,
               "type": [“null”,”string”]
           }
       ],
       "version": "1.0"
   }

7. Import the data using the following data file.
8. Create a solution using any recipe. In this post, we use the aws-user-personalization
9. Create a campaign.

Creating your filter

Now that you have set up your Amazon Personalize resources, you can define and test your custom filters.

Filter expression language

Amazon Personalize uses its own DSL called filter expressions to determine which items to exclude or include in a set of recommendations. Filter expressions can only filter recommendations based on datasets in the same dataset group, and you can only use them to filter results for solution versions (an Amazon Personalize model trained using your datasets in the dataset group) or campaigns (a deployed solution version for real-time recommendations). Amazon Personalize can filter items based on user-item interactions, item metadata, or user metadata datasets. For filter expression values, you can specify fixed values or add placeholder parameters, which allow you to set the filter criteria when you get recommendations from Amazon Personalize.

Dynamically passing values in filters is only supported for IN and = operations. For range queries, you need to continue to use static filters. Range queries use the following operations: NOT IN, <, >, <=, and >=.

The following are some examples of filter expressions.

Filtering by item

To remove all items in a genre chosen when you make your inference call with a filter applied, use the following filter expression:

EXCLUDE ItemId WHERE items.genre in ($GENRE)

To remove all items in the Comedy genre, use the following filter expression:

EXCLUDE ItemId WHERE items.genre in ("Comedy")

To include items with a number of downloads less than 20, use the following filter expression:

INCLUDE ItemId WHERE items.number_of_downloads < 20

This last filter expression includes a range query (items.number_of_downloads < 20). To perform this query in an Amazon Personalize recommendation filter, it needs to be predefined as a static filter.

Filtering by interaction

To remove items that have been clicked or streamed by a user, use the following filter expression:

EXCLUDE ItemId WHERE interactions.event_type in (“click”, “stream”)

To include items that a user has interacted with in any way, use the following filter expression:

INCLUDE ItemId WHERE interactions.event_type in ("*")

Filtering by item based on user properties

To exclude items where the number of downloads is less than 20 if the current user’s age is greater than 18 but less than 30, use the following filter expression:

EXCLUDE ItemId WHERE items.number_of_downloads < 20 IF CurrentUser.age > 18 AND CurrentUser.age < 30

You can also combine multiple expressions together using a pipe ( | ) to separate them. Each expression is evaluated independently and the recommendations you receive is the union of those expressions.

The following filter expression includes two expressions. The first expression includes items in the Comedy genre; the second expression excludes items with a description of classic, and the results are the union of the results of both filters.

INCLUDE Item.ID WHERE items.genre IN (“Comedy”) | EXCLUDE ItemID WHERE items.description IN ("classic”)

Filters can also use multiple filter expressions with dynamically passed values to be applied to your recommendations. The first expression includes a genre defined in the request for recommendations with the $GENRE value, and second expression excludes a description defined in the request for recommendations with the $DESC value. The result of the filter is the union of the results of both expressions that includes the $GENRE but excludes items with a description defined in the request for recommendations with the $DESC value:

INCLUDE Item.ID WHERE items.GENRE IN ($GENRE) | EXCLUDE ItemID WHERE item.DESCRIPTION IN ("$DESC”)

For more information, see Datasets and Schemas. For further details on filter definition DSL, see our documentation.

Creating a filter on the Amazon Personalize console

You can use the preceding DSL to create a customizable filter on the Amazon Personalize console. Complete the following steps:

1. On the Amazon Personalize console, on the Filters page, choose Create filter.
2. For Filter name, enter the name for your filter.
3. Select Build expression or add your expression manually to create your custom filter.
4. Enter a value of $ plus a parameter name that is similar to your property name and easy to remember (for example, $GENRE).
5. Optionally, to chain additional expressions with your filter choose, +.
6. To add additional filter expressions, choose Add expression.
7. Choose Finish.

Creating a filter takes you to a page containing detailed information about your filter. Here you can view more information about your filter, including the filter ARN and the corresponding filter expression you created (see the following screenshot). You can also delete filters on this page or create more filters from the summary page.

You can also create filters via the createFilter API in Amazon Personalize. For more information, see CreateFilter.

Range queries are not supported when dynamically passing values to recommendations filters. For example, filters excluding items with less than 20 views in the items metadata must be defined as static filters.

Applying your filter via the Amazon Personalize console

The Amazon Personalize console allows you to spot-check real-time recommendations from the Campaigns page. On this page, you can test your filters while retrieving recommendations for a specific user on demand. To do so, navigate to the Campaign tab; this should be in the same dataset group that you used to create the filter. You can then test the impact of applying the filter on the recommendations.

The following screenshot shows results when you pass the value Action as the parameter to a filter based on the items’ genre.

When applying filters to your recommendations in real time via the console, the Filter parameters section auto-populates with the filter parameter and expects a value to be passed to the filter when you choose Get recommendations button.

Applying your filter via the SDK

You can also apply filters to recommendations that are served through your SDK or APIs by supplying the filterArn as an additional and optional parameter to your GetRecommendations calls. Use filterArn as the parameter key and supply the filterArn as a string for the value. filterArn is a unique identifying key that the CreateFilter API call returns. You can also find a filter’s ARN on the filter’s detailed information page.

The following example code is a request body for GetRecommendations API that applies a filter to a recommendation:

{
    "campaignArn": "arn:aws:personalize:us-west-2:000000000000:campaign/test-campaign",
    "userId": "1",
    "itemId": "1",
    "numResults": 5,
    "filterArn": "arn:aws:personalize:us-west-2:000000000000:filter/test-filter"
    "filter-values": ‘[{
	"GENRE": ““ACTION”, “HORROR””
	}]

Summary

Recommendation filters in Amazon Personalize allow you to further customize your recommendations for each user in real time to provide an even more tailored experience that improves customer engagement and conversion. For more information about optimizing your user experience with Amazon Personalize, see What Is Amazon Personalize?

About the Author

Vaibhav Sethi is the Product Manager for Amazon Personalize. He focuses on delivering products that make it easier to build machine learning solutions. In his spare time, he enjoys hiking and reading.

 Samuel Ashman is a Technical Writer for Amazon Personalize. His goal is to make it easier for developers to use machine learning and AI in their applications. His studies in computer science allow him to understand the challenges developers face. In his free time, Samuel plays guitar and exercises.

Parth Pooniwala is a Senior Software Engineer with Amazon Personalize focused on building AI-powered recommender systems at scale. Outside of work he is a bibliophile, film enthusiast, and occasional armchair philosopher.

Read More

At AWS, our mission is to enable developers and businesses with no prior machine learning (ML) expertise to easily build sophisticated, scalable, ML-powered applications with our AI services. Today, we’re excited to announce that Amazon Lex and Amazon Polly are expanding language support. You can build ML-powered applications that fit the language preferences of your users. These easy-to-use services allow you to add intelligence to your business processes, automate workstreams, reduce costs, and improve the user experience for your customers and employees in a variety of languages.

New and improved features

Amazon Lex is a service for building conversational interfaces into any application using voice and text. Amazon Lex now supports French, Spanish, Italian and Canadian French. With the addition of these new languages, you can build and expand your conversational experiences to better understand and engage your customer base in a variety of different languages and accents. Amazon Lex can be applied to a diverse set of use cases such as virtual agents, conversational IVR systems, self-service chatbots, or application bots. For a full list of languages, please go to Amazon Lex languages.

Amazon Polly, a service that turns text into lifelike speech offers voices for all Amazon Lex languages. Our first Australian English voice, Olivia, is now generally available in Neural Text-to-Speech (NTTS). Olivia’s unique vocal personality and voice sounds expressive, natural and is easy to follow. You can now choose among three Australian English voices: Russell, Nicole and Olivia. For a full list of Amazon Polly’s voices, please go to Amazon Polly voices.

“Growing demand for conversational experiences led us to launch Amazon Lex and Amazon Polly to enable businesses to connect with their customers more effectively,” shares Julien Simon, AWS AIML evangelist.

“Amazon Lex uses automatic speech recognition and natural language understanding to help organizations understand a customer’s intent, fluidly manage conversations and create highly engaging and lifelike interactions. We are delighted to advance the language capabilities of Lex and Polly. These launches allow our customers to take advantage of AI in the area of conversational interfaces and voice AI,” Simon says.

“Amazon Lex is a core AWS service that enables Accenture to deliver next-generation, omnichannel contact center solutions, such as our Advanced Customer Engagement (ACE+) platform, to a diverse set of customers. The addition of French, Italian, and Spanish to Amazon Lex will further enhance the accessibility of our global customer engagement solutions, while also vastly enriching and personalizing the overall experience for people whose primary language is not English. Now, we can quickly build interactive digital solutions based on Amazon’s deep learning expertise to deflect more calls, reduce contact center costs and drive a better customer experience in French, Italian, and Spanish-speaking markets. Amazon Lex can now improve customer satisfaction and localized brand awareness even more effectively,” says J.C. Novoa, Global Technical Evangelist – Advanced Customer Engagement (ACE+) for Accenture.

Another example is Clevy, a French start-up and AWS customer. François Falala-Sechet, the CTO of Clevy adds, “At Clevy, we have been utilizing Amazon Lex’s best-in-class natural language processing services to help bring customers a scalable low-code approach to designing, developing, deploying and maintaining rich conversational experiences with more powerful and more integrated chatbots. With the addition of Spanish, Italian and French in Amazon Lex, Clevy can now help our developers deliver chatbot experiences to a more diverse audience in our core European markets.”

Eudata helps customers implement effective contact and management systems. Andrea Grompone, the Head of Contact Center Delivery at Eudata says, “Ora Amazon Lex parla in italiano! We are excited about the new opportunities this opens for Eudata. Amazon Lex simplifies the process of creating automated dialog-based interactions to address challenges we see in the market. The addition of Italian allows us to build a customer experience that ensures both service speed and quality in our markets.”

Using the new features

To use the new Amazon Lex languages, simply choose the language when creating a new bot via the  Amazon Lex console or AWS SDK. The following screenshot shows the console view.

To learn more, visit the Amazon Lex Development Guide.

You can use new Olivia voice in the Amazon Polly console, the AWS Command Line Interface (AWS CLI), or AWS SDK. The feature is available across all AWS Regions supporting NTTS. For the full list of available voices, see Voices in Amazon Polly, or log in to the Amazon Polly console to try it out for yourself.

Summary

Use Amazon Lex and Amazon Polly to build more self-service bots, to voice-enable applications, and to create an integrated voice and text experience for your customers and employees in a variety of languages. Try them out for yourself!

About the Author

Esther Lee is a Product Manager for AWS Language AI Services. She is passionate about the intersection of technology and education. Out of the office, Esther enjoys long walks along the beach, dinners with friends and friendly rounds of Mahjong.

Read More

AWS DeepRacer is the fastest way to get rolling with machine learning (ML). It’s a fully autonomous 1/18th scale race car driven by reinforcement learning, a 3D racing simulator, and a global racing league. Throughout 2020, tens of thousands of developers honed their ML skills and competed in the League’s virtual circuit via the AWS DeepRacer console and 14 AWS Summit online events.

The AWS DeepRacer League’s 2020 season is nearing the final lap with the Championship at AWS re:Invent 2020. From November 10 through December 15, there are three ways to join in the racing fun: learn how to develop a competitive reinforcement learning model through our sessions, enter and compete in the racing action for a chance to win prizes, and watch to cheer on other developers as they race for the cup. More than 100 racers have already qualified for the Championship Cup, but there is still time to compete. Log in today to qualify for a chance to win the Championship Cup by entering the Wildcard round, offering the top 5 racers spots in the Knockout Rounds. Starting December 1, it’s time for the Knockout Rounds to start – and for racers to compete all the way to the checkered flag and the Championship Cup. The Grand Prize winner will receive a choice of either 10,000 USD AWS promotional credits and a chance to win an expenses-paid trip to an F1 Grand Prix in the upcoming 2021 season or a Coursera online Machine Learning degree scholarship with a maximum value of up to 20,000 USD. See our AWS DeepRacer 2020 Championships Official Rules for more details.

Watch the latest episode of DRTV news to learn more about how the Championship at AWS re:Invent 2020 will work.

Congratulations to our 2020 AWS re:Invent Championship Finalists!

Thanks to the thousands of developers who competed in the 2020 AWS DeepRacer League. Below is the list of our Virtual and Summit Online Circuit winners who qualified for the Championship at AWS re:Invent 2020.

Last chance for the Championship: Enter the Wildcard

Are you yet to qualify for the Championship Cup this season? Are you brand new to the league and want to take a shot at the competition? Well, you have one last chance to qualify with the Wildcard. Through November, the open-play wildcard race will be open. This race is a traditional virtual circuit style time trial race, taking place in the AWS DeepRacer console. Participants have until 11:59pm UTC November 30 (6:59pm EST, 3:59pm PST) to submit their fastest model. The top five competitors from the wildcard race will advance to the Championship Cup knockout.

Don’t worry if you don’t advance to the next round. There are chances for developers of all skill levels to compete and win at AWS re:Invent, including the AWS DeepRacer League open racing and special live virtual races. Visit our DeepRacer page for complete race schedule and additional details.

Here’s an overview of how the Championships are organized and how many racers participate in each round from qualifying through to the Grand Prix Finale.

Round 1: Live Group Knockouts

On December 1, racers need to be ready for anything in the championships, no matter what road blocks they may come across. In Round 1, competitors have the opportunity to participate in a brand-new live racing format on the console. Racers submit their best models and control maximum speed remotely from anywhere in the world, while their autonomous models attempt to navigate the track, complete with objects to avoid. They’ll have 3 minutes to try to achieve their single best lap to top the leaderboard. Racers will be split into eight groups based on their time zone, with start order determined by the warmup round (with the fastest racers from round 1 getting to go last in their group). The top four times in each group will advance to our bracket round. Tune in to AWS DeepRacer TV  throughout AWS re:Invent to catch the championship action. 

Round 2: Bracket Elimination

The top 32 remaining competitors will be placed into a single elimination bracket, where they face off against one another in a head-to-head format in a five-lap race. Head-to-head virtual matchups will proceed until eight racers remain. Results will be released on the AWS DeepRacer League page and in the console. 

Round 3: Grand Prix Finale

The final race will take place before the closing keynote on December 15 as an eight-person virtual Grand Prix. Similar to the F1 ProAm in May, our eight finalists will submit their model on the console and the AWS DeepRacer team will run the Grand Prix, where the eight racers simultaneously face off on the track in simulation, to complete five laps. The first car to successfully complete 5 laps and cross the finish line will be crowned the 2020 AWS DeepRacer Champion and officially announced at the closing keynote.

More Options for your ML Journey

If you’re ready to get over the starting line on your ML journey, AWS DeepRacer re:Invent sessions are the best place to learn ML fast.  In 2020, we have not one, not two, but three levels of ML content for aspiring developers to go from zero to hero in no time! Register now for AWS re:Invent to learn more about session schedules when they become available.

• Get rolling with Machine Learning on AWS DeepRacer (200L). Get hands-on with AWS DeepRacer, including exciting announcements and enhancements coming to the league in 2021. Learn about the basics of machine learning and reinforcement learning (a machine learning technique ideal for autonomous driving). In this session, you can build a reinforcement learning model and submit that model to the AWS DeepRacer League for a chance to win prizes and glory.
• Shift your Machine Learning model into overdrive with AWS DeepRacer analysis tools (300L). Make your way from the middle of the pack to the top of the AWS DeepRacer podium! This session extends your machine learning skills by exploring how human analysis of reinforcement learning through logs will improve your performance through trend identification and optimization to better prepare for new racing divisions coming to the league in 2021.
• Replicate AWS DeepRacer architecture to master the track with SageMaker Notebooks (400L). Complete the final lap on your machine learning journey by demystifying the underlying architecture of AWS DeepRacer using Amazon SageMaker, AWS RoboMaker, and Amazon Kinesis Video Streams. Dive into SageMaker notebooks to learn how others have applied the skills acquired through AWS DeepRacer to real-world use cases and how you can apply your reinforcement learning models to relevant use cases.

You can take all the courses live during re:Invent or learn at your own speed on-demand. It’s up to you.  Visit the DeepRacer page at AWS re:Invent to register and find out more on when sessions will be available.

As you can see, there are many opportunities to up-level your ML skills, join in the racing action and cheer on developers as they go for the Championship Cup. Watch this page for schedule and video updates all through AWS re:Invent 2020!

About the Author

Dan McCorriston is a Senior Product Marketing Manager for AWS Machine Learning. He is passionate about technology, collaborating with developers, and creating new methods of expanding technology education. Out of the office he likes to hike, cook and spend time with his family.

Read More

We’re excited to announce that you can now measure the accuracy of your forecasting model to optimize the trade-offs between under-forecasting and over-forecasting costs, giving you flexibility in experimentation. Costs associated with under-forecasting and over-forecasting differ. Generally, over-forecasting leads to high inventory carrying costs and waste, whereas under-forecasting leads to stock-outs, unmet demand, and missed revenue opportunities. Amazon Forecast allows you to optimize these costs for your business objective by providing an average forecast as well as a distribution of forecasts that captures variability of demand from a minimum to maximum value. With this launch, Forecast now provides accuracy metrics for multiple distribution points when training a model, allowing you to quickly optimize for under-forecasting and over-forecasting without the need to manually calculate metrics.

Retailers rely on probabilistic forecasting to optimize their supply chain to balance the cost of under-forecasting, leading to stock-outs, with the cost of over-forecasting, leading to inventory carrying costs and waste. Depending on the product category, retailers may choose to generate forecasts at different distribution points. For example, grocery retailers choose to over-stock staples such as milk and eggs to meet variable demand. While these staple foods have relatively low carrying costs, stock-outs may not only lead to a lost sale, but also a complete cart abandonment. By maintaining high in-stock rates, retailers can improve customer satisfaction and customer loyalty. Conversely, retailers may choose to under-stock substitutable products with high carrying costs when the cost of markdowns and inventory disposal outweighs the occasional lost sale. The ability to forecast at different distribution points allows retailers to optimize these competing priorities when demand is variable.

Anaplan Inc., a cloud-native platform for orchestrating business performance, has integrated Forecast into its PlanIQ solution to bring predictive forecasting and agile scenario planning to enterprise customers. Evgy Kontorovich, Head of Product Management, says, “Each customer we work with has a unique operational and supply chain model that drives different business priorities. Some customers aim to reduce excess inventory by making more prudent forecasts, while others strive to improve in-stock availability rates to consistently meet customer demand. With forecast quantiles, planners can evaluate the accuracy of these models, evaluate model quality, and fine-tune them based on their business goals. The ability to assess the accuracy of forecast models at multiple custom quantile levels further empowers our customers to make highly informed decisions optimized for their business.”

Although Forecast provided the ability to forecast at the entire distribution of variability to manage the trade-offs of under-stocking and over-stocking, the accuracy metrics were only provided for the minimum, median, and maximum predicted demand, providing an 80% confidence band centered on the median. To evaluate the accuracy metric at a particular distribution point of interest, you first had to create forecasts at that point, and then manually calculate the accuracy metrics on your own.

With today’s launch, you can assess the strengths of your forecasting models at any distribution point within Forecast without needing to generate forecasts and manually calculate metrics. This capability enables you to experiment faster, and more cost-effectively arrive at a distribution point for your business needs.

To use this new capability, you select forecast types (or distribution points) of interest when creating the predictor. Forecast splits the input data into training and testing datasets, trains and tests the model created, and generates accuracy metrics for these distribution points. You can continue to experiment to further optimize forecast types without needing to create forecasts at each step.

Understanding Forecast accuracy metrics

Forecast provides different model accuracy metrics for you to assess the strength of your forecasting models. We provide the weighted quantile loss (wQL) metric for each selected distribution point, and weighted average percentage error (WAPE) and root mean square error (RMSE), calculated at the mean forecast. For each metric, a lower value indicates a smaller error and therefore a more accurate model. All these accuracy metrics are non-negative.

Let’s take an example of a retail dataset as shown in the following table to understand these different accuracy metrics. In this dataset, three items are being forecasted for 2 days.

The following table summarizes the calculated accuracy metrics for the retail dataset use case.

In the following sections, we explain how each metric was calculated and recommendations for the best use case for each metric.

Weighted quantile loss (wQL)

The wQL metric measures the accuracy of a model at specified distribution points called quantiles. This metric helps capture the bias inherent in each quantile. For a grocery retailer who prefers to over-stock staples such as milk, choosing a higher quantile such as 0.75 (P75) better captures spikes in demand and therefore is more informative than forecasting at the median quantile of 0.5 (P50).

In this example, more emphasis is given to over-forecasting than under-forecasting and suggests that a higher amount of stock is needed to satisfy customer demand with 75% probability of success. In other words, the actual demand is less than or equal to forecasted demand 75% of the time, allowing the grocer to maintain target in-stock rates with less safety stock.

We recommend using the wQL measure at different quantiles when the costs of under-forecasting and over-forecasting differ. If the difference in costs is negligible, you may consider forecasting at the median quantile of 0.5 (P50) or use the WAPE metric, which is evaluated using the mean forecast. The following figure illustrates the probability of meeting demand dependent on quantile.

For the retail dataset example, the P75 forecast indicates that we want to prioritize over-forecasting and penalize under-forecasting. To calculate the wQL[0.75], sum the positive values in the P75 Error column and multiply them by a smaller weight of 1-0.75 = 0.25, and sum the negative values in the P75 Error column and multiply them by a larger weight of 0.75 to penalize under-forecasting. The wQL[0.75] is as follows:

Weighted absolute percentage error (WAPE)

The WAPE metric is the sum of the absolute error normalized by the total demand. The WAPE doesn’t penalize for under-forecasting or over-forecasting, and uses the average (mean) expected value of forecasts to calculate the differences. We recommend using the WAPE metric when the difference in costs of under-forecasting or over-forecasting is negligible, or if you want to evaluate the model accuracy at the mean forecast. For example, to predict the amount of cash needed at a given time in an ATM, a bank may choose to meet average demand because there is less concern of losing a customer or having excess cash in the ATM machine. In this example, you may choose to forecast at the mean and choose the WAPE as your metric to evaluate the model accuracy.

The normalization or weighting helps compare models from different datasets. For example, if the absolute error summed over the entire dataset is 5, it’s hard to interpret the quality of this metric without knowing the scale of total demand. A high total demand of 1000 results in a low WAPE (0.005) and a small total demand of 10 results in a high WAPE (0.5). The weighting in the WAPE and wQL allows these metrics to be compared across datasets with different scales.

The normalization or weighting also helps evaluate datasets that contain a mix of items with large and small demand. The WAPE metric emphasizes accuracy for items with larger demand. You can use WAPE for datasets where forecasting for a small number of SKUs drives majority of sale. For example, a retailer may prefer to use the WAPE metric to put less emphasis on forecasting errors related to special edition items, and prioritize forecasting errors for standard items with most sales.

In our retail dataset use case, the WAPE is equal to the sum of the absolute error column divided by the sum of the actual demand column (total demand).

Because the sum of the total demand is mostly driven by Item₁, the WAPE gives more importance to the accuracy of the popular Item₁.

Many retail customers work with sparse datasets, where most of their SKUs are sold infrequently. For most of the historical data points, the demand is 0. For these datasets, it’s important to account for the scale of total demand, making wQL and WAPE a better metric over RMSE to evaluate sparse datasets. The RMSE metric doesn’t take into account the scale of total demand, and returns a lower RMSE value by considering the total number of historical data points and the total number of SKUs, giving you a false sense of security that you have an accurate model.

Root mean square error (RMSE)

The RMSE metric is the square root of the sum of the squared errors (differences of the mean forecasted and actual values) divided by the product of the number of items and number of time points. The RMSE also doesn’t penalize for under-forecasting or over-forecasting, and can be used when the trade-offs between under-forecasting or over-forecasting are negligible or if you prefer to forecast at mean. Because the RMSE is proportional to the square of the errors, it’s sensitive to large deviations between the actual demand and forecasted values.

However, you should use RMSE with caution, because a few large deviations in forecasting errors can severely punish an otherwise accurate model. For example, if one item in a large dataset is severely under-forecasted or over-forecasted, the error in that item skews the entire RMSE metric drastically, and may make you reject an otherwise accurate model prematurely. For use cases where a few large deviations are not of importance, consider using wQL or WAPE.

In our retail dataset example, the RMSE is equal to square root of the sum of the squared error column divided by the total number of points (3 items * 2 days = 6).

The RMSE gives more importance to the large deviation of forecasting error of Item₃, resulting in a higher RMSE value.

We recommend using the RMSE metric when a few large incorrect predictions from a model on some items can be very costly to the business. For example, a manufacturer who is predicting machine failure may prefer to use the RMSE metric. Because operating machinery is critical, any large deviation from the actual and forecasted demand, even infrequent, needs to be over-emphasized when assessing model accuracy.

The following table summarizes our discussion on selecting an accuracy metric dependent on your use case.

Selecting custom distribution points for assessing model accuracy in Forecast

You can measure model accuracy in Forecast through the CreatePredictor API and GetAccuracyMetrics API or using the Forecast console. In this section, we walk through the steps to use the console.

1. On the Forecast console, create a dataset group.

2. Upload your dataset.

3. In the navigation pane, choose Predictors.
4. Choose Train predictor.

5. Under Predictor Settings for Forecast types, you can enter up to five distribution points of your choosing. Mean is also accepted.

If unspecified, default quantiles of 0.1, 0.5, and 0.9 are used to train a predictor and calculate accuracy metrics.

6. For Algorithm selection, select Automatic (AutoML).

AutoML optimizes a model for the specified quantiles.

7. Optionally, for Number of backtest windows, you can select up to five to assess your model strength.

8. After your predictor is trained, choose your predictor on the Predictors page to view details of the accuracy metrics.

On the predictor’s details page, you can view the overall model accuracy metrics and the metrics from each backtest windows that you specified. Overall model accuracy metrics are calculating by averaging the individual backtest window metrics.

9. Now that your model is trained, choose Forecasts in the navigation pane.
10. Choose Create a forecast.
11. Select your trained predictor to create a forecast.

Forecasts are generated for the distribution points used during training a predictor. You also have the option to specify different distribution points for generating forecasts.

Tips and best practices

In this section, we share a few tips and best practices when using Forecast:

• Before experimenting with Forecast, define your business problem related to costs of under-forecasting or over-forecasting. Evaluate the trade-offs and prioritize if you would rather over-forecast than under.
• Experiment with multiple distribution points to optimize your forecast model to balance the costs associated with under-forecasting and over-forecasting.
• If you’re comparing different models, use the wQL metric at the same quantile for comparison. The lower the value, the more accurate the forecasting model.
• Forecast allows you to select up to five backtest windows. Forecast uses backtesting to tune predictors and produce accuracy metrics. To perform backtesting, Forecast automatically splits your time-series datasets into two sets: training and testing. The training set is used to train your model, and the testing set to evaluate the model’s predictive accuracy. We recommend choosing more than one backtest window to minimize selection bias that may make one window more or less accurate by chance. Assessing the overall model accuracy from multiple backtest windows provides a better measure of the strength of the model.

Conclusion

You can now measure the accuracy of your forecasting model at multiple distribution points of your choosing with Forecast. This gives you more flexibility to align Forecast with your business, because the impact of over-forecasting frequently differs from under-forecasting. You can use this capability in all Regions where Forecast is publicly available. For more information about Region availability, see Region Table. For more information about APIs specifying custom distribution points when creating a predictor and accessing accuracy metrics, see the documentation for the CreatePredictor API and GetAccuracyMetrics API. For more details on evaluating a predictor’s accuracy for multiple distribution points, see Evaluating Predictor Accuracy.

About the Authors

Namita Das is a Sr. Product Manager for Amazon Forecast. Her current focus is to democratize machine learning by building no-code/low-code ML services. On the side, she frequently advises startups and is raising a puppy named Imli.

Danielle Robinson is an Applied Scientist on the Amazon Forecast team. Her research is in time series forecasting and in particular how we can apply new neural network-based algorithms within Amazon Forecast. Her thesis research was focused on developing new, robust, and physically accurate numerical models for computational fluid dynamics. Her hobbies include cooking, swimming, and hiking.

Read More