Monthly Archives: February 2024

Enterprises are seeking to quickly unlock the potential of generative AI by providing access to foundation models (FMs) to different lines of business (LOBs). IT teams are responsible for helping the LOB innovate with speed and agility while providing centralized governance and observability. For example, they may need to track the usage of FMs across teams, chargeback costs and provide visibility to the relevant cost center in the LOB. Additionally, they may need to regulate access to different models per team. For example, if only specific FMs may be approved for use.

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon via a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI. Because Amazon Bedrock is serverless, you don’t have to manage any infrastructure, and you can securely integrate and deploy generative AI capabilities into your applications using the AWS services you are already familiar with.

A software as a service (SaaS) layer for foundation models can provide a simple and consistent interface for end-users, while maintaining centralized governance of access and consumption. API gateways can provide loose coupling between model consumers and the model endpoint service, and flexibility to adapt to changing model, architectures, and invocation methods.

In this post, we show you how to build an internal SaaS layer to access foundation models with Amazon Bedrock in a multi-tenant (team) architecture. We specifically focus on usage and cost tracking per tenant and also controls such as usage throttling per tenant. We describe how the solution and Amazon Bedrock consumption plans map to the general SaaS journey framework. The code for the solution and an AWS Cloud Development Kit (AWS CDK) template is available in the GitHub repository.

Challenges

An AI platform administrator needs to provide standardized and easy access to FMs to multiple development teams.

The following are some of the challenges to provide governed access to foundation models:

• Cost and usage tracking – Track and audit individual tenant costs and usage of foundation models, and provide chargeback costs to specific cost centers
• Budget and usage controls – Manage API quota, budget, and usage limits for the permitted use of foundation models over a defined frequency per tenant
• Access control and model governance – Define access controls for specific allow listed models per tenant
• Multi-tenant standardized API – Provide consistent access to foundation models with OpenAPI standards
• Centralized management of API – Provide a single layer to manage API keys for accessing models
• Model versions and updates – Handle new and updated model version rollouts

Solution overview

In this solution, we refer to a multi-tenant approach. A tenant here can range from an individual user, a specific project, team, or even an entire department. As we discuss the approach, we use the term team, because it’s the most common. We use API keys to restrict and monitor API access for teams. Each team is assigned an API key for access to the FMs. There can be different user authentication and authorization mechanisms deployed in an organization. For simplicity, we do not include these in this solution. You may also integrate existing identity providers with this solution.

The following diagram summarizes the solution architecture and key components. Teams (tenants) assigned to separate cost centers consume Amazon Bedrock FMs via an API service. To track consumption and cost per team, the solution logs data for each individual invocation, including the model invoked, number of tokens for text generation models, and image dimensions for multi-modal models. In addition, it aggregates the invocations per model and costs by each team.

You can deploy the solution in your own account using the AWS CDK. AWS CDK is an open source software development framework to model and provision your cloud application resources using familiar programming languages. The AWS CDK code is available in the GitHub repository.

In the following sections, we discuss the key components of the solution in more detail.

Capturing foundation model usage per team

The workflow to capture FM usage per team consists of the following steps (as numbered in the preceding diagram):

1. A team’s application sends a POST request to Amazon API Gateway with the model to be invoked in the model_id query parameter and the user prompt in the request body.
2. API Gateway routes the request to an AWS Lambda function (bedrock_invoke_model) that’s responsible for logging team usage information in Amazon CloudWatch and invoking the Amazon Bedrock model.
3. Amazon Bedrock provides a VPC endpoint powered by AWS PrivateLink. In this solution, the Lambda function sends the request to Amazon Bedrock using PrivateLink to establish a private connection between the VPC in your account and the Amazon Bedrock service account. To learn more about PrivateLink, see Use AWS PrivateLink to set up private access to Amazon Bedrock.
4. After the Amazon Bedrock invocation, Amazon CloudTrail generates a CloudTrail event.
5. If the Amazon Bedrock call is successful, the Lambda function logs the following information depending on the type of invoked model and returns the generated response to the application:
   • team_id – The unique identifier for the team issuing the request.
   • requestId – The unique identifier of the request.
   • model_id – The ID of the model to be invoked.
   • inputTokens – The number of tokens sent to the model as part of the prompt (for text generation and embeddings models).
   • outputTokens – The maximum number of tokens to be generated by the model (for text generation models).
   • height – The height of the requested image (for multi-modal models and multi-modal embeddings models).
   • width – The width of the requested image (for multi-modal models only).
   • steps – The steps requested (for Stability AI models).

Tracking costs per team

A different flow aggregates the usage information, then calculates and saves the on-demand costs per team on a daily basis. By having a separate flow, we ensure that cost tracking doesn’t impact the latency and throughput of the model invocation flow. The workflow steps are as follows:

1. An Amazon EventBridge rule triggers a Lambda function (bedrock_cost_tracking) daily.
2. The Lambda function gets the usage information from CloudWatch for the previous day, calculates the associated costs, and stores the data aggregated by team_id and model_id in Amazon Simple Storage Service (Amazon S3) in CSV format.

To query and visualize the data stored in Amazon S3, you have different options, including S3 Select, and Amazon Athena and Amazon QuickSight.

Controlling usage per team

A usage plan specifies who can access one or more deployed APIs and optionally sets the target request rate to start throttling requests. The plan uses API keys to identify API clients who can access the associated API for each key. You can use API Gateway usage plans to throttle requests that exceed predefined thresholds. You can also use API keys and quota limits, which enable you to set the maximum number of requests per API key each team is permitted to issue within a specified time interval. This is in addition to Amazon Bedrock service quotas that are assigned only at the account level.

Prerequisites

Before you deploy the solution, make sure you have the following:

• An AWS account. If you are new to AWS, see Creating an AWS account.
• The AWS Command Line Interface (AWS CLI) installed in your development environment and an AWS CLI profile configured with AWS Identity and Access Management (IAM) permissions to deploy the resources used in this solution.
• Python 3 installed in your development environment.
• AWS CDK installed in your development environment.

Deploy the AWS CDK stack

Follow the instructions in the README file of the GitHub repository to configure and deploy the AWS CDK stack.

The stack deploys the following resources:

• Private networking environment (VPC, private subnets, security group)
• IAM role for controlling model access
• Lambda layers for the necessary Python modules
• Lambda function invoke_model
• Lambda function list_foundation_models
• Lambda function cost_tracking
• Rest API (API Gateway)
• API Gateway usage plan
• API key associated to the usage plan

Onboard a new team

For providing access to new teams, you can either share the same API key across different teams and track the model consumptions by providing a different team_id for the API invocation, or create dedicated API keys used for accessing Amazon Bedrock resources by following the instructions provided in the README.

The stack deploys the following resources:

• API Gateway usage plan associated to the previously created REST API
• API key associated to the usage plan for the new team, with reserved throttling and burst configurations for the API

For more information about API Gateway throttling and burst configurations, refer to Throttle API requests for better throughput.

After you deploy the stack, you can see that the new API key for team-2 is created as well.

Configure model access control

The platform administrator can allow access to specific foundation models by editing the IAM policy associated to the Lambda function invoke_model. The

IAM permissions are defined in the file setup/stack_constructs/iam.py. See the following code:

self.bedrock_policy = iam.Policy(
            scope=self,
            id=f"{self.id}_policy_bedrock",
            policy_name="BedrockPolicy",
            statements=[
                iam.PolicyStatement(
                    effect=iam.Effect.ALLOW,
                    actions=[
                        "sts:AssumeRole",
                    ],
                    resources=["*"],
                ),
                iam.PolicyStatement(
                    effect=iam.Effect.ALLOW,
                    actions=[
                        "bedrock:InvokeModel",
				“bedrock:ListFoundationModels",

                    ],
                    resources=[
  	"arn:aws:bedrock:*::foundation-model/anthropic.claude-v2.1",
	"arn:aws:bedrock:*::foundation-model/amazon.titan-text-express-v1",
	"arn:aws:bedrock:*::foundation-model/amazon.titan-embed-text-v1"
],
                )
            ],
        )

…

self.bedrock_policy.attach_to_role(self.lambda_role)

Invoke the service

After you have deployed the solution, you can invoke the service directly from your code. The following

is an example in Python for consuming the invoke_model API for text generation through a POST request:

api_key=”abcd1234”

model_id = "amazon.titan-text-express-v1" #the model id for the Amazon Titan Express model
 
model_kwargs = { # inference configuration
    "maxTokenCount": 4096,
    "temperature": 0.2
}

prompt = "What is Amazon Bedrock?"

response = requests.post(
    f"{api_url}/invoke_model?model_id={model_id}",
    json={"inputs": prompt, "parameters": model_kwargs},
    headers={
        "x-api-key": api_key, #key for querying the API
        "team_id": team_id #unique tenant identifier 
    }
)

text = response.json()[0]["generated_text"]

print(text)

Output: Amazon Bedrock is an internal technology platform developed by Amazon to run and operate many of their services and products. Some key things about Bedrock …

The following is another example in Python for consuming the invoke_model API for embeddings generation through a POST request:

model_id = "amazon.titan-embed-text-v1" #the model id for the Amazon Titan Embeddings Text model

prompt = "What is Amazon Bedrock?"

response = requests.post(
    f"{api_url}/invoke_model?model_id={model_id}",
    json={"inputs": prompt, "parameters": model_kwargs},
    headers={
        "x-api-key": api_key, #key for querying the API
        "team_id": team_id #unique tenant identifier,
	"embeddings": "true" #boolean value for the embeddings model 
    }
)

text = response.json()[0]["embedding"]

Output: 0.91796875, 0.45117188, 0.52734375, -0.18652344, 0.06982422, 0.65234375, -0.13085938, 0.056884766, 0.092285156, 0.06982422, 1.03125, 0.8515625, 0.16308594, 0.079589844, -0.033935547, 0.796875, -0.15429688, -0.29882812, -0.25585938, 0.45703125, 0.044921875, 0.34570312 …

Access denied to foundation models

The following is an example in Python for consuming the invoke_model API for text generation through a POST request with an access denied response:

model_id = " anthropic.claude-v1" #the model id for Anthropic Claude V1 model
 
model_kwargs = { # inference configuration
    "maxTokenCount": 4096,
    "temperature": 0.2
}

prompt = "What is Amazon Bedrock?"

response = requests.post(
    f"{api_url}/invoke_model?model_id={model_id}",
    json={"inputs": prompt, "parameters": model_kwargs},
    headers={
        "x-api-key": api_key, #key for querying the API
        "team_id": team_id #unique tenant identifier 
    }
)

print(response)
print(response.text)

<Response [500]> “Traceback (most recent call last):n File ”/var/task/index.py”, line 213, in lambda_handlern response = _invoke_text(bedrock_client, model_id, body, model_kwargs)n File ”/var/task/index.py”, line 146, in _invoke_textn raise en File ”/var/task/index.py”, line 131, in _invoke_textn response = bedrock_client.invoke_model(n File ”/opt/python/botocore/client.py”, line 535, in _api_calln return self._make_api_call(operation_name, kwargs)n File ”/opt/python/botocore/client.py”, line 980, in _make_api_calln raise error_class(parsed_response, operation_name)nbotocore.errorfactory.AccessDeniedException: An error occurred (AccessDeniedException) when calling the InvokeModel operation: Your account is not authorized to invoke this API operation.n”

Cost estimation example

When invoking Amazon Bedrock models with on-demand pricing, the total cost is calculated as the sum of the input and output costs. Input costs are based on the number of input tokens sent to the model, and output costs are based on the tokens generated. The prices are per 1,000 input tokens and per 1,000 output tokens. For more details and specific model prices, refer to Amazon Bedrock Pricing.

Let’s look at an example where two teams, team1 and team2, access Amazon Bedrock through the solution in this post. The usage and cost data saved in Amazon S3 in a single day is shown in the following table.

The columns input_tokens and output_tokens store the total input and output tokens across model invocations per model and per team, respectively, for a given day.

The columns input_cost and output_cost store the respective costs per model and per team. These are calculated using the following formulas:

input_cost = input_token_count * model_pricing["input_cost"] / 1000
output_cost = output_token_count * model_pricing["output_cost"] / 1000

End-to-end view of a functional multi-tenant serverless SaaS environment

Let’s understand what an end-to-end functional multi-tenant serverless SaaS environment might look like. The following is a reference architecture diagram.

This architecture diagram is a zoomed-out version of the previous architecture diagram explained earlier in the post, where the previous architecture diagram explains the details of one of the microservices mentioned (foundational model service). This diagram explains that, apart from foundational model service, you need to have other components as well in your multi-tenant SaaS platform to implement a functional and scalable platform.

Let’s go through the details of the architecture.

Tenant applications

The tenant applications are the front end applications that interact with the environment. Here, we show multiple tenants accessing from different local or AWS environments. The front end applications can be extended to include a registration page for new tenants to register themselves and an admin console for administrators of the SaaS service layer. If the tenant applications require a custom logic to be implemented that needs interaction with the SaaS environment, they can implement the specifications of the application adaptor microservice. Example scenarios could be adding custom authorization logic while respecting the authorization specifications of the SaaS environment.

Shared services

The following are shared services:

• Tenant and user management services –These services are responsible for registering and managing the tenants. They provide the cross-cutting functionality that’s separate from application services and shared across all of the tenants.
• Foundation model service –The solution architecture diagram explained at the beginning of this post represents this microservice, where the interaction from API Gateway to Lambda functions is happening within the scope of this microservice. All tenants use this microservice to invoke the foundations models from Anthropic, AI21, Cohere, Stability, Meta, and Amazon, as well as fine-tuned models. It also captures the information needed for usage tracking in CloudWatch logs.
• Cost tracking service –This service tracks the cost and usage for each tenant. This microservice runs on a schedule to query the CloudWatch logs and output the aggregated usage tracking and inferred cost to the data storage. The cost tracking service can be extended to build further reports and visualization.

Application adaptor service

This service presents a set of specifications and APIs that a tenant may implement in order to integrate their custom logic to the SaaS environment. Based on how much custom integration is needed, this component can be optional for tenants.

Multi-tenant data store

The shared services store their data in a data store that can be a single shared Amazon DynamoDB table with a tenant partitioning key that associates DynamoDB items with individual tenants. The cost tracking shared service outputs the aggregated usage and cost tracking data to Amazon S3. Based on the use case, there can be an application-specific data store as well.

A multi-tenant SaaS environment can have a lot more components. For more information, refer to Building a Multi-Tenant SaaS Solution Using AWS Serverless Services.

Support for multiple deployment models

SaaS frameworks typically outline two deployment models: pool and silo. For the pool model, all tenants access FMs from a shared environment with common storage and compute infrastructure. In the silo model, each tenant has its own set of dedicated resources. You can read about isolation models in the SaaS Tenant Isolation Strategies whitepaper.

The proposed solution can be adopted for both SaaS deployment models. In the pool approach, a centralized AWS environment hosts the API, storage, and compute resources. In silo mode, each team accesses APIs, storage, and compute resources in a dedicated AWS environment.

The solution also fits with the available consumption plans provided by Amazon Bedrock. AWS provides a choice of two consumptions plan for inference:

• On-Demand – This mode allows you to use foundation models on a pay-as-you-go basis without having to make any time-based term commitments
• Provisioned Throughput – This mode allows you to provision sufficient throughput to meet your application’s performance requirements in exchange for a time-based term commitment

For more information about these options, refer to Amazon Bedrock Pricing.

The serverless SaaS reference solution described in this post can apply the Amazon Bedrock consumption plans to provide basic and premium tiering options to end-users. Basic could include On-Demand or Provisioned Throughput consumption of Amazon Bedrock and could include specific usage and budget limits. Tenant limits could be enabled by throttling requests based on requests, token sizes, or budget allocation. Premium tier tenants could have their own dedicated resources with provisioned throughput consumption of Amazon Bedrock. These tenants would typically be associated with production workloads that require high throughput and low latency access to Amazon Bedrock FMs.

Conclusion

In this post, we discussed how to build an internal SaaS platform to access foundation models with Amazon Bedrock in a multi-tenant setup with a focus on tracking costs and usage, and throttling limits for each tenant. Additional topics to explore include integrating existing authentication and authorization solutions in the organization, enhancing the API layer to include web sockets for bi-directional client server interactions, adding content filtering and other governance guardrails, designing multiple deployment tiers, integrating other microservices in the SaaS architecture, and many more.

The entire code for this solution is available in the GitHub repository.

For more information about SaaS-based frameworks, refer to SaaS Journey Framework: Building a New SaaS Solution on AWS.

About the Authors

Hasan Poonawala is a Senior AI/ML Specialist Solutions Architect at AWS, working with Healthcare and Life Sciences customers. Hasan helps design, deploy and scale Generative AI and Machine learning applications on AWS. He has over 15 years of combined work experience in machine learning, software development and data science on the cloud. In his spare time, Hasan loves to explore nature and spend time with friends and family.

Anastasia Tzeveleka is a Senior AI/ML Specialist Solutions Architect at AWS. As part of her work, she helps customers across EMEA build foundation models and create scalable generative AI and machine learning solutions using AWS services.

Bruno Pistone is a Generative AI and ML Specialist Solutions Architect for AWS based in Milan. He works with large customers helping them to deeply understand their technical needs and design AI and Machine Learning solutions that make the best use of the AWS Cloud and the Amazon Machine Learning stack. His expertise include: Machine Learning end to end, Machine Learning Industrialization, and Generative AI. He enjoys spending time with his friends and exploring new places, as well as travelling to new destinations.

Vikesh Pandey is a Generative AI/ML Solutions architect, specialising in financial services where he helps financial customers build and scale Generative AI/ML platforms and solution which scales to hundreds to even thousands of users. In his spare time, Vikesh likes to write on various blog forums and build legos with his kid.

Read More

Generative AI agents are a versatile and powerful tool for large enterprises. They can enhance operational efficiency, customer service, and decision-making while reducing costs and enabling innovation. These agents excel at automating a wide range of routine and repetitive tasks, such as data entry, customer support inquiries, and content generation. Moreover, they can orchestrate complex, multi-step workflows by breaking down tasks into smaller, manageable steps, coordinating various actions, and ensuring the efficient execution of processes within an organization. This significantly reduces the burden on human resources and allows employees to focus on more strategic and creative tasks.

As AI technology continues to evolve, the capabilities of generative AI agents are expected to expand, offering even more opportunities for customers to gain a competitive edge. At the forefront of this evolution sits Amazon Bedrock, a fully managed service that makes high-performing foundation models (FMs) from Amazon and other leading AI companies available through an API. With Amazon Bedrock, you can build and scale generative AI applications with security, privacy, and responsible AI. You can now use Agents for Amazon Bedrock and Knowledge Bases for Amazon Bedrock to configure specialized agents that seamlessly run actions based on natural language input and your organization’s data. These managed agents play conductor, orchestrating interactions between FMs, API integrations, user conversations, and knowledge sources loaded with your data.

This post highlights how you can use Agents and Knowledge Bases for Amazon Bedrock to build on existing enterprise resources to automate the tasks associated with the insurance claim lifecycle, efficiently scale and improve customer service, and enhance decision support through improved knowledge management. Your Amazon Bedrock-powered insurance agent can assist human agents by creating new claims, sending pending document reminders for open claims, gathering claims evidence, and searching for information across existing claims and customer knowledge repositories.

Solution overview

The objective of this solution is to act as a foundation for customers, empowering you to create your own specialized agents for various needs such as virtual assistants and automation tasks. The code and resources required for deployment are available in the amazon-bedrock-examples repository.

The following demo recording highlights Agents and Knowledge Bases for Amazon Bedrock functionality and technical implementation details.

Agents and Knowledge Bases for Amazon Bedrock work together to provide the following capabilities:

• Task orchestration – Agents use FMs to understand natural language inquiries and dissect multi-step tasks into smaller, executable steps.
• Interactive data collection – Agents engage in natural conversations to gather supplementary information from users.
• Task fulfillment – Agents complete customer requests through series of reasoning steps and corresponding actions based on ReAct prompting.
• System integration – Agents make API calls to integrated company systems to run specific actions.
• Data querying – Knowledge bases enhance accuracy and performance through fully managed Retrieval Augmented Generation (RAG) using customer-specific data sources.
• Source attribution – Agents conduct source attribution, identifying and tracing the origin of information or actions through chain-of-thought reasoning.

The following diagram illustrates the solution architecture.

The workflow consists of the following steps:

1. Users provide natural language inputs to the agent. The following are some example prompts:
   1. Create a new claim.
   2. Send a pending documents reminder to the policy holder of claim 2s34w-8x.
   3. Gather evidence for claim 5t16u-7v.
   4. What is the total claim amount for claim 3b45c-9d?
   5. What is the repair estimate total for that same claim?
   6. What factors determine my car insurance premium?
   7. How can I lower my car insurance rates?
   8. Which claims have open status?
   9. Send reminders to all policy holders with open claims.
2. During preprocessing, the agent validates, contextualizes, and categorizes user input. The user input (or task) is interpreted by the agent using chat history and the instructions and underlying FM that were specified during agent creation. The agent’s instructions are descriptive guidelines outlining the agent’s intended actions. Also, you can optionally configure advanced prompts, which allow you to boost your agent’s precision by employing more detailed configurations and offering manually selected examples for few-shot prompting. This method allows you to enhance the model’s performance by providing labeled examples associated with a particular task.
3. Action groups are a set of APIs and corresponding business logic, whose OpenAPI schema is defined as JSON files stored in Amazon Simple Storage Service (Amazon S3). The schema allows the agent to reason around the function of each API. Each action group can specify one or more API paths, whose business logic is run through the AWS Lambda function associated with the action group.
4. Knowledge Bases for Amazon Bedrock provides fully managed RAG to supply the agent with access to your data. You first configure the knowledge base by specifying a description that instructs the agent when to use your knowledge base. Then you point the knowledge base to your Amazon S3 data source. Finally, you specify an embedding model and choose to use your existing vector store or allow Amazon Bedrock to create the vector store on your behalf. After it’s configured, each data source sync creates vector embeddings of your data that the agent can use to return information to the user or augment subsequent FM prompts.
5. During orchestration, the agent develops a rationale with the logical steps of which action group API invocations and knowledge base queries are needed to generate an observation that can be used to augment the base prompt for the underlying FM. This ReAct style prompting serves as the input for activating the FM, which then anticipates the most optimal sequence of actions to complete the user’s task.
6. During postprocessing, after all orchestration iterations are complete, the agent curates a final response. Postprocessing is disabled by default.

In the following sections, we discuss the key steps to deploy the solution, including pre-implementation steps and testing and validation.

Create solution resources with AWS CloudFormation

Prior to creating your agent and knowledge base, it is essential to establish a simulated environment that closely mirrors the existing resources used by customers. Agents and Knowledge Bases for Amazon Bedrock are designed to build upon these resources, using Lambda-delivered business logic and customer data repositories stored in Amazon S3. This foundational alignment provides a seamless integration of your agent and knowledge base solutions with your established infrastructure.

To emulate the existing customer resources utilized by the agent, this solution uses the create-customer-resources.sh shell script to automate provisioning of the parameterized AWS CloudFormation template, bedrock-customer-resources.yml, to deploy the following resources:

• An Amazon DynamoDB table populated with synthetic claims data.
• Three Lambda functions that represent the customer business logic for creating claims, sending pending document reminders for open status claims, and gathering evidence on new and existing claims.
• An S3 bucket containing API documentation in OpenAPI schema format for the preceding Lambda functions and the repair estimates, claim amounts, company FAQs, and required claim document descriptions to be used as our knowledge base data source assets.
• An Amazon Simple Notification Service (Amazon SNS) topic to which policy holders’ emails are subscribed for email alerting of claim status and pending actions.
• AWS Identity and Access Management (IAM) permissions for the preceding resources.

AWS CloudFormation prepopulates the stack parameters with the default values provided in the template. To provide alternative input values, you can specify parameters as environment variables that are referenced in the ParameterKey=<ParameterKey>,ParameterValue=<Value> pairs in the following shell script’s aws cloudformation create-stack command.

Complete the following steps to provision your resources:

1. Create a local copy of the amazon-bedrock-samples repository using git clone:

   git clone https://github.com/aws-samples/amazon-bedrock-samples.git

2. Before you run the shell script, navigate to the directory where you cloned the amazon-bedrock-samples repository and modify the shell script permissions to executable:

   # If not already cloned, clone the remote repository (https://github.com/aws-samples/amazon-bedrock-samples) and change working directory to insurance agent shell folder
   cd amazon-bedrock-samples/agents/insurance-claim-lifecycle-automation/shell/
   chmod u+x create-customer-resources

3. Set your CloudFormation stack name, SNS email, and evidence upload URL environment variables. The SNS email will be used for policy holder notifications, and the evidence upload URL will be shared with policy holders to upload their claims evidence. The insurance claims processing sample provides an example front-end for the evidence upload URL.

   export STACK_NAME=<YOUR-STACK-NAME> # Stack name must be lower case for S3 bucket naming convention
   export SNS_EMAIL=<YOUR-POLICY-HOLDER-EMAIL> # Email used for SNS notifications
   export EVIDENCE_UPLOAD_URL=<YOUR-EVIDENCE-UPLOAD-URL> # URL provided by the agent to the policy holder for evidence upload

4. Run the create-customer-resources.sh shell script to deploy the emulated customer resources defined in the bedrock-insurance-agent.yml CloudFormation template. These are the resources on which the agent and knowledge base will be built.

   source ./create-customer-resources.sh

The preceding source ./create-customer-resources.sh shell command runs the following AWS Command Line Interface (AWS CLI) commands to deploy the emulated customer resources stack:

export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export ARTIFACT_BUCKET_NAME=$STACK_NAME-customer-resources
export DATA_LOADER_KEY="agent/lambda/data-loader/loader_deployment_package.zip"
export CREATE_CLAIM_KEY="agent/lambda/action-groups/create_claim.zip"
export GATHER_EVIDENCE_KEY="agent/lambda/action-groups/gather_evidence.zip"
export SEND_REMINDER_KEY="agent/lambda/action-groups/send_reminder.zip"

aws s3 mb s3://${ARTIFACT_BUCKET_NAME} --region us-east-1
aws s3 cp ../agent/ s3://${ARTIFACT_BUCKET_NAME}/agent/ --recursive --exclude ".DS_Store"

export BEDROCK_AGENTS_LAYER_ARN=$(aws lambda publish-layer-version 
--layer-name bedrock-agents 
--description "Agents for Bedrock Layer" 
--license-info "MIT" 
--content S3Bucket=${ARTIFACT_BUCKET_NAME},S3Key=agent/lambda/lambda-layer/bedrock-agents-layer.zip 
--compatible-runtimes python3.11 
--query LayerVersionArn --output text)

aws cloudformation create-stack 
--stack-name ${STACK_NAME} 
--template-body file://../cfn/bedrock-customer-resources.yml 
--parameters 
ParameterKey=ArtifactBucket,ParameterValue=${ARTIFACT_BUCKET_NAME} 
ParameterKey=DataLoaderKey,ParameterValue=${DATA_LOADER_KEY} 
ParameterKey=CreateClaimKey,ParameterValue=${CREATE_CLAIM_KEY} 
ParameterKey=GatherEvidenceKey,ParameterValue=${GATHER_EVIDENCE_KEY} 
ParameterKey=SendReminderKey,ParameterValue=${SEND_REMINDER_KEY} 
ParameterKey=BedrockAgentsLayerArn,ParameterValue=${BEDROCK_AGENTS_LAYER_ARN} 
ParameterKey=SNSEmail,ParameterValue=${SNS_EMAIL} 
ParameterKey=EvidenceUploadUrl,ParameterValue=${EVIDENCE_UPLOAD_URL} 
--capabilities CAPABILITY_NAMED_IAM

aws cloudformation describe-stacks --stack-name $STACK_NAME --query "Stacks[0].StackStatus"
aws cloudformation wait stack-create-complete --stack-name $STACK_NAME

Create a knowledge base

Knowledge Bases for Amazon Bedrock uses RAG, a technique that harnesses customer data stores to enhance responses generated by FMs. Knowledge bases allow agents to access existing customer data repositories without extensive administrator overhead. To connect a knowledge base to your data, you specify an S3 bucket as the data source. With knowledge bases, applications gain enriched contextual information, streamlining development through a fully managed RAG solution. This level of abstraction accelerates time-to-market by minimizing the effort of incorporating your data into agent functionality, and it optimizes cost by negating the necessity for continuous model retraining to use private data.

The following diagram illustrates the architecture for a knowledge base with an embeddings model.

Knowledge base functionality is delineated through two key processes: preprocessing (Steps 1-3) and runtime (Steps 4-7):

1. Documents undergo segmentation (chunking) into manageable sections.
2. Those chunks are converted into embeddings using an Amazon Bedrock embedding model.
3. The embeddings are used to create a vector index, enabling semantic similarity comparisons between user queries and data source text.
4. During runtime, users provide their text input as a prompt.
5. The input text is transformed into vectors using an Amazon Bedrock embedding model.
6. The vector index is queried for chunks related to the user’s query, augmenting the user prompt with additional context retrieved from the vector index.
7. The augmented prompt, coupled with the additional context, is used to generate a response for the user.

To create a knowledge base, complete the following steps:

1. On the Amazon Bedrock console, choose Knowledge base in the navigation pane.
2. Choose Create knowledge base.
3. Under Provide knowledge base details, enter a name and optional description, leaving all default settings. For this post, we enter the description:
   Use to retrieve claim amount and repair estimate information for claim ID, or answer general insurance questions about things like coverage, premium, policy, rate, deductible, accident, and documents.
4. Under Set up data source, enter a name.
5. Choose Browse S3 and select the knowledge-base-assets folder of the data source S3 bucket you deployed earlier (<YOUR-STACK-NAME>-customer-resources/agent/knowledge-base-assets/).
   
6. Under Select embeddings model and configure vector store, choose Titan Embeddings G1 – Text and leave the other default settings. An Amazon OpenSearch Serverless collection will be created for you. This vector store is where the knowledge base preprocessing embeddings are stored and later used for semantic similarity search between queries and data source text.
7. Under Review and create, confirm your configuration settings, then choose Create knowledge base.
   
8. After your knowledge base is created, a green “created successfully” banner will display with the option to sync your data source. Choose Sync to initiate the data source sync.
   
9. On the Amazon Bedrock console, navigate to the knowledge base you just created, then note the knowledge base ID under Knowledge base overview.
   
10. With your knowledge base still selected, choose your knowledge base data source listed under Data source, then note the data source ID under Data source overview.
    

The knowledge base ID and data source ID are used as environment variables in a later step when you deploy the Streamlit web UI for your agent.

Create an agent

Agents operate through a build-time run process, comprising several key components:

• Foundation model – Users select an FM that guides the agent in interpreting user inputs, generating responses, and directing subsequent actions during its orchestration process.
• Instructions – Users craft detailed instructions that outline the agent’s intended functionality. Optional advanced prompts allow customization at each orchestration step, incorporating Lambda functions to parse outputs.
• (Optional) Action groups – Users define actions for the agent, using an OpenAPI schema to define APIs for task runs and Lambda functions to process API inputs and outputs.
• (Optional) Knowledge bases – Users can associate agents with knowledge bases, granting access to additional context for response generation and orchestration steps.

The agent in this sample solution uses an Anthropic Claude V2.1 FM on Amazon Bedrock, a set of instructions, three action groups, and one knowledge base.

To create an agent, complete the following steps:

1. On the Amazon Bedrock console, choose Agents in the navigation pane.
2. Choose Create agent.
3. Under Provide Agent details, enter an agent name and optional description, leaving all other default settings.
4. Under Select model, choose Anthropic Claude V2.1 and specify the following instructions for the agent: You are an insurance agent that has access to domain-specific insurance knowledge. You can create new insurance claims, send pending document reminders to policy holders with open claims, and gather claim evidence. You can also retrieve claim amount and repair estimate information for a specific claim ID or answer general insurance questions about things like coverage, premium, policy, rate, deductible, accident, documents, resolution, and condition. You can answer internal questions about things like which steps an agent should follow and the company's internal processes. You can respond to questions about multiple claim IDs within a single conversation
5. Choose Next.
6. Under Add Action groups, add your first action group:
   1. For Enter Action group name, enter create-claim.
   2. For Description, enter Use this action group to create an insurance claim
   3. For Select Lambda function, choose <YOUR-STACK-NAME>-CreateClaimFunction.
   4. For Select API schema, choose Browse S3, choose the bucket created earlier (<YOUR-STACK-NAME>-customer-resources), then choose agent/api-schema/create_claim.json.
      
7. Create a second action group:
   1. For Enter Action group name, enter gather-evidence.
   2. For Description, enter Use this action group to send the user a URL for evidence upload on open status claims with pending documents. Return the documentUploadUrl to the user
   3. For Select Lambda function, choose <YOUR-STACK-NAME>-GatherEvidenceFunction.
   4. For Select API schema, choose Browse S3, choose the bucket created earlier, then choose agent/api-schema/gather_evidence.json.
8. Create a third action group:
   1. For Enter Action group name, enter send-reminder.
   2. For Description, enter Use this action group to check claim status, identify missing or pending documents, and send reminders to policy holders
   3. For Select Lambda function, choose <YOUR-STACK-NAME>-SendReminderFunction.
   4. For Select API schema, choose Browse S3, choose the bucket created earlier, then choose agent/api-schema/send_reminder.json.
9. Choose Next.
10. For Select knowledge base, choose the knowledge base you created earlier (claims-knowledge-base).
11. For Knowledge base instructions for Agent, enter the following: Use to retrieve claim amount and repair estimate information for claim ID, or answer general insurance questions about things like coverage, premium, policy, rate, deductible, accident, and documents
12. Choose Next.
    
13. Under Review and create, confirm your configuration settings, then choose Create agent.
    

After your agent is created, you will see a green “successfully created” banner.

Testing and validation

The following testing procedure aims to verify that the agent correctly identifies and understands user intents for creating new claims, sending pending document reminders for open claims, gathering claims evidence, and searching for information across existing claims and customer knowledge repositories. Response accuracy is determined by evaluating the relevancy, coherency, and human-like nature of the answers generated by Agents and Knowledge Bases for Amazon Bedrock.

Assessment measures and evaluation technique

User input and agent instruction validation includes the following:

• Preprocessing – Use sample prompts to assess the agent’s interpretation, understanding, and responsiveness to diverse user inputs. Validate the agent’s adherence to configured instructions for validating, contextualizing, and categorizing user input accurately.
• Orchestration – Evaluate the logical steps the agent follows (for example, “Trace”) for action group API invocations and knowledge base queries to enhance the base prompt for the FM.
• Postprocessing – Review the final responses generated by the agent after orchestration iterations to ensure accuracy and relevance. Postprocessing is inactive by default and therefore not included in our agent’s tracing.

Action group evaluation includes the following:

• API schema validation – Validate that the OpenAPI schema (defined as JSON files stored in Amazon S3) effectively guides the agent’s reasoning around each API’s purpose.
• Business logic Implementation – Test the implementation of business logic associated with API paths through Lambda functions linked with the action group.

Knowledge base evaluation includes the following:

• Configuration verification – Confirm that the knowledge base instructions correctly direct the agent on when to access the data.
• S3 data source integration – Validate the agent’s ability to access and use data stored in the specified S3 data source.

The end-to-end testing includes the following:

• Integrated workflow – Perform comprehensive tests involving both action groups and knowledge bases to simulate real-world scenarios.
• Response quality assessment – Evaluate the overall accuracy, relevancy, and coherence of the agent’s responses in diverse contexts and scenarios.

Test the knowledge base

After setting up your knowledge base in Amazon Bedrock, you can test its behavior directly to assess its responses before integrating it with an agent. This testing process enables you to evaluate the knowledge base’s performance, inspect responses, and troubleshoot by exploring the source chunks from which information is retrieved. Complete the following steps:

1. On the Amazon Bedrock console, choose Knowledge base in the navigation pane.
   
2. Select the knowledge base you want to test, then choose Test to expand a chat window.
   
3. In the test window, select your foundation model for response generation.
   
4. Test your knowledge base using the following sample queries and other inputs:
   1. What is the diagnosis on the repair estimate for claim ID 2s34w-8x?
   2. What is the resolution and repair estimate for that same claim?
   3. What should the driver do after an accident?
   4. What is recommended for the accident report and images?
   5. What is a deductible and how does it work?
      
      

You can toggle between generating responses and returning direct quotations in the chat window, and you have the option to clear the chat window or copy all output using the provided icons.

To inspect knowledge base responses and source chunks, you can select the corresponding footnote or choose Show result details. A source chunks window will appear, allowing you to search, copy chunk text, and navigate to the S3 data source.

Test the agent

Following the successful testing of your knowledge base, the next development phase involves the preparation and testing of your agent’s functionality. Preparing the agent involves packaging the latest changes, whereas testing provides a critical opportunity to interact with and evaluate the agent’s behavior. Through this process, you can refine agent capabilities, enhance its efficiency, and address any potential issues or improvements necessary for optimal performance. Complete the following steps:

1. On the Amazon Bedrock console, choose Agents in the navigation pane.
   
2. Choose your agent and note the agent ID.
   
   You use the agent ID as an environment variable in a later step when you deploy the Streamlit web UI for your agent.
3. Navigate to your Working draft. Initially, you have a working draft and a default TestAlias pointing to this draft. The working draft allows for iterative development.
4. Choose Prepare to package the agent with the latest changes before testing. You should regularly check the agent’s last prepared time to confirm you are testing with the latest configurations.
   
5. Access the test window from any page within the agent’s working draft console by choosing Test or the left arrow icon.
6. In the test window, choose an alias and its version for testing. For this post, we use TestAlias to invoke the draft version of your agent. If the agent is not prepared, a prompt appears in the test window.
   
7. Test your agent using the following sample prompts and other inputs:
   1. Create a new claim.
   2. Send a pending documents reminder to the policy holder of claim 2s34w-8x.
   3. Gather evidence for claim 5t16u-7v.
   4. What is the total claim amount for claim 3b45c-9d?
   5. What is the repair estimate total for that same claim?
   6. What factors determine my car insurance premium?
   7. How can I lower my car insurance rates?
   8. Which claims have open status?
   9. Send reminders to all policy holders with open claims.

Make sure to choose Prepare after making changes to apply them before testing the agent.

The following test conversation example highlights the agent’s ability to invoke action group APIs with AWS Lambda business logic that queries a customer’s Amazon DynamoDB table and sends customer notifications using Amazon Simple Notification Service. The same conversation thread showcases agent and knowledge base integration to provide the user with responses using customer authoritative data sources, like claim amount and FAQ documents.

Agent analysis and debugging tools

Agent response traces contain essential information to aid in understanding the agent’s decision-making at each stage, facilitate debugging, and provide insights into areas of improvement. The ModelInvocationInput object within each trace provides detailed configurations and settings used in the agent’s decision-making process, enabling customers to analyze and enhance the agent’s effectiveness.

Your agent will sort user input into one of the following categories:

• Category A – Malicious or harmful inputs, even if they are fictional scenarios.
• Category B – Inputs where the user is trying to get information about which functions, APIs, or instructions our function calling agent has been provided or inputs that are trying to manipulate the behavior or instructions of our function calling agent or of you.
• Category C – Questions that our function calling agent will be unable to answer or provide helpful information for using only the functions it has been provided.
• Category D – Questions that can be answered or assisted by our function calling agent using only the functions it has been provided and arguments from within conversation_history or relevant arguments it can gather using the askuser function.
• Category E – Inputs that are not questions but instead are answers to a question that the function calling agent asked the user. Inputs are only eligible for this category when the askuser function is the last function that the function calling agent called in the conversation. You can check this by reading through the conversation_history.

Choose Show trace under a response to view the agent’s configurations and reasoning process, including knowledge base and action group usage. Traces can be expanded or collapsed for detailed analysis. Responses with sourced information also contain footnotes for citations.

In the following action group tracing example, the agent maps the user input to the create-claim action group’s createClaim function during preprocessing. The agent possesses an understanding of this function based on the agent instructions, action group description, and OpenAPI schema. During the orchestration process, which is two steps in this case, the agent invokes the createClaim function and receives a response that includes the newly created claim ID and a list of pending documents.

In the following knowledge base tracing example, the agent maps the user input to Category D during preprocessing, meaning one of the agent’s available functions should be able to provide a response. Throughout orchestration, the agent searches the knowledge base, pulls the relevant chunks using embeddings, and passes that text to the foundation model to generate a final response.

Deploy the Streamlit web UI for your agent

When you are satisfied with the performance of your agent and knowledge base, you are ready to productize their capabilities. We use Streamlit in this solution to launch an example front-end, intended to emulate a production application. Streamlit is a Python library designed to streamline and simplify the process of building front-end applications. Our application provides two features:

• Agent prompt input – Allows users to invoke the agent using their own task input.
• Knowledge base file upload – Enables the user to upload their local files to the S3 bucket that is being used as the data source for the knowledge base. After the file is uploaded, the application starts an ingestion job to sync the knowledge base data source.

To isolate our Streamlit application dependencies and for ease of deployment, we use the setup-streamlit-env.sh shell script to create a virtual Python environment with the requirements installed. Complete the following steps:

1. Before you run the shell script, navigate to the directory where you cloned the amazon-bedrock-samples repository and modify the Streamlit shell script permissions to executable:

cd amazon-bedrock-samples/agents/insurance-claim-lifecycle-automation/agent/streamlit/
chmod u+x setup-streamlit-env.sh

2. Run the shell script to activate the virtual Python environment with the required dependencies:

source ./setup-streamlit-env.sh

3. Set your Amazon Bedrock agent ID, agent alias ID, knowledge base ID, data source ID, knowledge base bucket name, and AWS Region environment variables:

export BEDROCK_AGENT_ID=<YOUR-AGENT-ID>
export BEDROCK_AGENT_ALIAS_ID=<YOUR-AGENT-ALIAS-ID>
export BEDROCK_KB_ID=<YOUR-KNOWLEDGE-BASE-ID>
export BEDROCK_DS_ID=<YOUR-DATA-SOURCE-ID>
export KB_BUCKET_NAME=<YOUR-KNOWLEDGE-BASE-S3-BUCKET-NAME>
export AWS_REGION=<YOUR-STACK-REGION>

4. Run your Streamlit application and begin testing in your local web browser:

streamlit run agent_streamlit.py

Clean up

To avoid charges in your AWS account, clean up the solution’s provisioned resources

The delete-customer-resources.sh shell script empties and deletes the solution’s S3 bucket and deletes the resources that were originally provisioned from the bedrock-customer-resources.yml CloudFormation stack. The following commands use the default stack name. If you customized the stack name, adjust the commands accordingly.

# cd amazon-bedrock-samples/agents/insurance-claim-lifecycle-automation/shell/
# chmod u+x delete-customer-resources.sh
# export STACK_NAME=<YOUR-STACK-NAME>
./delete-customer-resources.sh

The preceding ./delete-customer-resources.sh shell command runs the following AWS CLI commands to delete the emulated customer resources stack and S3 bucket:

echo "Emptying and Deleting S3 Bucket: $ARTIFACT_BUCKET_NAME"
aws s3 rm s3://${ARTIFACT_BUCKET_NAME} --recursive
aws s3 rb s3://${ARTIFACT_BUCKET_NAME}

echo "Deleting CloudFormation Stack: $STACK_NAME"
aws cloudformation delete-stack --stack-name $STACK_NAME
aws cloudformation describe-stacks --stack-name $STACK_NAME --query "Stacks[0].StackStatus"
aws cloudformation wait stack-delete-complete --stack-name $STACK_NAME

To delete your agent and knowledge base, follow the instructions for deleting an agent and deleting a knowledge base, respectively.

Considerations

Although the demonstrated solution showcases the capabilities of Agents and Knowledge Bases for Amazon Bedrock, it’s important to understand that this solution is not production-ready. Rather, it serves as a conceptual guide for customers aiming to create personalized agents for their own specific tasks and automated workflows. Customers aiming for production deployment should refine and adapt this initial model, keeping in mind the following security factors:

• Secure access to APIs and data:
  • Restrict access to APIs, databases, and other agent-integrated systems.
  • Utilize access control, secrets management, and encryption to prevent unauthorized access.
• Input validation and sanitization:
  • Validate and sanitize user inputs to prevent injection attacks or attempts to manipulate the agent’s behavior.
  • Establish input rules and data validation mechanisms.
• Access controls for agent management and testing:
  • Implement proper access controls for consoles and tools used to edit, test, or configure the agent.
  • Limit access to authorized developers and testers.
• Infrastructure security:
  • Adhere to AWS security best practices regarding VPCs, subnets, security groups, logging, and monitoring for securing the underlying infrastructure.
• Agent instructions validation:
  • Establish a meticulous process to review and validate the agent’s instructions to prevent unintended behaviors.
• Testing and auditing:
  • Thoroughly test the agent and integrated components.
  • Implement auditing, logging, and regression testing of agent conversations to detect and address issues.
• Knowledge base security:
  • If users can augment the knowledge base, validate uploads to prevent poisoning attacks.

For other key considerations, refer to Build generative AI agents with Amazon Bedrock, Amazon DynamoDB, Amazon Kendra, Amazon Lex, and LangChain.

Conclusion

The implementation of generative AI agents using Agents and Knowledge Bases for Amazon Bedrock represents a significant advancement in the operational and automation capabilities of organizations. These tools not only streamline the insurance claim lifecycle, but also set a precedent for the application of AI in various other enterprise domains. By automating tasks, enhancing customer service, and improving decision-making processes, these AI agents empower organizations to focus on growth and innovation, while handling routine and complex tasks efficiently.

As we continue to witness the rapid evolution of AI, the potential of tools like Agents and Knowledge Bases for Amazon Bedrock in transforming business operations is immense. Enterprises that use these technologies stand to gain a significant competitive advantage, marked by improved efficiency, customer satisfaction, and decision-making. The future of enterprise data management and operations is undeniably leaning towards greater AI integration, and Amazon Bedrock is at the forefront of this transformation.

To learn more, visit Agents for Amazon Bedrock, consult the Amazon Bedrock documentation, explore the generative AI space at community.aws, and get hands-on with the Amazon Bedrock workshop.

About the Author

Kyle T. Blocksom is a Sr. Solutions Architect with AWS based in Southern California. Kyle’s passion is to bring people together and leverage technology to deliver solutions that customers love. Outside of work, he enjoys surfing, eating, wrestling with his dog, and spoiling his niece and nephew.

Read More

In the first post of this three-part series, we presented a solution that demonstrates how you can automate detecting document tampering and fraud at scale using AWS AI and machine learning (ML) services for a mortgage underwriting use case.

In the second post, we discussed an approach to develop a deep learning-based computer vision model to detect and highlight forged images in mortgage underwriting.

In this post, we present a solution to automate mortgage document fraud detection using an ML model and business-defined rules with Amazon Fraud Detector.

Solution overview

We use Amazon Fraud Detector, a fully managed fraud detection service, to automate the detection of fraudulent activities. With an objective to improve fraud prediction accuracies by proactively identifying document fraud, while improving underwriting accuracies, Amazon Fraud Detector helps you build customized fraud detection models using a historical dataset, configure customized decision logic using the built-in rules engine, and orchestrate risk decision workflows with the click of a button.

The following diagram represents each stage in a mortgage document fraud detection pipeline.

We will now be covering the third component of the mortgage document fraud detection pipeline. The steps to deploy this component are as follows:

1. Upload historical data to Amazon Simple Storage Service (Amazon S3).
2. Select your options and train the model.
3. Create the model.
4. Review model performance.
5. Deploy the model.
6. Create a detector.
7. Add rules to interpret model scores.
8. Deploy the API to make predictions.

Prerequisites

The following are prerequisite steps for this solution:

1. Sign up for an AWS account.
2. Set up permissions that allows your AWS account to access Amazon Fraud Detector.
3. Collect the historical fraud data to be used to train the fraud detector model, with the following requirements:
   1. Data must be in CSV format and have headers.
   2. Two headers are required: EVENT_TIMESTAMP and EVENT_LABEL.
   3. Data must reside in Amazon S3 in an AWS Region supported by the service.
   4. It’s highly recommended to run a data profile before you train (use an automated data profiler for Amazon Fraud Detector).
   5. It’s recommended to use at least 3–6 months of data.
   6. It takes time for fraud to mature; data that is 1–3 months old is recommended (not too recent).
   7. Some NULLs and missing values are acceptable (but too many and the variable is ignored, as discussed in Missing or incorrect variable type).

Upload historical data to Amazon S3

After you have the custom historical data files to train a fraud detector model, create an S3 bucket and upload the data to the bucket.

Select options and train the model

The next step towards building and training a fraud detector model is to define the business activity (event) to evaluate for the fraud. Defining an event involves setting the variables in your dataset, an entity initiating the event, and the labels that classify the event.

Complete the following steps to define a docfraud event to detect document fraud, which is initiated by the entity applicant mortgage, referring to a new mortgage application:

1. On the Amazon Fraud Detector console, choose Events in the navigation pane.
2. Choose Create.
3. Under Event type details, enter docfraud as the event type name and, optionally, enter a description of the event.
4. Choose Create entity.
5. On the Create entity page, enter applicant_mortgage as the entity type name and, optionally, enter a description of the entity type.
6. Choose Create entity.
7. Under Event variables, for Choose how to define this event’s variables, choose Select variables from a training dataset.
8. For IAM role, choose Create IAM role.
9. On the Create IAM role page, enter the name of the S3 bucket with your example data and choose Create role.
10. For Data location, enter the path to your historical data. This is the S3 URI path that you saved after uploading the historical data. The path is similar to S3://your-bucket-name/example dataset filename.csv.
11. Choose Upload.

Variables represent data elements that you want to use in a fraud prediction. These variables can be taken from the event dataset that you prepared for training your model, from your Amazon Fraud Detector model’s risk score outputs, or from Amazon SageMaker models. For more information about variables taken from the event dataset, see Get event dataset requirements using the Data models explorer.

1. Under Labels – optional, for Labels, choose Create new labels.
2. On the Create label page, enter fraud as the name. This label corresponds to the value that represents the fraudulent mortgage application in the example dataset.
3. Choose Create label.
4. Create a second label called legit. This label corresponds to the value that represents the legitimate mortgage application in the example dataset.
5. Choose Create event type.

The following screenshot shows our event type details.

The following screenshot shows our variables.

The following screenshot shows our labels.

Create the model

After you have loaded the historical data and selected the required options to train a model, complete the following steps to create a model:

1. On the Amazon Fraud Detector console, choose Models in the navigation pane.
2. Choose Add model, and then choose Create model.
3. On the Define model details page, enter mortgage_fraud_detection_model as the model’s name and an optional description of the model.
4. For Model type, choose the Online Fraud Insights model.
5. For Event type, choose docfraud. This is the event type that you created earlier.
6. In the Historical event data section, provide the following information:
   1. For Event data source, choose Event data stored uploaded to S3 (or AFD).
   2. For IAM role, choose the role that you created earlier.
   3. For Training data location, enter the S3 URI path to your example data file.
7. Choose Next.
8. In the Model inputs section, leave all checkboxes checked. By default, Amazon Fraud Detector uses all variables from your historical event dataset as model inputs.
9. In the Label classification section, for Fraud labels, choose fraud, which corresponds to the value that represents fraudulent events in the example dataset.
10. For Legitimate labels, choose legit, which corresponds to the value that represents legitimate events in the example dataset.
11. For Unlabeled events, keep the default selection Ignore unlabeled events for this example dataset.
12. Choose Next.
13. Review your settings, then choose Create and train model.

Amazon Fraud Detector creates a model and begins to train a new version of the model.

On the Model versions page, the Status column indicates the status of model training. Model training that uses the example dataset takes approximately 45 minutes to complete. The status changes to Ready to deploy after model training is complete.

Review model performance

After the model training is complete, Amazon Fraud Detector validates the model performance using 15% of your data that was not used to train the model and provides various tools, including a score distribution chart and confusion matrix, to assess model performance.

To view the model’s performance, complete the following steps:

1. On the Amazon Fraud Detector console, choose Models in the navigation pane.
2. Choose the model that you just trained (sample_fraud_detection_model), then choose 1.0. This is the version Amazon Fraud Detector created of your model.
3. Review the Model performance overall score and all other metrics that Amazon Fraud Detector generated for this model.

Deploy the model

After you have reviewed the performance metrics of your trained model and are ready to use it generate fraud predictions, you can deploy the model:

1. On the Amazon Fraud Detector console, choose Models in the navigation pane.
2. Choose the model sample_fraud_detection_model, and then choose the specific model version that you want to deploy. For this post, choose 1.0.
3. On the Model version page, on the Actions menu, choose Deploy model version.

On the Model versions page, the Status shows the status of the deployment. The status changes to Active when the deployment is complete. This indicates that the model version is activated and available to generate fraud predictions.

Create a detector

After you have deployed the model, you build a detector for the docfraud event type and add the deployed model. Complete the following steps:

1. On the Amazon Fraud Detector console, choose Detectors in the navigation pane.
2. Choose Create detector.
3. On the Define detector details page, enter fraud_detector for the detector name and, optionally, enter a description for the detector, such as my sample fraud detector.
4. For Event Type, choose docfraud. This is the event that you created in earlier.
5. Choose Next.

Add rules to interpret

After you have created the Amazon Fraud Detector model, you can use the Amazon Fraud Detector console or application programming interface (API) to define business-driven rules (conditions that tell Amazon Fraud Detector how to interpret model performance score when evaluating for fraud prediction). To align with the mortgage underwriting process, you may create rules to flag mortgage applications according to the risk levels associated and mapped as fraud, legitimate, or if a review is needed.

For example, you may want to automatically decline mortgage applications with a high fraud risk, considering parameters like tampered images of the required documents, missing documents like paystubs or income requirements, and so on. On the other hand, certain applications may need a human in the loop for making effective decisions.

Amazon Fraud Detector uses the aggregated value (calculated by combining a set of raw variables) and raw value (the value provided for the variable) to generate the model scores. The model scores can be between 0–1000, where 0 indicates low fraud risk and 1000 indicates high fraud risk.

To add the respective business-driven rules, complete the following steps:

1. On the Amazon Fraud Detector console, choose Rules in the navigation pane.
2. Choose Add rule.
3. In the Define a rule section, enter fraud for the rule name and, optionally, enter a description.
4. For Expression, enter the rule expression using the Amazon Fraud Detector simplified rule expression language $docdraud_insightscore >= 900
5. For Outcomes, choose Create a new outcome (An outcome is the result from a fraud prediction and is returned if the rule matches during an evaluation.)
6. In the Create a new outcome section, enter decline as the outcome name and an optional description.
7. Choose Save outcome
8. Choose Add rule to run the rule validation checker and save the rule.
9. After it’s created, Amazon Fraud Detector makes the following high_risk rule available for use in your detector.
   1. Rule name: fraud
   2. Outcome: decline
   3. Expression: $docdraud_insightscore >= 900
10. Choose Add another rule, and then choose the Create rule tab to add additional 2 rules as below:
11. Create a low_risk rule with the following details:
    1. Rule name: legit
    2. Outcome: approve
    3. Expression: $docdraud_insightscore <= 500
12. Create a medium_risk rule with the following details:
    1. Rule name: review needed
    2. Outcome: review
    3. Expression: $docdraud_insightscore <= 900 and docdraud_insightscore >=500

These values are examples used for this post. When you create rules for your own detector, use values that are appropriate for your model and use case.

1. After you have created all three rules, choose Next.

Deploy the API to make predictions

After the rules-based actions have been triggered, you can deploy an Amazon Fraud Detector API to evaluate the lending applications and predict potential fraud. The predictions can be performed in a batch or real time.

Integrate your SageMaker model (Optional)

If you already have a fraud detection model in SageMaker, you can integrate it with Amazon Fraud Detector for your preferred results.

This implies that you can use both SageMaker and Amazon Fraud Detector models in your application to detect different types of fraud. For example, your application can use the Amazon Fraud Detector model to assess the fraud risk of customer accounts, and simultaneously use your PageMaker model to check for account compromise risk.

Clean up

To avoid incurring any future charges, delete the resources created for the solution, including the following:

• S3 bucket
• Amazon Fraud Detector endpoint

Conclusion

This post walked you through an automated and customized solution to detect fraud in the mortgage underwriting process. This solution allows you to detect fraudulent attempts closer to the time of fraud occurrence and helps underwriters with an effective decision-making process. Additionally, the flexibility of the implementation allows you to define business-driven rules to classify and capture the fraudulent attempts customized to specific business needs.

For more information about building an end-to-end mortgage document fraud detection solution, refer to Part 1 and Part 2 in this series.

About the authors

Anup Ravindranath is a Senior Solutions Architect at Amazon Web Services (AWS) based in Toronto, Canada working with Financial Services organizations. He helps customers to transform their businesses and innovate on cloud.

Vinnie Saini is a Senior Solutions Architect at Amazon Web Services (AWS) based in Toronto, Canada. She has been helping Financial Services customers transform on cloud, with AI and ML driven solutions laid on strong foundational pillars of Architectural Excellence.

Read More