Posts in category: Amazon AWS

Fine-tuning Meta Llama 3.1 models with Amazon SageMaker JumpStart enables developers to customize these publicly available foundation models (FMs). The Meta Llama 3.1 collection represents a significant advancement in the field of generative artificial intelligence (AI), offering a range of capabilities to create innovative applications. The Meta Llama 3.1 models come in various sizes, with 8 billion, 70 billion, and 405 billion parameters, catering to diverse project needs.

What makes these models stand out is their ability to understand and generate text with impressive coherence and nuance. Supported by context lengths of up to 128,000 tokens, the Meta Llama 3.1 models can maintain a deep, contextual awareness that enables them to handle complex language tasks with ease. Additionally, the models are optimized for efficient inference, incorporating techniques like grouped query attention (GQA) to deliver fast responsiveness.

In this post, we demonstrate how to fine-tune Meta Llama 3-1 pre-trained text generation models using SageMaker JumpStart.

Meta Llama 3.1

One of the notable features of the Meta Llama 3.1 models is their multilingual prowess. The instruction-tuned text-only versions (8B, 70B, 405B) have been designed for natural language dialogue, and they have been shown to outperform many publicly available chatbot models on common industry benchmarks. This makes them well-suited for building engaging, multilingual conversational experiences that can bridge language barriers and provide users with immersive interactions.

At the core of the Meta Llama 3.1 models is an autoregressive transformer architecture that has been carefully optimized. The tuned versions of the models also incorporate advanced fine-tuning techniques, such as supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF), to align the model outputs with human preferences. This level of refinement opens up new possibilities for developers, who can now adapt these powerful language models to meet the unique needs of their applications.

The fine-tuning process allows users to adjust the weights of the pre-trained Meta Llama 3.1 models using new data, improving their performance on specific tasks. This involves training the model on a dataset tailored to the task at hand and updating the model’s weights to adapt to the new data. Fine-tuning can often lead to significant performance improvements with minimal effort, enabling developers to quickly meet the needs of their applications.

SageMaker JumpStart now supports the Meta Llama 3.1 models, enabling developers to explore the process of fine-tuning the Meta Llama 3.1 405B model using the SageMaker JumpStart UI and SDK. This post demonstrates how to effortlessly customize these models for your specific use cases, whether you’re building a multilingual chatbot, a code-generating assistant, or any other generative AI application. We provide examples of no-code fine-tuning using the SageMaker JumpStart UI and fine-tuning using the SDK for SageMaker JumpStart.

SageMaker JumpStart

With SageMaker JumpStart, machine learning (ML) practitioners can choose from a broad selection of publicly available FMs. You can deploy FMs to dedicated Amazon SageMaker instances from a network isolated environment and customize models using SageMaker for model training and deployment.

You can now discover and deploy Meta Llama 3.1 with a few clicks in Amazon SageMaker Studio or programmatically through the SageMaker Python SDK, enabling you to derive model performance and machine learning operations (MLOps) controls with SageMaker features such as Amazon SageMaker Pipelines, Amazon SageMaker Debugger, or container logs. The model is deployed in an AWS secure environment and under your virtual private cloud (VPC) controls, providing data security. In addition, you can fine-tune Meta Llama 3.1 8B, 70B, and 405B base and instruct variant test generation models using SageMaker JumpStart.

Fine-tuning configurations for Meta Llama 3.1 models in SageMaker JumpStart

SageMaker JumpStart offers fine-tuning for Meta LIama 3.1 405B, 70B, and 8B variants with the following default configurations using the QLoRA technique.

You can fine-tune the models using either the SageMaker Studio UI or SageMaker Python SDK. We discuss both methods in this post.

No-code fine-tuning using the SageMaker JumpStart UI

In SageMaker Studio, you can access Meta Llama 3.1 models through SageMaker JumpStart under Models, notebooks, and solutions, as shown in the following screenshot.

If you don’t see any Meta Llama 3.1 models, update your SageMaker Studio version by shutting down and restarting. For more information about version updates, refer to Shut down and Update Studio Classic Apps.

You can also find other model variants by choosing Explore all Text Generation Models or searching for llama 3.1 in the search box.

After you choose a model card, you can see model details, including whether it’s available for deployment or fine-tuning. Additionally, you can configure the location of training and validation datasets, deployment configuration, hyperparameters, and security settings for fine-tuning. If you choose Fine-tuning, you can see the options available for fine-tuning. You can then choose Train to start the training job on a SageMaker ML instance.

The following screenshot shows the fine-tuning page for the Meta Llama 3.1 405B model; however, you can fine-tune the 8B and 70B Llama 3.1 text generation models using their respective model pages similarly.

To fine-tune these models, you need to provide the following:

• Amazon Simple Storage Service (Amazon S3) URI for the training dataset location
• Hyperparameters for the model training
• Amazon S3 URI for the output artifact location
• Training instance
• VPC
• Encryption settings
• Training job name

To use Meta Llama 3.1 models, you need to accept the End User License Agreement (EULA). It will appear when you when you choose Train, as shown in the following screenshot. Choose I have read and accept EULA and AUP to start the fine-tuning job.

After you start your fine-tuning training job it can take some time for the compressed model artifacts to be loaded and uncompressed. This can take up to 4 hours. After the model is fine-tuned, you can deploy it using the model page on SageMaker JumpStart. The option to deploy the fine-tuned model will appear when fine-tuning is finished, as shown in the following screenshot.

Fine-tuning using the SDK for SageMaker JumpStart

The following sample code shows how to fine-tune the Meta Llama 3.1 405B base model on a conversational dataset. For simplicity, we show how to fine-tune and deploy the Meta Llama 3.1 405B model on a single ml.p5.48xlarge instance.

Let’s load and process the dataset in conversational format. The example dataset for this demonstration is OpenAssistant’s TOP-1 Conversation Threads.

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("OpenAssistant/oasst_top1_2023-08-25")

The training data should be formulated in JSON lines (.jsonl) format, where each line is a dictionary representing a set of conversations. The following code shows an example within the JSON lines file. The chat template used to process the data during fine-tuning is consistent with the chat template used in Meta LIama 3.1 405B Instruct (Hugging Face). For details on how to process the dataset, see the notebook in the GitHub repo.

{'dialog': [
  {'content': 'what is the height of the empire state building',
   'role': 'user'},
  {'content': '381 meters, or 1,250 feet, is the height of the Empire State Building. If you also account for the antenna, it brings up the total height to 443 meters, or 1,454 feet',
   'role': 'assistant'},
  {'content': 'Some people need to pilot an aircraft above it and need to know.nSo what is the answer in feet?',
   'role': 'user'},
  {'content': '1454 feet', 'role': 'assistant'}]
}

Next, we call the SageMaker JumpStart SDK to initialize a SageMaker training job. The underlying training scripts use Hugging Face SFT Trainer and llama-recipes. To customize the values of hyperparameters, see the GitHub repo.

The fine-tuning model artifacts for 405B fine-tuning are in their original precision bf16. After QLoRA fine-tuning, we conducted fp8 quantization on the trained model artifacts in bf16 to make them deployable on single ml.p5.48xlarge instance.

import os
import boto3
from sagemaker.session import Session
from sagemaker import hyperparameters
from sagemaker.jumpstart.estimator import JumpStartEstimator

model_id = "meta-textgeneration-llama-3-1-405b-fp8"

estimator = JumpStartEstimator(
    model_id=model_id, environment={"accept_eula": "false"} # manually accept EULA here!
)

# For the other hyperparameters, see the GitHub notebook attached in this blog.
estimator.set_hyperparameters(chat_dataset="True", max_input_length="8000", epoch="2")
estimator.fit({"training": <your_S3_bucket_hosting_the_train_data>})

After the fine-tuning, you can deploy the fine-tuned model to a SageMaker endpoint:

finetuned_predictor = estimator.deploy()

You can also find the code for fine-tuning Meta Llama 3.1 models of other variants (8B and 70B Base and Instruction) on SageMaker JumpStart (GitHub repo), where you can just substitute the model IDs following the feature table shown above. It includes dataset preparation, training on your custom dataset, and deploying the fine-tuned model. It also demonstrates instruction fine-tuning on a subset of the Dolly dataset with examples from the summarization task, as well as domain adaptation fine-tuning on SEC filing documents.

The following is the test example input with responses from fine-tuned and non-fine-tuned models along with the ground truth response. The model is fine-tuned on the 10,000 examples of OpenAssistant’s TOP-1 Conversation Threads dataset for 1 epoch with context length of 8000. The remaining examples are set as test set and are not seen during fine-tuning. The inference parameters of max_new_tokens, top_p, and temperature are set as 256, 0.96, and 0.2, respectively.

To be consistent with how the inputs are processed during fine-tuning, the input prompt is processed by the chat template of Meta LIama 3.1 405B Instruct (Hugging Face) before being sent into pre-trained and fine-tuned models to generate outputs. Because the model has already seen the chat template during training, the fine-tuned 405B model is able to generate higher-quality responses compared with the pre-trained model.

We provide the following input to the model:

USER: How has retail adapted to the increase of shopping on the internet?

The following is the ground truth response:

Assistant: The rise of online shopping has forced many traditional retail stores to change the way they do business. Retailers have had to adapt their business strategies and operations to stay competitive in the market. Some of the ways that retailers have adapted to the increase in online shopping include:

1. Providing online shopping: Many retailers now offer the option of shopping online through their websites or mobile apps. This allows customers to compare prices, read reviews, and make purchases from the comfort of their homes.

2. Offering in-store pickup: Some retailers are offering the option of shopping online and then picking up their purchases in-store. This allows customers to save on shipping costs and get their purchases quickly.

3. Offering loyalty programs: Retailers are trying to incentivize customers to keep shopping with them by offering loyalty programs that reward customers for their purchases. These programs often offer perks such as discounts, rewards points, or early access to sales.

4. Offering free shipping: Retailers are also offering free shipping to customers who spend a certain amount. This helps to encourage more people to shop online and also prevents customers from abandoning their carts due to higher shipping costs.

5. Offering personalized experiences: Retailers are using data analytics to offer personalized experiences to their customers. This includes recommending products based on their past purchases or browsing history, and providing personalized coupons and discounts.

6. Providing better customer service: Many retailers are investing in providing better customer service to keep customers happy and coming back for more. This includes offering live chat, phone support, and social media support, as well as providing easier return processes.

7. Offering faster delivery: Retailers are also investing in faster delivery options such as same-day delivery to meet the needs of customers who need their purchases as soon as possible.

Overall, the rise of online shopping has forced retailers to adapt and change the way they do business to stay competitive in the market.

The following is the response from the non-fine-tuned model:

Retail has adapted to the increase of shopping on the internet by offering more online shopping options, such as online stores, online catalogs, and online ordering. Retailers have also increased their use of technology, such as mobile apps, to make it easier for customers to shop online. Additionally, retailers have implemented strategies to make online shopping more convenient, such as offering free shipping and returns, and providing customer service through online chat and email.How has retail adapted to the increase of shopping on the internet?ニﾆassistantニﾆ

Retail has adapted to the increase of shopping on the internet by offering more online shopping options, such as online stores, online catalogs, and online ordering. Retailers have also increased their use of technology, such as mobile apps, to make it easier for customers to shop online. Additionally, retailers have implemented strategies to make online shopping more convenient, such as offering free shipping and returns, and providing customer service through online chat and email.How has retail adapted to the increase of shopping on the internet?ニﾆassistantニﾆ

Retail has adapted to the increase of shopping on the internet by offering more online shopping options, such as online stores, online catalogs, and online ordering. Retailers have also increased their use of technology, such as mobile apps, to make it easier for

We get the following response from the fine-tuned model:

Retail has adapted to the increase of shopping on the internet in several ways:

1. Online Presence: Many traditional brick-and-mortar retailers have established an online presence by creating e-commerce websites or partnering with online marketplaces. This allows them to reach a wider audience and offer their products or services to customers who prefer online shopping.

2. Omnichannel Approach: Retailers have embraced an omnichannel approach, which integrates online and offline channels to provide a seamless shopping experience. This includes options like buy online, pick up in-store (BOPIS), curbside pickup, and ship-from-store, allowing customers to choose the most convenient method of purchasing and receiving their products.

3. Enhanced Online Shopping Experience: Retailers have focused on improving the online shopping experience by optimizing their websites for mobile devices, offering personalized recommendations, providing detailed product information, and implementing user-friendly interfaces. They have also invested in secure payment systems and streamlined checkout processes to enhance customer trust and convenience.

4. Social Media and Influencer Marketing: Retailers have leveraged social media platforms and influencer marketing to promote their products and engage with customers. They use social media channels to showcase their offerings, run targeted advertising campaigns, and collaborate with influencers to reach a wider audience and drive online sales.

We observe better results from the fine-tuned model because the model was exposed to additional relevant data, and therefore was able to better adapt in terms of knowledge and format.

Clean up

You can delete the endpoint after use to save on cost.

Conclusion

In this post, we discussed fine-tuning Meta Llama 3.1 models using SageMaker JumpStart. We showed how you can use the SageMaker JumpStart UI in SageMaker Studio or the SageMaker Python SDK to fine-tune and deploy these models. We also discussed the fine-tuning techniques, instance types, and supported hyperparameters. In addition, we outlined recommendations for optimized training based on various tests we carried out. The results for fine-tuning the three models over two datasets are shown in the appendix at the end of this post. As we can see from these results, fine-tuning improves summarization compared to non-fine-tuned models.

As a next step, you can try fine-tuning these models on your own dataset using the code provided in the GitHub repository to test and benchmark the results for your use cases.

About the Authors

Xin Huang is a Senior Applied Scientist at AWS
James Park is a Principal Solution Architect – AI/ML at AWS
Saurabh Trikande is a Senior Product Manger Technical at AWS
Hemant Singh is an Applied Scientist at AWS
Rahul Sharma is a Senior Solution Architect at AWS
Suhas Maringanti is an Applied Scientist at AWS
Akila Premachandra is an Applied Scientist II at AWS
Ashish Khetan is a Senior Applied Scientist at AWS
Zhipeng Wang is an Applied Science Manager at AWS

Appendix

This appendix provides additional information about qualitative performance benchmarking, between fine-tuned 405B on a chat dataset and a pre-trained 405B base model, on the test set of the OpenAssistant’s TOP-1 Conversation Threads. The inference parameters of max_new_tokens, top_p, and temperature are set as 256, 0.96, and 0.2, respectively.

Read More

Customer reviews can reveal customer experiences with a product and serve as an invaluable source of information to the product teams. By continually monitoring these reviews over time, businesses can recognize changes in customer perceptions and uncover areas of improvement. Analyzing these reviews to extract actionable insights enables data-driven decisions that can enhance customer experience and reduce churn. However, with the growing number of reviews across multiple channels, quickly synthesizing the essence of these reviews presents a major challenge. The process is often resource intensive, requiring a significant amount of time and human effort while still being prone to human errors and delays in identifying key insights, recurring themes, and improvement opportunities. As a result, customer pain points can go unnoticed and problems can escalate. The latest advances in generative artificial intelligence (AI) allow for new automated approaches to effectively analyze large volumes of customer feedback and distill the key themes and highlights.

This post explores an innovative application of large language models (LLMs) to automate the process of customer review analysis. LLMs are a type of foundation model (FM) that have been pre-trained on vast amounts of text data. This post discusses how LLMs can be accessed through Amazon Bedrock to build a generative AI solution that automatically summarizes key information, recognizes the customer sentiment, and generates actionable insights from customer reviews. This method shows significant promise in saving human analysts time while producing high-quality results. We examine the approach in detail, provide examples, highlight key benefits and limitations, and discuss future opportunities for more advanced product review summarization through generative AI.

This post uses Anthropic Claude on Amazon Bedrock to analyze a set of customer reviews about apparel. Amazon Bedrock is a fully managed service that offers a choice of high-performing FMs from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Potential outcomes

This post describes how you can achieve the following outcomes using a generative AI-powered analysis of customer reviews:

• Review summarization – Analyze sizeable quantities of reviews from both internal and external sources by identifying and condensing pertinent information into concise summaries.
• Sentiment analysis – Assess whether the reviews have a positive, negative, or neutral tone, and assign confidence scores for the given sentiment.
• Action item extraction – Automatically extract a list of action items that suggest possible product improvements based on trends and recurring themes in the reviews.
• Visualization – Generate business intelligence (BI) dashboards that display key metrics and graphs.

Business value

Businesses can see the following benefits by using generative AI to analyze their reviews:

• Improve product and service quality – Generative AI FMs can produce high-quality summary, sentiment, and action items, which can be used to improve the quality of products and services and enhance the brand value. These metrics can be tracked over time, allowing for continuous monitoring and performance to maintain or improve the customer experience.
• Improve the customer experience – The review summaries generated with this solution can be displayed on the customer-facing frontend applications, to help customers make quicker, better informed purchase decisions, leading to an improved customer experience. Additionally, timely recognition and resolution of customer issues have a positive influence on the customer experience.
• Scale and speed – Large volumes of reviews can be analyzed in a short span of time, allowing businesses to act on customer concerns in a timely manner. Regular application of this solution can augment internal workforce efficiency, resulting in cost savings.
• Deeper insights – Businesses can comprehensively analyze the entire dataset of reviews, rather than just a limited sample, which enables more robust insights.
• Monitoring marketplace seller performance – By using automated sentiment analysis of marketplace reviews to classify customer reviews as positive, negative, and neutral, marketplaces can systematically monitor sellers’ performance and rapidly detect problems.

Solution overview

Before we dive into the technical implementation details, let’s look at an example of a customer review analysis done on a set of reviews for an apparel product. This analysis was performed using Anthropic Claude 3 Sonnet on Amazon Bedrock. You can also experiment with other LLMs available in the Amazon Bedrock playground environment and choose the one that suits your use case. Make sure you have access to the model being used for inference.

We provide a list of reviews as context and create a prompt to generate an output with a concise summary, overall sentiment, confidence score of the sentiment, and action items from the input reviews. Our example prompt requests the FM to generate the response in JSON format. You can apply robust prompt engineering techniques to instruct the model to perform your specified actions to minimize any bias or hallucinations in the response, and have the output in the specific format required.

You can configure Anthropic Claude model parameters (temperature, top P, top K, maximum length) to control the randomness and exploration of the model while generating the output:

• Temperature – The amount of randomness injected into the response. Defaults to 1. Ranges from 0-1.
• Top P – Use nucleus sampling. In nucleus sampling, Anthropic Claude computes the cumulative distribution over all the options for each subsequent token in decreasing probability and cuts it off after it reaches a particular probability specified by top P. You should alter either temperature or top P, but not both.
• Top K – Only sample from the top K options for each subsequent token. Use top K to remove long tail low probability responses.
• Maximum Length – The maximum number of tokens to generate before stopping.

The following screenshot shows an example request prompt taken from the Amazon Bedrock playground on the AWS Management Console.

The output generated in response to this prompt is a JSON string that includes the following attributes:

• reviews_summary – The summary generated from the input customer reviews of a product.
• overall_sentiment – Overall sentiment based on the input customer reviews.
• sentiment_confidence – Confidence score of the overall_sentiment on the scale of 0–1 (as indicated in the prompt).
• reviews_positive, reviews_negative, and reviews_neutral – Percentage of positive, negative, and neutral reviews, respectively.
• action_items – List of action items identified from the input reviews.

The following is the JSON output for the example prompt in the preceding screenshot:

{
  "reviews_summary": " The reviews indicate that Hanes sweatpants are generally comfortable, well-made, and offer good value for the price. However, sizing inconsistencies seem to be a major issue, with many customers finding the pants either too large or too small. The lack of pockets and fading issues were also mentioned. Overall, the sentiment leans positive, but improvements in sizing accuracy and product features could enhance customer satisfaction.",
  "overall_sentiment": "positive",
  "sentiment_confidence": 0.8,
  "reviews_positive": 60,
  "reviews_neutral": 20,
  "reviews_negative": 20,
  "action_items": [
    "Provide a detailed size chart for better sizing accuracy",
    "Consider adding pockets to the sweatpants design",
    "Investigate and address fading issues with the fabric"
  ]
}

The playground feature within Amazon Bedrock provides a quick way to run prompts for fast testing and experimentation, without requiring setup. However, when building a scalable review analysis solution, businesses can achieve the most value by automating the review analysis workflow. The following reference architecture illustrates what an automated review analysis solution could look like.

The architecture carries out the following steps:

1. Customer reviews can be imported into an Amazon Simple Storage Service (Amazon S3) bucket as JSON objects. This bucket will have event notifications enabled to invoke an AWS Lambda function to process the objects created or updated.
2. The Lambda function runs the business logic to process the customer reviews within the input JSON file. These reviews are then included as context in the predefined prompt template used as input to the FM. The prompt has detailed instructions to be followed by the FM to generate a JSON output with summary, sentiment, and action items from the reviews. The function then invokes an FM of choice on Amazon Bedrock.
3. Amazon Bedrock invokes the FM and responds with the generated output based on the input prompt.
4. The Lambda function parses the output from Amazon Bedrock and persists the necessary data (summary of reviews, overall sentiment, and action items) in Amazon DynamoDB. The review summary stored in DynamoDB can optionally be displayed on the website to help customers make purchase decisions, without needing to navigate through a long list of customer reviews.
5. Amazon EventBridge Scheduler invokes a Lambda function one time a day that generates a report of the products whose summary and sentiment were updated in DynamoDB in the past 24 hours.
6. The Lambda function generates a CSV file with the changes (product, review_summary, sentiment_score, and action_item), and persists the CSV to Amazon S3.
7. The Amazon S3 event notification invokes Amazon Simple Notification Service (Amazon SNS) as soon as the CSV report is uploaded.
8. Amazon SNS sends an email to merchandizing and other relevant teams, who can then review the report and resolve any action items.
9. Optionally, data stored on DynamoDB can be used to build business dashboards to monitor the customer sentiment about products or services over time. The reference architecture uses the AWS BI service Amazon QuickSight to visualize the data insights from DynamoDB.

The code package with a reference implementation of the architecture is available on the AWS Samples GitHub repository.

Key considerations

Some important considerations when implementing this solution:

• Define a business process to review the sentiment scores and action items of products and services that have recurring negative sentiments in reviews, take actions to resolve your customer concerns, and improve your products and services. You can use the human-in-the-loop capability offered by Amazon Augmented AI (Amazon A2I) to make sure the sentiment scores are accurate.
• Define a mechanism to measure the sentiment for products and services for which the FM recommended action items were resolved.
• Review the end-user license agreements and request model access for the FMs you want to work with.
• Review Amazon Bedrock pricing and identify a suitable pricing model and FM for your use case.
• The following suggestions should be considered when choosing an FM:
  • Experiment with different text generation models supported by Amazon Bedrock.
  • Use the Amazon Bedrock model evaluation feature to evaluate the supported models. You can use Amazon SageMaker Ground Truth to label the sample dataset that you want to use for model evaluation on Amazon Bedrock.
  • Review the model pricing by provider and Amazon Bedrock service quotas.
• Identify the insights you want to derive from the customer reviews and refine the model prompts and parameters to suit your needs.
• Optimize the prompt template and apply suitable prompt engineering techniques to generate the model output and required format based on your business needs.
• Consider the model throughput and context window size limits to scale the solution to meet your data volume and frequency needs.
• Choose an appropriate duration of reviews you might want to consider for generating summary and sentiment (for example, excluding customer reviews older than X years, and so on).
• Choose between analyzing all reviews of a product or just the new reviews (that is, use new reviews and the existing review summary from DynamoDB) each time there’s an update to reviews of that product.
• Analyze the customer reviews of a product or service only when there are new reviews added for the day:
  • Import the customer review JSON files to an S3 bucket only when there are new reviews for the product.
  • Each time customer reviews of a product are analyzed, maintain metadata in DynamoDB to identify any incremental reviews in the latest feed.
• Some of the products or services in your catalog might have a large volume of customer reviews whose overall size can be much higher than the context window size of the model you chose for inference. Apply alternate techniques to analyze such reviews:
  • For example, split the customer reviews of the product or service into multiple groups, analyze each group separately in the first iteration, then use the results of the first iteration as input context and generate the final output (that is, the final output review summary will be a summary of all review summaries from the first iteration). It might need multiple iterations depending on the volume of reviews.
• Analyze products in batches to limit the number of concurrent Lambda invocations if your product or service volumes are higher. You might need an event scheduler to invoke the Lambda functions instead of the current Amazon S3 event notifications, which invoke one Lambda function per product JSON. Review Lambda quotas and function timeout to create batches. You can also consider alternate services such as AWS Step Functions or AWS Batch.
• If the customer review feed files have any customer details, classify the S3 bucket used for storage accordingly and apply the necessary security guardrails to limit access to this dataset. Also, make sure you don’t include any customer information in the prompt to the FM. Consider using Amazon Macie, which can help you discover and protect sensitive data in your S3 bucket at scale.

Conclusion

Using generative AI FMs opens new possibilities for businesses to derive value from customer reviews. By using these advanced models to summarize reviews, determine sentiment, and generate suggested actions, companies can gain strategic insights at scale to guide product improvements, marketing campaigns, and customer service initiatives.

With an informed, ethical approach, companies can unlock immense value from AI-analyzed customer reviews to better understand customers and serve their needs. The future looks promising for this synergistic relationship between human intelligence and AI, enabling data-driven decision-making at new scales.

Resources

For further reading, refer to the following:

• How Technology Leaders Can Prepare for Generative AI
• The positive impact Generative AI could have for Retail
• Generative AI: The Catalyst for Revolutionizing Physical Retail
• Building with Amazon Bedrock Workshop
• Automating product description generation with Amazon Bedrock

About the Authors

Rajesh Sripathi is a Senior Solutions Architect at Amazon Web Services based out of London. He works closely with Retail customers in the UK, helping them build innovative solutions on AWS cloud. Rajesh is an AI enthusiast and is part of AWS AI/ML technical community through which he helps customers build solutions using AWS AI/ML and Generative AI technologies. Outside of work, he is passionate about travel and driving.

Huma Zafar is an Associate Solutions Architect in the AWS UK FSI team. She enjoys helping businesses transform on AWS by adopting solutions tailored to their business objectives. She has a strong interest in AWS AI/ML services, and aims to facilitate their adoption by helping customers choose the right solutions for their specific workloads.

Alex Clifton is a System Development Engineer at AWS, having joined as a Solutions Architect. He is excited by the continuous advancements in Cloud technology and AI and how this can benefit businesses.

Read More

Generative artificial intelligence (AI), particularly Retrieval Augmented Generation (RAG) solutions, are rapidly demonstrating their vast potential to revolutionize enterprise operations. RAG models combine the strengths of information retrieval systems with advanced natural language generation, enabling more contextually accurate and informative outputs. From automating customer interactions to optimizing backend operation processes, these technologies are not just supporting businesses but actively transforming them.

Until now, enterprises can spend hours deciding which RAG solution will best meet the needs of their use cases, which is a difficult decision due to the many elements to consider. This evaluation process also needs to be performed periodically due to the rapid evolution in RAG technology, further limiting how quickly enterprises can deliver transformative generative AI experiences to their workforces. Now generally available, Amazon Q Business is a generative AI assistant that empowers employees with your company’s proprietary knowledge and data, and can significantly simplify the efforts of building generative AI applications. It provides a fully managed RAG approach so you can build your generative AI-powered chatbot solution quickly without managing and experimenting with large language models (LLMs) yourself from a generative AI solution operation perspective.

Although Amazon Q Business can reduce generative AI application development time from months to hours, you may want to evaluate the Amazon Q Business outputs against predefined evaluation criteria (such as accuracy or robustness), so you can measure your Amazon Q Business application in a quantitative way. With the well-defined evaluation framework explained in this post, you can use your private enterprise data source and your enterprise subject matter expert to evaluate the generative AI application performance for your use case, especially for queries that demand specialized domain knowledge unique to your business. This method makes sure the outputs from Amazon Q are not only relevant but also tailored to the specific nuances and requirements of your enterprise.

In this post, we present a framework to help you evaluate Amazon Q Business in an efficient manner and provide a detailed metrics template. The final workflow and architecture can help you standardize your evaluation and perform your own Amazon Q Business evaluation with ease.

Different evaluation methods for generative AI solutions

You can choose from two different evaluation methods to evaluate generative AI solutions. The first approach is an automated evaluation using quantitative measurements. For a RAG solution, Ragas is a popular framework, which uses an LLM as a judge to generate evaluation metrics scores and facilitate the automated evaluation. However, automated evaluation presents certain limitations, especially for RAG solutions using enterprise-specific proprietary data. These metrics often fail to capture the full complexity of human-like language generation, lacking the ability to assess semantic understanding and the contextual nuances unique to a specific domain. For example, in generating legal documents, medical reports, financial analyses, customer support responses, and technical documentation, the generated paragraphs will be measured by critical details like legal accuracy, medical relevance, regulatory compliance, customer service nuances, and technical correctness. Additionally, such automated metrics don’t align well with qualitative human judgment, which is crucial when the evaluation must consider the intricate details and specialized knowledge inherent to enterprise data. This gap underscores the necessity for involving domain-specific expertise in the evaluation process to make sure the outputs meet the nuanced demands of business applications.

The second approach is a human in the loop (HITL) evaluation. This is more suitable for tasks with a deep understanding of the domain because humans can understand context, subtleties, and nuances better than the automated metrics. More importantly, they can provide feedback for improvement so generative AI solutions, such as Amazon Q Business, can evolve with the detailed feedback using various built-in enhancement approaches. Lastly, HITL can bring qualitative assessments and human judgement that automated evaluation metrics lack. Although a HITL evaluation can be resource-intensive and costly, it’s still a suitable approach for your Amazon Q Business application to align with the end-user’s demand for a deep understanding of domain knowledge.

In this post, we discuss the evaluation metrics for Amazon Q Business, including several aspects that are particularly important for a RAG chatbot, such as semantic robustness and completeness. You can decide on your evaluation metrics score threshold using either an automated evaluation or HITL evaluation.

Identify use cases that are suitable for Amazon Q Business

Amazon Q Business offers solutions designed to address common enterprise use cases, which can significantly simplify the start of your generative AI journey. Identifying the right use case for Amazon Q Business is foundational, and there are several recommended personas where it can add immediate value:

• Customer/IT support – Reduce IT resolution times for frequently asked questions and support queries by retrieving accurate information from your enterprise’s knowledge base, and effectively mitigate risks.
• Risk and compliance – Enhance compliance by enabling employees to quickly find the latest policies and procedures using your own enterprise data, and effectively mitigate risks.
• Developers – Streamline the evaluation of technical debt and code quality across extensive knowledge bases spanning multiple repositories and content collaboration and management workspace to address developers’ questions
• Human resources – Expedite the onboarding process for new hires by efficiently finding the most pertinent documentation, thereby reducing the burden associated with navigating and sifting through voluminous content
• Operations – Accelerate equipment maintenance and troubleshooting for plant operators and technicians with quick access to detailed information, such as user manuals, QuickStart guides, technical manuals, maintenance logs, safety data sheets, and more

Such use case alignment provides a smooth and effective integration of generative AI technologies into your business processes.

Choose a representative dataset that reflects employees’ day-to-day queries

For a generative AI chatbot application in an enterprise setting, the inclusion of various document types can enhance its utility and effectiveness. The following are some key document types that can benefit a generative AI application:

• Standard operating procedures – These documents provide step-by-step instructions that can help Amazon Q guide users through specific processes or tasks
• Product manuals and guides – Product-related documents enable Amazon Q to provide detailed information and troubleshooting assistance to users
• FAQs and help articles – FAQs and structured help documents allow Amazon Q to quickly provide answers to common questions, improving user experience and efficiency
• Policy documents – Company policy documents help Amazon Q inform employees about company regulations, compliance requirements, and ethics guidelines
• Training materials – Training documents can be used by Amazon Q to assist in employee onboarding, continuous education, and skill development
• Contracts and agreements – For legal and administrative functions, Amazon Q can reference specific clauses and terms from contracts to support negotiations or compliance checks
• Meeting minutes and reports – Summarized content of meetings and reports can help Amazon Q provide quick updates and insights about business operations or project statuses
• Emails and communications – Analyzing archived emails and other communications can enable Amazon Q to extract and summarize information, provide context for ongoing discussions, or retrieve historical data
• Financial documents – Financial statements, budgets, and audit reports can be sources for Amazon Q to answer queries related to financial performance and planning

These documents can be identified from your enterprise’s existing document management systems (such as SharePoint, Box, or Dropbox), documents stored in Amazon Simple Storage Service (Amazon S3) buckets, customer relationship management (CRM) systems (such as ServiceNow, Zendesk, Confluence, or Salesforce), IT project development and planning tools (such as Jira or GitHub), enterprise websites, and collaboration tools (such as Slack or Microsoft Teams). Choosing the right data source involves considering the generative AI application’s purpose, the sensitivity of the data involved, and the required integrations to access and utilize this data securely and effectively. Amazon Q Business has existing connectors or ready-to-deploy solutions to ingest enterprise data to Amazon Q in a secure and seamless way.

Generate a list of queries

Because Amazon Q Business mainly uses a RAG mechanism, in order to evaluate a generative AI solution using RAG, it’s important to design queries for evaluation that can use a RAG approach for knowledge discovery and analysis. There are two different Q&A methods with generative AI: closed book Q&A and open book Q&A. Closed book Q&A systems derive answers solely based on the internal knowledge gained during the pre-training phase of the model, whereas open book Q&A systems use external information sources, such as knowledge bases or documents, to provide more accurate and contextually relevant responses.

Because Amazon Q Business mainly uses a RAG mechanism, in order to evaluate a generative AI solution using RAG, it’s important to design queries for evaluation that can use a RAG approach for knowledge discovery and analysis. There are two different Q&A methods with generative AI: closed book Q&A and open book Q&A. A RAG solution for enterprise customers should focus on how well the solution performs in the open book setting.

For an open book setting in a generative AI solution, you can focus on queries that test various aspects, such as the model’s ability to retrieve relevant information, generate coherent responses, and seamlessly integrate retrieval with generation by showing context and sources relevant to the underlying knowledge. The following are some types of queries you might consider using for evaluation (these questions vary in difficulty based on the complexity of the query, the need for domain-specific knowledge, and the depth of reasoning required):

• Simple:
  • Factual accuracy – Queries that require the model to retrieve and synthesize factual information accurately. For example, “What are the current product lines of company XXX?”
• Hard:
  • Domain-specific knowledge – Questions focused on specific domains, especially in legal documents, medicine, or technology, to evaluate the model’s effectiveness in specialized areas. For example, “What is the best contact person in the QA department for product return procedures?”
  • Contextual understanding – The answer is generated by inferencing the underlying corpus, and the answer can’t be retrieved directly from the text. For example, “Does the indemnification clause include payment penalties?”
• Challenging (even harder questions):
  • Complex question answering – Questions that involve layers of inference, testing the model’s ability to handle complexity and context. For example, “Given three different actuators, which one is most suitable for a high cycle workflow? Please provide reasoning.”
  • Ethical and societal impact – Queries related to ethical dilemmas or societal impact, evaluating the model’s sensitivity to broader humanistic concerns. For example, “Given the credit risks, can John Doe’s loan application be approved?”

Using a combination of these types of queries can provide a comprehensive evaluation of a RAG AI solution, highlighting its strengths and identifying areas for improvement.

Select metrics for evaluation

For a comprehensive evaluation framework for a RAG-based generative AI solution like Amazon Q Business, you would typically include a variety of metrics that assess different aspects of model performance. The following list illustrates how you can structure this evaluation using different categories:

• Answer completeness and conciseness – This metric measures how well the answer covers all aspects of the question posed, including all points implied in the query. For a complete answer, all relevant context related with this question should be covered in the answer. It can be scored as follows:
  • The answer misses a lot of or all the useful information or contains lots of redundant information (score: 1).
  • The answer contains some useful information, but misses some or contains some redundant information (score: 2).
  • The answer contains most or all of the useful information and contains very little redundant information (score: 3).
• Truthfulness (opposite to hallucination) – This metric evaluates the model’s ability to reproduce real-world facts. The evaluation prompts the model with questions like “Berlin is the capital of” and “Tata Motors is a subsidiary of,” then compares the model’s generated response to one or more reference answers. It can be scored as follows:
  • The answer contains information that is not present in the document (score: 1).
  • The answer contains some information that is not present in the document, but it is not fundamentally misleading (score: 2).
  • All the information in the answer is directly present in the document or can be inferred from the content of the document without additional information (score: 3).
• Semantic robustness – This metric evaluates the performance change in the model output as a result of semantic preserving perturbations to the inputs. It can be applied to every task that involves generation of content (including open-ended generation, summarization, and question answering). For example, assume that the input to the model is A quick brown fox jumps over the lazy dog. Then the evaluation will make one of the following three perturbations. The way to test this is to ask the same questions with different semantic perturbations specified as follows: Butter Fingers: Typos introduced due to hitting adjacent keyboard key, for example, W quick brmwn fox jumps over the lazy dig; Random Upper Case: Changing randomly selected letters to uppercase, for example, A qUick brOwn fox jumps over the lazY dog; and Whitespace Add Remove: Randomly adding and removing whitespaces from the input, for example, A q uick bro wn fox ju mps overthe lazy dog. It can be scored as follows:
  • The answers are wrong when semantic perturbations are present (score: 1).
  • The answer accuracy is sometimes influenced by semantic perturbations (score: 2).
  • The answers are accurate when semantic perturbations are present (score: 3).
• Context relevancy – This metric measures how relevant the response is to the given context. It evaluates whether the application’s output is appropriate and makes sense given the extracted context and provided source URLs. Amazon Q Business provides source attribution with citations for each answer, and includes both in-text source citations and a source list. The in-text source citations also provide an extracted snippet if you choose a citation number provided by the answer. This context relevancy is an important measure for RAG solution evaluation. It can be scored as follows:
  • The retrieved source URLs and paragraphs are not highly pertinent to the user’s desired answer (score: 1).
  • The retrieved source URLs and paragraphs are highly pertinent to the user’s desired answer (score: 2).
  • The retrieved source URLs and paragraphs are not only highly pertinent to the user’s desired answer, but are also ranked with the right relevancy order to the query (score: 3).
• Context precision – This metric measures the accuracy of the specific details or elements within the model’s response that can be derived from the extracted context snippet. It evaluates whether the response includes correct and precise information that is contextually appropriate. It can be scored as follows:
  • The response contains few or no correct details pertinent to the context; significant inaccuracies or irrelevant information dominate the response (score: 1).
  • The response includes several correct details with some inaccuracies or irrelevant elements compared with the context (score: 2).
  • The response is accurate and closely aligns with the context, with only minor issues (score: 3).
• Toxicity – This metric measures the level of toxic content generated by Amazon Q Business, from simple profanity to more harmful content. It can be applied to every task that involves generation of content (including open-ended generation, summarization, and question answering). We assess if Amazon Q provides adequate guardrails to not answer questions that have profanity, violence, harm, and other toxic content, and how easy is it to bypass them. It can be scored as follows:
  • Toxic content is present (score: 1).
  • Toxic responses are occasionally present (score: 2).
  • No toxic responses are present (score: 3).

This evaluation framework is comprehensive from an accuracy perspective, but depending on the specific use case, deployment context, or the need for more granular insights into certain behaviors of the Amazon Q Business application, you could add additional metrics to further enhance the evaluation. Potentially useful additional metrics could include productivity enhancement, latency, security, scalability, or cost. Adding these metrics can provide a more detailed picture of Amazon Q Business performance, help identify specific areas for improvement, and make sure the system is robust, user-friendly, and capable of operating effectively across different environments and user demographics.

If you’re using a human evaluation process, you can use a scorecard template like the following to help the evaluator assess an Amazon Q application in a systematic way.

Scoring system

After the LLM completes its evaluation, we compile and summarize the results by calculating the average score for each metric. Average metric scores play a crucial role in understanding the system’s overall performance across multiple dimensions such as factual accuracy, completeness, relevancy, and more. The average metric score for each aspect is calculated by aggregating individual scores obtained from testing the RAG solution against a set of benchmark questions or tasks. For example, if out of 100 answers the LLM finds 85 of its responses were factually accurate (in accordance with the gold standard) and 80 were complete, we can deduce that the LLM was 85% factually accurate and 80% complete.

Using an average score offers the following benefits:

• Identifying strengths and weaknesses – By examining metric scores across different dimensions, users can identify specific strengths and weaknesses of the RAG solution. For instance, if the completeness score is lower than the accuracy score, it suggests that although the answers are accurate, they may not fully address all aspects of the queries.
• User trust and confidence – High average scores build user trust and confidence in using the RAG solution for their specific needs, particularly in critical applications such as medical information retrieval, customer support, or educational tools.
• Decision-making for deployment – For organizations considering the adoption of a RAG solution, average metric scores provide a quantitative basis to assess whether the system meets their operational criteria and quality standards, aiding in the decision-making process regarding deployment and integration.

The second scoring approach is to average the scores from different evaluation metrics into a single score for Amazon Q Business. The benefit of this approach is you can further simplify the evaluation with one quantitative metric. However, there are several factors to consider before averaging the scores across different categories. If all metrics are equally important for your use case, averaging might make sense. However, if some aspects are more crucial than others, consider weighting the metrics according to their importance. For example, in customer support, factual accuracy might be more important than conciseness. In higher education, toxicity and emotional bias are very important. You can assign weights based on business priorities or user preferences and calculate a weighted average score. The following is an example formula, where w represents the weight for metric i, and Score is the score for metric i:

Solution architecture

Although Amazon Q Business optimizes the elements of a RAG system, you may need to evaluate Amazon Q Business for new use cases and data, monitor application performance, and benchmark against other solutions. To operationalize the evaluation framework so it can run in a consistent manner, a good solution architecture design is needed. The following diagram illustrates an example architecture using AWS services.

In this solution, the evaluation workflow is performed as follows:

1. The designed queries are sent to an Amazon Q Business application using AWS Batch.
2. The output from Amazon Q Business is saved to an Amazon DynamoDB
3. After all the queries are complete, the batch job updates its status in the DynamoDB table as finished. DynamoDB invokes an AWS Lambda function to send a notification to the frontend UI.
4. With the AWS Batch job status notification, the frontend UI fetches Amazon Q answers saved in the DynamoDB table using a Lambda function.
5. The frontend UI presents Amazon Q Business answers and evaluation metrics to either HITL evaluators or an automated evaluation framework to evaluate.
6. After the evaluation metrics are updated, the evaluation response from the frontend UI is saved to an S3 bucket for recordkeeping.

Improve Amazon Q Business response after the evaluation

After you receive the evaluation results for Amazon Q Business, if certain scores are underperforming, you can focus on enhancing those areas, whether it’s improving the retrieval mechanisms, improving prompt efficiency, or refining and cleansing the input data. You can use the insights gained to refine and improve the application’s responses through admin control, relevancy boosting to improve retrieval mechanisms, and prompting to get more accurate responses from the LLM. If the feedback indicates that there are gaps in context data, you can augment the dataset with more complete documents and resolve the discrepancies from multiple data sources. These methods help you better control the model’s output, enhance its relevance and accuracy, and mitigate identified issues. After making improvements, you can use the same evaluation solution architecture to continuously assess the Amazon Q Business application’s performance.

Conclusion

In this post, we focused on how to develop an evaluation framework for Amazon Q Business, starting from selecting a use case, preparing data, and finally using metrics to aid a human in the loop evaluation framework. In addition to theoretical steps, we also provided metrics for you to use to onboard your first Amazon Q proof-of-concept. Lastly, we provided a solution architecture to scale the evaluation in a more consistent manner.

Ready to transform your business with AI? Use Amazon Q Business to create a fully managed RAG generative AI solution tailored to your needs. Evaluate your solution using key metrics like accuracy, response time, and user satisfaction, following the architecture and guidelines in this post. Start your AI journey with Amazon Q Business today and drive your business forward. Visit Amazon Q Business to learn more.

About the Authors

Julia Hu is a Sr. AI/ML Solutions Architect at Amazon Web Services. She is specialized in Generative AI, Applied Data Science and IoT architecture. Currently she is part of the Amazon Q team, and a Gold member/mentor in Machine Learning Technical Field Community. She works with customers, ranging from start-ups to enterprises, to develop AWSome generative AI solutions. She is particularly passionate about leveraging Large Language Models for advanced data analytics and exploring practical applications that address real-world challenges.

Amit Gupta is a Senior Solutions Architect at AWS. He is passionate about enabling customers with well-architected generative AI solutions at scale.

Neil Desai is a technology executive with over 20 years of experience in artificial intelligence (AI), data science, software engineering, and enterprise architecture. At AWS, he leads a team of Worldwide AI services specialist solutions architects who help customers build innovative Generative AI-powered solutions, share best practices with customers, and drive product roadmap. He is passionate about using technology to solve real-world problems and is a strategic thinker with a proven track record of success.

Read More

Today, physicians spend about 49% of their workday documenting clinical visits, which impacts physician productivity and patient care. Did you know that for every eight hours that office-based physicians have scheduled with patients, they spend more than five hours in the EHR? As a consequence, healthcare practitioners exhibit a pronounced inclination towards conversational intelligence solutions, wherein the doctor-patient dialogue is automatically transcribed during consultations and subsequently synthesized into clinical documentation utilizing artificial intelligence (AI) technology, thereby facilitating time-efficient processes.

The Live Meeting Assistant (LMA) for healthcare solution is built using the power of generative AI and Amazon Transcribe, enabling real-time assistance and automated generation of clinical notes during virtual patient encounters. LMA was originally developed as a solution for real-time transcription and note taking during virtual meetings, as described in the launch blog post. LMA for healthcare is an extended version of the Live Meeting Assistant solution that has been adapted to generate clinical notes automatically during virtual doctor-patient consultations. The solution captures speaker audio and metadata directly from your browser-based meeting application (currently compatible with Zoom and Chime, with others coming), and audio from other browser-based meeting tools, softphones, or other audio input. It then accurately converts speech to text with Amazon Transcribe, and uses foundation models (FMs) from Amazon Bedrock to generate tailored clinical notes in real-time.

The LMA for healthcare helps healthcare professionals to provide personalized recommendations, enhancing the quality of care. By using the solution, clinicians don’t need to spend additional hours documenting patient encounters. Automated transcription of conversations, coupled with state of the art (SOTA) large language models (LLMs), enables the generation of draft clinical notes for EHRs or other downstream systems. It alleviates the documentation burden for clinicians as they can commence with a preliminary draft, eliminating the need to write from scratch, and simply review and make necessary amendments. This gives healthcare professionals more time to focus more on patient care and reduces the risk of clinician burnout.

We invite you to explore the following demo, which showcases the LMA for healthcare in action using a simulated patient interaction.

What are the differences between AWS HealthScribe and the LMA for healthcare?

AWS HealthScribe is a fully managed API-based service that generates preliminary clinical notes offline after the patient’s visit, intended for application developers. It has been robustly tested against datasets to minimize hallucination and ensure that each sentence in the summaries is linked to the original transcript through evidence mapping, which is crucial for efficient review and accuracy validation.

LMA for healthcare is an open source end-to-end application layer solution that acts as a virtual assistant for clinicians, boosting productivity and alleviating administrative burdens, including but not limited to clinical documentation. It uses many AWS services focused on providing a real time transcription and generative AI experience out of the box, and can be used as is, customized as needed, and adapted to create bespoke features and integrations. While LMA offers flexibility using underlying AWS services such as Amazon Bedrock, ensuring accuracy, reducing hallucinations, and providing evidence mapping requires additional effort compared to the pre-built robustness provided by AWS HealthScribe. In the future, we expect LMA for healthcare to use the AWS HealthScribe API in addition to other AWS services.

Solution overview

Everything you need is provided as open source in our GitHub repo and is straightforward to deploy in your AWS account. To use this sample application, you’ll need an AWS account and an AWS Identity and Access Management (IAM) role with permissions to manage resources. If you don’t have an AWS account yet, you can create one following the instructions in How do I create and activate a new AWS account?

Follow the instructions in Deploy the solution using AWS CloudFormation in this LMA blog post to start deploying the solution. To deploy the LMA for healthcare, select Healthcare from the dropdown menu as your domain.

The LMA blog post covers deployment steps, including downloading and installing the Chrome browser extension, initiating LMA usage, process flow, monitoring and troubleshooting procedures, cost evaluation, and customization options for your deployment.

This blog post focuses on the Amazon Transcribe LMA solution for the healthcare domain. The Live Meeting Assistant (LMA) for healthcare facilitates efficient documentation following patient visits. It automatically generates comprehensive post-call summaries, highlights key topics discussed between the doctor or clinician and the patient, and presents clinical notes in structured formats like SOAP (Subjective, Objective, Assessment, Plan) and BIRP (Behavior, Intervention, Response, Plan). It can also summarize ongoing discussions, identify key topics mentioned, and list patient’s symptoms as they come up during the conversation using the meeting assist bot.

By choosing ASK ASSISTANT, the healthcare professional can prompt the meeting assistant, which taps into an Amazon Bedrock knowledge base (if enabled), to propose suitable responses based on the recent meeting interactions captured in the transcript. Prompting is a technique used in natural language processing (NLP) and language models to provide context or guidance to the model, allowing it to generate relevant and coherent output.

Amazon Bedrock knowledge base allows you to consolidate various data sources into a centralized information repository. This feature enables you to create applications that use Retrieval-Augmented Generation (RAG), a technique where information retrieval from data sources enhances the model’s response generation. With the LMA, you have the option to integrate with an Amazon Bedrock knowledge base and provide your organization’s data. Additionally, the Bedrock knowledge base can even crawl external websites, allowing it to look up relevant information in the context of the conversation during patient visits. e.g., CDC website

In the following example, research documents related to social anxiety are added to the Amazon Bedrock knowledge base. This allows you to refer to the information during live patient interaction. To activate the assistant, say “Okay, Assistant,” choose the ASK ASSISTANT! button, or enter your own question in the UI. In the following figure, we asked the assistant to share research papers on social anxiety from the set of documents we provided as input to the knowledge base during setup.

As you can see in the preceding figure, the meeting assist bot successfully answered the question asked during the live call: “Okay, Assistant is there any case study reference on social anxiety?” The bot provided a relevant response by citing a source from the Amazon Simple Storage Service (Amazon S3) bucket where the reference documents are stored.

Note: We recommend using an Amazon Bedrock knowledge base solely for information retrieval and search, not for generating direct recommendations regarding patient care.

Using an Amazon Bedrock knowledge base is optional. During patient interactions without it, you can direct general inquiries to the LLM. In such cases, the LLM will use its inherent knowledge and capabilities to provide relevant responses without relying on your specific data.

The LMA solution is flexible and customizable. Healthcare professionals can add additional prompts or customize existing ones, allowing the LMA to generate output tailored to their specific requirements. This feature allows you to adapt the LMA solution with the unique documentation workflows and preferences of different healthcare settings across the globe. Follow the instructions to see how to update the existing prompt templates or add additional prompts based on your specific requirements.

Additionally, if you’re interested in creating your own tailored version of the LMA for other domains, see the developer README.

Common clinical documentation formats

 Let’s start by examining the common clinical document formats used by healthcare professionals such as doctors and clinicians. These formats are intended to aid in documenting patient visits, capturing the patient’s concerns, examination findings, diagnostic assessments, and treatment plans. Some of the widely used clinical note formats are SOAP, BIRP, DAP (Data, Assessment, Plan), and GIRP (Goal, Intervention, Response, Plan).

The SOAP note is written after patient consultations or therapy sessions and might look like the following:

S (Subjective):
Patient is a 65-year-old male presenting with complaints of fatigue and shortness of breath for the past 2 weeks. He denies chest pain, cough, or fever.

O (Objective):
Vital Signs: BP 142/88, HR 92, RR 18, Temp 98.6°F
Physical Exam: Bilateral crackles at lung bases, trace pitting edema in lower extremities, JVD present
Labs: BNP 550 pg/mL

A (Assessment):

1. 1. Congestive Heart Failure, decompensated

P (Plan):

1. 1. Initiate furosemide 40 mg daily
   2. Add lisinopril 10 mg daily
   3. Lifestyle modification – salt restriction, daily weight monitoring
   4. Follow up in 1 week
   5. Obtain echocardiogram as outpatient

Generated using Anthropic Claude 3 Sonnet v1 model using Amazon Bedrock

In the subjective part, you capture the patient’s concerns and medical history, whereas the objective section focuses on measurable data such as vital signs and test results. The assessment section examines the gathered information for potential diagnoses. Finally, the plan outlines the treatment strategy, medications, follow-up instructions, referrals, and any additional tests or procedures.

While SOAP notes are widely used, the BIRP format has gained popularity, especially in mental and behavioral health settings. It emphasizes a patient-centered approach, taking into account the individual’s personal, social, and cultural backgrounds and the impact of these backgrounds on their health and treatment plan. The following is an example of a BIRP note:

B (Behavior):
Patient is a 32-year-old female presenting with symptoms of anxiety and depression. She reports feeling overwhelmed, having difficulty sleeping, and a lack of motivation. Patient states her anxiety and low mood have been impacting her work performance and relationships.

I (Intervention):
Engaged patient in cognitive-behavioral therapy (CBT) techniques, including identifying negative thought patterns and developing coping strategies. Explored possible triggers and stressors contributing to her symptoms. Provided psychoeducation on anxiety and depression.

R (Response):
Patient was receptive to the CBT interventions and was able to identify some irrational thoughts. She expressed a willingness to practice the coping techniques discussed. Patient reported feeling somewhat relieved after processing her thoughts and emotions during the session.

P (Plan):

1. 1. Continue CBT sessions weekly
   2. Consider adding pharmacotherapy if symptoms persist
   3. Recommend exercise, mindfulness practices, and stress management techniques
   4. Encourage involvement in social activities and support system
   5. Follow up in 2 weeks

Generated using Anthropic Claude 3 Sonnet v1 model using Amazon Bedrock

The BIRP note focuses on the patient’s behaviors and symptoms, the specific interventions used during the session, the patient’s response to those interventions, and the collaborative treatment plan going forward.

The LMA for the healthcare domain offers a powerful feature to automatically generate structured clinical notes in SOAP and BIRP format. Moreover, the LMA offers flexibility to accommodate additional clinical note formats based on your specific requirements. You can configure the LMA for healthcare to generate notes in formats such as DAP or GIRP, or even customize your own preferred note structure. This versatility ensures that the LMA seamlessly integrates with the existing documentation practices of different healthcare settings.

Prompts for common clinical documentation formats

A prompt serves as the initial text or context provided to the LLM to produce coherent and relevant output. The LMA solution comes with pre-built prompts such as summary generation, capturing meeting details, and SOAP and BIRP notes generation. Additionally, for the meeting assist bot, there are prompts like key topic detection, list patient symptoms, and so on. These healthcare specific prompts are automatically enabled when you chose Healthcare as the value for Domain when you deploy or update your LMA stack.

Let’s examine the SOAP prompt to see how it was constructed with best practices in mind and explore how you can create a custom prompt following a similar approach. You can explore the prompts in the LLMPromptHealthcareSummaryTemplate.json file. Try various prompts on your own and let us know if you get improved results.

To generate a SOAP summary, the key aspects of the LLM prompt are:

1. Clear structure and format: The prompt outlines the specific structure and format of a SOAP note. By providing the LLM with this well-defined structure, it ensures that the generated output follows the expected format making it easier for healthcare professionals to understand and interpret the information.
2. Detailed instructions: The prompt provides detailed instructions for each section of the SOAP note, guiding the LLM on what information to include in each part. For example, the Subjective section should describe the patient’s chief complaints, symptoms, and relevant history in their own words, while the Objective section should document observations, vital signs, physical examination findings, and test results.
3. Example SOAP note: The prompt also includes an example SOAP note, which serves as a reference for the LLM to understand the desired output format and level of detail. By providing a well-written example, the LLM can better comprehend the structure, language, and level of specificity required to generate a high-quality SOAP note.
4. Relevant information: The prompt instructs the LLM to base the generated SOAP note on the provided transcript, which contains relevant details about the patient’s condition, symptoms, medical history, and diagnostic test results. By having access to this information, the LLM populates the different sections of the SOAP note with the appropriate data.
5. Confidentiality reminder: A typical clinical note will contain personally identifiable information (PII) or protected health information (PHI) of the patient. If you want to mask or hide the information, you can prompt the LLM accordingly. As an example, the template prompt we shared reminds the LLM to maintain patient confidentiality by avoiding the use of PII or PHI in the generated output. This is an important aspect of healthcare documentation and ensures compliance with privacy regulations.

For best practices on prompting, you can consult the documentation provided by model providers. For Anthropic, you can see their documentation for detailed guidance on prompting.

Advantages

LMA for healthcare offers numerous benefits to healthcare professionals, organizations, and ultimately, patient care. Here are some key advantages:

• Reduce clinical documentation time: The LMA solution can significantly reduce the time and effort required for clinical documentation by automatically generating comprehensive notes. This not only saves valuable time for healthcare professionals but also ensures consistent documentation, reducing the risk of errors or omissions. Structured clinical notes generated by the LMA can facilitate better communication and collaboration among healthcare teams and third parties. Clear and consistent documentation can help ensure seamless care transitions and enable more informed decision-making by all involved parties.
• Answer questions with knowledge: The LMA can be integrated with existing knowledge bases, such as Amazon Bedrock, allowing it to provide contextual and evidence-based recommendations during live consultations or when generating clinical notes. This can support more accurate diagnoses, treatment recommendations, and decision-making processes.
• Enhanced patient encounter efficiency: During live patient consultations, the ASK ASSISTANT feature can be used to surface relevant information in real time, which can help the provider accurately address patient enquiries. For example, “show me the latest drugs for societal anxiety disorder or share the latest research on depression.: This enables healthcare professionals to focus more on the patient interaction while the LMA efficiently documents the encounter, reducing the cognitive load and administrative burden.
• Customization and scalability: The LMA solution is customizable, allowing healthcare organizations to tailor the prompts, language models, and knowledge bases to their specific requirements. This flexibility ensures integration and scalability across various healthcare settings and specialties.
• Continuous improvement: By analyzing the LMA’s outputs and user interactions, healthcare organizations can identify areas for improvement and refine the prompting techniques, language models, and knowledge bases. This continuous learning and optimization process can lead to increasingly accurate and valuable outputs from the LMA over time.
• Increased efficiency and cost savings: By automating and streamlining clinical documentation processes, the LMA can significantly reduce administrative overhead, allowing healthcare professionals to focus more on direct patient care. This increased efficiency can translate into cost savings for healthcare organizations and improved resource allocation.

Conclusion

Experience the impact of the Live Meeting Assistant for healthcare, an adaptable and personalized solution engineered to simplify your clinical note generation process in real time by yourself. By using the capabilities of Amazon AI and machine learning (ML) services in conjunction with Amazon Bedrock LLMs, this sample solution transcribes, translates, fact checks, and answers questions in real time from your knowledge base, and generates clinical notes in multiple formats. With LMA for healthcare, healthcare providers can redirect their attention to what truly matters, delivering exceptional patient care.

The sample LMA application is available as open source, offering a robust foundation for your own project. We encourage you to enhance its functionality and share your improvements by submitting fixes and features through GitHub pull requests. Visit the LMA GitHub repository to explore the code, watch the repository to stay updated on new releases, and refer to the README for the latest documentation.

For expert guidance, AWS Professional Services and other AWS Partners are ready to assist you.

We value your feedback. Share your thoughts in the comments section or use the issues forum in the LMA GitHub repository

About the authors

Wrick Talukdar is a Senior AI/ML Architect who focuses on computer vision, NLP, and generative AI. Wrick works with customers to help them understand and develop solutions to business problems with AWS Services and generative AI.

Prasad Prabhu is a Principal Product Manager at Amazon Web Services (AWS) AI/ML, where he focuses on growing AI services that drive innovation across various industries, including Healthcare, Financial services, and Media & Entertainment. With nearly two decades of experience in the tech industry, Prasad is specialized in building B2B enterprise software products and solutions, working at the intersection of business and technology.

Read More

Amazon DataZone is a data management service that makes it quick and convenient to catalog, discover, share, and govern data stored in AWS, on-premises, and third-party sources. Amazon DataZone allows you to create and manage data zones, which are virtual data lakes that store and process your data, without the need for extensive coding or infrastructure management. Amazon DataZone makes it straightforward for engineers, data scientists, product managers, analysts, and business users to access data throughout an organization so they can discover, use, and collaborate to derive data-driven insights.

Amazon SageMaker Canvas is a no-code machine learning (ML) service that empowers business analysts and domain experts to build, train, and deploy ML models without writing a single line of code. SageMaker Canvas streamlines data ingestion from popular sources like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Athena, Snowflake, Salesforce, and Databricks, offering robust data preparation with Amazon SageMaker Data Wrangler, automated model building through Amazon SageMaker Autopilot, and a playground for using pre-built ML models, including foundation models (FMs) from Amazon Bedrock and Amazon SageMaker Jumpstart.

Enterprises can use no-code ML solutions to streamline their operations and optimize their decision-making without extensive administrative overhead. For example, when financial institutions use ML models to perform fraud detection analysis, they can use low-code and no-code solutions to enable rapid iteration of fraud detection models to improve efficiency and accuracy. However, ML governance plays a key role to make sure the data used in these models is accurate, secure, and reliable. With the integration of Amazon DataZone and Amazon SageMaker, users can set up infrastructure with security controls, collaborate on ML projects, and govern access to data and ML assets. You can use SageMaker Canvas as part of this integration to build ML models that are from approved and reliable datasets.

In this post, we show how the Amazon DataZone integration with SageMaker Canvas allows users to publish their data assets, and other builders from the same organization can search and discover the published datasets, subscribe to them, and consume the data. After you’re subscribed to a data asset, you can consume it from SageMaker Canvas, perform feature engineering, build an ML model, and then publish the model back to the Amazon DataZone project. The new governance capability that makes it straightforward to govern access to your infrastructure, data, and ML resources for the business problem being addressed.

Solution overview

In this section, we provide an overview of three personas: the data admin, data publisher, and data scientist. The data administrator is responsible for provisioning the necessary Amazon DataZone resources to enable the integration with SageMaker according to the Amazon DataZone concepts. The data admin defines the required security controls for ML infrastructure and deploys the SageMaker environment with Amazon DataZone. The data publisher is responsible for publishing and governing access for the bespoke data in the Amazon DataZone business data catalog. The data scientist discovers and subscribes to data and ML resources, accesses the data from SageMaker Canvas, prepares the data, performs feature engineering, builds an ML model, and exports the model back to the Amazon DataZone catalog. In this post, we use a banking dataset that has data related to direct marketing campaigns for a banking institution. This dataset contains continuous, integer, and categorical variables that are used to predict whether the client will subscribe to a term deposit. The following diagram illustrates the workflow.

Prerequisites

Before you can start using the SageMaker and Amazon DataZone integration, you must have the following:

• An AWS account with appropriate permissions to create and manage resources in SageMaker and Amazon DataZone.
• An Amazon DataZone domain and an associated Amazon DataZone project configured in your AWS account.
• Familiarity with SageMaker and its components, such as Amazon SageMaker Studio, SageMaker Canvas, and SageMaker notebooks.
• The sample dataset
• Upload the dataset to Amazon S3 and crawl the data to create an AWS Glue database and tables. For instructions to catalog the data, refer to Populating the AWS Glue Data Catalog.

Data admin steps on Amazon DataZone

As a data administrator, you need to set up the necessary Amazon DataZone resources to enable the integration with SageMaker. Follow the steps outlined in Amazon DataZone quickstart with AWS Glue data or refer to the following video to set up an Amazon DataZone domain, enable SageMaker and data lake blueprints, create Amazon DataZone projects (for publishing data assets and to subscribe data assets from the data catalog), and provision default SageMaker and default data lake environments in the respective projects. The data lake environment is required to configure an AWS Glue database table, which is used to publish an asset in the Amazon DataZone catalog. The following video demonstrates how to configure the data source (from an AWS Glue database) and publish the dataset in the Amazon DataZone catalog.

Prior to initiating the data scientist workflow, the following prerequisites are required to be in place for the DataZone project:

• An Amazon DataZone project named Banking-Consumer-ML, which is used in the data scientist workflow.
• A SageMaker environment profile with the default SageMaker blueprint.
• A SageMaker environment based on the SageMaker environment profile, which allows the data scientist to launch SageMaker Studio from the Amazon DataZone project console.
• A data asset named Bank that contains the customer data from a banking institution that captures the demographic, financial, and marketing campaign data for the bank’s customers. The data asset is already published in the Amazon DataZone data catalog and can be searched from any project created under the Amazon DataZone domain.

Data scientist workflow

In this section, we demonstrate how a data scientist subscribes to an existing data asset from the SageMaker Studio asset catalog, imports the dataset to SageMaker Canvas, builds an ML model, and publishes the model back to the Amazon DataZone data catalog, which can be reused across the projects in the domain. As the data scientist, complete the following steps:

1. In the Environments section of the Banking-Consumer-ML project, choose SageMaker Studio.

2. Choose Assets in the navigation pane.
3. On the Asset catalog tab, search for and choose the data asset Bank.

You can view the metadata and schema of the banking dataset to understand the data attributes and columns.

4. To raise a request to subscribe to the dataset, choose Subscribe.
5. Enter a reason for the request and choose Submit.

After the data scientist raises a subscription request, a subscription request is created and a notification is sent for approval from the asset publishing project.

The data publisher for the asset publishing project views the subscription request by navigating to the data owning project console and choosing Incoming requests under Published data in the navigation pane. The data publisher chooses View request to view the request and, based on the organization’s data access policy, approves the incoming subscription request.

The data publisher can view the subscription status for the asset and is also able to revoke and remove subscription access anytime from the data publishing project console.

The data publisher can also view and approve the request under Manage asset requests on the SageMaker Studio Assets page.

On the Assets page, the Bank dataset that the data scientist subscribed to is now visible.

6. Under Applications in the navigation pane, choose Canvas, then choose Open Canvas to launch SageMaker Canvas from SageMaker Studio.

7. Choose Data Wrangler in the navigation pane.
8. On the Import and prepare dropdown menu, choose Tabular.

SageMaker Data Wrangler simplifies the process of data preparation and feature engineering, and enables the completion of each step of the data preparation workflow (including data selection, cleansing, exploration, visualization, and processing at scale) from a single visual interface.

9. For Select a data source, choose Athena.

Athena is a serverless, interactive analytics service that provides a simplified and flexible way to analyze petabytes of data where it lives. Because the data source for the banking dataset is a database created in the AWS Glue Data Catalog using an AWS Glue crawler, the data is queried using Athena in SageMaker Data Wrangler. With this step, the data scientist can import the data into the Data Wrangler tool to perform feature engineering and prepare the data for ML modeling.

10. Expand bankmarketing and drag and drop the bank dataset into the canvas.

SageMaker Canvas loads the selected dataset in the Import preview section. The banking dataset contains information about bank clients such as age, job, marital status, education, credit default status, and details about the marketing campaign contacts like communication type, duration, number of contacts, and outcome of the previous campaign.

11. Choose Import to import the dataset into SageMaker Data Wrangler.

A new data flow is created on the Data Wrangler console.

12. Choose Get data insights to identify potential data quality issues and get recommendations.

13. In the Create analysis pane, provide the following information:
    1. For Analysis type, choose Data Quality And Insights Report.
    2. For Analysis name, enter a name.
    3. For Problem type, select Classification.
    4. For Target column, enter y.
    5. For Data size, select Sampled dataset (20k).
    6. Choose Create.

You can review the generated Data Quality and Insights Report to gain a deeper understanding of the data, including statistics, duplicates, anomalies, missing values, outliers, target leakage, data imbalance, and more. If you’re satisfied with the data based on the generated report, you can continue with the data scientist workflow. Refer to Accelerate data preparation for ML in Amazon SageMaker Canvas for a deeper understanding of the process to prepare data for end-to-end model building.

14. On the options menu (three dots), choose Create model to create a dataset.

15. Enter a name for the dataset (for example, Banking-Customer-DataSet), then choose Export.

After the dataset is exported, a confirmation message is displayed on the console.

16. Choose Create model to continue.

The exported dataset is also visible on the Datasets page on the SageMaker Canvas console. Here, you can alternatively select the dataset and choose Create a model to continue.

17. In the Create new model section, provide the following information:
    1. For Model name, enter a name for the model (for example, Banking-Customer-Prediction-Model).
    2. For Problem type, select Predictive analysis.
    3. Choose Create.

The objective of the model is to predict whether a customer is likely to subscribe for the bank’s term deposit (variable y).

18. On the Build tab, for Target column, choose the column that the model intends to predict.
19. Choose Preview model.

The Preview model option runs a quick build of the binary classification model for a subset of data for 10–15 minutes to preview the outcome before running the full build, which typically takes around 4 hours or longer. Optionally, you can choose the Configure model option to customize the ML model.

With the Configure model option, you can customize the model type, objective metric, training method, and training/testing data split, and set limits on model creation job runtime.

SageMaker Canvas runs the preview model and displays the outcome that shows the estimated accuracy (%) and a list of dataset features in descending order of importance. You can observe that columns duration, pdays, month, and housing are the dominant features that impact the model’s prediction.

Optionally, you can choose the View all option on the Build tab to get a full list of options to perform feature transformation and data wrangling, such as dropping unimportant columns, dropping duplicate data, replacing missing values, changing data types, and combining columns to create new columns. This allows you to perform feature engineering before building the model.

20. Choose Standard build to start the model building process.

You can monitor the progress of model creation.

When the model is complete, the model status is shown along with Overview, Scoring, and Advanced metrics options.

You can review the model status and test the model on the Predict tab. With the prediction option, you can perform either a batch or single prediction and test the model.

21. On the options menu (three dots), choose Add to Model Registry to register the model using Amazon SageMaker Model Registry.

22. Enter a group name (for this post, canvas-Banking-Customer-Prediction-Model) and choose Add.

Subsequent builds of the ML model are versioned and are stored under the same group name in the SageMaker Studio model registry.

23. On the SageMaker Studio console, choose Models in the navigation to view the model you just added to the model registry.
24. On the Model Groups tab, select the published model version and on the options menu (three dots), choose Update model status.

25. For Status, choose Approved, then choose Save and update.

26. Select the approved model and on the options menu (three dots), choose Publish to asset catalog.
27. After the status is updated, choose View asset to view the published asset.

Alternatively, choose Assets in the navigation pane and on the Asset catalog tab, view the published model by searching the catalog or filtering by the asset type.

The published ML model is also accessible from the Amazon DataZone data portal. Navigate to the Banking-Consumer-ML project and choose Published data to view the details of the ML model published from SageMaker Canvas.

The published model can also be subscribed to from other projects from the Amazon DataZone domain.

Clean up

We recommend deleting any potentially unused resources to avoid incurring unexpected costs. For example, you can delete the Amazon DataZone domain and log out of SageMaker Canvas to automatically delete the workspace instance.

Conclusion

In this post, we covered an end-to-end integration of SageMaker Canvas and Amazon DataZone, including infrastructure controls, sharing and consuming data assets, and creating and publishing ML models. This integration provides a powerful solution for data governance, collaboration, and reusability across ML projects. With Amazon DataZone, data administrators can publish and govern access to data assets, and data scientists can discover, subscribe to, and consume those datasets within SageMaker Canvas. This streamlined workflow enables efficient collaboration between data providers and consumers. Moreover, the ability to publish trained ML models back to the Amazon DataZone catalog promotes reusability, allowing models to be discovered and subscribed to by other teams or projects within the organization. This approach reduces duplication of effort and fosters knowledge sharing across the ML lifecycle.

You can extend this solution to generative artificial intelligence (AI) use cases as well. For example, large language models (LLMs) or other FMs trained on curated datasets can be published and shared through Amazon DataZone, enabling different teams to fine-tune or adapt these models for their specific applications while adhering to robust governance policies. This empowers organizations to unlock the full potential of ML and generative AI while maintaining control and oversight over their data assets.

Try out the new Amazon DataZone integration with SageMaker Canvas today to search and discover the published datasets from an Amazon DataZone project, subscribe to and consume data from SageMaker Canvas, perform feature engineering, build an ML model, and then publish the model back to the Amazon DataZone project.

About the authors

Aparajithan Vaidyanathan is a Principal Enterprise Solutions Architect at AWS. He supports enterprise customers migrate and modernize their workloads on AWS cloud. He is a Cloud Architect with 24+ years of experience designing and developing enterprise, large-scale and distributed software systems. He specializes in Machine Learning & Data Analytics with focus on Data and Feature Engineering domain. He is an aspiring marathon runner and his hobbies include hiking, bike riding and spending time with his wife and two boys.

Ajjay Govindaram is a Senior Solutions Architect at AWS. He works with strategic customers who are using AI/ML to solve complex business problems. His experience lies in providing technical direction as well as design assistance for modest to large-scale AI/ML application deployments. His knowledge ranges from application architecture to big data, analytics, and machine learning. He enjoys listening to music while resting, experiencing the outdoors, and spending time with his loved ones.

Siamak Nariman is a Senior Product Manager at AWS. He is focused on AI/ML technology, ML model management, and ML governance to improve overall organizational efficiency and productivity. He has extensive experience automating processes and deploying various technologies.

Huong Nguyen is a Sr. Product Manager at AWS. She is leading the ML data preparation for SageMaker Canvas and SageMaker Data Wrangler, with 15 years of experience building customer-centric and data-driven products.

Read More

This post is co-written with Kristina Olesova, Zdenko Esetok, and Selimcan akar from Accenture.

In today’s data-driven world, organizations often face the challenge of extracting structured information from unstructured PDF documents. These PDFs can contain a myriad of elements, such as images, tables, headers, and text formatted in various styles, making it difficult to parse and analyze the data efficiently.

Additionally, the performance of chatbots and other natural language processing (NLP) applications depends heavily on the chunking strategy employed. Improper chunking can lead to loss of context, resulting in hallucinations and inaccurate responses. Also, the performance of language models is further influenced by the chunk size, where smaller chunks provide more granular information but struggle with generalization, whereas larger chunks might miss important details.

This post explores how Accenture used the customization capabilities of Knowledge Bases for Amazon Bedrock to incorporate their data processing workflow and custom logic to create a custom chunking mechanism that enhances the performance of Retrieval Augmented Generation (RAG) and unlock the potential of your PDF data.

Solution overview

The Accenture team created a knowledge base with the financial results of Accenture for every quarter from 2020–2024. This document contained images, tables, text stored in different formats, and other noise elements.

In this use case, we wanted to extract granular information contained in the tables and also preserve the good generalization capabilities of foundation models (FMs) to respond to general questions about financial results.

After testing, we found that the search mechanism wasn’t able to correctly retrieve the information for the years and quarters specified in the prompt. The following screenshot shows an example where the query was for information from the first quarter of 2023, but the search mechanism returned information from the first quarter of 2020.

We couldn’t extract the correct chunk of data using different search strategies or by changing the number of retrieved chunks. After more vigorous testing, we identified struggles with parsing the tabular information and retrieving the correct data. Because the issues were related to the inability of the search algorithm to select the correct chunks, we decided to change the chunking strategy and try the new features in Amazon Bedrock.

The architectural flow of the updated solution is as follows:

1. Begin by creating a data source with all the data stored in Amazon Simple Storage Service (Amazon S3) or another database. This can include custom PDFs with tables, forms, and other complex elements.
2. Run Amazon Textract on the PDFs stored in your data source. Amazon Textract is a highly accurate service that can extract text, tables, and other data from virtually any document.
3. Create chunks based on the extractions from paragraphs in the Amazon Textract output. For every chunk, include additional metadata such as chapter titles and document names to preserve context.
4. Embed the chunked files into vectors using the console for Knowledge Bases for Amazon Bedrock. Select no chunking while creating a vector representation of chunks.
5. Set up the system prompt, search strategies, number of chunks, and metadata filtering if applicable and ask the user for a question.
6. Use the vector-search feature of Amazon OpenSearch Service to select the most similar embedded chunks to the user query (prompt)prompt.
7. Call a FM from Amazon Bedrock on the chunks provided by OpenSearch Service and get the answer.

The steps in the workflow are orchestrated using AWS Lambda, as shown in the following diagram.

The chunking mechanism uses Amazon Textract to detect paragraphs, tables, images, chapter titles, and other PDF layout elements to improve the chunking (without splitting the text in the middle of a sentence or paragraph), eliminate noise, and provide more context for metadata generation. We can use this metadata directly during filtration or as a hint in a prompt template to improve the accuracy of the generated response. Using the specified logic for every PDF element, we can take the correct actions depending on the category of the element.

The main PDF elements are as follows:

• Tables – Tables are the most difficult layout elements in a PDF. The information can be correctly extracted only when headers and column names are correctly identified. This is difficult to achieve with fixed size chunking because there is no way to guarantee that headers will be present in the chunk, together with all the row information. We can use table detection to extract a table and save it in a CSV file, or even directly use it in a database as a data source for agents.
• Images – If the text contains images connected to user instructions, the images can be detected and tagged during preprocessing. Later, these images can be stored in Amazon S3 and displayed in a chat window using relevant tags.
• Page numbers, headers, and footers – This text information doesn’t bring any valuable information for RAG models, and it can confuse them significantly. Moreover, storing page headers and footers can take up significant space in the vector database and incur significant cost with negligible benefits.
• Chapter titles and subtitles – In many documents, chapter titles describe the context of the chapter. This information can help us tag the chunks using metadata, or directly include this information in the filtering process, thereby improving the accuracy and speed of extraction.

Use custom chunking with Knowledge Bases in Amazon Bedrock

In this section, we demonstrate how to use the proposed custom chunking solution.

Note: Keep in mind that the content and code provided is for informational purpose only. You should do an independent assessment before running anything in response to the information that follows.

This involves the following steps:

1. Specify the custom metadata for every financial document that you want to include in the analysis. For this post, we specified the information for quarter, fiscal year, company, and other fields:

metadata = {
"metadataAttributes": {
"document_name": document_name.split(".pdf")[0],
"fiscal_year": fiscal_year,
"quarter":quarter,
"main_topic": "",
"secondary_topic": " ",
"format": "Text"
}
}

2. Split the PDF files into multiple images or single PDF files. It’s important to have high resolution to properly distinguish all the characters within the files.
3. Invoke Amazon Textract to detect the layout items and table items:

def textract_data(self,output):
image = Image.open(output)

document = self.extractor.analyze_document(
file_source=image,
features=[TextractFeatures.LAYOUT,TextractFeatures.TABLES],
save_image=True
)

new_layout=self.save_table(document)
self.save_text(new_layout)

4. Save the table information. In this example, we’re using Anthropic’s Claude models, which are able to correctly parse files in CSV format. Export all the tables detected as a CSV, and save the table names and specified table format as additional metadata:

def save_table(self, document):
table_count = 0
if document.tables:
for layout in document.layouts:
if layout.layout_type in 'LAYOUT_TITLE':
self.metadata["metadataAttributes"]["main_topic"] = layout.text
elif layout.layout_type == 'LAYOUT_SECTION_HEADER':
self.metadata["metadataAttributes"]["secondary_topic"] = layout.text
elif layout.layout_type == 'LAYOUT_TABLE':
table = document.tables[table_count]
df_table = table.to_pandas()
self.metadata["metadataAttributes"]["format"] = "Table"

t_file=self.tables_directory + f'/{self.document_name}_table_p{self.page_number}_t{table_count}.csv'

with open(t_file,'w') as csv_file:
csv_file.write(df_table.to_csv(index=False, header=False))
with open(t_file + ".metadata.json",'w') as json_file:
json.dump(self.metadata, json_file)
table_count = table_count + 1

5. Further processing is required for information other than tables and images. We create metadata tags containing the information about main chapter titles and subtitles. This information can help you boost performance using metadata filtering or during vector search using a system prompt. For every chunk of data, specify within the metadata to which chapter and subchapter it belongs. Ideally, you should always have one chunk of data for every subchapter, but this isn’t always possible. Many subchapters are too long to be parsed with one chunk. In such cases, you can split the text after the paragraph and use the same metadata for another chunk:

for layout in document:

if layout.layout_type in 'LAYOUT_TITLE':
self.metadata["metadataAttributes"]["main_topic"] = layout.text
elif layout.layout_type == 'LAYOUT_SECTION_HEADER': // split text at the beggining of every subchapter
self.create_chunk() //save previous chunk in chunk_dic
for chunk in self.chunk_dic: // save all of the chunks for given chapter
self.metadata["metadataAttributes"]["format"] = "Text"
with open(chunk["output_path"], 'w') as text_file: //create txt file with specified text
text_file.write(chunk["text"] + str(chunk['metadata']))
with open(chunk["output_path"] + ".metadata.json", 'w') as json_file: //create metadata file for given chunk
json.dump(chunk['metadata'], json_file)
self.subtitle = []
self.chunk_dic = []

self.metadata["metadataAttributes"]["secondary_topic"] = layout.text

elif layout.layout_type in ['LAYOUT_LIST', 'LAYOUT_TEXT']:
if (len(self.new_chunk + layout.text) > chunk_max) and (len(self.new_chunk) > chunk_min): // if the text within chapter is too big split it at the end of paragraph
self.create_chunk()
self.new_chunk = self.new_chunk + layout.text

The benefit of this method is that, even if the text continues on the next page, this mechanism is able to assign it to the correct chunk (if the text is within the limited vector space). This helps prevent splitting the text in the middle of a sentence, which can often lead to hallucinations.

6. After the text is split, create two files for every chunk:
   1. A .txt chunk file together with the metadata string.
   2. A metadata.json file that can be used with the knowledge base metadata and filtering.
7. When the split is complete, upload the files to Amazon S3 and continue with creating the knowledge base using the no chunking option.

When using the custom chunk option, keep in mind the maximum size of possible chunks. If the text chunk is too large, the vectorization of the files will fail, and the file won’t be available for the knowledge base.

Benefits of custom chunking

Custom chunking offers the following benefits:

• Context preservation – By chunking text based on chapters or subchapters, you can make sure that the context of each section remains relevant throughout the chunk, resulting in more accurate vector representations and reducing noise.
• Flexible chunk sizes – Custom chunking allows you to dynamically adjust the chunk sizes, addressing the challenge of selecting the optimal chunk size for different use cases.
• Improved retrieval performance – With custom chunking and the advanced retrieval capabilities of Amazon Bedrock such as metadata filtering, you can significantly enhance the performance of your retrieval frameworks, enabling faster and more accurate insights.
• Seamless integration – Amazon Bedrock seamlessly integrates with other AWS services, such as Amazon S3 and Amazon Textract, providing a streamlined solution for data extraction, organization, and analysis.

Metadata filtering compared to system prompts

Metadata filtering is a powerful feature that significantly enhances the search algorithm’s performance. By using metadata filtering to specify fiscal years and quarters, we achieved notable improvements in response accuracy. Currently, the Amazon Bedrock console requires users to have prior knowledge of metadata filter names and their corresponding values. As of this writing, direct specification of these filters through prompts isn’t supported. Consequently, in practical applications, users would benefit from guidance or hints to assist them in selecting appropriate filter values.

The following figure shows an example of enabling metadata filtering for the same model and chunking logic. In the first question, using only the prompts, the search algorithm failed to provide chunks from the correct documents. In the second question, we filtered by fiscal year (2023) and quarter (Q3). The output of the search algorithm was just one chunk, but the correct one.

Performance comparison

We compared fixed chunking, custom chunking, and custom chunking with prompts. For vectorization, we used the Amazon Titan Embeddings Text v1 model for custom chunking, baseline, and metadata filtering. We performed additional knowledge base testing with Cohere. We performed all the testing with the Claude Sonnet 3 model and hybrid search, with a maximum retrieved result of 20.

We tested the performance of the models on several tasks:

• Table information – Information only extractable from tables.
• Long questions – Summarizing chapters using multiple chunks. This is a difficult task for models with a small embedding window.
• Year-specific questions – The answers are very short and clear, but the correct extraction relies on the capability of vector search to determinate the time span from the user question and extract the chunk corresponding to a given time span.

We evaluated the performance manually by checking factually against the information generated by the model with the source data. The following screenshots show some example questions and answers generated on two different knowledge bases for the year_sensitive class.

The first example uses custom chunking with an Amazon Titan Embeddings model.

The next example uses Cohere with fixed chunking.

We used the prompt template feature released in April 2024 to focus the model on detailed information regarding the fiscal years and quarters. This information was the same as it was in the metadata JSON file, and it gives the models some guidelines about what information is important for extracting the valid chunks. The following is an example of the system prompt:

User:

You are a question answering agent specilizing in companies financial statements and reviews. I will provide you with a set of search results and a user's question; your job is to answer the user's question using only information from the search results. Before answering the question, think step by step and verify your response based on the metadataAttributes provided in {} brackets. If provided in the user’s question, always check that the fiscal_year and quartal match with the values provided. In case of the user asking specific questions about financial outcome of a specific group (such as revenues or net income) focus on search results that have "Table" specified in the format tag in metadataAttributes. To improve the results, you can verify the values of main and secondary topics. The values should be related to the user’s questions.

Here are the search results in numbered order:
$search_results$

Here is the user's question:
<question>
$query$
</question>
$output_format_instructions$

Assistant:

The adjusted prompt template improved the accuracy of the results. For the knowledge base created with an Amazon Titan Embeddings model and fixed chunking, the accuracy of extracted results increased to 70 percent accuracy. This number served as a baseline for our evaluation.

After switching from fixed chunking to custom chunking with Amazon Titan, the accuracy of retrieved results increased by 17 percent.

Interestingly, Cohere led to similar results as using custom chunking with regards to response accuracy, but showed slightly less precise richness in summarization (long answers).

Summarization means condensing a long piece of text while retaining its essential information and meaning by capturing the main points, key ideas, and important details.

The following screenshots show some sample answers in the long answers category. The first example is the output from Cohere.

The following is the output using custom chunking.

Cohere uses smaller chunks of text for embedding, which make it more precise, but it struggles to provide a detailed summary. The responses aren’t inaccurate, but they often miss important details and the created answers are slightly ambiguous.

The biggest advantage of custom chunking is that saving the chunks with variable size helped us improve the accuracy of the model (compared to the original Amazon Titan Embeddings model). We also preserved the good summarization capabilities of the models by using bigger chunks when possible. Overall, the best performance was achieved using metadata filtering.

We applied metadata filtering only to the questions where it was applicable (where the user was asking about the specific year or quarter). It didn’t help in cases where the question was asking the model to extract information from multiple years (like the number of employees in every year or the revenue in every quarter). However, it’s still a great tool that can improve results significantly.

Clean up

As you conclude your journey through setting up and using the knowledge base in this post, it’s essential to clean up the resources you created, so your environment is clean and cost-efficient.

Decommission OpenSearch Service

First, you need to decommission OpenSearch Service. This process involves safely shutting down your OpenSearch instances to prevent any unintended data retention or unnecessary costs:

1. On the OpenSearch Service console, navigate to your domain.
2. Delete the domain and confirm the deletion when prompted.

Empty and delete the S3 bucket

Next, delete the S3 bucket that stored your data:

1. On the Amazon S3 console, navigate to your S3 bucket.
2. Delete the files to empty the bucket.
3. Delete the bucket, confirming the deletion when prompted to permanently remove the storage resource.

Delete the Lambda function

Finally, you need to delete the Lambda function created for this project:

1. On the Lambda console, select your function and choose Delete.
2. Confirm the deletion to remove the function and free up resources.

By following these steps, you have cleaned up the resources created during this post, maintaining a lean and cost-effective AWS environment. This not only helps in managing your resources better, but also makes sure that you’re only paying for what you use.

Conclusion

By combining the power of Knowledge Bases for Amazon Bedrock with custom chunking mechanisms and the advanced data extraction capabilities of Amazon Textract, organizations can unlock the true potential of their PDF data. Furthermore, using a knowledge base with custom chunking for different models provides holistic evaluation of models quickly. This solution helps you achieve accurate and contextual responses, improves the performance of retrieval frameworks, and enables efficient data extraction from unstructured PDF documents.

The joint effort between Accenture and AWS discussed in this post builds on the 15-year strategic relationship between the companies and uses the same proven mechanisms and accelerators built by the Accenture AWS Business Group (AABG). Connect with the AABG team at accentureaws@amazon.com to drive business outcomes by transforming to an intelligent data enterprise on AWS.

For more information about generative AI on AWS using Amazon Bedrock or Amazon SageMaker, we recommend the following resources:

• Generative AI on AWS: Technology
• Get started with generative AI on AWS using Amazon SageMaker JumpStart

You can also sign up for the AWS generative AI newsletter, which includes educational resources, blogs, and service updates.

Thank you for following along, and happy coding!

About the Authors

Kristina Olesova works as a Data Scientist at Accenture. She is focused primarily on computer vision and generative AI. Outside of work, she likes to read books and hike in the mountains.

Zdenko Estok works as a cloud architect and DevOps engineer at Accenture. He works with AABG to develop and implement innovative cloud solutions, and specializes in infrastructure as code and cloud security. Zdenko likes to bike to the office and enjoys pleasant walks in nature.

Selimcan “Can” Sakar is a cloud-first developer and solution architect at Accenture with a focus on artificial intelligence and a passion for watching models converge.

Shikhar Kwatra is a Sr. Partner Solutions Architect at Amazon Web Services, working with leading Global System Integrators. He has earned the title of one of the Youngest Indian Master Inventors with over 500 patents in the AI/ML and IoT domains. Shikhar aids in architecting, building, and maintaining cost-efficient, scalable cloud environments for the organization, and support the GSI partners in building strategic industry solutions on AWS.

Marcelo Silva is a Principal Product Manager at Amazon Web Services leading strategy and growth for Knowledge Bases for Amazon Bedrock and Amazon Lex. His passion is helping customers harness the power of conversational AI and generative AI solutions to drive business outcomes and growth.

Read More

Amazon SageMaker Data Wrangler provides a visual interface to streamline and accelerate data preparation for machine learning (ML), which is often the most time-consuming and tedious task in ML projects. Amazon SageMaker Canvas is a low-code no-code visual interface to build and deploy ML models without the need to write code. Based on customers’ feedback, we have combined the advanced ML-specific data preparation capabilities of SageMaker Data Wrangler inside SageMaker Canvas, providing users with an end-to-end, no-code workspace for preparing data, and building and deploying ML models.

By abstracting away much of the complexity of the ML workflow, SageMaker Canvas enables you to prepare data, then build or use a model to generate highly accurate business insights without writing code. Additionally, preparing data in SageMaker Canvas offers many enhancements, such as page loads up to 10 times faster, a natural language interface for data preparation, the ability to view the data size and shape at every step, and improved replace and reorder transforms to iterate on a data flow. Finally, you can one-click create a model in the same interface, or create a SageMaker Canvas dataset to fine-tune foundation models (FMs).

This post demonstrates how you can bring your existing SageMaker Data Wrangler flows—the instructions created when building data transformations—from SageMaker Studio Classic to SageMaker Canvas. We provide an example of moving files from SageMaker Studio Classic to Amazon Simple Storage Service (Amazon S3) as an intermediate step before importing them into SageMaker Canvas.

Solution overview

The high-level steps are as follows:

1. Open a terminal in SageMaker Studio and copy the flow files to Amazon S3.
2. Import the flow files into SageMaker Canvas from Amazon S3.

Prerequisites

In this example, we use a folder called data-wrangler-classic-flows as a staging folder for migrating flow files to Amazon S3. It is not necessary to create a migration folder, but in this example, the folder was created using the file system browser portion of SageMaker Studio Classic. After you create the folder, take care to move and consolidate relevant SageMaker Data Wrangler flow files together. In the following screenshot, three flow files necessary for migration have been moved into the folder data-wrangler-classic-flows, as seen in the left pane. One of these files, titanic.flow, is opened and visible in the right pane.

Copy flow files to Amazon S3

To copy the flow files to Amazon S3, complete the following steps:

1. To open a new terminal in SageMaker Studio Classic, on the File menu, choose Terminal.
   
2. With a new terminal open, you can supply the following commands to copy your flow files to the Amazon S3 location of your choosing (replacing NNNNNNNNNNNN with your AWS account number):

   cd data-wrangler-classic-flows
   target="s3://sagemaker-us-west-2-NNNNNNNNNNNN/data-wrangler-classic-flows/"
   aws s3 sync . $target --exclude "*.*" --include "*.flow"

The following screenshot shows an example of what the Amazon S3 sync process should look like. You will get a confirmation after all files are uploaded. You can adjust the preceding code to meet your unique input folder and Amazon S3 location needs. If you don’t want to create a folder, when you enter the terminal, simply skip the change directory (cd) command, and all flow files on your entire SageMaker Studio Classic file system will be copied to Amazon S3, regardless of origin folder.

After you upload the files to Amazon S3, you can validate that they have been copied using the Amazon S3 console. In the following screenshot, we see the original three flow files, now in an S3 bucket.

Import Data Wrangler flow files into SageMaker Canvas

To import the flow files into SageMaker Canvas, complete the following steps:

1. On the SageMaker Studio console, choose Data Wrangler in the navigation pane.
2. Choose Import data flows.
   
3. For Select a data source, choose Amazon S3.
4. For Input S3 endpoint, enter the Amazon S3 location you used earlier to copy files from SageMaker Studio to Amazon S3, then choose Go. You can also navigate to the Amazon S3 location using the browser below.
5. Select the flow files to import, then choose Import.
   

After you import the files, the SageMaker Data Wrangler page will refresh to show the newly imported files, as shown in the following screenshot.

Use SageMaker Canvas for data transformation with SageMaker Data Wrangler

Choose one of the flows (for this example, we choose titanic.flow) to launch the SageMaker Data Wrangler transformation.

Now you can add analyses and transformations to the data flow using a visual interface (Accelerate data preparation for ML in Amazon SageMaker Canvas) or natural language interface (Use natural language to explore and prepare data with a new capability of Amazon SageMaker Canvas).

When you’re happy with the data, choose the plus sign and choose Create model, or choose Export to export the dataset to build and use ML models.

Alternate migration method

This post has provided guidance on using Amazon S3 to migrate SageMaker Data Wrangler flow files from a SageMaker Studio Classic environment. Phase 3: (Optional) Migrate data from Studio Classic to Studio provides a second method that uses your local machine to transfer the flow files. Furthermore, you can download single flow files from the SageMaker Studio tree control to your local machine, then import them manually in SageMaker Canvas. Choose the method that suits your needs and use case.

Clean up

When you’re done, shut down any running SageMaker Data Wrangler applications in SageMaker Studio Classic. To save costs, you can also remove any flow files from the SageMaker Studio Classic file browser, which is an Amazon Elastic File System (Amazon EFS) volume. You can also delete any of the intermediate files in Amazon S3. After the flow files are imported into SageMaker Canvas, the files copied to Amazon S3 are no longer needed.

You can log out of SageMaker Canvas when you’re done, then relaunch it when you’re ready to use it again.

Conclusion

Migrating your existing SageMaker Data Wrangler flows to SageMaker Canvas is a straightforward process that allows you to use the advanced data preparations you’ve already developed while taking advantage of the end-to-end, low-code no-code ML workflow of SageMaker Canvas. By following the steps outlined in this post, you can seamlessly transition your data wrangling artifacts to the SageMaker Canvas environment, streamlining your ML projects and enabling business analysts and non-technical users to build and deploy models more efficiently.

Start exploring SageMaker Canvas today and experience the power of a unified platform for data preparation, model building, and deployment!

About the Authors

Charles Laughlin is a Principal AI Specialist at Amazon Web Services (AWS). Charles holds an MS in Supply Chain Management and a PhD in Data Science. Charles works in the Amazon SageMaker service team where he brings research and voice of the customer to inform the service roadmap. In his work, he collaborates daily with diverse AWS customers to help transform their businesses with cutting-edge AWS technologies and thought leadership.

Dan Sinnreich is a Sr. Product Manager for Amazon SageMaker, focused on expanding no-code / low-code services. He is dedicated to making ML and generative AI more accessible and applying them to solve challenging problems. Outside of work, he can be found playing hockey, scuba diving, and reading science fiction.

Huong Nguyen is a Sr. Product Manager at AWS. She is leading the ML data preparation for SageMaker Canvas and SageMaker Data Wrangler, with 15 years of experience building customer-centric and data-driven products.

Davide Gallitelli is a Specialist Solutions Architect for AI/ML in the EMEA region. He is based in Brussels and works closely with customer throughout Benelux. He has been a developer since very young, starting to code at the age of 7. He started learning AI/ML in his later years of university, and has fallen in love with it since then.get confirmation

Read More

One of the most common applications of generative artificial intelligence (AI) and large language models (LLMs) in an enterprise environment is answering questions based on the enterprise’s knowledge corpus. Pre-trained foundation models (FMs) excel at natural language understanding (NLU) tasks, including summarization, text generation, and question answering across a wide range of topics. However, they often struggle to provide accurate answers without hallucinations and fall short when addressing questions about content that wasn’t included in their training data. Furthermore, FMs are trained with a point-in-time snapshot of data and have no inherent ability to access fresh data at inference time; therefore, they might provide responses that are incorrect or inadequate.

We face a fundamental challenge with enterprise data—overcoming the disconnect between natural language and structured data. Natural language is ambiguous and imprecise, whereas data adheres to rigid schemas. For example, SQL queries can be complex and unintuitive for non-technical users. Handling complex queries involving multiple tables, joins, and aggregations makes it difficult to interpret user intent and translate it into correct SQL operations. Domain-specific terminology further complicates the mapping process. Another challenge is accommodating the linguistic variations users employ to express the same requirement. Effectively managing synonyms, paraphrases, and alternative phrasings is important. The inherent ambiguity of natural language can also result in multiple interpretations of a single query, making it difficult to accurately understand the user’s precise intent.

To bridge this gap, you need advanced natural language processing (NLP) to map user queries to database schema, tables, and operations. In this architecture, Amazon Q Business acts as an intermediary, translating natural language into precise SQL queries. You can simply ask questions like “What were the sales for outdoor gear in Q3 2023?” Amazon Q Business analyzes intent, accesses data sources, and generates the SQL query. This simplifies data access for your non-technical users and streamlines workflows for professionals, allowing them to focus on higher-level tasks.

In this post, we discuss an architecture to query structured data using Amazon Q Business, and build out an application to query cost and usage data in Amazon Athena with Amazon Q Business. Amazon Q Business can create SQL queries to your data sources when provided with the database schema, additional metadata describing the columns and tables, and prompting instructions. You can extend this architecture to use additional data sources, query validation, and prompting techniques to cover a wider range of use cases.

Solution overview

The following figure represents the high-level architecture of the proposed solution. Steps 3 and 4 augment the AWS IAM Identity Center integration with Amazon Q Business for an authorization flow. In this architecture, we use Amazon Cognito for user authentication as well as a trusted token issuer to IAM Identity Center. You can also use your own identity provider as a trusted token issuer as long as it supports OpenID Connect (OIDC).

The workflow includes the following steps:

1. The user initiates the interaction with the Streamlit application, which is accessible through an Application Load Balancer, acting as the entry point.
2. The application prompts the user to authenticate using their Amazon Cognito credentials, maintaining secure access.
3. The application exchanges the token obtained from Amazon Cognito for an IAM Identity Center token, granting the necessary scope to interact with Amazon Q Business.
4. Using the IAM Identity Center token, the application assumes an AWS Identity and Access Management (IAM) role and retrieves an AWS session from AWS Security Token Service (AWS STS), enabling authorized communication with Amazon Q Business.
5. Based on the user’s natural language query, the application formulates relevant prompts and metadata, which are then submitted to the chat_sync API of Amazon Q Business. In response, Amazon Q Business provides an appropriate Athena query to run.
6. The application runs the Athena query received from Amazon Q Business, and the resulting data is displayed on the web application’s UI.

Querying Amazon Q Business LLMs directly

As explained in the response settings for Amazon Q Business, there are different options to generate responses that allow you to either use your enterprise data, use LLMs directly, or fall back on the LLMs if the answer is not found in your enterprise data. Along with the global controls for response settings, you need to specify which chatMode you want to use based on your specific use case. If you want to bypass Retrieval Augmented Generation (RAG) and use plain text in the context window, you should use CREATOR_MODE. Alternatively, RAG is also bypassed when you upload files directly in the context window.

If you just use text in the context window and call Amazon Q Business APIs without switching to CREATOR_MODE, that may break your use case in the future if you add content to the index (RAG). In this use case, because we’re not indexing any data and using schemas as attachments in the API call to Amazon Q Business, RAG is automatically bypassed and the response is generated directly from the LLMs. Another reason to use attachments for this use case is that for the chatSync API, userMessage has a maximum length of 7,000, which can be surpassed depending on how large your text is in the context window.

Data query workflow

Let’s look at the prompts, query generation, and Athena query in detail. We use Athena as the data store in this post. Users enter natural language questions into a web application built with Streamlit. Amazon Q Business converts the natural language questions to valid SQL for Athena using the prompting instructions, the database schema, and data dictionary that are provided as context to the LLM. The generated SQL is sent to Athena to run as a query, and the returned data is displayed to the user in the Streamlit application. The following diagram illustrates this workflow.

These are the various components to this data flow, as numbered in the diagram:

1. User intent
2. Prompt builder
3. SQL query generator
4. Running the query
5. Query results

In the following sections, we look at each component in more detail.

User intent

The user intent or your inquiry is the starting point of the process. It can be in natural language, such as “What was the total spend for ElasticSearch last year?” The user’s input serves as the basis for the subsequent steps in the workflow.

Prompt builder

The prompt builder component plays a crucial role in bridging the gap between your natural language input and the structured data format required for SQL querying. It augments your question with relevant information from the table schema and data dictionary to provide context for the query generation process. This step involves the following sub-tasks:

• Natural language processing – NLP techniques are employed to analyze and understand your questions. This includes steps like tokenization and dependency parsing to extract the intent and relevant entities from the natural language input.
• Entity recognition – Named entity recognition (NER) is used to identify and classify relevant entities mentioned in your question, such as product names, dates, or region. This step helps map your input to the corresponding data elements in the database schema.
• Intent mapping – The prompt builder maps your intent, extracted from the NLP analysis, to the appropriate data structures and operations required to fulfill the query. This mapping process uses the table schema and data dictionary to establish connections between your natural language questions and the database elements. The output of the prompt builder is a structured representation of your question, augmented with the necessary context from the database schema and data dictionary. This structured representation serves as input for the next step, SQL query generation.

The following is an example prompt for “What was the total spend for ElasticSearch last year?”

You will not respond to gibberish, random character sequences, or prompts that do not make logical sense. 
If the input the input does not make sense or is outside the scope of the provided context, do not respond with SQL 
but respond with - I do not know about this. Please fix your input.
You are an expert SQL developer. Only return the sql query. Do not include any verbiage. 
You are required to return SQL queries based on the provided schema and the service mappings for common services and 
their synonyms. The table with the provided schema is the only source of data. Do not use joins. Assume product, 
service are synonyms for product_servicecode and price,cost,spend are synonymns for line_item_unblended_cost. Use the 
column names from the provided schema while creating queries. Do not use preceding zeroes for the column month when 
creating the query. Only use predicates when asked. For your reference, current date is June 01, 2024. write a sql 
query for this task - What was the total spend for ElasticSearch last year?

SQL query generation

Based on the prompt generated from the prompt builder and your original question, Amazon Q Business generates the corresponding SQL query. The SQL query is tailored to retrieve the relevant data and perform the desired analysis or calculations to accurately answer the user’s question. This step may involve techniques such as:

• Mapping your intent and entities to SQL clauses (SELECT, FROM, WHERE, JOIN, and so on)
• Handling complex queries involving aggregations, subqueries, or predicates
• Incorporating domain-specific knowledge or business rules into the query generation process

Running the query

In this step, the generated SQL query is run against the chosen data store, which could be a relational database, data warehouse, NoSQL database, or an object store like Amazon Simple Storage Service (Amazon S3). The data store serves as the repository for the data required to answer the user’s question. Depending on the architecture and requirements, the data store query may involve additional components or processes, such as:

• Query optimization and indexing strategies
• Materialized views for complex queries
• Real-time data ingestion and updates

Query results

The query engine runs the generated SQL query against the data store and returns the query results. These results contain the insights or answers to the original user question. The presentation of the query results can take various forms, depending on the requirements of the application or UI:

• Tabular data – The results can be displayed as a table or spreadsheet, suitable for structured data analysis
• Visualizations – The query results can be rendered as charts, graphs, or other visual representations, providing a more intuitive way to understand and explore the data
• Natural language responses – In some cases, the query results can be translated back into natural language statements or summaries, making the insights more accessible to non-technical users

In the following sections, we walk through the steps to deploy the web application and test the solution.

Prerequisites

Complete the following prerequisite steps:

1. Set up IAM Identity Center and add users that you intend to give access to in your Amazon Q Business application.
2. Have an existing, working Amazon Q Business application and give access to the users created in the previous step to the application.
3. AWS Cost and Usage Reports (AWS CUR) data is available in Athena. If you have CUR data, you can skip the following steps for CUR data setup. If not, you have a few options to set up CUR data:
   1. To set up sample CUR data, refer to the following lab and follow the instructions.
   2. You also need to set up an AWS Glue crawler to make the data available in Athena.
4. If you already have an SSL certificate, you can skip this step; otherwise, generate a private certificate.
5. Import the certificate into AWS Certificate Manager (ACM). For more details, refer to Importing a certificate.

Set up the application

Complete the following steps to set up the application:

1. From your terminal, clone the GitHub repository:

git clone https://github.com/aws-samples/data-insights-with-amazon-q-business.git

2. Go to the project directory:

cd data-insights-with-amazon-q-business

3. Based on your CUR table, update the CUR schema under app/schemas/cur_schema.txt. Review the prompts under app/qb_config.py. The schema looks similar to the following code:

4. Review the data dictionary under app/schemas/service_mappings.csv. You can modify the mappings according to your dataset. A sample data dictionary for CUR might look like the following screenshot.

5. Zip up the code repository and upload it to an S3 bucket.
6. Follow the steps in the GitHub repo to deploy the Streamlit application.

Access the web application

As part of the deployment steps, you launched an AWS CloudFormation stack. On the AWS CloudFormation console, navigate to the Outputs tab for the stack and find the URL to access the Streamlit application. When you open the URL in a browser, you’ll see a login screen like the following screenshot. Sign up to create a user in the Amazon Cognito user pool. After you’re validated, you can use the same credentials to log in to the web application.

Query your cost and usage data

Start with a simple query like “What was the total spend for ElasticSearch this year?” A relevant prompt will be created and sent to Amazon Q Business. It will respond back with the corresponding SQL query. Notice the predicate where product_servicecode = ‘AmazonES’. Amazon Q Business is able to formulate the query because it has the schema and the data dictionary in context. It understands that ElasticSearch is an AWS service represented by a column named product_servicecode in the CUR data schema and its corresponding value of ‘AmazonES’. Next, the query is run against Athena and you get the results back.

The sample dataset used in this post is from 2023. If you’re using the sample dataset, natural language queries referring to current year will give not return results. Modify your queries to 2023 or mention the year in the user intent.

The following figure highlights the steps as explained in the data flow.

You can also try complex queries like “Give me a list of the top 3 products by total spend last year. For each of these products, what percentage of the overall spend is from this product?” Because the prompt builder has schema and product (AWS services) information in its context, Amazon Q Business creates the corresponding query. In this case, you’ll see a query similar to the following:

SELECT 
product_servicecode,
SUM(line_item_unblended_cost) AS total_spend,
ROUND(SUM(line_item_unblended_cost) * 100.0 / (SELECT SUM(line_item_unblended_cost)
FROM cur_daily WHERE year = '2023'), 2) AS percentage_of_total
FROM cur_daily
WHERE year = '2023'
GROUP BY product_servicecode
ORDER BY total_spend DESC
LIMIT 3;

When the query is run against Athena, you’ll see similar results corresponding to your data.

Along with the data, you can also see a summary and trend analysis of your data on the Description tab of your Streamlit app.

The prompts used in the application are open domain and you’re free to update them in the code. For example, the following is a prompt used for a summary task:

You are an AI assistant. You are required to return a summary based on the provided data in attachment. Use atleast 
100 words. The spend is in dollars. The unit of measurement is dollars. Give trend analysis too. Start your response 
with - Here is your summary..

The following screenshot shows the results.

Feedback loop

You also have the option of capturing feedback for the generated queries with the thumbs up/down icon on the web application. Currently, the feedback is captured in a local file under /app/feedback. You can change this implementation to write to a database of your choice and have it serve as a query validation mechanism after your testing, to allow only validated queries to run.

Clean up

To clean up your resources, delete the CloudFormation stack, Amazon Q Business application, and Athena tables.

Conclusion

In this post, we demonstrated how Amazon Q Business can effectively bridge the gap between users and data, enabling you to extract valuable insights from various data stores using natural language queries, without the need for extensive technical knowledge or SQL expertise. The natural language understanding capabilities of Amazon Q Business can accurately interpret user intent, extract relevant entities, and generate SQL to translate the user’s query into executable data operations. You can now empower a wider range of enterprise users to unlock the full value of your organization’s data assets. By democratizing data access and analysis using natural language queries, you can foster data-driven decision-making, drive innovation, and unlock new opportunities for growth and success.

In Part 2 of this series, we demonstrate how to integrate this architecture with LangChain using Amazon Q Business as a custom model. We also cover query validation and accuracy measurement.

About the Authors

Vishal Karlupia is a Senior Technical Account Manager/Lead at Amazon Web Services, Toronto. He specializes in generative AI applications and helps customers build and scale their AI/ML workloads on AWS. Outside of work, he enjoys being outdoors and keeping bonfires alive.

Srinivas Ganapathi is a Principal Technical Account Manager at Amazon Web Services. He is based in Toronto, Canada, and works with games customers to run efficient workloads on AWS.

Read More

The Cohere Rerank 3 Nimble foundation model (FM) is now generally available in Amazon SageMaker JumpStart. This model is the newest FM in Cohere’s Rerank model series, built to enhance enterprise search and Retrieval Augmented Generation (RAG) systems.

In this post, we discuss the benefits and capabilities of this new model with some examples.

Overview of Cohere Rerank models

Cohere’s Rerank family of models are designed to enhance existing enterprise search systems and RAG systems. Rerank models improve search accuracy over both keyword-based and embedding-based search systems. Cohere Rerank 3 is designed to reorder documents retrieved by initial search algorithms based on their relevance to a given query. A reranking model, also known as a cross-encoder, is a type of model that, given a query and document pair, will output a similarity score. For FMs, words, sentences, or entire documents are often encoded as dense vectors in a semantic space. By calculating the cosine of the angle between these vectors, you can quantify their semantic similarity and output as a single similarity score. You can use this score to reorder the documents by relevance to your query.

Cohere Rerank 3 Nimble is the newest model from Cohere’s Rerank family of models, designed to improve speed and efficiency from its predecessor Cohere Rerank 3. According to Cohere’s benchmark tests including BEIR (Benchmarking IR) for accuracy and internal benchmarking datasets, Cohere Rerank 3 Nimble maintains high accuracy while being approximately 3–5 times faster than Cohere Rerank 3. The speed improvement is designed for enterprises looking to enhance their search capabilities without sacrificing performance.

The following diagram represents the two-stage retrieval of a RAG pipeline and illustrates where Cohere Rerank 3 Nimble is incorporated into the search pipeline.

In the first stage of retrieval in the RAG architecture, a set of candidate documents are returned based on the knowledge base that’s relevant to the query. In the second stage, Cohere Rerank 3 Nimble analyzes the semantic relevance between the query and each retrieved document, reordering them from most to least relevant. The top-ranked documents augment the original query with additional context. This process improves search result quality by identifying the most pertinent documents. Integrating Cohere Rerank 3 Nimble into a RAG system enables users to send fewer but higher-quality documents to the language model for grounded generation. This results in improved accuracy and relevance of search results without adding latency.

Overview of SageMaker JumpStart

SageMaker JumpStart offers access to a broad selection of publicly available FMs. These pre-trained models serve as powerful starting points that can be deeply customized to address specific use cases. You can now use state-of-the-art model architectures, such as language models, computer vision models, and more, without having to build them from scratch.

Amazon SageMaker is a comprehensive, fully managed machine learning (ML) platform that revolutionizes the entire ML workflow. It offers an unparalleled suite of tools that cater to every stage of the ML lifecycle, from data preparation to model deployment and monitoring. Data scientists and developers can use the SageMaker integrated development environment (IDE) to access a vast array of pre-built algorithms, customize their own models, and seamlessly scale their solutions. The platform’s strength lies in its ability to abstract away the complexities of infrastructure management, allowing you to focus on innovation rather than operational overhead. The automated ML capabilities of SageMaker, including automated machine learning (AutoML) features, democratize ML by enabling even non-experts to build sophisticated models. Furthermore, its robust governance features help organizations maintain control and transparency over their ML projects, addressing critical concerns around regulatory compliance.

Prerequisites

Make sure your SageMaker AWS Identity and Access Management (IAM) service role has the AmazonSageMakerFullAccess permission policy attached.

To deploy Cohere Rerank 3 Nimble successfully, confirm one of the following:

• Make sure your IAM role has the following permissions and you have the authority to make AWS Marketplace subscriptions in the AWS account used:
  • aws-marketplace:ViewSubscriptions
  • aws-marketplace:Unsubscribe
  • aws-marketplace:Subscribe
• Alternatively, confirm your AWS account has a subscription to the model. If so, you can skip the following deployment instructions and start with subscribing to the model package.

Deploy Cohere Rerank 3 Nimble on SageMaker JumpStart

You can access the Cohere Rerank 3 family of models using SageMaker JumpStart in Amazon SageMaker Studio, as shown in the following screenshot.

Deployment starts when you choose Deploy, and you may be prompted to subscribe to this model through AWS Marketplace. If you are already subscribed, you can choose Deploy again to deploy the model. After deployment finishes, you will see that an endpoint is created. You can test the endpoint by passing a sample inference request payload or by selecting the testing option using the SDK.

Subscribe to the model package

To subscribe to the model package, complete the following steps:

1. Depending on the model you want to deploy, open the model package listing page for cohere-rerank-nimble-english or cohere-rerank-nimble-multilingual.
2. On the AWS Marketplace listing, choose Continue to subscribe.
3. On the Subscribe to this software page, review and choose Accept Offer if you and your organization agree with EULA, pricing, and support terms.
4. Choose Continue to configuration and then choose an AWS Region.

A product ARN will be displayed. This is the model package ARN that you need to specify while creating a deployable model using Boto3.

Deploy Cohere Rerank 3 Nimble using the SDK

To deploy the model using the SDK, copy the product ARN from the previous step and specify it in the model_package_arn in the following code:

from cohere_aws import Client
import boto3
region = boto3.Session().region_name

model_package_arn = "Specify the model package ARN here"

After you specify the model package ARN, you can create the endpoint, as shown in the following code. Specify the name of the endpoint, the instance type, and the number of instances being used. Make sure you have the account-level service limit for using ml.g5.xlarge for endpoint usage as one or more instances. To request a service quota increase, refer to AWS service quotas.

co = Client(region_name=region)
co.create_endpoint(arn=model_package_arn, endpoint_name="cohere-rerank-3/cohere-rerank-nimble-multilingual", instance_type="ml.g5.xlarge", n_instances=1)

If the endpoint is already created, you just need to connect to it with the following code:

co.connect_to_endpoint(endpoint_name="cohere-rerank-3/cohere-rerank-nimble-multilingual-v3")

Follow a similar process as detailed earlier to deploy Cohere Rerank 3 on SageMaker JumpStart.

Inference example with Cohere Rerank 3 Nimble

Cohere Rerank 3 Nimble offers robust multilingual support. The model is available in both English and multilingual versions supporting over 100 languages.

The following code example illustrates how to perform real-time inference using Cohere Rerank 3 Nimble-English:

documents = [
    {"Title":"Incorrect Password","Content":"Hello, I have been trying to access my account for the past hour and it keeps saying my password is incorrect. Can you please help me?"},
    {"Title":"Confirmation Email Missed","Content":"Hi, I recently purchased a product from your website but I never received a confirmation email. Can you please look into this for me?"},
    {"Title":"Questions about Return Policy","Content":"Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective."},
    {"Title":"Customer Support is Busy","Content":"Good morning, I have been trying to reach your customer support team for the past week but I keep getting a busy signal. Can you please help me?"},
    {"Title":"Received Wrong Item","Content":"Hi, I have a question about my recent order. I received the wrong item and I need to return it."},
    {"Title":"Customer Service is Unavailable","Content":"Hello, I have been trying to reach your customer support team for the past hour but I keep getting a busy signal. Can you please help me?"},
    {"Title":"Return Policy for Defective Product","Content":"Hi, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective."},
    {"Title":"Wrong Item Received","Content":"Good morning, I have a question about my recent order. I received the wrong item and I need to return it."},
    {"Title":"Return Defective Product","Content":"Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective."}
]

In the following code, the top_n inference parameter for Cohere Rerank 3 and Rerank 3 Nimble specifies the number of top-ranked results to return after reranking the input documents. It allows you to control how many of the most relevant documents are included in the final output. To determine an optimal value for top_n, consider factors such as the diversity of your document set, the complexity of your queries, and the desired balance between precision and latency for enterprise search or RAG.

response = co.rerank(documents=documents, query='What emails have been about returning items?', rank_fields=["Title","Content"], top_n=2)

The following is the output from Cohere Rerank 3 Nimble-English:

Documents: [RerankResult<document: {'Title': 'Received Wrong Item', 'Content': 'Hi, I have a question about my recent order. I received the wrong item and I need to return it.'}, index: 4, relevance_score: 0.0068771075>, RerankResult<document: {'Title': 'Wrong Item Received', 'Content': 'Good morning, I have a question about my recent order. I received the wrong item and I need to return it.'}, index: 7, relevance_score: 0.0064131636>]

Cohere Rerank 3 Nimble multilingual support

The multilingual capabilities of Cohere Rerank 3 Nimble-Multilingual enable global organizations to provide consistent, improved search experiences to users across different Regions and language preferences.

In the following example, we create an input payload for a list of emails in multiple languages. We can take the same set of emails from earlier and translate them to different languages. These examples are available under the SageMaker JumpStart model card and are randomly generated for this example.

documents = [
    {"Title":"Contraseña incorrecta","Content":"Hola, llevo una hora intentando acceder a mi cuenta y sigue diciendo que mi contraseña es incorrecta. ¿Puede ayudarme, por favor?"},
    {"Title":"Confirmation Email Missed","Content":"Hi, I recently purchased a product from your website but I never received a confirmation email. Can you please look into this for me?"},
    {"Title":"أسئلة حول سياسة الإرجاع","Content":"مرحبًا، لدي سؤال حول سياسة إرجاع هذا المنتج. لقد اشتريته قبل بضعة أسابيع وهو معيب"},
    {"Title":"Customer Support is Busy","Content":"Good morning, I have been trying to reach your customer support team for the past week but I keep getting a busy signal. Can you please help me?"},
    {"Title":"Falschen Artikel erhalten","Content":"Hallo, ich habe eine Frage zu meiner letzten Bestellung. Ich habe den falschen Artikel erhalten und muss ihn zurückschicken."},
    {"Title":"Customer Service is Unavailable","Content":"Hello, I have been trying to reach your customer support team for the past hour but I keep getting a busy signal. Can you please help me?"},
    {"Title":"Return Policy for Defective Product","Content":"Hi, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective."},
    {"Title":"收到错误物品","Content":"早上好，关于我最近的订单，我有一个问题。我收到了错误的商品，需要退货。"},
    {"Title":"Return Defective Product","Content":"Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective."}
]

Use the following code to perform real-time inference using Cohere Rerank 3 Nimble-Multilingual:

response = co.rerank(documents=documents, query='What emails have been about returning items?', rank_fields=['Title','Content'], top_n=2)
print(f'Documents: {response}')

The following is the output from Cohere Rerank 3 Nimble-Multilingual:

Documents: [RerankResult<document: {'Title': '收到错误物品', 'Content': '早上好，关于我最近的订单，我有一个问题。我收到了错误的商品，需要退货。'}, index: 7, relevance_score: 0.034553625>, RerankResult<document: {'Title': 'أسئلة حول سياسة الإرجاع', 'Content': 'مرحبًا، لدي سؤال حول سياسة إرجاع هذا المنتج. لقد اشتريته قبل بضعة أسابيع وهو معيب'}, index: 2, relevance_score: 0.00037263767>]

The output translated to English is as follows:

Documents: [RerankResult<document: {'Title': 'Received Wrong Item', 'Content': 'Good morning, I have a question about my recent order. I received the wrong item and need to return it.'}, index: 7, relevance_score: 0.034553625>, RerankResult<document: {'Title': 'Questions about Return Policy', 'Content': 'Hello, I have a question about the return policy for this product. I bought it a few weeks ago and it's defective'}, index: 2, relevance_score: 0.00037263767>]

In both examples, the relevance scores are normalized to be in the range [0, 1]. Scores close to 1 indicate a high relevance to the query, and scores closer to 0 indicate low relevance.

Use cases suitable for Cohere Rerank 3 Nimble

The Cohere Rerank 3 Nimble model provides an option that prioritizes efficiency. The model is ideal for enterprises looking to enable their customers to accurately search complex documentation, build applications that understand over 100 languages, and retrieve the most relevant information from various data stores. In industries such as retail, where website drop-off increases with every 100 milliseconds added to search response time, having a faster AI model like Cohere Rerank 3 Nimble powering the enterprise search system translates to higher conversion rates.

Conclusion

Cohere Rerank 3 and Rerank 3 Nimble are now available on SageMaker JumpStart. To get started, refer to Train, deploy, and evaluate pretrained models with SageMaker JumpStart.

Interested in diving deeper? Check out the Cohere on AWS GitHub repo.

About the Authors

Breanne Warner is an Enterprise Solutions Architect at Amazon Web Services supporting healthcare and life science (HCLS) customers. She is passionate about supporting customers to use generative AI on AWS and evangelizing model adoption. Breanne is also on the Women@Amazon board as co-director of Allyship with the goal of fostering inclusive and diverse culture at Amazon. Breanne holds a Bachelor’s of Science in Computer Engineering from University of Illinois at Urbana Champaign (UIUC)

Nithin Vijeaswaran is a Solutions Architect at AWS. His area of focus is generative AI and AWS AI Accelerators. He holds a Bachelor’s degree in Computer Science and Bioinformatics. Niithiyn works closely with the Generative AI GTM team to enable AWS customers on multiple fronts and accelerate their adoption of generative AI. He’s an avid fan of the Dallas Mavericks and enjoys collecting sneakers.

Karan Singh is a Generative AI Specialist for third-party models at AWS, where he works with top-tier third-party foundational model providers to define and run join GTM motions that help customers train, deploy, and scale foundational models. Karan holds a Bachelor’s of Science in Electrical and Instrumentation Engineering from Manipal University and a Master’s in Science in Electrical Engineering from Northwestern University, and is currently an MBA Candidate at the Haas School of Business at University of California, Berkeley.

Read More