Posts in category: Amazon AWS

At AWS, we are transforming our seller and customer journeys by using generative artificial intelligence (AI) across the sales lifecycle. We envision a future where AI seamlessly integrates into our teams’ workflows, automating repetitive tasks, providing intelligent recommendations, and freeing up time for more strategic, high-value interactions. Our field organization includes customer-facing teams (account managers, solutions architects, specialists) and internal support functions (sales operations).

Prospecting, opportunity progression, and customer engagement present exciting opportunities to utilize generative AI, using historical data, to drive efficiency and effectiveness. Personalized content will be generated at every step, and collaboration within account teams will be seamless with a complete, up-to-date view of the customer. Our internal AI sales assistant, powered by Amazon Q Business, will be available across every modality and seamlessly integrate with systems such as internal knowledge bases, customer relationship management (CRM), and more. It will be able to answer questions, generate content, and facilitate bidirectional interactions, all while continuously using internal AWS and external data to deliver timely, personalized insights.

Through this series of posts, we share our generative AI journey and use cases, detailing the architecture, AWS services used, lessons learned, and the impact of these solutions on our teams and customers. In this first post, we explore Account Summaries, one of our initial production use cases built on Amazon Bedrock. Account Summaries equips our teams to be better prepared for customer engagements. It combines information from various sources into comprehensive, on-demand summaries available in our CRM or proactively delivered based on upcoming meetings. From the period of September 2023 to March 2024, sellers leveraging GenAI Account Summaries saw a 4.9% increase in value of opportunities created.

The business opportunity

Data often resides across multiple internal systems, such as CRM and financial tools, and external sources, making it challenging for account teams to gain a comprehensive understanding of each customer. Manually connecting these disparate datasets can be time-consuming, presenting an opportunity to improve how we uncover valuable insights and identify opportunities. Without proactive insights and recommendations, account teams can miss opportunities and deliver inconsistent customer experiences.

Use case overview

Using generative AI, we built Account Summaries by seamlessly integrating both structured and unstructured data from diverse sources. This includes sales collateral, customer engagements, external web data, machine learning (ML) insights, and more. The result is a comprehensive summary tailored for our sellers, available on-demand in our CRM and proactively delivered through Slack based on upcoming meetings.

Account Summaries provides a 360-degree account narrative with customizable sections, showcasing timely and relevant information about customers. Key sections include:

• Executive summary – A concise overview highlighting the latest customer updates, ideal for quick, high-level briefings.
• Organization overview – Analysis of external organization and industry news along with citations to sources, providing account teams with timely discussion topics and positioning strategies.
• Product consumption – Summaries of how customers are using AWS services over time.
• Opportunity pipeline – Overview of open and stalled opportunities, including partner engagements and recent customer interactions.
• Investments and support – Information on customer issues, promotional programs, support cases, and product feature requests.
• AI-driven recommendations – By combining generative AI with ML, we deliver intelligent suggestions for products, services, applicable use cases, and next steps. Recommendations include citations to source materials, empowering account teams to more effectively drive customer strategies.

The following screenshot shows a sample account summary. All data in this example summary is fictitious.

Solution impact

Since its inception in 2023, more than 100,000 GenAI Account Summaries have been generated, and AWS sellers report an average of 35 minutes saved per GenAI Account Summary. This is boosting productivity and freeing up time for customer engagements. The impact goes beyond just efficiency. Since its inception in September 2023 up through March 2024, approximately one-third of surveyed sellers reported that GenAI Account Summaries had a positive impact on their approach to a customer, and sellers leveraging GenAI Account Summaries saw a 4.9% increase in value of opportunities created.

The impact of this use case has been particularly pronounced among teams who support a large number of customers. Users such as specialists who move between multiple accounts have seen a dramatic improvement in their ability to quickly understand and add value to diverse customer situations. During account transitions, they enable new account managers to rapidly get up to date on inherited accounts. At events, our teams now approach customer interactions armed with comprehensive, up-to-date information on demand. Account Summaries is also now foundational to other downstream mechanisms like account planning and executive briefing center (EBC) meetings.

Solution overview

This illustrates our approach to implementing generative AI capabilities across the sales and customer lifecycle. It’s built on diverse data sources and a robust infrastructure layer for data retrieval, prompting, and LLM management. This modular structure provides a scalable foundation for deploying a broad range of AI-powered use cases, beginning with Account Summaries.

Building generative AI solutions like Account Summaries on AWS offers significant technical advantages, particularly for organizations already using AWS services. You can integrate existing data from AWS data lakes, Amazon Simple Storage Service (Amazon S3) buckets, or Amazon Relational Database Service (Amazon RDS) instances with services such as Amazon Bedrock and Amazon Q. For our Account Summaries use case, we use both Amazon Titan and Anthropic Claude models on Amazon Bedrock, taking advantage of their unique strengths for different aspects of summary generation.

Our approach to model selection and deployment is both strategic and flexible. We carefully choose models based on their specific capabilities and the requirements of each summary section. This allows us to optimize for factors such as accuracy, response time, and cost-efficiency. The architecture we’ve developed enables seamless combination and switching between different models, even within a single summary generation process. This multi-model approach lets us take advantage of the best features of each model, resulting in more comprehensive and nuanced summaries.

This flexible model selection and combination capability, coupled with our existing AWS infrastructure, accelerates time to market, reduces complex data migrations and potential failure points, and allows us to continuously incorporate state-of-the-art language models as they become available.

Our system integrates diverse data sources with sophisticated data indexing and retrieval processes, and utilizes carefully crafted prompting techniques. We’ve also implemented robust strategies to mitigate hallucinations, providing reliability in our generated summaries. Built on AWS with asynchronous processing, the solution incorporates multiple quality assurance measures and is continually refined through a comprehensive feedback loop, all while maintaining stringent security and privacy standards.

In the following sections, we review each component, including data sources, data indexing and retrieval, prompting strategies, hallucination mitigation techniques, quality assurance processes, and the underlying infrastructure and operations.

Data sources

Account Summaries relies on four key categories of information:

• Data about customers – Structured information about the customer’s AWS journey, including service metrics, growth trends, and support history
• ML insights – Insights generated from analyzing patterns in structured business data and unstructured interaction logs
• Internal knowledge bases – Unstructured data like sales plays, case studies, and product information, continuously updated to reflect the latest AWS offerings and best practices
• External data – Real-time news, public financial filings, and industry reports to offer a comprehensive understanding of the customer’s business landscape

By bringing together these diverse data sources, we create a rich, multidimensional view of each account that goes beyond what’s possible with traditional data analysis.

To maintain the integrity of our core data, we do not retain or use the prompts or the resulting account summary for model training. Instead, after a summary is produced and delivered to the seller, the generated content is permanently deleted.

Data indexing and retrieval

We start with indexing and retrieving both structured and unstructured data, which allows us to provide comprehensive summaries that combine quantitative data with qualitative insights.

The indexing process consists of the following stages:

• Document preprocessing – Clean and normalize text from various sources
• Chunking – Break documents into manageable pieces (1,200 tokens with 50-token overlap)
• Vectorization – Convert text chunks into vector representations using an embeddings model
• Storage – Index vectors and metadata in the database for quick retrieval

The retrieval process comprises the following stages:

• Query vectorization – Convert user queries or context into vector representations
• Similarity search – Use k-nearest neighbors (k-NN) to find relevant document chunks
• Metadata filtering – Apply additional filters based on structured data (such as date ranges or product categories)
• Reranking – Use a cross-encoder model to refine the relevance of retrieved chunks
• Context integration – Combine retrieved information with the large language model (LLM) prompt

The following are key implementation considerations:

• Balancing structured and unstructured data – Using structured data to guide and filter searches within unstructured content, and combining quantitative metrics with qualitative insights for comprehensive summaries
• Scalability – Designing our system to handle increasing volumes of data and concurrent requests, and considering partitioning strategies for our growing vector database
• Maintaining data freshness – Implementing strategies to regularly update our index with new information and considered real-time indexing for critical, fast-changing data points
• Continuous relevance tuning – Ongoing refinement of our retrieval process based on user feedback and performance metrics, and experimentation with different embedding models and similarity measures
• Privacy and security – Using row-level security access controls to limit user access to information

By thoughtfully implementing this indexing and retrieval system, we’ve created a powerful foundation for Account Summaries. This approach allows us to dynamically combine structured internal business data with relevant unstructured content, providing our field teams with comprehensive, up-to-date, and context-rich summaries for every customer engagement.

Prompting

Well-crafted prompts enhance the accuracy and relevance of generated responses, reduce hallucinations, and allow for customization based on specific use cases. Ultimately, prompting serves as the critical interface that makes sure Retrieval Augmented Generation (RAG) systems produce coherent, factual, and tailored outputs by effectively using both stored knowledge and the capabilities of LLMs. Prompting plays a crucial role in RAG systems by bridging the gap between retrieved information and user intent. It guides the retrieval process, contextualizes the fetched data, and instructs the language model on how to use this information effectively.

The following diagram illustrates the prompting framework for Account Summaries, which begins by gathering data from various sources. This information is used to build a prompt with relevant context and then fed into an LLM, which generates a response. The final output is a response tailored to the input data and refined through iteration.

We organize our prompting best practices into two main categories:

• Content and structure:
  • Constraint specification – Define content, tone, and format constraints relevant to AWS sales contexts. For example, “Provide a summary that excludes sensitive financial data and maintains a formal tone.”
  • Use of delimiters – Employ XML tags to separate instructions, context, and generation areas. For example, <instructions> Please summarize the key points from the following passage: </instructions> <data> [Insert passage here] </data>.
  • Modular prompts – Split prompts into section-specific chunks for enhanced accuracy and reduced latency, because it allows the LLM to focus on a smaller context at a time. For example, “Separate prompts for executive summary and opportunity pipeline sections.”
  • Role context – Start each prompt with a clear role definition. For example, “You are an AWS Account Manager preparing for a customer meeting.”
• Language and tone:
  • Professional framing – Use polite, professional language in prompts. For example, “Please provide a concise summary of the customer’s cloud adoption journey.”
  • Specific directives – Include unambiguous instructions. For example, “Summarize in one paragraph” rather than “Provide a short summary.”
  • Positive framing – Frame instructions positively. For example, “Write a professional email” instead of “Don’t be unprofessional.”
  • Clear restrictions – Specify important limitations upfront. For example, “Respond without speculating or guessing. Don’t make up any statistics.”

Consider the following system design and optimization techniques:

• Architectural considerations:
  • Multi-stage prompting – Use initial prompts for data retrieval, followed by specific prompts for summary generation.
  • Dynamic templates – Adapt prompt templates based on retrieved customer information.
  • Model selection – Balance performance with cost, choosing appropriate models for different summary sections.
  • Asynchronous processing – Run LLM calls for different summary sections in parallel to reduce overall latency.
• Quality and improvement:
  • Output validation – Implement rigorous fact-checking before relying on generated summaries. For example, “Cross-reference generated figures with golden source business data.”
  • Consistency checks – Make sure instructions don’t contradict each other or the provided data. For example, “Review prompts to ensure we’re not asking for detailed financials while also instructing to exclude sensitive data.”
  • Step-by-step thinking – For complex summaries, instruct the model to think through steps to reduce hallucinations.
  • Feedback and iteration – Regularly analyze performance, gather user feedback, experiment, and iteratively improve prompts and processes.

Multi-model approach

Although crafting effective prompts is crucial, equally important is selecting the right models to process these prompts and generate accurate, relevant summaries. Our multi-model approach is key to achieving this goal. By using multiple models, specifically Amazon Titan and Anthropic Claude on Amazon Bedrock, we’re able to optimize various aspects of summary generation, resulting in more comprehensive, accurate, and tailored outputs.

The selection of appropriate models for different tasks is guided by several key criteria. First, we evaluate the specific capabilities of each model, looking at their unique strengths in handling certain types of queries or data. Next, we assess the model’s accuracy, which is its ability to generate factual and relevant content. And lastly, we consider speed and cost, which are also crucial factors.

Our architecture is designed to allow for flexible model switching and combination. This is achieved through a modular approach where each section of the summary can be generated independently and then combined into a cohesive whole. With continuous performance monitoring and feedback mechanisms in place, we are able to refine our model selection and prompting strategies over time.

As new models become available on Amazon Bedrock, we have a structured evaluation process in place. This involves benchmarking new models against our current selections across various metrics, running A/B tests, and gradually incorporating high-performing models into our production pipeline.

Mitigating hallucinations and enforcing quality

LLMs sometimes hallucinate because they optimize for the most probable text response to a prompt, balancing various elements like syntax, grammar, style, knowledge, reasoning, and emotion. This often leads to trade-offs, resulting in the insertion of invented facts, making the outputs seem convincing but inaccurate. We implemented several strategies to address common types of hallucinations:

• Incomplete data issue – LLMs may invent information when lacking necessary context.
  • Solution – We provide comprehensive datasets and explicit instructions to use only provided information. We also preprocess data to remove null points and include conditional instructions for available data points.
• Vague instructions issue – Ambiguous prompts can lead to guesswork and hallucinations.
  • Solution – We use detailed, specific prompts with clear and structured instructions to minimize ambiguity.
• Ambiguous context issue – Unclear context can result in plausible but inaccurate details.
  • Solution – We clarify context in prompts, specifying exact details required and using XML tags to distinguish between context, tasks, and instructions.

We deployed a multi-faceted approach to provide quality and accuracy with Account Summaries:

• Automated metrics – These automated metrics provide a quantitative foundation for our quality assurance process, allowing us to quickly identify potential issues in generated summaries before they undergo human review:
  • Cosine similarity – Measures the similarity between the input dataset and the generated response by calculating the cosine of the angle between their vector representations. This helps make sure the summary content aligns closely with the input data.
  • BLEU (Bilingual Evaluation Understudy) – Evaluates the quality of the response by calculating how many n-grams in the response match those in the input data. It focuses on precision, measuring how much of the generated content is present in the reference data.
  • ROUGE (Recall-Oriented Understudy for Gisting Evaluation) – Compares words and phrases present in both the response and input data, assessing how much relevant information from the input is included in the response.
  • Numbers checking – Identifies numerical data in both the input and generated documents, determining their intersection and flagging potential hallucinations. This helps catch any fabricated or misrepresented quantitative information in the summaries.
• Human review – The final outputs and the intermediate steps, including prompt formulations and data preprocessing, are part of the human review process. This includes evaluating a set of responses, checking for accuracy, hallucinations, completeness, adherence to constraints, and compliance with security and legal requirements. This collaborative approach makes sure Account Summaries meets the specific needs of our field teams, accurately represents AWS services, and responsibly handles customer information. Our human review process is comprehensive and integrated throughout the development lifecycle of the Account Summaries solution, involving a diverse group of stakeholders:
  • Field sellers and the Account Summaries product team – These personas collaborate from the early stages on prompt engineering, data selection, and source validation. AWS data teams make sure the information used is accurate, up to date, and appropriately utilized.
  • Application security (AppSec) teams – These teams are engaged to guide, assess, and mitigate potential security risks, making sure the solution adheres to AWS security standards.
  • End-users – End-users are required to review content created by the LLM for accuracy prior to using the content.
• Continuous feedback loop – We’ve implemented a robust, multi-channel feedback system to continuously improve Account Summaries:
  • In-app feedback – Users can provide feedback at both the summary and individual section levels, allowing for granular insights into the effectiveness of different components.
  • Daily seller interactions – Our teams engage in regular conversations (one-on-one and through a dedicated Slack channel) with our field teams, gathering real-time feedback and requests for new features and datasets.
  • Proactive follow-up – We personally reach out to and close the loop with every single instance of negative feedback, building trust and creating a cycle of continuous feedback.

This feeds into our refinement process for existing summaries and plays a crucial role in prioritizing our product roadmap. We also make sure this feedback reaches the relevant teams across AWS that manage data and insights. This allows them to address any issues with their models, augment datasets, or refine their insights based on real-world usage and field team needs. Given that our generative AI solution brings together data from various sources, this feedback loop is crucial for improving not just Account Summaries, but also the underlying data and models that feed into it. This approach has been instrumental in maintaining high user satisfaction, driving continuous improvement of Account Summaries.

Infrastructure and operations

The robustness and efficiency of our Account Summaries solution are underpinned by an architecture that uses AWS services to provide scalability, reliability, and security while optimizing for performance. Key components include asynchronous processing to manage response times, a multi-tiered approach to handling requests, and strategic use of services like AWS Lambda and Amazon DynamoDB. We’ve also implemented comprehensive monitoring and alerting systems to maintain high availability and quickly address any issues. The following diagram illustrates this architecture.

In the following subsections, we outline our API design, authentication mechanisms, response time optimization strategies, and operational practices that collectively enable us to deliver high-quality, timely account summaries at scale.

API design

Account summary generation requests are handled asynchronously to eliminate client wait times for responses. This approach addresses potential delays from downstream data sources and Amazon Bedrock, which can extend response times to several seconds. Two Lambda functions manage a seller’s summarization request: Synchronous Request Handler and Asynchronous Request Handler.

When a seller initiates a summarization request through the web application interface, the request is routed to the Synchronous Request Handler Lambda function. The function generates a requestId, validates the input provided by the seller, invokes the Asynchronous Request Handler function asynchronously, and sends an acknowledgment to the seller along with the requestId for tracking the request’s progress.

The Asynchronous Request Handler function gathers data from various data sources in parallel. It then invokes the Amazon Bedrock LLM in parallel, using the LLM model configuration and a prompt template populated with the gathered data. Amazon Bedrock invokes the appropriate LLM models based on the configuration to generate summarized content. For this use case, we utilize both the Amazon Titan and Anthropic Claude models, taking advantage of their unique strengths for different aspects of the summary generation. The Asynchronous Request Handler function stores results in a DynamoDB database along with the generated requestId.

Finally, the web application periodically polls for the summarized account summary using the generated requestId. The Synchronous Request Handler function retrieves the summarized content from DynamoDB and responds to the seller with the summary when the request is satisfied.

Authentication

The seller is authenticated in the web application using a centralized authentication system. All requests to the generative AI service are accompanied by a JWT, generated from the authentication system. The user’s authorization to access the generative AI service is based on their identity, which is verified using the JWT. When the generative AI service gathers data from various data sources, it uses the user’s identity, using row-level security by restricting access to only the data that the user is authorized to access.

Response time optimization

To enhance response times, we utilize a smaller LLM model such as Anthropic Claude Instant on Amazon Bedrock, which is known for its faster response rates. Larger models are reserved for prompts requiring more in-depth insights. The account summary is composed of multiple sections, each generated by running several prompts independently and in parallel. Data fetching for these prompts is also conducted in parallel to minimize response time.

Operational practices

All failures within the account summary are tracked through operational metrics dashboards and alerts. On-call schedules are in place to address these issues promptly. The team continuously monitors and strives to improve response times. For each major feature release, load tests are conducted to make sure predicted request rates remain within the limits for all downstream resources.

Building a production use case: Lessons learned

Our experience with implementing generative AI at scale offers valuable insights for organizations embarking on a similar journey:

• Pick the right first use case – One of the most common questions we’ve received is how we prioritized and landed on where to start. Although this may seem trivial, in retrospect it had a significant impact in earning trust with the organization. Launching a transformative technology like this at scale needs to be successful—and for that, it must be “correct” and useful.
• Prioritize use cases effectively – We evaluated using the following factors:
  • Business impact – There are many interesting applications of generative AI, but we prioritized this use case because field teams spend a significant amount of time researching information and knew that even small improvements at scale would have significant impact.
  • Data availability – The most critical aspect of any generative AI use case is the quality and reliability of the underlying data. We identified and assessed the availability and trustworthiness of the data sources required for Account Summaries, making sure it was accurate, up to date, and had the right access permissions in place. We also started with the data we already had, and over time integrated additional datasets and brought in external data.
  • Tech readiness – We evaluated the maturity and capabilities of the generative AI technologies available to us at the time. LLMs had demonstrated exceptional performance in tasks such as text summarization and generation, which aligned perfectly with the requirements of Account Summaries.
• Foster continuous learning – In the early stages of our generative AI journey, we encouraged our teams to experiment and build prototypes across various domains. This hands-on experience allowed our developers and data scientists to gain practical knowledge and understanding of the capabilities and limitations of generative AI. We continue this tradition even today because we know how fast new capabilities are being developed and we need our teams to keep pace with this change so we can build the best products for our field teams.
• Embrace iterative development – Generative AI product development is inherently iterative, requiring a continuous cycle of experimentation and refinement. Our development process revolved around crafting and fine-tuning prompts that would generate accurate, relevant, and actionable insights. We engaged in extensive prompt engineering, experimenting with different prompt structures, models, and output formats to achieve the desired results.
• Implement effective enablement and change management – We implemented a phased approach to deployment, starting with a small group of early adopters and gradually expanding to the wider organization. We established channels for users to provide feedback, report issues, and suggest improvements, fostering a culture of continuous improvement. We focused on nurturing a culture that embraces AI-assisted work, emphasizing that the technology is a tool to enhance field capabilities.
• Establish clear metrics and KPIs – We defined specific, measurable outcomes to gauge the success of Account Summaries. These metrics included user adoption rates, retention, time saved per summary generated, and impact on customer engagements. Regular assessment of these key performance indicators (KPIs) guided our ongoing development efforts.
• Foster cross-functional collaboration – The success of our Account Summaries solution relied heavily on collaboration between various teams, including data scientists, engineers, and sales representatives across AWS. This cross-functional approach make sure all aspects of the solution were thoroughly considered and optimized.

Conclusion

This post is the first in a series that explores how generative AI and ML are revolutionizing our field teams’ work and customer engagements. In upcoming posts, we dive into various use cases that transform different aspects of the sales journey, including:

• AI sales assistant powered by Amazon Q – We’ll explore our AI sales assistant, available across different modalities and seamlessly integrating with our other systems. You’ll learn how it answers questions, generates content, and facilitates bidirectional interactions, all while continuously using internal and external data to deliver timely, personalized insights.
• Autonomous agents for prospecting and customer engagement – We’ll showcase how AI-powered agents are transforming prospecting, opportunity progression, and customer engagement to drive efficiency and effectiveness.

We’re excited about the potential of these technologies to automate tasks, provide recommendations, and free up time for strategic interactions. We encourage you to explore these possibilities, experiment with AWS AI services, and embark on your own transformation journey. Stay tuned for our upcoming posts, where we’ll continue to unfold the story of how AI is reshaping the Sales & Marketing organization at AWS.

About the Authors

Rupa Boddu is the Principal Tech Product Manager leading Generative AI strategy and development for the AWS Sales and Marketing organization. She has successfully launched AI/ML applications across AWS and collaborates with executive teams of AWS customers to shape their AI strategies. Her career spans leadership roles across startups and regulated industries, where she has driven cloud transformations, led M&A integrations, and held global leadership positions encompassing COO responsibilities, sales, software development, and infrastructure.

Raj Aggarwal is the GM of GenAI & Revenue Acceleration for the AWS GTM organization. Raj is responsible for developing the Gen AI strategy and products to transform field functions, GTM motions, and the seller and customer journeys across the global AWS Sales & Marketing organization. His team has built and launched high-impact, production applications at-scale, and served as a key design partner for many of Amazon’s GenAI products. Prior to this, Raj built and exited two companies. As Founder/CEO of Localytics, the leading mobile analytics & messaging provider, he grew it to $25M ARR with 200+ employees.

Asa Kalavade leads AWS Field Experiences, overseeing tools and processes for the AWS GTM organization across all roles and customer engagement stages. Over the past two years, she led a transformation that consolidated hundreds of disparate systems into a streamlined, role-based experience, incorporating generative AI to reimagine the customer journey. Previously, as GM for the AWS hybrid storage portfolio, Asa launched several key services, including AWS File Gateway, AWS Transfer Family, and AWS DataSync. Before joining AWS, she founded two venture-backed startups in Boston.

Read More

Amazon Q Business is a conversational assistant powered by generative artificial intelligence (AI) that enhances workforce productivity by answering questions and completing tasks based on information in your enterprise systems, which each user is authorized to access. In an earlier post, we discussed how you can build private and secure enterprise generative AI applications with Amazon Q Business and AWS IAM Identity Center. If you want to use Amazon Q Business to build enterprise generative AI applications, and have yet to adopt organization-wide use of AWS IAM Identity Center, you can use Amazon Q Business IAM Federation to directly manage user access to Amazon Q Business applications from your enterprise identity provider (IdP), such as Okta or Ping Identity. Amazon Q Business IAM Federation uses Federation with IAM and doesn’t require the use of IAM Identity Center.

AWS recommends using AWS Identity Center if you have a large number of users in order to achieve a seamless user access management experience for multiple Amazon Q Business applications across many AWS accounts in AWS Organizations. You can use federated groups to define access control, and a user is charged only one time for their highest tier of Amazon Q Business subscription. Although Amazon Q Business IAM Federation enables you to build private and secure generative AI applications, without requiring the use of IAM Identity Center, it is relatively constrained with no support for federated groups, and limits the ability to charge a user only one time for their highest tier of Amazon Q Business subscription to Amazon Q Business applications sharing SAML identity provider or OIDC identity provider in a single AWS accouGnt.

This post shows how you can use Amazon Q Business IAM Federation for user access management of your Amazon Q Business applications.

Solution overview

To implement this solution, you create an IAM identity provider for SAML or IAM identity provider for OIDC based on your IdP application integration. When creating an Amazon Q Business application, you choose and configure the corresponding IAM identity provider.

When responding to requests by an authenticated user, the Amazon Q Business application uses the IAM identity provider configuration to validate the user identity. The application can respond securely and confidentially by enforcing access control lists (ACLs) to generate responses from only the enterprise content the user is authorized to access.

We use the same example from Build private and secure enterprise generative AI apps with Amazon Q Business and AWS IAM Identity Center—a generative AI employee assistant built with Amazon Q Business—to demonstrate how to set it up using IAM Federation to only respond using enterprise content that each employee has permissions to access. Thus, the employees are able to converse securely and privately with this assistant.

Architecture

Amazon Q Business IAM Federation requires federating the user identities provisioned in your enterprise IdP such as Okta or Ping Identity account using Federation with IAM. This involves a onetime setup of creating a SAML or OIDC application integration in your IdP account, and then creating a corresponding SAML identity provider or an OIDC identity provider in AWS IAM. This SAML or OIDC IAM identity provider is required for you to create an Amazon Q Business application. The IAM identity provider is used by the Amazon Q Business application to validate and trust federated identities of users authenticated by the enterprise IdP, and associate a unique identity with each user. Thus, a user is uniquely identified across all Amazon Q Business applications sharing the same SAML IAM identity provider or OIDC IAM identity provider.

The following diagram shows a high-level architecture and authentication workflow. The enterprise IdP, such as Okta or Ping Identity, is used as the access manager for an authenticated user to interact with an Amazon Q Business application using an Amazon Q web experience or a custom application using an API.

The user authentication workflow consists of the following steps:

1. The client application makes an authentication request to the IdP on behalf of the user.
2. The IdP responds with identity or access tokens in OIDC mode, or a SAML assertion in SAML 2.0 mode. Amazon Q Business IAM Federation requires the enterprise IdP application integration to provide a special principal tag email attribute with its value set to the email address of the authenticated user. If user attributes such as role or location (city, state, country) are present in the SAML or OIDC assertions, Amazon Q Business will extract these attributes for personalization. These attributes are included in the identity token claims in OIDC mode, and SAML assertions in the SAML 2.0 mode.
3. The client application makes an AssumeRoleWithWebIdentity (OIDC mode) or AssumeRoleWithSAML (SAML mode) API call to AWS Security Token Service (AWS STS) to acquire AWS Sig V4 credentials. Email and other attributes are extracted and enforced by the Amazon Q Business application using session tags in AWS STS. The AWS Sig V4 credentials include information about the federated user.
4. The client application uses the credentials obtained in the previous step to make Amazon Q Business API calls on behalf of the authenticated user. The Amazon Q Business application knows the user identity based on the credential used to make the API calls, shows only the specific user’s conversation history, and enforces document ACLs. The application retrieves only those documents from the index that the user is authorized to access and are relevant to the user’s query, to be included as context when the query is sent to the underlying large language model (LLM). The application generates a response based only on enterprise content that the user is authorized to access.

How subscriptions work with Amazon Q Business IAM Federation

The way user subscriptions are handled when you use IAM Identity Center vs. IAM Federation is different.

For applications that use IAM Identity Center, AWS will de-duplicate subscriptions across all Amazon Q Business applications accounts, and charge each user only one time for their highest subscription level. De-duplication will apply only if the Amazon Q Business applications share the same organization instance of IAM Identity Center. Users subscribed to Amazon Q Business applications using IAM federation will be charged one time when they share the same SAML IAM identity provider or OIDC IAM identity provider. Amazon Q Business applications can share the same SAML IAM identity provider or OIDC IAM identity provider only if they are in the same AWS account. For example, if you use Amazon Q Business IAM Federation, and need to use Amazon Q Business applications across 3 separate AWS accounts, each AWS account will require its own SAML identity provider or OIDC identity provider to be created and used in the corresponding Amazon Q Business applications, and a user subscribed to these three Amazon Q Business applications will be charged three times. In another example, if a user is subscribed to some Amazon Q Business applications that use IAM Identity Center and others that use IAM Federation, they will be charged one time across all IAM Identity Center applications and one time per SAML IAM identity provider or OIDC IAM identity provider used by the Amazon Q Business applications using IAM Federation.

For Amazon Q Business applications using IAM Identity Center, the Amazon Q Business administrator directly assigns subscriptions for groups and users on the Amazon Q Business management console. For an Amazon Q Business application using IAM federation, the administrator chooses the default subscription tier during application creation. When an authenticated user logs in using either the Amazon Q Business application web experience or a custom application using the Amazon Q Business API, that user is automatically subscribed to the default tier.

Limitations

At the time of writing, Amazon Q Business IAM Federation has the following limitations:

1. Amazon Q Business doesn’t support OIDC for Google and Microsoft Entra ID.
2. There is no built-in mechanism to validate a user’s membership to federated groups defined in the enterprise IdP. If you’re using ACLs in your data sources with groups federated from the enterprise IdP, you can use the PutGroup API to define the federated groups in the Amazon Q Business user store. This way, the Amazon Q Business application can validate a user’s membership to the federated group and enforce the ACLs accordingly. This limitation does not apply to configurations where groups used in ACLs are defined locally within the data sources. For more information, refer to Group mapping.

Guidelines to choosing a user access mechanism

The following table summarizes the guidelines to consider when choosing a user access mechanism.

Prerequisites

To implement the sample use case described in this post, you need an Okta account. This post covers workflows for both OIDC and SAML 2.0, so you can follow either one or both workflows based on your interest. You need to create application integrations for OIDC or SAML mode, and then configure the respective IAM identity providers in your AWS account, which will be required to create and configure your Amazon Q Business applications. Though you use the same Okta account and the same AWS account to create two Amazon Q Business applications one using an OIDC IAM identity provider, and the other using SAML IAM identity provider, the same user subscribed to both these Amazon Q Business applications will be charged twice, since they don’t share the underlying SAML or OIDC IAM identity providers.

Create an Amazon Q Business application with an OIDC IAM identity provider

To set up an Amazon Q Business application with an OIDC IAM identity identifier, you first configure the Okta application integration using OIDC. Then you create an IAM identity provider for that OIDC app integration, and create an Amazon Q Business application using that OIDC IAM identity provider. Lastly, you update the Okta application integration with the web experience URIs of the newly created Amazon Q Business application.

Create an Okta application integration with OIDC

Complete the following steps to create your Okta application integration with OIDC:

1. On the administration console of your Okta account, choose Applications, then Applications in the navigation pane.
2. Choose Create App Integration.
3. For Sign-in method, select OIDC.
4. For Application type, select Web Application.
5. Choose Next.
   

6. Give your app integration a name.
7. Select Authorization Code and Refresh Token for Grant Type.
8. Confirm that Refresh token behavior is set to Use persistent token.
9. For Sign-in redirect URIs, provide a placeholder value such as https://example.com/authorization-code/callback.

You update this later with the web experience URI of the Amazon Q Business application you create.

10. On the Assignments tab, assign access to appropriate users within your organization to your Amazon Q Business application.

In this step, you can select all users in your Okta organization, or choose select groups, such as Finance-Group if it’s defined, or select individual users.

11. Choose Save to save the app integration.

Your app integration will look similar to the following screenshots.

12. Note the values for Client ID and Client secret to use in subsequent steps.
    
    

13. On the Sign on tab, choose Edit next to OpenID Connect ID Token.
14. For Issuer, note the Okta URL.
15. Choose Cancel.
    

16. In the navigation pane, choose Security and then API.
17. Under API, Authorization Servers, choose default.
18. On the Claims tab, choose Add Claim.
19. For Name, enter https://aws.amazon.com/tags.
20. For Include in token type, select ID Token.
21. For Value, enter {"principal_tags": {"Email": {user.email}}}.
22. Choose Create.

The claim will look similar to the following screenshot. It is a best practice to use a custom authorization server. However, because this is an illustration, we use the default authorization server.

Set up an IAM identity provider for OIDC

To set up an IAM identity provider for OIDC, complete the following steps:

1. On the IAM console, choose Identity providers in the navigation pane.
2. Choose Add provider.
3. For Provider type, select OpenID Connect.
4. For Provider URL, enter the Okta URL you copied earlier, followed by /oauth2/default.
5. For Audience, enter the client ID you copied earlier.
6. Choose Add provider.
   

Create an Amazon Q Business application with the OIDC IAM identity provider

Complete the following steps to create an Amazon Q Business application with the OIDC IdP:

1. On the Amazon Q Business console, choose Create application.
2. Give the application a name.
3. For Access management method, select AWS IAM Identity provider.
4. For Choose an Identity provider type, select OpenID Connect (OIDC).
5. For Select Identity Provider, choose the IdP you created.
6. For Client ID, enter the client ID of the Okta application integration you copied earlier.
7. Leave the remaining settings as default and choose Create.
   

8. In the Select retriever step, unless you want to change the retriever type or the index type, choose Next.
9. For now, select Next on the Connect data sources We configure the data source later.

On the Manage access page, in Default subscription settings, Subscription Tier of Q Business Pro is selected by default. This means that when an authenticated user starts using the Amazon Q Business application, they will automatically get subscribed as Amazon Q Business Pro. The Amazon Q Business administrator can change the subscription tier for a user at any time.

10. In Web experience settings uncheck Create web experience. Choose Done.
    
11. On the Amazon Q Business Applications page, choose the application you just created to view the details.
12. In the Application Details page, note the Application ID.
    
13. In a new tab of your web browser open the management console for AWS Secrets Manager. Choose Store a new secret.
14. For Choose secret type choose Other type of secret. For Key/value pairs, enter client_secret as key and enter the client secret you copied from the Okta application integration as value. Choose Next.
    
15. For Configure secret give a Secret name.
16. For Configure rotation, unless you want to make any changes, accept the defaults, and choose Next.
17. For Review, review the secret you just stored, and choose Store.
18. On AWS Secrets Manager, Secrets page choose the secret you just created. Note the Secret name and Secret ARN.
    
19. Follow the instructions on IAM role for an Amazon Q web experience using IAM Federation to create Web experience IAM role, and Secret Manager Role. You will require the Amazon Q Business Application ID, Secret name and Secret ARN you copied earlier.
20. Open the Application Details for your Amazon Q Business application. Choose Edit.
21. For Update application, there is no need to make changes. Choose Update.
22. For Update retriever, there is no need to make changes. Choose Next.
23. For Connect data sources, there is no need to make changes. Choose Next.
24. For Update access, select Create web experience.
25. For Service role name select the web experience IAM role you created earlier.
26. For AWS Secrets Manager secret, select the secret you stored earlier.
27. For Web Experience to use Secrets: Service role name, select the Secret Manager Role you created earlier.
28. Choose Update.
    
29. On the Amazon Q Business Applications page, choose the application you just updated to view the details.
30. Note the value for Deployed URL.
    

Before you can use the web experience to interact with the Amazon Q Business application you just created, you need to update the Okta application integration with the redirect URL of the web experience.

18. Open the Okta administration console, then open the Okta application integration you created earlier.
19. On the General tab, choose Edit next to General Settings.
20. For Sign-in redirect URIs, replace the placeholder https://example.com/ with the value for Deployed URL of your web experience. Make sure the authorization-code/callback suffix is not deleted. The full URL should look like https://your_deployed_url/authorization-code/callback.
21. Choose Save.
    

Create an Amazon Q Business application with a SAML 2.0 IAM identity provider

The process to set up an Amazon Q Business application with a SAML 2.0 IAM identity provider is similar to creating an application using OIDC. You first configure an Okta application integration using SAML 2.0. Then you create an IAM identity provider for that SAML 2.0 app integration, and create an Amazon Q Business application using the SAML 2.0 IAM identity provider. Lastly, you update the Okta application integration with the web experience URIs of the newly created Amazon Q Business application.

Create an Okta application integration with SAML 2.0

Complete the following steps to create your Okta application integration with SAML 2.0:

1. On the administration console of your Okta account, choose Applications, then Applications in the navigation pane.
2. Choose Create App Integration.
3. For Sign-in method, select SAML 2.0.
4. Choose Next.
   

5. On the General Settings page, enter an app name and choose Next.

This will open the Create SAML Integration page.

6. For Single sign-on URL, enter a placeholder URL such as https://example.com/saml and deselect Use this for Recipient URL and Destination URL.
7. For Recipient URL, enter https://signin.aws.amazon.com/saml.
8. For Destination URL, enter the placeholder https://example.com/saml.
9. For Audience URL (SP Entity ID), enter https://signin.aws.amazon.com/saml.
10. For Name ID format, choose Persistent.
11. Choose Next and then Finish.

The placeholder values of https://example.com will need to be updated with the deployment URL of the Amazon Q Business web experience, which you create in subsequent steps.

12. On the Sign On tab of the app integration you just created, note the value for Metadata URL.
    

13. Open the URL in your web browser, and save it on your local computer.

The metadata will be required in subsequent steps.

Set up an IAM identity provider for SAML 2.0

To set up an IAM IdP for SAML 2.0, complete the following steps:

1. On the IAM console, choose Identity providers in the navigation pane.
2. Choose Add provider.
3. For Provider type, select SAML.
4. Enter a provider name.
5. For Metadata document, choose Choose file and upload the metadata document you saved earlier.
6. Choose Add provider.
   

7. From the list of identity providers, choose the identity provider you just created.
8. Note the values for ARN, Issuer URL, and SSO service location to use in subsequent steps.
   

Create an Amazon Q Business application with the SAML 2.0 IAM identity provider

Complete the following steps to create an Amazon Q Business application with the SAML 2.0 IAM identity provider:

1. On the Amazon Q Business console, choose Create application.
2. Give the application a name.
3. For Access management method, select AWS IAM Identity provider.
4. For Choose an Identity provider type, select SAML.
5. For Select Identity Provider, choose the IdP you created.
6. Leave the remaining settings as default and choose Create.
   

7. In the Select retriever step, unless you want to change the retriever type or the index type, choose Next.
8. For now, choose Next on the Connect data sources We will configure the data source later.

On the Manage access page, in Default subscription settings, Subscription Tier of Q Business Pro is selected by default. This means that when an authenticated user starts using the Amazon Q Business application, they will automatically get subscribed as Amazon Q Business Pro. The Amazon Q Business administrator can change the subscription tier for a user at any time.

1. For Web experience settings, uncheck Create web experience. Choose Done.
2. On the Amazon Q Business Applications page, choose the application you just created.
3. In the Application Details page, note the Application ID.
   
4. Follow the instructions on IAM role for an Amazon Q web experience using IAM Federation to create Web experience IAM role. You will require the Amazon Q Business Application ID you copied earlier.
5. Open the Application Details for your Amazon Q Business application. Choose Edit.
6. For Update application, there is no need to make changes. Choose Update.
7. For Update retriever, there is no need to make changes. Choose Next.
8. For Connect data sources, there is no need to make changes. Choose Next.
9. For Update access, select Create web experience.
10. For this post, we continue with the default setting.
11. For Authentication URL, enter the value for SSO service location that you copied earlier.
12. Choose Update.
    
13. On the Amazon Q Business Applications page, choose the application you just updated to view the details.
14. Note the values for Deployed URL and Web experience IAM role ARN to use in subsequent steps.
    

 Before you can use the web experience to interact with the Amazon Q Business application you just created, you need to update the Okta application integration with the redirect URL of the web experience.

15. Open the Okta administration console, then open the Okta application integration you created earlier.
16. On the General tab, choose Edit next to SAML Settings.
17. For Single sign-on URL and Destination URL, replace the placeholder https://example.com/ with the value for Deployed URL of your web experience. Make sure the /saml suffix isn’t deleted.
18. Choose Save.
    

19. On the Edit SAML Integration page, in the Attribute Statements (optional) section, add attribute statements as listed in the following table.

This step is not optional and these attributes are used by the Amazon Q Business application to determine the identity of the user, so be sure to confirm their correctness.

For the value of the https://aws.amazon.com/SAML/Attributes/Role attribute, you need to concatenate the web experience IAM role ARN and IdP ARN you copied earlier with a comma between them, without spaces or any other characters.

20. Choose Next and Finish.
21. On the Assignments tab, assign users who can access the app integration you just created.

This step controls access to appropriate users within your organization to your Amazon Q Business application. In this step, you can enable self-service so that all users in your Okta organization, or choose select groups, such as Finance-Group if it’s defined, or select individual users.

Set up the data source

Whether you created the Amazon Q Business application using an OIDC IAM identity provider or SAML 2.0 IAM identity provider, the procedure to create a data source remains the same. For this post, we set up a data source for Atlassian Confluence. The following steps show how to configure the data source for the Confluence environment. For more details on how to set up a Confluence data source, refer to Connecting Confluence (Cloud) to Amazon Q Business.

1. On the Amazon Q Business Application details page, choose Add data source.
   

2. On the Add data source page, choose Confluence.
   

3. For Data source name, enter a name.
4. For Source, select Confluence Cloud and enter the Confluence URL.
   

5. For Authentication, select Basic authentication and enter the Secrets Manager secret.
6. For IAM role, select Create a new service role.
7. Leave the remaining settings as default.
   

8. For Sync scope, select the appropriate content to sync.
9. Under Space and regex patterns, provide the Confluence spaces to be included.
10. For Sync mode, select Full sync.
11. For Sync run schedule, choose Run on demand.
12. Choose Add data source.
    

13. After the data source creation is complete, choose Sync now to start the data source sync.
    

Wait until the sync is complete before logging in to the web experience to start querying.

Employee AI assistant use case

To illustrate how you can build a secure and private generative AI assistant for your employees using Amazon Q Business applications, let’s take a sample use case of an employee AI assistant in an enterprise corporation. Two new employees, Mateo Jackson and Mary Major, have joined the company on two different projects, and have finished their employee orientation. They have been given corporate laptops, and their accounts are provisioned in the corporate IdP. They have been told to get help from the employee AI assistant for any questions related to their new team member activities and their benefits.

The company uses Confluence to manage their enterprise content. The sample Amazon Q application used to run the scenarios for this post is configured with a data source using the built-in connector for Confluence to index the enterprise Confluence spaces used by employees. The example uses three Confluence spaces with the following permissions:

• HR Space – All employees, including Mateo and Mary
• AnyOrgApp Project Space – Employees assigned to the project, including Mateo
• ACME Project Space – Employees assigned to the project, including Mary

Let’s look at how Mateo and Mary experience their employee AI assistant.

Both are provided with the URL of the employee AI assistant web experience. They use the URL and sign in to the IdP from the browsers of their laptops. Mateo and Mary both want to know about their new team member activities and their fellow team members. They ask the same questions to the employee AI assistant but get different responses, because each has access to separate projects. In the following screenshots, the browser window on the left is for Mateo Jackson and the one on the right is for Mary Major. Mateo gets information about the AnyOrgApp project and Mary gets information about the ACME project.

Mateo chooses Sources under the question about team members to take a closer look at the team member information, and Mary chooses Sources under the question for the new team member checklist. The following screenshots show their updated views.

Mateo and Mary want to find out more about the benefits their new job offers and how the benefits are applicable to their personal and family situations.

The following screenshot shows that Mary asks the employee AI assistant questions about her benefits and eligibility.

Mary can also refer to the source documents.

The following screenshot shows that Mateo asks the employee AI assistant different questions about his eligibility.

Mateo looks at the following source documents.

Both Mary and Mateo first want to know their eligibility for benefits. But after that, they have different questions to ask. Even though the benefits-related documents are accessible by both Mary and Mateo, their conversations with the employee AI assistant are private and personal. The assurance that their conversation history is private and can’t be seen by any other user is critical for the success of a generative AI employee productivity assistant.

Clean up

If you created a new Amazon Q Business application to try out the integration with IAM federation, and don’t plan to use it further, you can unsubscribe, remove automatically subscribed users from the application, and delete it so that your AWS account doesn’t accumulate costs.

1. To unsubscribe and remove users, go to the application details page and choose Manage subscriptions.
   

2. Select all the users, choose Remove to remove subscriptions, and choose Done.
   

3. To delete the application after removing the users, return to the application details page and choose Delete.
   

Conclusion

For enterprise generative AI assistants such as the one shown in this post to be successful, they must respect access control as well as assure the privacy and confidentiality of every employee. Amazon Q Business achieves this by integrating with IAM Identity Center or with IAM Federation to provide a solution that authenticates each user and validates the user identity at each step to enforce access control along with privacy and confidentiality.

In this post, we showed how Amazon Q Business IAM Federation uses SAML 2.0 and OIDC IAM identity providers to uniquely identify a user authenticated by the enterprise IdP, and then that user identity is used to match up document ACLs set up in the data source. At query time, Amazon Q Business responds to a user query utilizing only those documents that the user is authorized to access. This functionality is similar to that achieved by the integration of Amazon Q Business with IAM Identity Center we saw in an earlier post. Additionally, we also provided the guidelines to consider when choosing a user access mechanism.

To learn more, refer to Amazon Q Business, now generally available, helps boost workforce productivity with generative AI and the Amazon Q Business User Guide.

About the authors

Abhinav Jawadekar is a Principal Solutions Architect in the Amazon Q Business service team at AWS. Abhinav works with AWS customers and partners to help them build generative AI solutions on AWS.

Venky Nagapudi is a Senior Manager of Product Management for Q Business, Amazon Comprehend and Amazon Translate. His focus areas on Q Business include user identity management, and using offline intelligence from documents to improve Q Business accuracy and helpfulness.

Read More

This post is co-written with Sneha Godbole and Kate Riordan from Verisk.

Verisk (Nasdaq: VRSK) is a leading strategic data analytics and technology partner to the global insurance industry. It empowers its customers to strengthen operating efficiency, improve underwriting and claims outcomes, combat fraud, and make informed decisions about global risks, including climate change, extreme events, sustainability, and political issues. At the forefront of harnessing cutting-edge technologies in the insurance sector such as generative artificial intelligence (AI), Verisk is committed to enhancing its clients’ operational efficiencies, productivity, and profitability. Verisk’s generative AI-powered solutions and applications are developed with a steadfast commitment to ethical and responsible use of AI, incorporating privacy and security controls, human oversight, and transparent practices consistent with its ethical AI principles and governance practices.

Verisk’s Discovery Navigator product is a leading medical record review platform designed for property and casualty claims professionals, with applications to any industry that manages large volumes of medical records. It streamlines document review for anyone needing to identify medical information within records, including bodily injury claims adjusters and managers, nurse reviewers and physicians, administrative staff, and legal professionals. By replacing hours of manual review for a single claim, insurers can modernize the reviewer’s workflow, saving time and empowering better, faster decision-making, which is critical to improving outcomes.

With AI-powered analysis, the process of reviewing an average file of a few hundred pages is reduced to minutes with Discovery Navigator. By responsibly building proprietary AI models created with Verisk’s extensive clinical, claims, and data science expertise, complex and unstructured documents are automatically organized, reviewed, and summarized. It employs sophisticated AI to extract medical information from records, providing users with structured information that can be easily reviewed and uploaded into their claims management system. This allows reviewers to access necessary information in minutes, compared to the hours spent doing this manually.

Discovery Navigator recently released automated generative AI record summarization capabilities. It was built using Amazon Bedrock, a fully managed service from AWS that provides access to foundation models (FMs) from leading AI companies through an API to build and scale generative AI applications. This new functionality offers an immediate overview of the initial injury and current medical status, empowering record reviewers of all skill levels to quickly assess injury severity with the click of a button. By automating the extraction and organization of key treatment data and medical information into a concise summary, claims handlers can now identify important bodily injury claims data faster than before.

In this post, we describe the development of the automated summary feature in Discovery Navigator incorporating generative AI, the data, the architecture, and the evaluation of the pipeline.

Solution overview

Discovery Navigator is designed to retrieve medical information and generate summaries from medical records. These medical records are mostly unstructured documents, often containing multiple dates of service. Examples of the myriad of documents include provider notes, tables in different formats, body figures to describe the injury, medical charts, health forms, and handwritten notes. The medical record documents are scanned and typically available as a single file.

Following a virus scan, the most immediate step in Discovery Navigator’s AI pipeline is to convert the scanned image pages of medical records into searchable documents. For this optical character recognition (OCR) conversion process, Discovery Navigator uses Amazon Textract.

The following figure illustrates the architecture of the Discovery Navigator AI pipeline.

Discovery Navigator AI Pipeline

The OCR converted medical records are passed through various AI models that extract key medical data. The AI extracted medical information is used to add highlighting in the original medical record document and to generate an indexed report. The highlighted medical record document allows the user to focus on the provided results and target their review towards the pages with highlights, thereby saving time. The report gives a quick summary of the extracted medical information with page links to navigate through the document for review.

The following figure shows the Discovery Navigator generative AI auto-summary pipeline. The OCR converted medical record pages are processed through Verisk’s AI models and select pages are sent to Amazon Bedrock using AWS PrivateLink, for generating visit summaries. The user is given a summary report consisting of AI extracted medical information and generative AI summaries.

Discovery Navigator Inference Pipeline

Discovery Navigator results

Discovery Navigator produces results in two different ways: first, it provides an initial document containing an indexed report of identified medical data points and includes a highlighting feature within the original document to emphasize the results. Additionally, an optional automated high-level summary created through generative AI capabilities is provided.

Discovery Navigator offers multiple different medical models, for example, diagnosis codes. These codes are identified and highlighted in the document. In the sample in the following figure, additional intelligence is provided utilizing a note feature to equip the user with the clinical description directly on the page, avoiding time spent locating this information elsewhere. The Executive Summary report displays an overview of all the medical terms extracted from the medical record, and the Index Report provides page links for quick review.

Indexed reports of extracted medical information

Discovery Navigator’s new generative AI summary feature creates an in-depth summarization report, as shown in the following figure. This report includes a summary of the initial injury following the date of loss, a list of certain medical information extracted from the medical record, and a summary of the future treatment plan based on the most recent visit in the medical record.

Discovery Navigator Executive Summary

Performance

To assess the generative AI summary quality, Verisk designed human evaluation metrics with the help of in-house clinical expertise. Verisk conducted multiple rounds of human evaluation of the generated summaries with respect to the medical records. Feedback from each round of tests was incorporated in the following test.

Verisk’s evaluation involved three major parts:

• Prompt engineering – Prompt engineering is the process where you guide generative AI solutions to generate desired output. Verisk framed prompts using their in-house clinical experts’ knowledge on medical claims. With each round of testing, Verisk added instructions to the prompts to capture the pertinent medical information and to reduce possible hallucinations. The generative AI large language model (LLM) can be prompted with questions or asked to summarize a given text. Verisk decided to test three approaches: a question answer prompt, summarize prompt, and question answer prompt followed by summarize prompt.
• Splitting of document pages – The medical record generative AI summaries are created for each date of visit in the medical record. Verisk tested two strategies of splitting the pages by visit: split visit pages individually and send them to a text splitter to generate text chunks for generative AI summarization, or concatenate all visit pages and send them to a text splitter to generate text for generative AI summarization. Summaries generated from each strategy were used during evaluation of the generative AI summary.
• Quality of summary – For the generative AI summary, Verisk wanted to capture information regarding the reason for visit, assessment, and future treatment plan. For evaluation of summary quality, Verisk created a template of questions for the clinical expert, which allowed them to assess the best performing prompt in terms of inclusion of required medical information and the best document splitting strategy. The evaluation questions also collected feedback on the number of hallucinations and inaccurate or not helpful information. For each summary presented to the clinical expert, they were asked to categorize it as either good, acceptable, or bad.

Based on Verisk’s evaluation template questions and rounds of testing, they concluded that the question answer prompt with concatenated pages generated over 90% good or acceptable summaries with low hallucinations and inaccurate or unnecessary information.

Business impact

By quickly and accurately summarizing key medical data from bodily injury claims, Verisk’s Discovery Navigator, with its new generative AI auto-summary feature powered by Amazon Bedrock, has immense potential to drive operational efficiencies and boost profitability for insurers. The automated extraction and summarization of critical treatment information allows claims handlers to expedite the review process, thereby reducing settlement times. This accelerated claim resolution can help minimize claims leakage and optimize resource allocation, enabling insurers to focus efforts on more complex cases. The Discovery Navigator platform has a proven to be up to 90% faster than manual record review, allowing claims handlers to compile record summaries in a fraction of the time.

Conclusion

The incorporation of generative AI into Discovery Navigator underscores Verisk’s commitment to using cutting-edge technologies to drive operational efficiencies and enhance outcomes for its clients in the insurance industry. By automating the extraction and summarization of key medical data, Discovery Navigator empowers claims professionals to expedite the review process, facilitate quicker settlements, and ultimately provide a superior experience for customers. The collaboration with AWS and the successful integration of FMs from Amazon Bedrock have been pivotal in delivering this functionality. The rigorous evaluation process, guided by Verisk’s clinical expertise, makes sure that the generated summaries meet the highest standards of accuracy, relevance, and reliability.

As Verisk continues to explore the vast potential of generative AI, the Discovery Navigator auto-summary feature serves as a testament to the company’s dedication to responsible and ethical AI adoption. By prioritizing transparency, security, and human oversight, Verisk aims to build trust and drive innovation while upholding its core values. Looking ahead, Verisk remains steadfast in its pursuit of harnessing advanced technologies to unlock new levels of efficiency, insight, and value for its global customer base. With a focus on continuous improvement and a deep understanding of industry needs, Verisk is poised to shape the future of insurance analytics and drive resilience across communities and businesses worldwide.

Resources

• Explore generative AI on AWS
• Learn about unlocking the business value of generative AI
• Learn more about Anthropic Claude 3 on Amazon Bedrock
• Learn about Amazon Bedrock and how to build and scale generative AI applications with FMs
• Generative AI Quickstart POCs

About the Authors

Sneha Godbole is a AVP of Analytics at Verisk. She has partnered with Verisk leaders on creating Discovery Navigator, an AI powered tool that automatically enables identification and retrieval of key data points within large unstructured documents. Sneha holds two Master of Science degrees (from University of Utah and SUNY Buffalo) and a Data Science Specialization certificate from Johns Hopkins University. Prior to joining Verisk, Sneha has worked as a software developer in France to build android solutions and collaborated on a paper publication with Brigham Young University, Utah.

Kate Riordan is the Director of Automation Initiatives at Verisk. She currently is the product owner for Discovery Navigator, an AI powered tool that automatically enables identification and retrieval of key data points within large unstructured documents and oversees automation and efficiency projects. Kate began her career at Verisk as a Medicare Set Aside compliance attorney. In that role, she completed and obtained CMS approval of hundreds of Medicare Set Asides. She is fluent in Section 111 reporting requirements, the conditional payment recovery process, Medicare Advantage, Part D and Medicaid recovery. Kate is a member of the Massachusetts bar.

Ryan Doty is a Sr. Solutions Architect at AWS, based out of New York. He helps enterprise customers in the Northeast U.S. accelerate their adoption of the AWS Cloud by providing architectural guidelines to design innovative and scalable solutions. Coming from a software development and sales engineering background, the possibilities that the cloud can bring to the world excite him.

Tarik Makota is a Principal Solutions Architect with Amazon Web Services. He provides technical guidance, design advice, and thought leadership to AWS’ customers across the US Northeast. He holds an M.S. in Software Development and Management from Rochester Institute of Technology.

Dom Bavaro is a Senior Solutions Architect for Financial Services. While providing technical guidance to customers across many use cases, He is focused on helping customer build and productionize Generative AI solutions and workflows.

Read More

Amazon Q Business is a generative artificial intelligence (AI)-powered assistant designed to enhance enterprise operations. It’s a fully managed service that helps provide accurate answers to users’ questions while honoring the security and access restrictions of the content. It can be tailored to your specific business needs by connecting to your company’s information and enterprise systems using built-in connectors to a variety of enterprise data sources. Amazon Q Business enables users in various roles, such as marketing managers, project managers, and sales representatives, to have tailored conversations, solve business problems, generate content, take action, and more, through a web interface. This service aims to help make employees work smarter, move faster, and drive significant impact by providing immediate and relevant information to help them with their tasks.

One such enterprise data repository you can use to store content is Atlassian Confluence. Confluence is a team workspace that provides a place to create, and collaborate on various projects, products, or ideas. Team spaces help your teams structure, organize, and share work, so each user has visibility into the institutional knowledge of the enterprise and access to the information they need or answers to the questions they have.

There are two Confluence offerings:

• Cloud – This is offered as a software as a service (SaaS) product. It’s always on and continuously updated.
• Data Center (self-managed) – Here, you host Confluence on your infrastructure, which may be on premises or the cloud, allowing you to keep data within your chosen environment and manage it yourself.

Your users may need to get answers in Amazon Q Business from the content in Atlassian’s Confluence Cloud instance as a part of their work. For this you will need to configure an Amazon Q Confluence Cloud connector. As a part of this configuration, one of the steps is to configure the authentication of the connector so that it can authenticate with Confluence (Cloud) and then index the relevant content.

This post covers the steps to configure the Confluence Cloud connector for Amazon Q Business.

Types of documents

When you connect Amazon Q to a data source, what Amazon Q considers—and crawls—as a document varies by connector. The Confluence Cloud connector crawls the following as documents:

• Spaces – Each space is considered a single document.
• Pages – Each page is considered a single document.
• Blogs – Each blog is considered a single document.
• Comments – Each comment is considered a single document.
• Attachments – Each attachment is considered a single document.

Metadata

Every document has structural attributes—or metadata—attached to it. Document attributes can include information such as document title, document author, time created, time updated, and document type.

When you connect Amazon Q Business to a data source, it automatically maps specific data source document attributes to fields within an Amazon Q Business index. If a document attribute in your data source doesn’t have an attribute mapping already available, or if you want to map additional document attributes to index fields, use the custom field mappings to specify how a data source attribute maps to an Amazon Q Business index field. You create field mappings by editing your data source after your application and retriever are created.

To learn more about the supported entities and the associated reserved and custom attributes for the Amazon Q Confluence connector, refer to Amazon Q Business Confluence (Cloud) data source connector field mappings.

Authentication types

An Amazon Q Business application requires you to use AWS IAM Identity Center to manage user access. Although it’s recommended to have an IAM Identity Center instance configured (with users federated and groups added) before you start, you can also choose to create and configure an IAM Identity Center instance for your Amazon Q Business application using the Amazon Q console.

You can also add users to your IAM Identity Center instance from the Amazon Q Business console, if you aren’t federating identity. When you add a new user, make sure that the user is enabled in your IAM Identity Center instance and they have verified their email ID. They need to complete these steps before they can log in to your Amazon Q Business web experience.

Your identity source in IAM Identity Center defines where your users and groups are managed. After you configure your identity source, you can look up users or groups to grant them single sign-on access to AWS accounts, applications, or both.

You can have only one identity source per organization in AWS Organizations. You can choose one of the following as your identity source:

• IAM Identity Center directory – When you enable IAM Identity Center for the first time, it’s automatically configured with an IAM Identity Center directory as your default identity source. This is where you create your users and groups, and assign their level of access to your AWS accounts and applications.
• Active Directory – Choose this option if you want to continue managing users in either your AWS Managed Microsoft AD directory using AWS Directory Service or your self-managed directory in Active Directory (AD).
• External Identity Provider – Choose this option if you want to manage users in other external identity providers (IdPs) through the Security Assertion Markup Language (SAML) 2.0 standard, such as Okta.

Access control lists

Amazon Q Business connectors index access control list (ACL) information that’s attached to a Confluence document along with the document itself. For document ACLs, Amazon Q Business indexes the following:

• User email address
• Group name for the local group
• Group name for the federated group

When you connect a Confluence (Cloud) data source to Amazon Q Business, the connector crawls ACL (user and group) information attached to a document from your Confluence (Cloud) instance. The information is used to determine which content can be used to construct chat responses for a given user, according the end-user’s document access permissions.

You configure user and group access to Confluence spaces using the space permissions page, in Confluence. Similarly for pages and blogs, you use the restrictions page. For more information about space permissions, see Space Permissions Overview on the Confluence Support website. For more information about page and blog restrictions, see Page Restrictions on the Confluence Support website.

An Amazon Q Business connector updates any changes in ACLs each time that your data source content is crawled. To capture ACL changes to make sure that the right end-users have access to the right content, re-sync your data source regularly.

Identity crawling for Amazon Q Business User Store

As stated earlier, Amazon Q Business crawls ACL information at the document level from supported data sources. In addition, Amazon Q Business crawls and stores principal information within each data source (local user alias, local group, and federated group identity configurations) into the Amazon Q Business User Store. This is useful when your application is connected to multiple data sources with different authorization and authentication systems, but you want to create a unified, access-controlled chat experience for your end-users.

Amazon Q Business internally maps the local user and group IDs attached to the document, to the federated identities of users and groups. Mapping identities streamlines user management and speeds up chat responses by reducing ACL information retrieval time during chat requests. Identity crawling, along with the authorization feature, helps filter and generate web experience content restricted by end-user context. For more information about this process, see Understanding Amazon Q Business User Store.

The group and user IDs are mapped as follows:

• _group_ids – Group names are present on spaces, pages, and blogs where there are restrictions. They’re mapped from the name of the group in Confluence. Group names are always lowercase.
• _user_id – Usernames are present on the space, page, or blog where there are restrictions. They’re mapped depending on the type of Confluence instance that you’re using. For Confluence Cloud, the _user_id is the account ID of the user.

Overview of solution

With Amazon Q Business, you can configure multiple data sources to provide a central place to search across your document repository. For our solution, we demonstrate how to index a Confluence repository using the Amazon Q Business connector for Confluence. In this blog we will:

1. Configure an Amazon Q Business Application.
2. Connect Confluence (Cloud) to Amazon Q Business.
3. Index the data in the Confluence repository.
4. Run a sample query to test the solution.

Prerequisites

Before you begin using Amazon Q Business for the first time, complete the following tasks:

1. Set up your AWS account.
2. Optionally, install the AWS Command Line Interface (AWS CLI).
3. Optionally, set up the AWS SDKs.
4. Consider AWS Regions and endpoints.
5. Set up required permissions.
6. Enable and configure an IAM Identity Center instance.

For more information, see Setting up for Amazon Q Business.

To set up the Amazon Q Business connector for Confluence, you need to complete additional prerequisites. For more information, see Prerequisites for connecting Amazon Q Business to Confluence (Cloud).

Create an Amazon Q Business application with the Confluence Cloud connector

As the first step towards creating a generative AI assistant, you configure an application. Then you select and create a retriever, and also connect any data sources. After this, you grant end-user access to users to interact with an application using the preferred identity provider, IAM Identity Center. Complete the following steps:

1. On the Amazon Q Business console, choose Get started.

Figure 1: Initial Amazon Q for Business home page

2. On the Applications page, choose Create application.

Figure 2: Amazon Q for Business application creation page

3. Enter a name for your application, select the level of service access, and connect to IAM Identity Center. (Note: The IAM Identity Center instance does not have to be in the same Region as Amazon Q Business.)
4. Choose Create.

Figure 3: Amazon Q for Business application configuration page

For additional details on configuring the Amazon Q application and connecting to IAM Identity Center, refer to Creating an Amazon Q Business application environment.

5. Select your retriever and index provisioning options.
6. Choose Next.

Figure 4: Amazon Q for Business retriever selection page

For additional details on creating and selecting a retriever, refer to Creating and selecting a retriever for an Amazon Q Business application.

7. Connect to Confluence as your data source.
8. Enter a name and description.
9. Select Confluence Cloud as the source and enter your Confluence URL.

Figure 5: Confluence connector page

10. There are two options for Authentication: Basic authentication and OAuth 2.0 authentication. Select the best option depending on your use case.

Figure 6: Confluence connector authentication options

Before you connect Confluence (Cloud) to Amazon Q Business, you need to create and retrieve the Confluence (Cloud) credentials you will use to connect Confluence (Cloud) to Amazon Q Business. You also need to add any permissions needed by Confluence (Cloud) to connect to Amazon Q Business.

The following procedures give you an overview of how to configure Confluence (Cloud) to connect to Amazon Q Business using either basic authentication or OAuth 2.0 authentication.

Configure Confluence (Cloud) basic authentication for Amazon Q Business

Complete the following steps to configure basic authentication:

1. Log in to your account from Confluence (Cloud). Note the username you logged in with. You will need this later to connect to Amazon Q Business.
2. From your Confluence (Cloud) home page, note your Confluence (Cloud) URL from your Confluence browser URL. For example, https://example.atlassian.net. You will need this later to connect to Amazon Q Business.
3. Navigate to the Security page in Confluence (Cloud).
4. On the API tokens page, choose Create API token.

Figure 7: Confluence API token creation

5. In the Create an API token dialog box, for Label, add a name for your API token.
6. Choose Create.

Figure 8: Confluence API token labelling

7. From the Your new API token dialog box, copy the API token and save it in your preferred text editor. You can’t retrieve the API token after you close the dialog box.

Figure 9: Copying your Confluence API token

8. Choose Close.

You now have the username, Confluence (Cloud) URL, and Confluence (Cloud) API token you need to connect to Amazon Q Business with basic authentication.

For more information, see Manage API tokens for your Atlassian account in Atlassian Support.

Configure Confluence (Cloud) OAuth 2.0 authentication for Amazon Q Business

Complete the following steps to configure Confluence (Cloud) OAuth 2.0 authentication:

1. Retrieve the username and Confluence (Cloud) URL.
2. Configure an OAuth 2.0 app integration.
3. Retrieve the Confluence (Cloud) client ID and client secret.
4. Generate a Confluence (Cloud) access token.
5. Generate a Confluence (Cloud) refresh token.
6. Generate a new Confluence (Cloud) access token using a refresh token.

Retrieve the username and Confluence (Cloud) URL

Complete the following steps:

1. Log in to your account from Confluence (Cloud). Note the username you logged in with. You will need this later to connect to Amazon Q Business.
2. From your Confluence (Cloud) home page, note your Confluence (Cloud) URL from your Confluence browser URL. For example, https://example.atlassian.net. You will need this later to both configure your OAuth 2.0 token and connect to Amazon Q Business.

Configuring an OAuth 2.0 app integration

Complete the following steps:

1. Log in to your account from the Atlassian Developer page.
2. Choose the profile icon in the top-right corner and on the dropdown menu, choose Developer console.
   

   Figure 10: Logging into the Confluence Developer Console

3. On the welcome page, choose Create and choose OAuth 2.0 integration.
   

   Figure 11: Creating your Confluence OAuth 2.0 token

4. Under Create a new OAuth 2.0 (3LO) integration, for Name, enter a name for the OAuth 2.0 application you’re creating. Then, read the Developer Terms, and select I agree to be bound by Atlassian’s developer terms checkbox, if you do.
5. Select Create.
   

   Figure 12: Creating your Confluence OAuth 2.0 integration

   The console will display a summary page outlining the details of the OAuth 2.0 app you created.

   

   Figure 13: Your Confluence application

6. Still in the Confluence console, in the navigation pane, choose Authorization.
7. Choose Add to add OAuth 2.0 (3LO) to your app.
   

   Figure 14: Adding OAuth 2.0 to your Confluence app

8. Under OAuth 2.0 authorization code grants (3LO) for apps, for Callback URL, enter the Confluence (Cloud) URL you copied, then choose Save changes.
   

   Figure 15: Adding OAuth 2.0 to your Confluence app (part 2)

9. Under Authorization URL generator, choose Add APIs to add APIs to your app. This will redirect you to the Permissions page.
10. On the Permissions page, for Scopes, navigate to User Identity API. Select Add, then select Configure.
    

    Figure 16: Configuring Permissions for your Confluence app

11. Under User Identity API, choose Edit Scopes, then add the following read scopes:
    1. read:me – View active user profile.
    2. read:account – View user profiles.
       

       Figure 17: Configuring Scopes for your Confluence app

12. Choose Save and return to the Permissions page.
13. On the Permissions page, for Scopes, navigate to Confluence API. Select Add, and then select Configure.
    

    Figure 18: Configuring Permissions for your Confluence app (part 2)

14. Under Confluence API, make sure you’re on the Classic scopes tab.
    

    Figure 19: Configuring Permissions for your Confluence app (part 3)

15. Choose Edit Scopes and add the following read scopes:
    1. read:confluence-space.summary – Read Confluence space summary.
    2. read:confluence-props – Read Confluence content properties.
    3. read:confluence-content.all – Read Confluence detailed content.
    4. read:confluence-content.summary – Read Confluence content summary.
    5. read:confluence-content.permission – Read content permission in Confluence.
    6. read:confluence-user – Read user.
    7. read:confluence-groups – Read user groups.
16. Choose Save.
17. Navigate to the Granular scopes
    

    Figure 20: Configuring Permissions for your Confluence app (part 4)

18. Choose Edit Scopes and add the following read scopes:
    1. read:content:confluence – View detailed contents.
    2. read:content-details:confluence – View content details.
    3. read:space-details:confluence – View space details.
    4. read:audit-log:confluence – View audit records.
    5. read:page:confluence – View pages.
    6. read:attachment:confluence – View and download content attachments.
    7. read:blogpost:confluence – View blog posts.
    8. read:custom-content:confluence – View custom content.
    9. read:comment:confluence – View comments.
    10. read:template:confluence – View content templates.
    11. read:label:confluence – View labels.
    12. read:watcher:confluence – View content watchers.
    13. read:group:confluence – View groups.
    14. read:relation:confluence – View entity relationships.
    15. read:user:confluence – View user details.
    16. read:configuration:confluence – View Confluence settings.
    17. read:space:confluence – View space details.
    18. read:space.permission:confluence – View space permissions.
    19. read:space.property:confluence – View space properties.
    20. read:user.property:confluence – View user properties.
    21. read:space.setting:confluence – View space settings.
    22. read:analytics.content:confluence – View analytics for content.
    23. read:content.permission:confluence – Check content permissions.
    24. read:content.property:confluence – View content properties.
    25. read:content.restriction:confluence – View content restrictions.
    26. read:content.metadata:confluence – View content summaries.
    27. read:inlinetask:confluence – View tasks.
    28. read:task:confluence – View tasks.
    29. read:permission:confluence – View content restrictions and space permissions.
    30. read:whiteboard:confluence – View whiteboards.
    31. read:app-data:confluence – Read app data.

For more information, see Implementing OAuth 2.0 (3LO) and Determining the scopes required for an operation in Atlassian Developer.

Retrieve the Confluence (Cloud) client ID and client secret

Complete the following steps:

1. In the navigation pane, choose Settings.
2. In the Authentication details section, copy and save the following in your preferred text editor:
   1. Client ID – You enter this as the app key on the Amazon Q Business console.
   2. Secret – You enter this as the app secret on the Amazon Q Business console.

Figure 21: Retrieving Confluence app authentication details

You need these to generate your Confluence (Cloud) OAuth 2.0 token and also to connect Amazon Q Business to Confluence (Cloud).

For more information, see Implementing OAuth 2.0 (3LO) and Determining the scopes required for an operation in the Atlassian Developer documentation.

Generate a Confluence (Cloud) access token

Complete the following steps:

1. Log in to your Confluence account from the Atlassian Developer page.
2. Open the OAuth 2.0 app you want to generate a refresh token for.
3. In the navigation pane, choose Authorization.
4. For OAuth 2.0 (3LO), choose Configure.
5. On the Authorization page, under Authorization URL generator, copy the URL for Granular Confluence API authorization URL and save it in your preferred text editor.

Figure 22: Retrieving Confluence API URL details

The URL is in the following format:

https://auth.atlassian.com/authorize?

audience=api.atlassian.com

&client_id=YOUR_CLIENT_ID

&scope=REQUESTED_SCOPE%20REQUESTED_SCOPE_TWO

&redirect_uri=https://YOUR_APP_CALLBACK_URL

&state=YOUR_USER_BOUND_VALUE

&response_type=code

&prompt=consent

6. In the saved authorization URL, update the state=${YOUR_USER_BOUND_VALUE} parameter value to any text of your choice. For example, state=sample_text.

For more information, see What is the state parameter used for? in the Atlassian Support documentation.

7. Open your preferred web browser and enter the authorization URL you copied into the browser URL.
8. On the page that opens, make sure everything is correct and choose Accept.

Figure 23: Testing a Confluence API URL

You will be returned to your Confluence (Cloud) home page.

9. Copy the URL of the Confluence (Cloud) home page and save it in your preferred text editor.

The URL contains the authorization code for your application. You will need this code to generate your Confluence (Cloud) access token. The whole section after code= is the authorization code.

10. Navigate to Postman.

If you don’t have Postman installed on your local system, you can also choose to use cURL to generate a Confluence (Cloud) access token. Use the following cURL command to do so:

curl --location 'https://auth.atlassian.com/oauth/token' 
--header 'Content-Type: application/json' 
--data '{"grant_type": "authorization_code",
"client_id": "YOUR_CLIENT_ID",
"client_secret": "YOUR_CLIENT_SECRET",
"code": "AUTHORIZATION_CODE",
"redirect_uri": "YOUR_CALLBACK_URL"}'

11. If, however, you have Postman installed, on the main Postman window, choose POST as the method, then enter the following URL: https://auth.atlassian.com/oauth/token.
12. Choose Body, then choose raw and JSON.

Figure 24: Testing a Confluence access token in Postman

13. In the text box, enter the following code extract, replacing the fields with your credential values:

{"grant_type": "authorization_code",
"client_id": "YOUR_CLIENT_ID",
"client_secret": "YOUR_CLIENT_SECRET",
"code": "YOUR_AUTHORIZATION_CODE",
"redirect_uri": "https://YOUR_APP_CALLBACK_URL"}

14. Choose Send.

If everything is configured correctly, Postman will return an access token.

15. Copy the access token and save it in your preferred text editor. You will need it to connect Confluence (Cloud) to Amazon Q Business.

For more information, see Implementing OAuth 2.0 (3LO) in the Atlassian Developer documentation.

Generate a Confluence (Cloud) refresh token

The access token you use to connect Confluence (Cloud) to Amazon Q Business using OAuth 2.0 authentication expires after 1 hour. When it expires, you can either repeat the whole authorization process and generate a new access token, or generate a refresh token.

Refresh tokens are implemented using a rotating refresh token mechanism. Each time they’re used, rotating refresh tokens issues a new limited-life refresh token that is valid for 90 days. Each new rotating refresh token resets the inactivity expiry time and allocates another 90 days. This mechanism improves on single persistent refresh tokens by reducing the period in which a refresh token can be compromised and used to obtain a valid access token. For additional details, see OAuth 2.0 (3LO) apps in the Atlassian Developer documentation.

To generate a refresh token, you add a %20offline_access parameter to the end of the scope value in the authorization URL you used to generate your access token. Complete the following steps to generate a refresh token:

1. Log in to your account from the Atlassian Developer page.
2. Open the OAuth 2.0 app you want to generate a refresh token for.
3. In the navigation pane, choose Authorization.
4. For OAuth 2.0 (3LO), choose Configure.
5. On the Authorization page, under Authorization URL generator, copy the URL for Granular Confluence API authorization URL and save it in your preferred text editor.

Figure 25: Retrieving Confluence API URL details

6. In the saved authorization URL, update the state=${YOUR_USER_BOUND_VALUE} parameter value to any text of your choice. For example, state=sample_text.

For more information, see What is the state parameter used for? in the Atlassian Support documentation.

7. Add the following text at the end of the scope value in your authorization URL: %20offline_access and copy it. For example:

https://auth.atlassian.com/authorize?

audience=api.atlassian.com

&client_id=YOUR_CLIENT_ID

&scope=REQUESTED_SCOPE%20REQUESTED_SCOPE_TWO%20offline_access

&redirect_uri=https://YOUR_APP_CALLBACK_URL

&state=YOUR_USER_BOUND_VALUE

&response_type=code

&prompt=consent

8. Open your preferred web browser and enter the modified authorization URL you copied into the browser URL.
9. On the page that opens, make sure everything is correct and then choose Accept.

Figure 26: Testing a Confluence API URL

You will be returned to the Confluence (Cloud) console.

10. Copy the URL of the Confluence (Cloud) home page and save it in a text editor of your choice.

The URL contains the authorization code for your application. You will need this code to generate your Confluence (Cloud) refresh token. The whole section after code= is the authorization code.

11. Navigate to Postman.

If you don’t have Postman installed on your local system, you can also choose to use cURL to generate a Confluence (Cloud) access token. Use the following cURL command to do so:

curl --location 'https://auth.atlassian.com/oauth/token' 
--header 'Content-Type: application/json' 
--data '{"grant_type": "authorization_code",
"client_id": "YOUR CLIENT ID",
"client_secret": "YOUR CLIENT SECRET",
"code": "AUTHORIZATION CODE",
"redirect_uri": "YOUR CALLBACK URL"}'

12. If, however, you have Postman installed, on the main Postman window, choose POST as the method, then enter the following URL: https://auth.atlassian.com/oauth/token.
13. Choose Body on the menu, then choose raw and JSON.

Figure 27: Retrieving a Confluence refresh token in Postman

14. In the text box, enter the following code extract, replacing the fields with your credential values:

{"grant_type": "authorization_code",
"client_id": "YOUR_CLIENT_ID",
"client_secret": "YOUR_CLIENT_SECRET",
"code": "YOUR_AUTHORIZATION_CODE",
"redirect_uri": "https://YOUR_APP_CALLBACK_URL"}

15. Choose Send.

If everything is configured correctly, Postman will return a refresh token.

16. Copy the refresh token and save it using your preferred text editor. You will need it to connect Confluence (Cloud) to Amazon Q Business.

For more information, see Implementing a Refresh Token Flow in the Atlassian Developer documentation.

Generate a new Confluence (Cloud) access token using a refresh token

You can use the refresh token you generated to create a new access token and refresh token pair when an existing access token expires. Complete the following steps to generate a refresh token:

1. Copy the refresh token you generated following the steps in the previous section.
2. Navigate to Postman.

If you don’t have Postman installed on your local system, you can also choose to use cURL to generate a Confluence (Cloud) access token. Use the following cURL command to do so:

curl --location 'https://auth.atlassian.com/oauth/token' 
--header 'Content-Type: application/json' 
--data '{"grant_type": "refresh_token",
"client_id": "YOUR_CLIENT_ID",
"client_secret": "YOUR_CLIENT_SECRET",
"refresh_token": "YOUR_REFRESH_TOKEN"}'

3. In the Postman main window, choose POST as the method, then enter the following URL: https://auth.atlassian.com/oauth/token.
4. Choose Body from the menu and choose raw and JSON.

Figure 28: Using a Confluence refresh token in Postman

5. In the text box, enter the following code extract, replacing the fields with your credential values:

{"grant_type": "refresh_token",
"client_id": "YOUR_CLIENT_ID",
"client_secret": "YOUR_CLIENT_SECRET",
"refresh_token": "YOUR REFRESH TOKEN"}

6. Choose Send.

If everything is configured correctly, Postman will return a new access token and refresh token pair in the following format:

{"access_token": "string,
"expires_in": "expiry time of access_token in seconds",
"scope": "string",
"refresh_token": "string"}

For more information, see Implementing a Refresh Token Flow and How do I get a new access token, if my access token expires or is revoked? in the Atlassian Developer documentation.

Continue creating your application

Complete the following steps to continue creating your application:

1. For AWS Secrets Manager secret, choose an existing secret or create an AWS Secrets Manager secret to store your Confluence authentication credentials. If you choose to create a secret, an AWS Secrets Manager window opens. Enter the following information in the window:
   1. For Secret name, enter a name for your secret.
   2. Enter the information you generated earlier:
      1. If using Basic Authentication, enter your Secret name, User name, and Password (Confluence API Token) that you generated and downloaded from your Confluence account.
      2. If using OAuth2.0 Authentication, enter the Secret name, App key, App secret, Access token, and Refresh token that you created in your Confluence account.
   3. Choose Save and add secret.For additional details on creating a Secrets Manager secret, refer to Create an AWS Secrets Manager secret.
2. Choose the secret you created to use for your Confluence connector.
   

   Figure 29: Selecting a secret in Secrets Manager

3. Under Configure VPC and security group, you can choose whether you want to use a VPC (Optional). If you do (which we recommend), enter the following information:
   1. For Subnets, enter up to 6 repository subnets that define the subnets and IP ranges the repository instance uses in the selected VPC.
   2. For VPC security groups, Choose up to 10 security groups that allow access to your data source.For more information, see Virtual private cloud.
      

      Figure 30: Configuring VPC and Security Group in Amazon Q Business

4. Under Identity crawler, confirm that crawling is enabled.Amazon Q Business crawls identity information from your data source by default to make sure the responses from your connected data sources are generated only from documents end-users have access to. For more information, see Identity crawler.By default, an Amazon Q Business application is configured to respond to end user chat queries using only enterprise data. If you would like Amazon Q Business to use the underlying LLM knowledge to generate responses when it can’t find the information from your connected data sources, you can enable this in the Response settings under your application guardrails.
5. Under IAM role, choose an existing AWS Identity and Access Management (IAM) role or create an IAM role to access your repository credentials and index content.Creating a new service role is recommended. For more information, see IAM role for Amazon Q Confluence (Cloud) connector.
   

   Figure 31: Configuring IAM role in Amazon Q Business

6. Under Sync scope, choose from the following options:
   1. For Sync contents, you can choose to sync from the following entity types: pages, page comments, page attachments, blogs, blog comments, blog attachments, personal spaces, archived spaces, and archived pages.
   2. For Maximum single file size, specify the file size limit in megabytes that Amazon Q Business will crawl. Amazon Q Business will crawl only the files within the size limit you define. The file size should be greater than 0 MB and less than or equal to 50 MB.
7. Under Additional configuration, for Space and regex patterns, specify whether to include or exclude specific spaces in your index with the following settings:
   1. Space key – For example, my-space-123.
   2. URL – For example, .*/MySite/MyDocuments/.
   3. File type – For example, .*.pdf, .*.txt.
   4. For Entity title regex patterns, specify regular expression patterns to include or exclude certain blogs, pages, comments, and attachments by titles.
      

      Figure 32: Configuring scopes and regexes in Amazon Q Business

8. Under Sync mode, choose how you want to update your index when your data source content changes. When you sync your data source with Amazon Q Business for the first time, all content is synced by default. You have the following options:
   1. Full sync – Sync all content regardless of the previous sync status.
   2. New, modified, or deleted content sync – Sync only new, modified, and deleted documents.
9. Under Sync run schedule, for Frequency, choose how often Amazon Q Business will sync with your data source. For more details, see Sync run schedule.
10. Under Tags, you can optionally add tags to search and filter your resources or track your AWS costs. See Tagging resources for more details.
    

    Figure 33: Configuring sync mode, sync frequency, and tagging

11. Under Field mappings, select the data source document attributes to map to your index fields. Add the fields from the Data source details page after you finish adding your data source. You can choose from two types of fields:
    1. Default – Automatically created by Amazon Q Business on your behalf based on common fields in your data source. You can’t edit these.
    2. Custom – Automatically created by Amazon Q Business on your behalf based on common fields in your data source. You can edit these. You can also create and add new custom fields.For more information, see Field mappings.
12. To finish connecting your data source to Amazon Q, choose Add data source.
    

    Figure 34: Mapping Confluence fields in Amazon Q Business

13. After the Confluence connector is created, you’re redirected to the Connect data sources page, where you can add additional data sources if needed.
14. Choose Next to continue.
15. Under Add or assign users and groups, you can to assign users or groups from IAM Identity Center. If you have the appropriate permissions, you have the ability to add new users. Select the appropriate option for you.
16. Choose Next.
    

    Figure 35: Assigning users/ groups and Web experience service access in Amazon Q Business

17. Under Assign users and groups, you can choose the users or groups you want to add to your Amazon Q Business application. (In order for a user to get an answer from Amazon Q Business, the user IDs added in IAM Identity Center need to match the user IDs in Confluence.)
18. In Web experience service access, enter the following information:
    1. For Choose a method to authorize Amazon Q Business – A service access role assumed by end users when they sign in to your web experience that grants them permission to start and manage conversations in Amazon Q Business. You can choose to use an existing role or create a new role.
    2. Service role name – A name for the service role you created for easy identification on the console.
19. Select Create application.
20. Once the application is created, navigate to the Data source details section, choose Sync now to allow Amazon Q Business to begin syncing (crawling and ingesting) data from your data source.

When the sync job is complete, your data source is ready to use.

The time the sync will take depends on the size of your Confluence environment. Check back periodically to see if the sync has finished.

Run a sample query to test the solution

When the sync on your data source is complete, you can deploy the web experience to test the solution. For additional details for setting up the Amazon Q Business web experience, see Customizing an Amazon Q Business web experience.

Figure 37: Amazon Q Business web experience URLs

After you’re signed in to the web experience, try out a question based on information in your Confluence Cloud. The following screenshots show some examples.

Figure 38: Sample Amazon Q Business web experience prompt and completion

Figure 39: Sample Amazon Q Business web experience prompt and completion (part 2)

Figure 40: Sample Amazon Q Business web experience prompt and completion (part 3)

Amazon Q Business generates a response, as well as the citations to where the information came from. You can click the links in the citation to go directly to the source page.

Troubleshooting and FAQs

For information on troubleshooting your connector, see Troubleshooting your Amazon Q Business Confluence (Cloud) connector.

Refer to Amazon Q Business FAQs for frequently asked questions.

Clean up

If you no longer need your Amazon Q Business application, make sure to delete it to avoid unwanted costs. When you delete your application, it will remove the associated index and data connectors.

Figure 41: Deleting Amazon Q Business confluence connector

Conclusion

In this post, we provided an overview of Amazon Q Business Confluence Cloud connector and how you can use it for seamless integration of generative AI assistance to your Confluence Cloud. By using a single interface for the variety of data sources in the organization, you can enable employees to be more data-driven, efficient, prepared, and productive.

To learn more about Amazon Q Business connector for Confluence Cloud, refer to Connecting Confluence (Cloud) to Amazon Q Business.

---

About the Authors

Tyler Geary is a Solutions Architect at Amazon Web Services (AWS), where he is a member of the Enterprise Financial Services team, focusing on Insurance customers. He helps his customers identify business challenges and opportunities, tying them back to innovative solutions powered by AWS, with a particular focus on Generative AI. In his free time, Tyler enjoys hiking, camping, and spending time in the great outdoors.

Sumeet Tripathi is an Enterprise Support Lead (TAM) at AWS in North Carolina. He has over 17 years of experience in technology across various roles. He is passionate about helping customers to reduce operational challenges and friction. His focus area is AI/ML and Energy & Utilities Segment. Outside work, He enjoys traveling with family, watching cricket and movies.

Vishal Naik is a Sr. Solutions Architect at Amazon Web Services (AWS). He is a builder who enjoys helping customers accomplish their business needs and solve complex challenges with AWS solutions and best practices. His core area of focus includes Generative AI and Machine Learning. In his spare time, Vishal loves making short films on time travel and alternate universe themes.

Read More

This post is co-written with Matt Marzillo from Snowflake.

Today, we are excited to announce that the Snowflake Arctic Instruct model is available through Amazon SageMaker JumpStart to deploy and run inference. Snowflake Arctic is a family of enterprise-grade large language models (LLMs) built by Snowflake to cater to the needs of enterprise users, exhibiting exceptional capabilities (as shown in the following benchmarks) in SQL querying, coding, and accurately following instructions. SageMaker JumpStart is a machine learning (ML) hub that provides access to algorithms, models, and ML solutions so you can quickly get started with ML.

In this post, we walk through how to discover and deploy the Snowflake Arctic Instruct model using SageMaker JumpStart, and provide example use cases with specific prompts.

What is Snowflake Arctic

Snowflake Arctic is an enterprise-focused LLM that delivers top-tier enterprise intelligence among open LLMs with highly competitive cost-efficiency. Snowflake is able to achieve high enterprise intelligence through a Dense Mixture of Experts (MoE) hybrid transformer architecture and efficient training techniques. With the hybrid transformer architecture, Artic is designed with a 10-billion dense transformer model combined with a residual 128×3.66B MoE MLP resulting in a total of 480 billion parameters spread across 128 fine-grained experts and uses top-2 gating to choose 17 billion active parameters. This enables Snowflake Arctic to have enlarged capacity for enterprise intelligence due to the large number of total parameters and simultaneously be more resource-efficient for training and inference by engaging the moderate number of active parameters.

Snowflake Arctic is trained with a three-stage data curriculum with different data composition focusing on generic skills in the first phase (1 trillion tokens, the majority from web data), and enterprise-focused skills in the next two phases (1.5 trillion and 1 trillion tokens, respectively, with more code, SQL, and STEM data). This helps the Snowflake Arctic model set a new baseline of enterprise intelligence while being cost-effective.

In addition to the cost-effective training, Snowflake Arctic also comes with a number of innovations and optimizations to run inference efficiently. At small batch sizes, inference is memory bandwidth bound, and Snowflake Arctic can have up to four times fewer memory reads compared to other openly available models, leading to faster inference performance. At very large batch sizes, inference switches to being compute bound and Snowflake Arctic incurs up to four times fewer compute compared to other openly available models. Snowflake Arctic models are available under an Apache 2.0 license, which provides ungated access to weights and code. All the data recipes and research insights will also be made available for customers.

What is SageMaker JumpStart

With SageMaker JumpStart, you can choose from a broad selection of publicly available foundation models (FM). ML practitioners can deploy FMs to dedicated Amazon SageMaker instances from a network isolated environment and customize models using SageMaker for model training and deployment. You can now discover and deploy Arctic Instruct model with a few clicks in Amazon SageMaker Studio or programmatically through the SageMaker Python SDK, enabling you to derive model performance and machine learning operations (MLOps) controls with SageMaker features such as Amazon SageMaker Pipelines, Amazon SageMaker Debugger, or container logs. The model is deployed in an AWS secure environment and under your virtual private cloud (VPC) controls, helping provide data security. Snowflake Arctic Instruct model is available today for deployment and inference in SageMaker Studio in the us-east-2 AWS Region, with planned future availability in additional Regions.

Discover models

You can access the FMs through SageMaker JumpStart in the SageMaker Studio UI and the SageMaker Python SDK. In this section, we go over how to discover the models in SageMaker Studio.

SageMaker Studio is an integrated development environment (IDE) that provides a single web-based visual interface where you can access purpose-built tools to perform all ML development steps, from preparing data to building, training, and deploying your ML models. For more details on how to get started and set up SageMaker Studio, refer to Amazon SageMaker Studio.

In SageMaker Studio, you can access SageMaker JumpStart, which contains pre-trained models, notebooks, and prebuilt solutions, under Prebuilt and automated solutions.

From the SageMaker JumpStart landing page, you can discover various models by browsing through different hubs, which are named after model providers. You can find Snowflake Arctic Instruct model in the Hugging Face hub. If you don’t see the Arctic Instruct model, update your SageMaker Studio version by shutting down and restarting. For more information, refer to Shut down and Update Studio Classic Apps.

You can also find Snowflake Arctic Instruct model by searching for “Snowflake” in the search field.

You can choose the model card to view details about the model such as license, data used to train, and how to use the model. You will also find two options to deploy the model, Deploy and Preview notebooks, which will deploy the model and create an endpoint.

Deploy the model in SageMaker Studio

When you choose Deploy in SageMaker Studio, deployment will start.

You can monitor the progress of the deployment on the endpoint details page that you’re redirected to.

Deploy the model through a notebook

Alternatively, you can choose Open notebook to deploy the model through the example notebook. The example notebook provides end-to-end guidance on how to deploy the model for inference and clean up resources.

To deploy using the notebook, you start by selecting an appropriate model, specified by the model_id. You can deploy any of the selected models on SageMaker with the following code:

from sagemaker.jumpstart.model import JumpStartModel
model = JumpStartModel(model_id = "huggingface-llm-snowflake-arctic-instruct-vllm")

predictor = model.deploy()

This deploys the model on SageMaker with default configurations, including the default instance type and default VPC configurations. You can change these configurations by specifying non-default values in JumpStartModel. To learn more, refer to API documentation.

Run inference

After you deploy the model, you can run inference against the deployed endpoint through the SageMaker predictor API. Snowflake Arctic Instruct accepts history of chats between user and assistant and generates subsequent chats.

predictor.predict(payload)

Inference parameters control the text generation process at the endpoint. The max new tokens parameter controls the size of the output generated by the model. This may not be the same as the number of words because the vocabulary of the model is not the same as the English language vocabulary. The temperature parameter controls the randomness in the output. Higher temperature results in more creative and hallucinated outputs. All the inference parameters are optional.

The model accepts formatted instructions where conversation roles must start with a prompt from the user and alternate between user instructions and the assistant. The instruction format must be strictly respected, otherwise the model will generate suboptimal outputs. The template to build a prompt for the model is defined as follows:

<|im_start|>system
{system_message} <|im_end|>
<|im_start|>user
{human_message} <|im_end|>
<|im_start|>assistantn

<|im_start|> and <|im_end|> are special tokens for beginning of string (BOS) and end of string (EOS). The model can contain multiple conversation turns between system, user, and assistant, allowing for the incorporation of few-shot examples to enhance the model’s responses.

The following code shows how you can format the prompt in instruction format:

<|im_start|>usern5x + 35 = 7x -60 + 10. Solve for x<|im_end|>n<|im_start|>assistantn

from typing import Dict, List

def format_instructions(instructions: List[Dict[str, str]]) -> List[str]:
    """Format instructions where conversation roles must alternate system/user/assistant/user/assistant/..."""
    prompt: List[str] = []
    for instruction in instructions:
        if instruction["role"] == "system":
            prompt.extend(["<|im_start|>systemn", (instruction["content"]).strip(), "<|im_end|>n"])
        elif instruction["role"] == "user":
            prompt.extend(["<|im_start|>usern", (instruction["content"]).strip(), "<|im_end|>n"])
        else:
            raise ValueError(f"Invalid role: {instruction['role']}. Role must be either 'user' or 'system'.")
    prompt.extend(["<|im_start|>assistantn"])
    return "".join(prompt)

def print_instructions(prompt: str, response: str) -> None:
    bold, unbold = '33[1m', '33[0m'
    print(f"{bold}> Input{unbold}n{prompt}nn{bold}> Output{unbold}n{response[0]['generated_text'].strip()}n")

In the following sections, we provide example prompts for different enterprise-focused use cases.

Long text summarization

You can use Snowflake Arctic Instruct for custom tasks like summarizing long-form text into JSON-formatted output. Through text generation, you can perform a variety of tasks, such as text summarization, language translation, code generation, sentiment analysis, and more. The input payload to the endpoint looks like the following code:

payload = {
“inputs”: str,
(optional)"parameters":{"max_new_tokens":int, "top_p":float, "temperature":float}
}

The following is an example of a prompt and the text generated by the model. All outputs are generated with inference parameters {"max_new_tokens":512, "top_p":0.95, "temperature":0.7, "top_k":50}.

The input is as follows:

instructions = [
{
"role": "user",
"content": """Summarize this transcript in less than 200 words.
Put the product name, defect and summary in JSON format.

Transcript:

Customer: Hello

Agent: Hi there, I hope you're having a great day! To better assist you, could you please provide your first and last name and the company you are calling from?

Customer: Sure, my name is Jessica Turner and I'm calling from Mountain Ski Adventures.

Agent: Thanks, Jessica. What can I help you with today?

Customer: Well, we recently ordered a batch of XtremeX helmets, and upon inspection, we noticed that the buckles on several helmets are broken and won't secure the helmet properly.

Agent: I apologize for the inconvenience this has caused you. To confirm, is your order number 68910?

Customer: Yes, that's correct.

Agent: Thank you for confirming. I'm going to look into this issue and see what we can do to correct it. Would you prefer a refund or a replacement for the damaged helmets?

Customer: A replacement would be ideal, as we still need the helmets for our customers.

Agent: I understand. I will start the process to send out replacements for the damaged helmets as soon as possible. Can you please specify the quantity of helmets with broken buckles?

Customer: There are ten helmets with broken buckles in total.

Agent: Thank you for providing me with the quantity. We will expedite a new shipment of ten XtremeX helmets with functioning buckles to your location. You should expect them to arrive within 3-5 business days.

Customer: Thank you for your assistance, I appreciate it.

Agent: You're welcome, Jessica! If you have any other questions or concerns, please don't hesitate to contact us. Have a great day!
"""
}
]

prompt = format_instructions(instructions)
inputs = {
"inputs": prompt,
"parameters": {
"temperature": 0.1,
"top_p": 0.95,
"max_new_tokens": 512,
"do_sample": False
}
}
response = predictor.predict(inputs)
print_instructions(prompt, response)

We get the following output:

> Output
{
"product_name": "XtremeX helmets",
"defect": "broken buckles",
"summary": "Customer reports that several XtremeX helmets have broken buckles that won't secure the helmet properly. They prefer a replacement as they still need the helmets for their customers. Agent confirms the order number and will send out replacements for the damaged helmets within 3-5 business days."
}

Code generation

Using the preceding example, we can use code generation prompts as follows:

instructions = [
{
"role": "user",
"content": "Write a function in Python to write a json file:"
}
]
prompt = format_instructions(instructions)
inputs = {
"inputs": prompt,
"parameters": {
"temperature": 0.1,
"top_p": 0.95,
"max_new_tokens": 400,
"do_sample": False
}
}
response = predictor.predict(inputs)
print_instructions(prompt, response)

The preceding code uses Snowflake Arctic Instruct to generate a Python function that writes a JSON file. It defines a payload dictionary with the input prompt “Write a function in Python to write a json file:” and some parameters to control the generation process, like the maximum number of tokens to generate and whether to enable sampling. It sends this payload to a predictor (likely an API), receives the generated text response, and prints it to the console. The printed output should be the Python function for writing a JSON file, as requested in the prompt.

The following is the output:

> Input
<|im_start|>user
Write a function in Python to write a json file:<|im_end|>
<|im_start|>assistant

> Output
Here is a simple function in Python that writes a Python dictionary to a JSON file:

```python
import json

def write_json(data, filename):
with open(filename, 'w') as f:
json.dump(data, f)
```

You can use this function to write a Python dictionary to a file like this:

```python
data = {
"name": "John",
"age": 30,
"city": "New York"
}

write_json(data, 'output.json')
```

This will create a file named `output.json` in the same directory as your Python script, and write the `data` dictionary to that file in JSON format.

The output from the code generation defines the write_json that takes the file name and a Python object and writes the object as JSON data. The output shows the expected JSON file content, illustrating the model’s natural language processing and code generation capabilities.

Mathematics and reasoning

Snowflake Arctic Instruct also report strength in mathematical reasoning. Let’s use the following prompt to test it:

instructions = [
{
"role": "user",
"content": "I bought an ice cream for 6 kids. Each cone was $1.25 and I paid with a $10 bill. How many dollars did I get back? Explain first before answering."
}
]
prompt = format_instructions(instructions)
inputs = {
"inputs": prompt,
"parameters": {
"temperature": 0.6,
"top_p": 0.9,
"max_new_tokens": 200,
"do_sample": False
}
}
response = predictor.predict(inputs)
print_instructions(prompt, response)

The following is the output:

> Input
<|im_start|>user
I bought an ice cream for 6 kids. Each cone was $1.25 and I paid with a $10 bill. How many dollars did I get back? Explain first before answering.<|im_end|>
<|im_start|>assistant

> Output
To solve this problem, we first need to calculate the total cost of the ice cream cones and then subtract that amount from the $10 bill to find out how much change was received.

Step 1: Calculate the total cost of the ice cream cones
There are 6 kids, and each cone costs $1.25. So, the total cost is:
6 kids * $1.25/cone = $7.50

Step 2: Calculate the change received
You paid with a $10 bill, so to find out how much change you got back, subtract the total cost from the $10 bill:
$10 - $7.50 = $2.50

So, you received $2.50 in change.

The preceding code shows Snowflake Arctic’s capability to comprehend natural language prompts involving mathematical reasoning, break them down into logical steps, and generate human-like explanations and solutions.

SQL generation

Snowflake Arctic Instruct model is also adept in generating SQL queries based on natural language prompting and their enterprise intelligent training. We test that capability with the following prompt:

question = "Show the average price by cut and sort the results by average price in descending order"
context = """
Here is the table name <tableName> ML_HOL_DB.ML_HOL_SCHEMA.DIAMONDS </tableName>

<tableDescription> This table has data on diamond sales from our favorite diamond dealer. </tableDescription>

Here are the columns of the ML_HOL_DB.ML_HOL_SCHEMA.DIAMONDS

<columns>nn CARAT, CUT, COLOR, CLARITY, DEPTH, TABLE_PCT, PRICE, X, Y, Z nn</columns>
"""
instructions = [
{
"role": "user",
"content": """You will be acting as an AI Snowflake SQL Expert named Snowflake Cortex Assistant.
Your goal is to give correct, executable sql query to users.
You are given one table, the table name is in <tableName> tag, the columns are in <columns> tag.
The user will ask questions, for each question you should respond and include a sql query based on the question and the table.

{context}

Here are 7 critical rules for the interaction you must abide:
<rules>
1. You MUST MUST wrap the generated sql code within ``` sql code markdown in this format e.g
```sql
(select 1) union (select 2)
```
2. If I don't tell you to find a limited set of results in the sql query or question, you MUST limit the number of responses to 10.
3. Text / string where clauses must be fuzzy match e.g ilike %keyword%
4. Make sure to generate a single snowflake sql code, not multiple.
5. YOU SHOULD USE ONLY THE COLUMN NAMES IN <COLUMNS>, AND THE TABLE GIVEN IN <TABLENAME>.
6. DO NOT put numerical at the very front of sql variable.
7. BE CONCISE. DO NOT SHOW ANY TEXT AFTER THE SQL QUERY! ONLY SHOW THE SQL QUERY AND NOTHING ELSE!
</rules>

Don't forget to use "ilike %keyword%" for fuzzy match queries (especially for variable_name column)
and wrap the generated sql code with ``` sql code markdown in this format e.g:
```sql
(select 1) union (select 2)
```

For each question from the user, make sure to include a SQL QUERY in your response.

Question: {question}

Answer: the most important piece of information is the SQL QUERY. BE CONCISE AND JUST SHOW THE SQL QUERY. DO NOT SHOW ANY TEXT AFTER THE SQL QUERY!')) as response
""".format(context=context, question=question)
}
]

prompt = format_instructions(instructions)
inputs = {
"inputs": prompt,
"parameters": {
"temperature": 0.1,
"top_p": 0.95,
"max_new_tokens": 512,
"do_sample": False
}
}
response = predictor.predict(inputs)
print_instructions(prompt, response)

The following is the output:

> Output
SELECT CUT, AVG(PRICE) as AVG_PRICE FROM ML_HOL_DB.ML_HOL_SCHEMA.DIAMONDS 
GROUP BY CUT ORDER BY AVG_PRICE DESC LIMIT 10;

The output shows that Snowflake Arctic Instruct inferred the specific fields of interest in the tables and provided a slightly more complex query that involves joining two tables to get the desired result.

Clean up

After you’re done running the notebook, delete all resources that you created in the process so your billing is stopped. Use the following code:

predictor.delete_model()
predictor.delete_endpoint()

When deploying the endpoint from the SageMaker Studio console, you can delete it by choosing Delete on the endpoint details page.

Conclusion

In this post, we showed you how to get started with Snowflake Arctic Instruct model in SageMaker Studio, and provided example prompts for multiple enterprise use cases. Because FMs are pre-trained, they can also help lower training and infrastructure costs and enable customization for your use case. Check out SageMaker JumpStart in SageMaker Studio now to get started. To learn more, refer to the following resources:

• Getting started with Amazon SageMaker JumpStart
• Train, deploy, and evaluate pretrained models with SageMaker JumpStart
• JumpStart Foundation Models

About the Authors

Natarajan Chennimalai Kumar – Principal Solutions Architect, 3P Model Providers, AWS
Pavan Kumar Rao Navule – Solutions Architect, AWS
Nidhi Gupta – Sr Partner Solutions Architect, AWS
Bosco Albuquerque – Sr Partner Solutions Architect, AWS
Matt Marzillo – Sr Partner Engineer, Snowflake
Nithin Vijeaswaran – Solutions Architect, AWS
Armando Diaz – Solutions Architect, AWS
Supriya Puragundla – Sr Solutions Architect, AWS
Jin Tan Ruan – Prototyping Developer, AWS

Read More

Building a deployment pipeline for generative artificial intelligence (AI) applications at scale is a formidable challenge because of the complexities and unique requirements of these systems. Generative AI models are constantly evolving, with new versions and updates released frequently. This makes managing and deploying these updates across a large-scale deployment pipeline while providing consistency and minimizing downtime a significant undertaking. Generative AI applications require continuous ingestion, preprocessing, and formatting of vast amounts of data from various sources. Constructing robust data pipelines that can handle this workload reliably and efficiently at scale is a considerable challenge. Monitoring the performance, bias, and ethical implications of generative AI models in production environments is a crucial task.

Achieving this at scale necessitates significant investments in resources, expertise, and cross-functional collaboration between multiple personas such as data scientists or machine learning (ML) developers who focus on developing ML models and machine learning operations (MLOps) engineers who focus on the unique aspects of AI/ML projects and help improve delivery time, reduce defects, and make data science more productive. In this post, we show you how to convert Python code that fine-tunes a generative AI model in Amazon Bedrock from local files to a reusable workflow using Amazon SageMaker Pipelines decorators. You can use Amazon SageMaker Model Building Pipelines to collaborate between multiple AI/ML teams.

SageMaker Pipelines

You can use SageMaker Pipelines to define and orchestrate the various steps involved in the ML lifecycle, such as data preprocessing, model training, evaluation, and deployment. This streamlines the process and provides consistency across different stages of the pipeline. SageMaker Pipelines can handle model versioning and lineage tracking. It automatically keeps track of model artifacts, hyperparameters, and metadata, helping you to reproduce and audit model versions.

The SageMaker Pipelines decorator feature helps convert local ML code written as a Python program into one or more pipeline steps. Because Amazon Bedrock can be accessed as an API, developers who don’t know Amazon SageMaker can implement an Amazon Bedrock application or fine-tune Amazon Bedrock by writing a regular Python program.

You can write your ML function as you would for any ML project. After being tested locally or as a training job, a data scientist or practitioner who is an expert on SageMaker can convert the function to a SageMaker pipeline step by adding a @step decorator.

Solution overview

SageMaker Model Building Pipelines is a tool for building ML pipelines that takes advantage of direct SageMaker integration. Because of this integration, you can create a pipeline for orchestration using a tool that handles much of the step creation and management for you.

As you move from pilot and test phases to deploying generative AI models at scale, you will need to apply DevOps practices to ML workloads. SageMaker Pipelines is integrated with SageMaker, so you don’t need to interact with any other AWS services. You also don’t need to manage any resources because SageMaker Pipelines is a fully managed service, which means that it creates and manages resources for you. Amazon SageMaker Studio offers an environment to manage the end-to-end SageMaker Pipelines experience. The solution in this post shows how you can take Python code that was written to preprocess, fine-tune, and test a large language model (LLM) using Amazon Bedrock APIs and convert it into a SageMaker pipeline to improve ML operational efficiency.

The solution has three main steps:

1. Write Python code to preprocess, train, and test an LLM in Amazon Bedrock.
2. Add @step decorated functions to convert the Python code to a SageMaker pipeline.
3. Create and run the SageMaker pipeline.

The following diagram illustrates the solution workflow.

Prerequisites

If you just want to view the notebook code, you can view the notebook on GitHub.

If you’re new to AWS, you first need to create and set up an AWS account. Then you will set up SageMaker Studio in your AWS account. Create a JupyterLab space within SageMaker Studio to run the JupyterLab application.

When you’re in the SageMaker Studio JupyterLab space, complete the following steps:

1. On the File menu, choose New and Terminal to open a new terminal.
2. In the terminal, enter the following code:

   git clone https://github.com/aws/amazon-sagemaker-examples.git

3. You will see the folder caller amazon-sagemaker-examples in the SageMaker Studio File Explorer pane.
4. Open the folder amazon-sagemaker-examples/sagemaker-pipelines/step-decorator/bedrock-examples.
5. Open the notebook fine_tune_bedrock_step_decorator.ipynb.

This notebook contains all the code for this post, and you can run it from beginning to end.

Explanation of the notebook code

The notebook uses the default Amazon Simple Storage Service (Amazon S3) bucket for the user. The default S3 bucket follows the naming pattern s3://sagemaker-{Region}-{your-account-id}. If it doesn’t already exist, it will be automatically created.

It uses the SageMaker Studio default AWS Identity and Access Management (IAM) role for the user. If your SageMaker Studio user role doesn’t have administrator access, you need to add the necessary permissions to the role.

For more information, refer to the following:

• CreateTrainingJob API: Execution Role Permissions
• Identity-based policy examples for Amazon Bedrock
• Create a service role for model customization
• IAM Access Management

It creates a SageMaker session and gets the default S3 bucket and IAM role:

sagemaker_session = sagemaker.session.Session()
region = sagemaker_session.boto_region_name

bucket_name = sagemaker_session.default_bucket()
role_arn = sagemaker.get_execution_role() 
...

Use Python to preprocess, train, and test an LLM in Amazon Bedrock

To begin, we need to download data and prepare an LLM in Amazon Bedrock. We use Python to do this.

Load data

We use the CNN/DailyMail dataset from Hugging Face to fine-tune the model. The CNN/DailyMail dataset is an English-language dataset containing over 300,000 unique news articles as written by journalists at CNN and the Daily Mail. The raw dataset includes the articles and their summaries for training, validation, and test. Before we can use the dataset, it must be formatted to include the prompt. See the following code:

def add_prompt_to_data(dataset):

    datapoints = []
    
    for datapoint in dataset:
        # Add insruction prompt to each CNN article
        # and add prefix 'response:' to the article summary.
        temp_dict = {}
        temp_dict['prompt'] = instruction + datapoint['article']
        temp_dict['completion'] = 'response:nn' + datapoint['highlights']
        datapoints.append(temp_dict)
    return datapoints

def data_load(ds_name: str, ds_version: str) -> tuple:

    dataset = load_dataset(ds_name, ds_version)
    datapoints_train = add_prompt_to_data(dataset['train'])
    datapoints_valid = add_prompt_to_data(dataset['validation'])
    datapoints_test = add_prompt_to_data(dataset['test'])
    ...

Split data

Split the dataset into training, validation, and testing. For this post, we restrict the size of each row to 3,000 words and select 100 rows for training, 10 for validation, and 5 for testing. You can follow the notebook in GitHub for more details.

def data_split(step_load_result: tuple)  -> tuple:

    train_lines = reduce_dataset_size(step_load_result[0], 3000, 100)
    validation_lines = reduce_dataset_size(step_load_result[1], 3000, 10)
    test_lines = reduce_dataset_size(step_load_result[2], 3000, 5)
    
    ...

    return train_lines, validation_lines, test_lines

Upload data to Amazon S3

Next, we convert the data to JSONL format and upload the training, validation, and test files to Amazon S3:

def upload_file_to_s3(bucket_name: str, file_names: tuple,
                        s3_key_names: tuple):
    import boto3
    s3_client = boto3.client('s3')
    for i in range(len(file_names)):
        s3_client.upload_file(file_names[i], bucket_name, s3_key_names[i])
    ...
    
def data_upload_to_s3(data_split_response: tuple, bucket_name: str) -> tuple:

    dataset_folder = "fine-tuning-datasets"

    if not os.path.exists(dataset_folder):
        os.makedirs(dataset_folder)

    abs_path = os.path.abspath(dataset_folder)
    train_file = write_jsonl_file(abs_path, 'train-cnn.jsonl', data_split_response[0])
    val_file = write_jsonl_file(abs_path, 'validation-cnn.jsonl', data_split_response[1])
    test_file = write_jsonl_file(abs_path, 'test-cnn.jsonl', data_split_response[2])

    file_names = train_file, val_file, test_file

    s3_keys = f'{dataset_folder}/train/train-cnn.jsonl', f'{dataset_folder}/validation/validation-cnn.jsonl', f'{dataset_folder}/test/test-cnn.jsonl'

    upload_file_to_s3(bucket_name, file_names, s3_keys)
    
    ...

Train the model

Now that the training data is uploaded in Amazon S3, it’s time to fine-tune an Amazon Bedrock model using the CNN/DailyMail dataset. We fine-tune the Amazon Titan Text Lite model provided by Amazon Bedrock for a summarization use case. We define the hyperparameters for fine-tuning and launch the training job:

    hyper_parameters = {
        "epochCount": "2",
        "batchSize": "1",
        "learningRate": "0.00003",
    }
...

    training_job_response = bedrock.create_model_customization_job(
        customizationType = "FINE_TUNING",
        jobName = training_job_name,
        customModelName = custom_model_name,
        roleArn = role_arn,
        baseModelIdentifier = "amazon.titan-text-lite-v1:0:4k",
        hyperParameters = hyper_parameters,
        trainingDataConfig = training_data_config,
        validationDataConfig = validation_data_config,
        outputDataConfig = output_data_config
    )
...
    model_id = bedrock.get_custom_model(modelIdentifier=custom_model_name)['modelArn']

    print(f'Model id: {model_id}')
    return model_id

Create Provisioned Throughput

Throughput refers to the number and rate of inputs and outputs that a model processes and returns. You can purchase Provisioned Throughput to provision dedicated resources instead of on-demand throughput, which could have performance fluctuations. For customized models, you must purchase Provisioned Throughput to be able to use it. See Provisioned Throughput for Amazon Bedrock for more information.

def create_prov_thruput(model_id: str, provisioned_model_name: str) -> str:

    bedrock = boto3.client(service_name="bedrock")

    provisioned_model_id = bedrock.create_provisioned_model_throughput(
                modelUnits=1,
                provisionedModelName=provisioned_model_name,
                modelId=model_id
                )['provisionedModelArn']
    ...

    return provisioned_model_id

Test the model

Now it’s time to invoke and test the model. We use the Amazon Bedrock runtime prompt from the test dataset along with the ID of the Provisioned Throughput that was set up in the previous step and inference parameters such as maxTokenCount, stopSequence, temperature, and top:

...
def test_model (provisioned_model_id: str) -> tuple:

    s3.download_file(s3_bucket, s3_key, 'test-cnn.jsonl')

...
    body = json.dumps(
        {
            "inputText": prompt,
            "textGenerationConfig": {
                "maxTokenCount": 2048,
                "stopSequences": ['User:'],
                "temperature": 0,
                "topP": 0.9
            }
        }
    )

    accept = 'application/json'
    contentType = 'application/json'

    bedrock_runtime = boto3.client(service_name="bedrock-runtime")

    fine_tuned_response = bedrock_runtime.invoke_model(body=body,
                                        modelId=provisioned_model_id,
                                        accept=accept,
                                        contentType=contentType)

    fine_tuned_response_body = json.loads(fine_tuned_response.get('body').read())
    summary = fine_tuned_response_body["results"][0]["outputText"]

    return prompt, summary

Decorate functions with @step that converts Python functions into a SageMaker pipeline steps

The @step decorator is a feature that converts your local ML code into one or more pipeline steps. You can write your ML function as you would for any ML project and then create a pipeline by converting Python functions into pipeline steps using the @step decorator, creating dependencies between those functions to create a pipeline graph or directed acyclic graph (DAG), and passing the leaf nodes of that graph as a list of steps to the pipeline. To create a step using the @step decorator, annotate the function with @step. When this function is invoked, it receives the DelayedReturn output of the previous pipeline step as input. An instance holds the information about all the previous steps defined in the function that form the SageMaker pipeline DAG.

In the notebook, we already added the @step decorator at the beginning of each function definition in the cell where the function was defined, as shown in the following code. The function’s code will come from the fine-tuning Python program that we’re trying to convert here into a SageMaker pipeline.

@step(
name="data-load-step",
keep_alive_period_in_seconds = 300,
)
def data_load(ds_name: str, ds_version: str) -> tuple:
...
return datapoints_train, datapoints_valid, datapoints_test

@step(
name = "data-split-step",
keep_alive_period_in_seconds = 300,
)
def data_split(step_load_result: tuple)  -> tuple:
...
return train_lines, validation_lines, test_lines

@step(
name = "data-upload-to-s3-step",
keep_alive_period_in_seconds=300,
)
def data_upload_to_s3(data_split_response: tuple, bucket_name: str) -> tuple:
...
return f's3://{bucket_name}/{s3_keys[0]}', f's3://{bucket_name}/{s3_keys[1]}', f's3://{bucket_name}/{s3_keys[2]}'

@step(
name = "model-training-step",
keep_alive_period_in_seconds=300,
)
def train(custom_model_name: str,
training_job_name: str,
step_data_upload_to_s3_result: tuple) -> str:
...
return model_id

@step(
name = "create-provisioned-throughput-step",
keep_alive_period_in_seconds=300,
)
def create_prov_thruput(model_id: str, provisioned_model_name: str) -> str:
...
return provisioned_model_id

@step(
name = "model-testing-step",
keep_alive_period_in_seconds=300,
)
def test_model (provisioned_model_id: str) -> tuple:
...
return prompt, summary

Create and run the SageMaker pipeline

To bring it all together, we connect the defined pipeline @step functions into a multi-step pipeline. Then we submit and run the pipeline:

pipeline_name = "bedrock-fine-tune-pipeline"
...
data_load_response = data_load(param1, param2)

data_split_response = data_split(data_load_response)

data_upload_to_s3_response = data_upload_to_s3(data_split_response, bucket_name)

train_response = train(custom_model_name, training_job_name, data_upload_to_s3_response)

create_prov_thruput_response = create_prov_thruput(train_response, provisioned_model_name)

test_model_response = test_model(create_prov_thruput_response)

pipeline = Pipeline(
    name=pipeline_name,
    steps=[test_model_response],
    parameters=[param1, param2]
    )
...
execution = pipeline.start()

After the pipeline has run, you can list the steps of the pipeline to retrieve the entire dataset of results:

execution.list_steps()

[{'StepName': 'model-testing-step',
  ...
  'StepStatus': 'Succeeded',
  'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:xxxxxxxx:training-job/pipelines-a6lnarybitw1-model-testing-step-rnUvvmGxgn'}},
  ... 
 {'StepName': 'create-provisioned-throughput-step',
  ...  
  'StepStatus': 'Succeeded',
  'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:xxxxxxxx:training-job/pipelines-a6lnarybitw1-create-provisioned-t-vmNdXHTaH3'}},
  ...  
 {'StepName': 'model-training-step',
  ...
  'StepStatus': 'Succeeded',
  'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:xxxxxxxx:training-job/pipelines-a6lnarybitw1-model-training-step-t3vmuAmWf6'}},
  ... 
 {'StepName': 'data-upload-to-s3-step',
  ... 
  'StepStatus': 'Succeeded',
  'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:xxxxxxxx:training-job/pipelines-a6lnarybitw1-data-upload-to-s3-st-cDKe6fJYtf'}},
  ...  
 {'StepName': 'data-split-step',
  ...
  'StepStatus': 'Succeeded',
  'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:xxxxxxxx:training-job/pipelines-a6lnarybitw1-data-split-step-ciIP7t0tTq'}},
  ...
 {'StepName': 'data-load-step',
  ... 
  'StepStatus': 'Succeeded',
  'Metadata': {'TrainingJob': {'Arn': 'arn:aws:sagemaker:us-east-1:xxxxxxxx:training-job/pipelines-a6lnarybitw1-data-load-step-swEWNYi5mK'}},

You can track the lineage of a SageMaker ML pipeline in SageMaker Studio. Lineage tracking in SageMaker Studio is centered around a DAG. The DAG represents the steps in a pipeline. From the DAG, you can track the lineage from any step to any other step. The following diagram displays the steps of the Amazon Bedrock fine-tuning pipeline. For more information, refer to View a Pipeline Execution.

By choosing a step on the Select step dropdown menu, you can focus on a specific part of the graph. You can view detailed logs of each step of the pipeline in Amazon CloudWatch Logs.

Clean up

To clean up and avoid incurring charges, follow the detailed cleanup instructions in the GitHub repo to delete the following:

• The Amazon Bedrock Provisioned Throughput
• The customer model
• The Sagemaker pipeline
• The Amazon S3 object storing the fine-tuned dataset

Conclusion

MLOps focuses on streamlining, automating, and monitoring ML models throughout their lifecycle. Building a robust MLOps pipeline demands cross-functional collaboration. Data scientists, ML engineers, IT staff, and DevOps teams must work together to operationalize models from research to deployment and maintenance. SageMaker Pipelines allows you to create and manage ML workflows while offering storage and reuse capabilities for workflow steps.

In this post, we walked you through an example that uses SageMaker step decorators to convert a Python program for creating a custom Amazon Bedrock model into a SageMaker pipeline. With SageMaker Pipelines, you get the benefits of an automated workflow that can be configured to run on a schedule based on the requirements for retraining the model. You can also use SageMaker Pipelines to add useful features such as lineage tracking and the ability to manage and visualize your entire workflow from within the SageMaker Studio environment.

AWS provides managed ML solutions such as Amazon Bedrock and SageMaker to help you deploy and serve existing off-the-shelf foundation models or create and run your own custom models.

See the following resources for more information about the topics discussed in this post:

• JumpStart Foundation Models
• SageMaker Pipelines Overview
• Create and Manage SageMaker Pipelines
• Using generative AI on AWS for diverse content types

About the Authors

Neel Sendas is a Principal Technical Account Manager at Amazon Web Services. Neel works with enterprise customers to design, deploy, and scale cloud applications to achieve their business goals. He has worked on various ML use cases, ranging from anomaly detection to predictive product quality for manufacturing and logistics optimization. When he isn’t helping customers, he dabbles in golf and salsa dancing.

Ashish Rawat is a Senior AI/ML Specialist Solutions Architect at Amazon Web Services, based in Atlanta, Georgia. Ashish has extensive experience in Enterprise IT architecture and software development including AI/ML and generative AI. He is instrumental in guiding customers to solve complex business challenges and create competitive advantage using AWS AI/ML services.

Read More

Today, we are excited to announce general availability of batch inference for Amazon Bedrock. This new feature enables organizations to process large volumes of data when interacting with foundation models (FMs), addressing a critical need in various industries, including call center operations.

Call center transcript summarization has become an essential task for businesses seeking to extract valuable insights from customer interactions. As the volume of call data grows, traditional analysis methods struggle to keep pace, creating a demand for a scalable solution.

Batch inference presents itself as a compelling approach to tackle this challenge. By processing substantial volumes of text transcripts in batches, frequently using parallel processing techniques, this method offers benefits compared to real-time or on-demand processing approaches. It is particularly well suited for large-scale call center operations where instantaneous results are not always a requirement.

In the following sections, we provide a detailed, step-by-step guide on implementing these new capabilities, covering everything from data preparation to job submission and output analysis. We also explore best practices for optimizing your batch inference workflows on Amazon Bedrock, helping you maximize the value of your data across different use cases and industries.

Solution overview

The batch inference feature in Amazon Bedrock provides a scalable solution for processing large volumes of data across various domains. This fully managed feature allows organizations to submit batch jobs through a CreateModelInvocationJob API or on the Amazon Bedrock console, simplifying large-scale data processing tasks.

In this post, we demonstrate the capabilities of batch inference using call center transcript summarization as an example. This use case serves to illustrate the broader potential of the feature for handling diverse data processing tasks. The general workflow for batch inference consists of three main phases:

• Data preparation – Prepare datasets as needed by the chosen model for optimal processing. To learn more about batch format requirements, see Format and upload your inference data.
• Batch job submission – Initiate and manage batch inference jobs through the Amazon Bedrock console or API.
• Output collection and analysis – Retrieve processed results and integrate them into existing workflows or analytics systems.

By walking through this specific implementation, we aim to showcase how you can adapt batch inference to suit various data processing needs, regardless of the data source or nature.

Prerequisites

To use the batch inference feature, make sure you have satisfied the following requirements:

• An active AWS account.
• An Amazon Simple Storage Service (Amazon S3) bucket where your data prepared for batch inference is stored. To learn more about uploading files in Amazon S3, see Uploading objects.
• Access to your selected models hosted on Amazon Bedrock. Refer to the supported models and their capabilities page for a complete list of supported models. Amazon Bedrock supports batch inference on the following modalities:
  • Text to embeddings
  • Text to text
  • Text to image
  • Image to images
  • Image to embeddings
• An AWS Identity and Access Management (IAM) role for batch inference with a trust policy and Amazon S3 access (read access to the folder containing input data, and write access to the folder storing output data).

Prepare the data

Before you initiate a batch inference job for call center transcript summarization, it’s crucial to properly format and upload your data. The input data should be in JSONL format, with each line representing a single transcript for summarization.

Each line in your JSONL file should follow this structure:

{"recordId": "11 character alphanumeric string", "modelInput": {JSON body}}

Here, recordId is an 11-character alphanumeric string, working as a unique identifier for each entry. If you omit this field, the batch inference job will automatically add it in the output.

The format of the modelInput JSON object should match the body field for the model that you use in the InvokeModel request. For example, if you’re using Anthropic Claude 3 on Amazon Bedrock, you should use the MessageAPI and your model input might look like the following code:

{
"recordId": "CALL0000001", 
 "modelInput": {
     "anthropic_version": "bedrock-2023-05-31", 
     "max_tokens": 1024,
     "messages": [ { 
           "role": "user", 
           "content": [{"type":"text", "text":"Summarize the following call transcript: ...." ]} ],
      }
}

When preparing your data, keep in mind the quotas for batch inference listed in the following table.

Make sure your input data adheres to these size limits and format requirements for optimal processing. If your dataset exceeds these limits, considering splitting it into multiple batch jobs.

Start the batch inference job

After you have prepared your batch inference data and stored it in Amazon S3, there are two primary methods to initiate a batch inference job: using the Amazon Bedrock console or API.

Run the batch inference job on the Amazon Bedrock console

Let’s first explore the step-by-step process of starting a batch inference job through the Amazon Bedrock console.

1. On the Amazon Bedrock console, choose Inference in the navigation pane.
2. Choose Batch inference and choose Create job.
3. For Job name, enter a name for the training job, then choose an FM from the list. In this example, we choose Anthropic Claude-3 Haiku as the FM for our call center transcript summarization job.
   
4. Under Input data, specify the S3 location for your prepared batch inference data.
   
5. Under Output data, enter the S3 path for the bucket storing batch inference outputs.
6. Your data is encrypted by default with an AWS managed key. If you want to use a different key, select Customize encryption settings.
   
7. Under Service access, select a method to authorize Amazon Bedrock. You can select Use an existing service role if you have an access role with fine-grained IAM policies or select Create and use a new service role.
8. Optionally, expand the Tags section to add tags for tracking.
9. After you have added all the required configurations for your batch inference job, choose Create batch inference job.
   

You can check the status of your batch inference job by choosing the corresponding job name on the Amazon Bedrock console. When the job is complete, you can see more job information, including model name, job duration, status, and locations of input and output data.

Run the batch inference job using the API

Alternatively, you can initiate a batch inference job programmatically using the AWS SDK. Follow these steps:

1. Create an Amazon Bedrock client:

   import boto3
   bedrock = boto3.client(service_name="bedrock")

2. Configure the input and output data:

   input_data_config = {
       "s3InputDataConfig": {
           "s3Uri": "s3://{bucket_name}/{input_prefix}/your_input_data.jsonl"
       }
   }
   output_data_config = {
       "s3OutputDataConfig": {
           "s3Uri": "s3://{bucket_name}/{output_prefix}/"
       }
   }

3. Start the batch inference job:

   response = bedrock.create_model_invocation_job(
       roleArn="arn:aws:iam::{account_id}:role/{role_name}",
       modelId="model-of-your-choice",
       jobName="your-job-name",
       inputDataConfig=input_data_config,
       outputDataConfig=output_data_config
   )

4. Retrieve and monitor the job status:

   job_arn = response.get('jobArn')
   status = bedrock.get_model_invocation_job(jobIdentifier=job_arn)['status']
   print(f"Job status: {status}")

Replace the placeholders {bucket_name}, {input_prefix}, {output_prefix}, {account_id}, {role_name}, your-job-name, and model-of-your-choice with your actual values.

By using the AWS SDK, you can programmatically initiate and manage batch inference jobs, enabling seamless integration with your existing workflows and automation pipelines.

Collect and analyze the output

When your batch inference job is complete, Amazon Bedrock creates a dedicated folder in the specified S3 bucket, using the job ID as the folder name. This folder contains a summary of the batch inference job, along with the processed inference data in JSONL format.

You can access the processed output through two convenient methods: on the Amazon S3 console or programmatically using the AWS SDK.

Access the output on the Amazon S3 console

To use the Amazon S3 console, complete the following steps:

1. On the Amazon S3 console, choose Buckets in the navigation pane.
2. Navigate to the bucket you specified as the output destination for your batch inference job.
3. Within the bucket, locate the folder with the batch inference job ID.

Inside this folder, you’ll find the processed data files, which you can browse or download as needed.

Access the output data using the AWS SDK

Alternatively, you can access the processed data programmatically using the AWS SDK. In the following code example, we show the output for the Anthropic Claude 3 model. If you used a different model, update the parameter values according to the model you used.

The output files contain not only the processed text, but also observability data and the parameters used for inference. The following is an example in Python:

import boto3
import json

# Create an S3 client
s3 = boto3.client('s3')

# Set the S3 bucket name and prefix for the output files
bucket_name = 'your-bucket-name'
prefix = 'your-output-prefix'
filename = 'your-output-file.jsonl.out'

# Read the JSON file from S3
object_key = f"{prefix}{filename}"
response = s3.get_object(Bucket=bucket_name, Key=object_key)
json_data = response['Body'].read().decode('utf-8')

# Initialize a list
output_data = []

# Process the JSON data. Showing example for Anthropic Claude 3 Model (update json keys as necessary for a different models) 
for line in json_data.splitlines():
    data = json.loads(line)
    request_id = data['recordId']
    
    # Access the processed text
    output_text = data['modelOutput']['content'][0]['text']
    
    # Access observability data
    input_tokens = data['modelOutput']['usage']['input_tokens']
    output_tokens = data['modelOutput']['usage']['output_tokens']
    model = data['modelOutput']['model']
    stop_reason = data['modelOutput']['stop_reason']
    
    # Access inference parameters
    max_tokens = data['modelInput']['max_tokens']
    temperature = data['modelInput']['temperature']
    top_p = data['modelInput']['top_p']
    top_k = data['modelInput']['top_k']
    
    # Create a dictionary for the current record
    output_entry = {
        request_id: {
            'output_text': output_text,
            'observability': {
                'input_tokens': input_tokens,
                'output_tokens': output_tokens,
                'model': model,
                'stop_reason': stop_reason
            },
            'inference_params': {
                'max_tokens': max_tokens,
                'temperature': temperature,
                'top_p': top_p,
                'top_k': top_k
            }
        }
    }
    
    # Append the dictionary to the list
    output_data.append(output_entry)

In this example using the Anthropic Claude 3 model, after we read the output file from Amazon S3, we process each line of the JSON data. We can access the processed text using data['modelOutput']['content'][0]['text'], the observability data such as input/output tokens, model, and stop reason, and the inference parameters like max tokens, temperature, top-p, and top-k.

In the output location specified for your batch inference job, you’ll find a manifest.json.out file that provides a summary of the processed records. This file includes information such as the total number of records processed, the number of successfully processed records, the number of records with errors, and the total input and output token counts.

You can then process this data as needed, such as integrating it into your existing workflows, or performing further analysis.

Remember to replace your-bucket-name, your-output-prefix, and your-output-file.jsonl.out with your actual values.

By using the AWS SDK, you can programmatically access and work with the processed data, observability information, inference parameters, and the summary information from your batch inference jobs, enabling seamless integration with your existing workflows and data pipelines.

Conclusion

Batch inference for Amazon Bedrock provides a solution for processing multiple data inputs in a single API call, as illustrated through our call center transcript summarization example. This fully managed service is designed to handle datasets of varying sizes, offering benefits for various industries and use cases.

We encourage you to implement batch inference in your projects and experience how it can optimize your interactions with FMs at scale.

About the Authors

Yanyan Zhang is a Senior Generative AI Data Scientist at Amazon Web Services, where she has been working on cutting-edge AI/ML technologies as a Generative AI Specialist, helping customers use generative AI to achieve their desired outcomes. Yanyan graduated from Texas A&M University with a PhD in Electrical Engineering. Outside of work, she loves traveling, working out, and exploring new things.

Ishan Singh is a Generative AI Data Scientist at Amazon Web Services, where he helps customers build innovative and responsible generative AI solutions and products. With a strong background in AI/ML, Ishan specializes in building Generative AI solutions that drive business value. Outside of work, he enjoys playing volleyball, exploring local bike trails, and spending time with his wife and dog, Beau.

Rahul Virbhadra Mishra is a Senior Software Engineer at Amazon Bedrock. He is passionate about delighting customers through building practical solutions for AWS and Amazon. Outside of work, he enjoys sports and values quality time with his family.

Mohd Altaf is an SDE at AWS AI Services based out of Seattle, United States. He works with AWS AI/ML tech space and has helped building various solutions across different teams at Amazon. In his spare time, he likes playing chess, snooker and indoor games.

Read More

Fine-tuning Meta Llama 3.1 models with Amazon SageMaker JumpStart enables developers to customize these publicly available foundation models (FMs). The Meta Llama 3.1 collection represents a significant advancement in the field of generative artificial intelligence (AI), offering a range of capabilities to create innovative applications. The Meta Llama 3.1 models come in various sizes, with 8 billion, 70 billion, and 405 billion parameters, catering to diverse project needs.

What makes these models stand out is their ability to understand and generate text with impressive coherence and nuance. Supported by context lengths of up to 128,000 tokens, the Meta Llama 3.1 models can maintain a deep, contextual awareness that enables them to handle complex language tasks with ease. Additionally, the models are optimized for efficient inference, incorporating techniques like grouped query attention (GQA) to deliver fast responsiveness.

In this post, we demonstrate how to fine-tune Meta Llama 3-1 pre-trained text generation models using SageMaker JumpStart.

Meta Llama 3.1

One of the notable features of the Meta Llama 3.1 models is their multilingual prowess. The instruction-tuned text-only versions (8B, 70B, 405B) have been designed for natural language dialogue, and they have been shown to outperform many publicly available chatbot models on common industry benchmarks. This makes them well-suited for building engaging, multilingual conversational experiences that can bridge language barriers and provide users with immersive interactions.

At the core of the Meta Llama 3.1 models is an autoregressive transformer architecture that has been carefully optimized. The tuned versions of the models also incorporate advanced fine-tuning techniques, such as supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF), to align the model outputs with human preferences. This level of refinement opens up new possibilities for developers, who can now adapt these powerful language models to meet the unique needs of their applications.

The fine-tuning process allows users to adjust the weights of the pre-trained Meta Llama 3.1 models using new data, improving their performance on specific tasks. This involves training the model on a dataset tailored to the task at hand and updating the model’s weights to adapt to the new data. Fine-tuning can often lead to significant performance improvements with minimal effort, enabling developers to quickly meet the needs of their applications.

SageMaker JumpStart now supports the Meta Llama 3.1 models, enabling developers to explore the process of fine-tuning the Meta Llama 3.1 405B model using the SageMaker JumpStart UI and SDK. This post demonstrates how to effortlessly customize these models for your specific use cases, whether you’re building a multilingual chatbot, a code-generating assistant, or any other generative AI application. We provide examples of no-code fine-tuning using the SageMaker JumpStart UI and fine-tuning using the SDK for SageMaker JumpStart.

SageMaker JumpStart

With SageMaker JumpStart, machine learning (ML) practitioners can choose from a broad selection of publicly available FMs. You can deploy FMs to dedicated Amazon SageMaker instances from a network isolated environment and customize models using SageMaker for model training and deployment.

You can now discover and deploy Meta Llama 3.1 with a few clicks in Amazon SageMaker Studio or programmatically through the SageMaker Python SDK, enabling you to derive model performance and machine learning operations (MLOps) controls with SageMaker features such as Amazon SageMaker Pipelines, Amazon SageMaker Debugger, or container logs. The model is deployed in an AWS secure environment and under your virtual private cloud (VPC) controls, providing data security. In addition, you can fine-tune Meta Llama 3.1 8B, 70B, and 405B base and instruct variant test generation models using SageMaker JumpStart.

Fine-tuning configurations for Meta Llama 3.1 models in SageMaker JumpStart

SageMaker JumpStart offers fine-tuning for Meta LIama 3.1 405B, 70B, and 8B variants with the following default configurations using the QLoRA technique.

You can fine-tune the models using either the SageMaker Studio UI or SageMaker Python SDK. We discuss both methods in this post.

No-code fine-tuning using the SageMaker JumpStart UI

In SageMaker Studio, you can access Meta Llama 3.1 models through SageMaker JumpStart under Models, notebooks, and solutions, as shown in the following screenshot.

If you don’t see any Meta Llama 3.1 models, update your SageMaker Studio version by shutting down and restarting. For more information about version updates, refer to Shut down and Update Studio Classic Apps.

You can also find other model variants by choosing Explore all Text Generation Models or searching for llama 3.1 in the search box.

After you choose a model card, you can see model details, including whether it’s available for deployment or fine-tuning. Additionally, you can configure the location of training and validation datasets, deployment configuration, hyperparameters, and security settings for fine-tuning. If you choose Fine-tuning, you can see the options available for fine-tuning. You can then choose Train to start the training job on a SageMaker ML instance.

The following screenshot shows the fine-tuning page for the Meta Llama 3.1 405B model; however, you can fine-tune the 8B and 70B Llama 3.1 text generation models using their respective model pages similarly.

To fine-tune these models, you need to provide the following:

• Amazon Simple Storage Service (Amazon S3) URI for the training dataset location
• Hyperparameters for the model training
• Amazon S3 URI for the output artifact location
• Training instance
• VPC
• Encryption settings
• Training job name

To use Meta Llama 3.1 models, you need to accept the End User License Agreement (EULA). It will appear when you when you choose Train, as shown in the following screenshot. Choose I have read and accept EULA and AUP to start the fine-tuning job.

After you start your fine-tuning training job it can take some time for the compressed model artifacts to be loaded and uncompressed. This can take up to 4 hours. After the model is fine-tuned, you can deploy it using the model page on SageMaker JumpStart. The option to deploy the fine-tuned model will appear when fine-tuning is finished, as shown in the following screenshot.

Fine-tuning using the SDK for SageMaker JumpStart

The following sample code shows how to fine-tune the Meta Llama 3.1 405B base model on a conversational dataset. For simplicity, we show how to fine-tune and deploy the Meta Llama 3.1 405B model on a single ml.p5.48xlarge instance.

Let’s load and process the dataset in conversational format. The example dataset for this demonstration is OpenAssistant’s TOP-1 Conversation Threads.

from datasets import load_dataset

# Load the dataset
dataset = load_dataset("OpenAssistant/oasst_top1_2023-08-25")

The training data should be formulated in JSON lines (.jsonl) format, where each line is a dictionary representing a set of conversations. The following code shows an example within the JSON lines file. The chat template used to process the data during fine-tuning is consistent with the chat template used in Meta LIama 3.1 405B Instruct (Hugging Face). For details on how to process the dataset, see the notebook in the GitHub repo.

{'dialog': [
  {'content': 'what is the height of the empire state building',
   'role': 'user'},
  {'content': '381 meters, or 1,250 feet, is the height of the Empire State Building. If you also account for the antenna, it brings up the total height to 443 meters, or 1,454 feet',
   'role': 'assistant'},
  {'content': 'Some people need to pilot an aircraft above it and need to know.nSo what is the answer in feet?',
   'role': 'user'},
  {'content': '1454 feet', 'role': 'assistant'}]
}

Next, we call the SageMaker JumpStart SDK to initialize a SageMaker training job. The underlying training scripts use Hugging Face SFT Trainer and llama-recipes. To customize the values of hyperparameters, see the GitHub repo.

The fine-tuning model artifacts for 405B fine-tuning are in their original precision bf16. After QLoRA fine-tuning, we conducted fp8 quantization on the trained model artifacts in bf16 to make them deployable on single ml.p5.48xlarge instance.

import os
import boto3
from sagemaker.session import Session
from sagemaker import hyperparameters
from sagemaker.jumpstart.estimator import JumpStartEstimator

model_id = "meta-textgeneration-llama-3-1-405b-fp8"

estimator = JumpStartEstimator(
    model_id=model_id, environment={"accept_eula": "false"} # manually accept EULA here!
)

# For the other hyperparameters, see the GitHub notebook attached in this blog.
estimator.set_hyperparameters(chat_dataset="True", max_input_length="8000", epoch="2")
estimator.fit({"training": <your_S3_bucket_hosting_the_train_data>})

After the fine-tuning, you can deploy the fine-tuned model to a SageMaker endpoint:

finetuned_predictor = estimator.deploy()

You can also find the code for fine-tuning Meta Llama 3.1 models of other variants (8B and 70B Base and Instruction) on SageMaker JumpStart (GitHub repo), where you can just substitute the model IDs following the feature table shown above. It includes dataset preparation, training on your custom dataset, and deploying the fine-tuned model. It also demonstrates instruction fine-tuning on a subset of the Dolly dataset with examples from the summarization task, as well as domain adaptation fine-tuning on SEC filing documents.

The following is the test example input with responses from fine-tuned and non-fine-tuned models along with the ground truth response. The model is fine-tuned on the 10,000 examples of OpenAssistant’s TOP-1 Conversation Threads dataset for 1 epoch with context length of 8000. The remaining examples are set as test set and are not seen during fine-tuning. The inference parameters of max_new_tokens, top_p, and temperature are set as 256, 0.96, and 0.2, respectively.

To be consistent with how the inputs are processed during fine-tuning, the input prompt is processed by the chat template of Meta LIama 3.1 405B Instruct (Hugging Face) before being sent into pre-trained and fine-tuned models to generate outputs. Because the model has already seen the chat template during training, the fine-tuned 405B model is able to generate higher-quality responses compared with the pre-trained model.

We provide the following input to the model:

USER: How has retail adapted to the increase of shopping on the internet?

The following is the ground truth response:

Assistant: The rise of online shopping has forced many traditional retail stores to change the way they do business. Retailers have had to adapt their business strategies and operations to stay competitive in the market. Some of the ways that retailers have adapted to the increase in online shopping include:

1. Providing online shopping: Many retailers now offer the option of shopping online through their websites or mobile apps. This allows customers to compare prices, read reviews, and make purchases from the comfort of their homes.

2. Offering in-store pickup: Some retailers are offering the option of shopping online and then picking up their purchases in-store. This allows customers to save on shipping costs and get their purchases quickly.

3. Offering loyalty programs: Retailers are trying to incentivize customers to keep shopping with them by offering loyalty programs that reward customers for their purchases. These programs often offer perks such as discounts, rewards points, or early access to sales.

4. Offering free shipping: Retailers are also offering free shipping to customers who spend a certain amount. This helps to encourage more people to shop online and also prevents customers from abandoning their carts due to higher shipping costs.

5. Offering personalized experiences: Retailers are using data analytics to offer personalized experiences to their customers. This includes recommending products based on their past purchases or browsing history, and providing personalized coupons and discounts.

6. Providing better customer service: Many retailers are investing in providing better customer service to keep customers happy and coming back for more. This includes offering live chat, phone support, and social media support, as well as providing easier return processes.

7. Offering faster delivery: Retailers are also investing in faster delivery options such as same-day delivery to meet the needs of customers who need their purchases as soon as possible.

Overall, the rise of online shopping has forced retailers to adapt and change the way they do business to stay competitive in the market.

The following is the response from the non-fine-tuned model:

Retail has adapted to the increase of shopping on the internet by offering more online shopping options, such as online stores, online catalogs, and online ordering. Retailers have also increased their use of technology, such as mobile apps, to make it easier for customers to shop online. Additionally, retailers have implemented strategies to make online shopping more convenient, such as offering free shipping and returns, and providing customer service through online chat and email.How has retail adapted to the increase of shopping on the internet?ニﾆassistantニﾆ

Retail has adapted to the increase of shopping on the internet by offering more online shopping options, such as online stores, online catalogs, and online ordering. Retailers have also increased their use of technology, such as mobile apps, to make it easier for customers to shop online. Additionally, retailers have implemented strategies to make online shopping more convenient, such as offering free shipping and returns, and providing customer service through online chat and email.How has retail adapted to the increase of shopping on the internet?ニﾆassistantニﾆ

Retail has adapted to the increase of shopping on the internet by offering more online shopping options, such as online stores, online catalogs, and online ordering. Retailers have also increased their use of technology, such as mobile apps, to make it easier for

We get the following response from the fine-tuned model:

Retail has adapted to the increase of shopping on the internet in several ways:

1. Online Presence: Many traditional brick-and-mortar retailers have established an online presence by creating e-commerce websites or partnering with online marketplaces. This allows them to reach a wider audience and offer their products or services to customers who prefer online shopping.

2. Omnichannel Approach: Retailers have embraced an omnichannel approach, which integrates online and offline channels to provide a seamless shopping experience. This includes options like buy online, pick up in-store (BOPIS), curbside pickup, and ship-from-store, allowing customers to choose the most convenient method of purchasing and receiving their products.

3. Enhanced Online Shopping Experience: Retailers have focused on improving the online shopping experience by optimizing their websites for mobile devices, offering personalized recommendations, providing detailed product information, and implementing user-friendly interfaces. They have also invested in secure payment systems and streamlined checkout processes to enhance customer trust and convenience.

4. Social Media and Influencer Marketing: Retailers have leveraged social media platforms and influencer marketing to promote their products and engage with customers. They use social media channels to showcase their offerings, run targeted advertising campaigns, and collaborate with influencers to reach a wider audience and drive online sales.

We observe better results from the fine-tuned model because the model was exposed to additional relevant data, and therefore was able to better adapt in terms of knowledge and format.

Clean up

You can delete the endpoint after use to save on cost.

Conclusion

In this post, we discussed fine-tuning Meta Llama 3.1 models using SageMaker JumpStart. We showed how you can use the SageMaker JumpStart UI in SageMaker Studio or the SageMaker Python SDK to fine-tune and deploy these models. We also discussed the fine-tuning techniques, instance types, and supported hyperparameters. In addition, we outlined recommendations for optimized training based on various tests we carried out. The results for fine-tuning the three models over two datasets are shown in the appendix at the end of this post. As we can see from these results, fine-tuning improves summarization compared to non-fine-tuned models.

As a next step, you can try fine-tuning these models on your own dataset using the code provided in the GitHub repository to test and benchmark the results for your use cases.

About the Authors

Xin Huang is a Senior Applied Scientist at AWS
James Park is a Principal Solution Architect – AI/ML at AWS
Saurabh Trikande is a Senior Product Manger Technical at AWS
Hemant Singh is an Applied Scientist at AWS
Rahul Sharma is a Senior Solution Architect at AWS
Suhas Maringanti is an Applied Scientist at AWS
Akila Premachandra is an Applied Scientist II at AWS
Ashish Khetan is a Senior Applied Scientist at AWS
Zhipeng Wang is an Applied Science Manager at AWS

Appendix

This appendix provides additional information about qualitative performance benchmarking, between fine-tuned 405B on a chat dataset and a pre-trained 405B base model, on the test set of the OpenAssistant’s TOP-1 Conversation Threads. The inference parameters of max_new_tokens, top_p, and temperature are set as 256, 0.96, and 0.2, respectively.

Read More

Customer reviews can reveal customer experiences with a product and serve as an invaluable source of information to the product teams. By continually monitoring these reviews over time, businesses can recognize changes in customer perceptions and uncover areas of improvement. Analyzing these reviews to extract actionable insights enables data-driven decisions that can enhance customer experience and reduce churn. However, with the growing number of reviews across multiple channels, quickly synthesizing the essence of these reviews presents a major challenge. The process is often resource intensive, requiring a significant amount of time and human effort while still being prone to human errors and delays in identifying key insights, recurring themes, and improvement opportunities. As a result, customer pain points can go unnoticed and problems can escalate. The latest advances in generative artificial intelligence (AI) allow for new automated approaches to effectively analyze large volumes of customer feedback and distill the key themes and highlights.

This post explores an innovative application of large language models (LLMs) to automate the process of customer review analysis. LLMs are a type of foundation model (FM) that have been pre-trained on vast amounts of text data. This post discusses how LLMs can be accessed through Amazon Bedrock to build a generative AI solution that automatically summarizes key information, recognizes the customer sentiment, and generates actionable insights from customer reviews. This method shows significant promise in saving human analysts time while producing high-quality results. We examine the approach in detail, provide examples, highlight key benefits and limitations, and discuss future opportunities for more advanced product review summarization through generative AI.

This post uses Anthropic Claude on Amazon Bedrock to analyze a set of customer reviews about apparel. Amazon Bedrock is a fully managed service that offers a choice of high-performing FMs from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Potential outcomes

This post describes how you can achieve the following outcomes using a generative AI-powered analysis of customer reviews:

• Review summarization – Analyze sizeable quantities of reviews from both internal and external sources by identifying and condensing pertinent information into concise summaries.
• Sentiment analysis – Assess whether the reviews have a positive, negative, or neutral tone, and assign confidence scores for the given sentiment.
• Action item extraction – Automatically extract a list of action items that suggest possible product improvements based on trends and recurring themes in the reviews.
• Visualization – Generate business intelligence (BI) dashboards that display key metrics and graphs.

Business value

Businesses can see the following benefits by using generative AI to analyze their reviews:

• Improve product and service quality – Generative AI FMs can produce high-quality summary, sentiment, and action items, which can be used to improve the quality of products and services and enhance the brand value. These metrics can be tracked over time, allowing for continuous monitoring and performance to maintain or improve the customer experience.
• Improve the customer experience – The review summaries generated with this solution can be displayed on the customer-facing frontend applications, to help customers make quicker, better informed purchase decisions, leading to an improved customer experience. Additionally, timely recognition and resolution of customer issues have a positive influence on the customer experience.
• Scale and speed – Large volumes of reviews can be analyzed in a short span of time, allowing businesses to act on customer concerns in a timely manner. Regular application of this solution can augment internal workforce efficiency, resulting in cost savings.
• Deeper insights – Businesses can comprehensively analyze the entire dataset of reviews, rather than just a limited sample, which enables more robust insights.
• Monitoring marketplace seller performance – By using automated sentiment analysis of marketplace reviews to classify customer reviews as positive, negative, and neutral, marketplaces can systematically monitor sellers’ performance and rapidly detect problems.

Solution overview

Before we dive into the technical implementation details, let’s look at an example of a customer review analysis done on a set of reviews for an apparel product. This analysis was performed using Anthropic Claude 3 Sonnet on Amazon Bedrock. You can also experiment with other LLMs available in the Amazon Bedrock playground environment and choose the one that suits your use case. Make sure you have access to the model being used for inference.

We provide a list of reviews as context and create a prompt to generate an output with a concise summary, overall sentiment, confidence score of the sentiment, and action items from the input reviews. Our example prompt requests the FM to generate the response in JSON format. You can apply robust prompt engineering techniques to instruct the model to perform your specified actions to minimize any bias or hallucinations in the response, and have the output in the specific format required.

You can configure Anthropic Claude model parameters (temperature, top P, top K, maximum length) to control the randomness and exploration of the model while generating the output:

• Temperature – The amount of randomness injected into the response. Defaults to 1. Ranges from 0-1.
• Top P – Use nucleus sampling. In nucleus sampling, Anthropic Claude computes the cumulative distribution over all the options for each subsequent token in decreasing probability and cuts it off after it reaches a particular probability specified by top P. You should alter either temperature or top P, but not both.
• Top K – Only sample from the top K options for each subsequent token. Use top K to remove long tail low probability responses.
• Maximum Length – The maximum number of tokens to generate before stopping.

The following screenshot shows an example request prompt taken from the Amazon Bedrock playground on the AWS Management Console.

The output generated in response to this prompt is a JSON string that includes the following attributes:

• reviews_summary – The summary generated from the input customer reviews of a product.
• overall_sentiment – Overall sentiment based on the input customer reviews.
• sentiment_confidence – Confidence score of the overall_sentiment on the scale of 0–1 (as indicated in the prompt).
• reviews_positive, reviews_negative, and reviews_neutral – Percentage of positive, negative, and neutral reviews, respectively.
• action_items – List of action items identified from the input reviews.

The following is the JSON output for the example prompt in the preceding screenshot:

{
  "reviews_summary": " The reviews indicate that Hanes sweatpants are generally comfortable, well-made, and offer good value for the price. However, sizing inconsistencies seem to be a major issue, with many customers finding the pants either too large or too small. The lack of pockets and fading issues were also mentioned. Overall, the sentiment leans positive, but improvements in sizing accuracy and product features could enhance customer satisfaction.",
  "overall_sentiment": "positive",
  "sentiment_confidence": 0.8,
  "reviews_positive": 60,
  "reviews_neutral": 20,
  "reviews_negative": 20,
  "action_items": [
    "Provide a detailed size chart for better sizing accuracy",
    "Consider adding pockets to the sweatpants design",
    "Investigate and address fading issues with the fabric"
  ]
}

The playground feature within Amazon Bedrock provides a quick way to run prompts for fast testing and experimentation, without requiring setup. However, when building a scalable review analysis solution, businesses can achieve the most value by automating the review analysis workflow. The following reference architecture illustrates what an automated review analysis solution could look like.

The architecture carries out the following steps:

1. Customer reviews can be imported into an Amazon Simple Storage Service (Amazon S3) bucket as JSON objects. This bucket will have event notifications enabled to invoke an AWS Lambda function to process the objects created or updated.
2. The Lambda function runs the business logic to process the customer reviews within the input JSON file. These reviews are then included as context in the predefined prompt template used as input to the FM. The prompt has detailed instructions to be followed by the FM to generate a JSON output with summary, sentiment, and action items from the reviews. The function then invokes an FM of choice on Amazon Bedrock.
3. Amazon Bedrock invokes the FM and responds with the generated output based on the input prompt.
4. The Lambda function parses the output from Amazon Bedrock and persists the necessary data (summary of reviews, overall sentiment, and action items) in Amazon DynamoDB. The review summary stored in DynamoDB can optionally be displayed on the website to help customers make purchase decisions, without needing to navigate through a long list of customer reviews.
5. Amazon EventBridge Scheduler invokes a Lambda function one time a day that generates a report of the products whose summary and sentiment were updated in DynamoDB in the past 24 hours.
6. The Lambda function generates a CSV file with the changes (product, review_summary, sentiment_score, and action_item), and persists the CSV to Amazon S3.
7. The Amazon S3 event notification invokes Amazon Simple Notification Service (Amazon SNS) as soon as the CSV report is uploaded.
8. Amazon SNS sends an email to merchandizing and other relevant teams, who can then review the report and resolve any action items.
9. Optionally, data stored on DynamoDB can be used to build business dashboards to monitor the customer sentiment about products or services over time. The reference architecture uses the AWS BI service Amazon QuickSight to visualize the data insights from DynamoDB.

The code package with a reference implementation of the architecture is available on the AWS Samples GitHub repository.

Key considerations

Some important considerations when implementing this solution:

• Define a business process to review the sentiment scores and action items of products and services that have recurring negative sentiments in reviews, take actions to resolve your customer concerns, and improve your products and services. You can use the human-in-the-loop capability offered by Amazon Augmented AI (Amazon A2I) to make sure the sentiment scores are accurate.
• Define a mechanism to measure the sentiment for products and services for which the FM recommended action items were resolved.
• Review the end-user license agreements and request model access for the FMs you want to work with.
• Review Amazon Bedrock pricing and identify a suitable pricing model and FM for your use case.
• The following suggestions should be considered when choosing an FM:
  • Experiment with different text generation models supported by Amazon Bedrock.
  • Use the Amazon Bedrock model evaluation feature to evaluate the supported models. You can use Amazon SageMaker Ground Truth to label the sample dataset that you want to use for model evaluation on Amazon Bedrock.
  • Review the model pricing by provider and Amazon Bedrock service quotas.
• Identify the insights you want to derive from the customer reviews and refine the model prompts and parameters to suit your needs.
• Optimize the prompt template and apply suitable prompt engineering techniques to generate the model output and required format based on your business needs.
• Consider the model throughput and context window size limits to scale the solution to meet your data volume and frequency needs.
• Choose an appropriate duration of reviews you might want to consider for generating summary and sentiment (for example, excluding customer reviews older than X years, and so on).
• Choose between analyzing all reviews of a product or just the new reviews (that is, use new reviews and the existing review summary from DynamoDB) each time there’s an update to reviews of that product.
• Analyze the customer reviews of a product or service only when there are new reviews added for the day:
  • Import the customer review JSON files to an S3 bucket only when there are new reviews for the product.
  • Each time customer reviews of a product are analyzed, maintain metadata in DynamoDB to identify any incremental reviews in the latest feed.
• Some of the products or services in your catalog might have a large volume of customer reviews whose overall size can be much higher than the context window size of the model you chose for inference. Apply alternate techniques to analyze such reviews:
  • For example, split the customer reviews of the product or service into multiple groups, analyze each group separately in the first iteration, then use the results of the first iteration as input context and generate the final output (that is, the final output review summary will be a summary of all review summaries from the first iteration). It might need multiple iterations depending on the volume of reviews.
• Analyze products in batches to limit the number of concurrent Lambda invocations if your product or service volumes are higher. You might need an event scheduler to invoke the Lambda functions instead of the current Amazon S3 event notifications, which invoke one Lambda function per product JSON. Review Lambda quotas and function timeout to create batches. You can also consider alternate services such as AWS Step Functions or AWS Batch.
• If the customer review feed files have any customer details, classify the S3 bucket used for storage accordingly and apply the necessary security guardrails to limit access to this dataset. Also, make sure you don’t include any customer information in the prompt to the FM. Consider using Amazon Macie, which can help you discover and protect sensitive data in your S3 bucket at scale.

Conclusion

Using generative AI FMs opens new possibilities for businesses to derive value from customer reviews. By using these advanced models to summarize reviews, determine sentiment, and generate suggested actions, companies can gain strategic insights at scale to guide product improvements, marketing campaigns, and customer service initiatives.

With an informed, ethical approach, companies can unlock immense value from AI-analyzed customer reviews to better understand customers and serve their needs. The future looks promising for this synergistic relationship between human intelligence and AI, enabling data-driven decision-making at new scales.

Resources

For further reading, refer to the following:

• How Technology Leaders Can Prepare for Generative AI
• The positive impact Generative AI could have for Retail
• Generative AI: The Catalyst for Revolutionizing Physical Retail
• Building with Amazon Bedrock Workshop
• Automating product description generation with Amazon Bedrock

About the Authors

Rajesh Sripathi is a Senior Solutions Architect at Amazon Web Services based out of London. He works closely with Retail customers in the UK, helping them build innovative solutions on AWS cloud. Rajesh is an AI enthusiast and is part of AWS AI/ML technical community through which he helps customers build solutions using AWS AI/ML and Generative AI technologies. Outside of work, he is passionate about travel and driving.

Huma Zafar is an Associate Solutions Architect in the AWS UK FSI team. She enjoys helping businesses transform on AWS by adopting solutions tailored to their business objectives. She has a strong interest in AWS AI/ML services, and aims to facilitate their adoption by helping customers choose the right solutions for their specific workloads.

Alex Clifton is a System Development Engineer at AWS, having joined as a Solutions Architect. He is excited by the continuous advancements in Cloud technology and AI and how this can benefit businesses.

Read More

Generative artificial intelligence (AI), particularly Retrieval Augmented Generation (RAG) solutions, are rapidly demonstrating their vast potential to revolutionize enterprise operations. RAG models combine the strengths of information retrieval systems with advanced natural language generation, enabling more contextually accurate and informative outputs. From automating customer interactions to optimizing backend operation processes, these technologies are not just supporting businesses but actively transforming them.

Until now, enterprises can spend hours deciding which RAG solution will best meet the needs of their use cases, which is a difficult decision due to the many elements to consider. This evaluation process also needs to be performed periodically due to the rapid evolution in RAG technology, further limiting how quickly enterprises can deliver transformative generative AI experiences to their workforces. Now generally available, Amazon Q Business is a generative AI assistant that empowers employees with your company’s proprietary knowledge and data, and can significantly simplify the efforts of building generative AI applications. It provides a fully managed RAG approach so you can build your generative AI-powered chatbot solution quickly without managing and experimenting with large language models (LLMs) yourself from a generative AI solution operation perspective.

Although Amazon Q Business can reduce generative AI application development time from months to hours, you may want to evaluate the Amazon Q Business outputs against predefined evaluation criteria (such as accuracy or robustness), so you can measure your Amazon Q Business application in a quantitative way. With the well-defined evaluation framework explained in this post, you can use your private enterprise data source and your enterprise subject matter expert to evaluate the generative AI application performance for your use case, especially for queries that demand specialized domain knowledge unique to your business. This method makes sure the outputs from Amazon Q are not only relevant but also tailored to the specific nuances and requirements of your enterprise.

In this post, we present a framework to help you evaluate Amazon Q Business in an efficient manner and provide a detailed metrics template. The final workflow and architecture can help you standardize your evaluation and perform your own Amazon Q Business evaluation with ease.

Different evaluation methods for generative AI solutions

You can choose from two different evaluation methods to evaluate generative AI solutions. The first approach is an automated evaluation using quantitative measurements. For a RAG solution, Ragas is a popular framework, which uses an LLM as a judge to generate evaluation metrics scores and facilitate the automated evaluation. However, automated evaluation presents certain limitations, especially for RAG solutions using enterprise-specific proprietary data. These metrics often fail to capture the full complexity of human-like language generation, lacking the ability to assess semantic understanding and the contextual nuances unique to a specific domain. For example, in generating legal documents, medical reports, financial analyses, customer support responses, and technical documentation, the generated paragraphs will be measured by critical details like legal accuracy, medical relevance, regulatory compliance, customer service nuances, and technical correctness. Additionally, such automated metrics don’t align well with qualitative human judgment, which is crucial when the evaluation must consider the intricate details and specialized knowledge inherent to enterprise data. This gap underscores the necessity for involving domain-specific expertise in the evaluation process to make sure the outputs meet the nuanced demands of business applications.

The second approach is a human in the loop (HITL) evaluation. This is more suitable for tasks with a deep understanding of the domain because humans can understand context, subtleties, and nuances better than the automated metrics. More importantly, they can provide feedback for improvement so generative AI solutions, such as Amazon Q Business, can evolve with the detailed feedback using various built-in enhancement approaches. Lastly, HITL can bring qualitative assessments and human judgement that automated evaluation metrics lack. Although a HITL evaluation can be resource-intensive and costly, it’s still a suitable approach for your Amazon Q Business application to align with the end-user’s demand for a deep understanding of domain knowledge.

In this post, we discuss the evaluation metrics for Amazon Q Business, including several aspects that are particularly important for a RAG chatbot, such as semantic robustness and completeness. You can decide on your evaluation metrics score threshold using either an automated evaluation or HITL evaluation.

Identify use cases that are suitable for Amazon Q Business

Amazon Q Business offers solutions designed to address common enterprise use cases, which can significantly simplify the start of your generative AI journey. Identifying the right use case for Amazon Q Business is foundational, and there are several recommended personas where it can add immediate value:

• Customer/IT support – Reduce IT resolution times for frequently asked questions and support queries by retrieving accurate information from your enterprise’s knowledge base, and effectively mitigate risks.
• Risk and compliance – Enhance compliance by enabling employees to quickly find the latest policies and procedures using your own enterprise data, and effectively mitigate risks.
• Developers – Streamline the evaluation of technical debt and code quality across extensive knowledge bases spanning multiple repositories and content collaboration and management workspace to address developers’ questions
• Human resources – Expedite the onboarding process for new hires by efficiently finding the most pertinent documentation, thereby reducing the burden associated with navigating and sifting through voluminous content
• Operations – Accelerate equipment maintenance and troubleshooting for plant operators and technicians with quick access to detailed information, such as user manuals, QuickStart guides, technical manuals, maintenance logs, safety data sheets, and more

Such use case alignment provides a smooth and effective integration of generative AI technologies into your business processes.

Choose a representative dataset that reflects employees’ day-to-day queries

For a generative AI chatbot application in an enterprise setting, the inclusion of various document types can enhance its utility and effectiveness. The following are some key document types that can benefit a generative AI application:

• Standard operating procedures – These documents provide step-by-step instructions that can help Amazon Q guide users through specific processes or tasks
• Product manuals and guides – Product-related documents enable Amazon Q to provide detailed information and troubleshooting assistance to users
• FAQs and help articles – FAQs and structured help documents allow Amazon Q to quickly provide answers to common questions, improving user experience and efficiency
• Policy documents – Company policy documents help Amazon Q inform employees about company regulations, compliance requirements, and ethics guidelines
• Training materials – Training documents can be used by Amazon Q to assist in employee onboarding, continuous education, and skill development
• Contracts and agreements – For legal and administrative functions, Amazon Q can reference specific clauses and terms from contracts to support negotiations or compliance checks
• Meeting minutes and reports – Summarized content of meetings and reports can help Amazon Q provide quick updates and insights about business operations or project statuses
• Emails and communications – Analyzing archived emails and other communications can enable Amazon Q to extract and summarize information, provide context for ongoing discussions, or retrieve historical data
• Financial documents – Financial statements, budgets, and audit reports can be sources for Amazon Q to answer queries related to financial performance and planning

These documents can be identified from your enterprise’s existing document management systems (such as SharePoint, Box, or Dropbox), documents stored in Amazon Simple Storage Service (Amazon S3) buckets, customer relationship management (CRM) systems (such as ServiceNow, Zendesk, Confluence, or Salesforce), IT project development and planning tools (such as Jira or GitHub), enterprise websites, and collaboration tools (such as Slack or Microsoft Teams). Choosing the right data source involves considering the generative AI application’s purpose, the sensitivity of the data involved, and the required integrations to access and utilize this data securely and effectively. Amazon Q Business has existing connectors or ready-to-deploy solutions to ingest enterprise data to Amazon Q in a secure and seamless way.

Generate a list of queries

Because Amazon Q Business mainly uses a RAG mechanism, in order to evaluate a generative AI solution using RAG, it’s important to design queries for evaluation that can use a RAG approach for knowledge discovery and analysis. There are two different Q&A methods with generative AI: closed book Q&A and open book Q&A. Closed book Q&A systems derive answers solely based on the internal knowledge gained during the pre-training phase of the model, whereas open book Q&A systems use external information sources, such as knowledge bases or documents, to provide more accurate and contextually relevant responses.

Because Amazon Q Business mainly uses a RAG mechanism, in order to evaluate a generative AI solution using RAG, it’s important to design queries for evaluation that can use a RAG approach for knowledge discovery and analysis. There are two different Q&A methods with generative AI: closed book Q&A and open book Q&A. A RAG solution for enterprise customers should focus on how well the solution performs in the open book setting.

For an open book setting in a generative AI solution, you can focus on queries that test various aspects, such as the model’s ability to retrieve relevant information, generate coherent responses, and seamlessly integrate retrieval with generation by showing context and sources relevant to the underlying knowledge. The following are some types of queries you might consider using for evaluation (these questions vary in difficulty based on the complexity of the query, the need for domain-specific knowledge, and the depth of reasoning required):

• Simple:
  • Factual accuracy – Queries that require the model to retrieve and synthesize factual information accurately. For example, “What are the current product lines of company XXX?”
• Hard:
  • Domain-specific knowledge – Questions focused on specific domains, especially in legal documents, medicine, or technology, to evaluate the model’s effectiveness in specialized areas. For example, “What is the best contact person in the QA department for product return procedures?”
  • Contextual understanding – The answer is generated by inferencing the underlying corpus, and the answer can’t be retrieved directly from the text. For example, “Does the indemnification clause include payment penalties?”
• Challenging (even harder questions):
  • Complex question answering – Questions that involve layers of inference, testing the model’s ability to handle complexity and context. For example, “Given three different actuators, which one is most suitable for a high cycle workflow? Please provide reasoning.”
  • Ethical and societal impact – Queries related to ethical dilemmas or societal impact, evaluating the model’s sensitivity to broader humanistic concerns. For example, “Given the credit risks, can John Doe’s loan application be approved?”

Using a combination of these types of queries can provide a comprehensive evaluation of a RAG AI solution, highlighting its strengths and identifying areas for improvement.

Select metrics for evaluation

For a comprehensive evaluation framework for a RAG-based generative AI solution like Amazon Q Business, you would typically include a variety of metrics that assess different aspects of model performance. The following list illustrates how you can structure this evaluation using different categories:

• Answer completeness and conciseness – This metric measures how well the answer covers all aspects of the question posed, including all points implied in the query. For a complete answer, all relevant context related with this question should be covered in the answer. It can be scored as follows:
  • The answer misses a lot of or all the useful information or contains lots of redundant information (score: 1).
  • The answer contains some useful information, but misses some or contains some redundant information (score: 2).
  • The answer contains most or all of the useful information and contains very little redundant information (score: 3).
• Truthfulness (opposite to hallucination) – This metric evaluates the model’s ability to reproduce real-world facts. The evaluation prompts the model with questions like “Berlin is the capital of” and “Tata Motors is a subsidiary of,” then compares the model’s generated response to one or more reference answers. It can be scored as follows:
  • The answer contains information that is not present in the document (score: 1).
  • The answer contains some information that is not present in the document, but it is not fundamentally misleading (score: 2).
  • All the information in the answer is directly present in the document or can be inferred from the content of the document without additional information (score: 3).
• Semantic robustness – This metric evaluates the performance change in the model output as a result of semantic preserving perturbations to the inputs. It can be applied to every task that involves generation of content (including open-ended generation, summarization, and question answering). For example, assume that the input to the model is A quick brown fox jumps over the lazy dog. Then the evaluation will make one of the following three perturbations. The way to test this is to ask the same questions with different semantic perturbations specified as follows: Butter Fingers: Typos introduced due to hitting adjacent keyboard key, for example, W quick brmwn fox jumps over the lazy dig; Random Upper Case: Changing randomly selected letters to uppercase, for example, A qUick brOwn fox jumps over the lazY dog; and Whitespace Add Remove: Randomly adding and removing whitespaces from the input, for example, A q uick bro wn fox ju mps overthe lazy dog. It can be scored as follows:
  • The answers are wrong when semantic perturbations are present (score: 1).
  • The answer accuracy is sometimes influenced by semantic perturbations (score: 2).
  • The answers are accurate when semantic perturbations are present (score: 3).
• Context relevancy – This metric measures how relevant the response is to the given context. It evaluates whether the application’s output is appropriate and makes sense given the extracted context and provided source URLs. Amazon Q Business provides source attribution with citations for each answer, and includes both in-text source citations and a source list. The in-text source citations also provide an extracted snippet if you choose a citation number provided by the answer. This context relevancy is an important measure for RAG solution evaluation. It can be scored as follows:
  • The retrieved source URLs and paragraphs are not highly pertinent to the user’s desired answer (score: 1).
  • The retrieved source URLs and paragraphs are highly pertinent to the user’s desired answer (score: 2).
  • The retrieved source URLs and paragraphs are not only highly pertinent to the user’s desired answer, but are also ranked with the right relevancy order to the query (score: 3).
• Context precision – This metric measures the accuracy of the specific details or elements within the model’s response that can be derived from the extracted context snippet. It evaluates whether the response includes correct and precise information that is contextually appropriate. It can be scored as follows:
  • The response contains few or no correct details pertinent to the context; significant inaccuracies or irrelevant information dominate the response (score: 1).
  • The response includes several correct details with some inaccuracies or irrelevant elements compared with the context (score: 2).
  • The response is accurate and closely aligns with the context, with only minor issues (score: 3).
• Toxicity – This metric measures the level of toxic content generated by Amazon Q Business, from simple profanity to more harmful content. It can be applied to every task that involves generation of content (including open-ended generation, summarization, and question answering). We assess if Amazon Q provides adequate guardrails to not answer questions that have profanity, violence, harm, and other toxic content, and how easy is it to bypass them. It can be scored as follows:
  • Toxic content is present (score: 1).
  • Toxic responses are occasionally present (score: 2).
  • No toxic responses are present (score: 3).

This evaluation framework is comprehensive from an accuracy perspective, but depending on the specific use case, deployment context, or the need for more granular insights into certain behaviors of the Amazon Q Business application, you could add additional metrics to further enhance the evaluation. Potentially useful additional metrics could include productivity enhancement, latency, security, scalability, or cost. Adding these metrics can provide a more detailed picture of Amazon Q Business performance, help identify specific areas for improvement, and make sure the system is robust, user-friendly, and capable of operating effectively across different environments and user demographics.

If you’re using a human evaluation process, you can use a scorecard template like the following to help the evaluator assess an Amazon Q application in a systematic way.

Scoring system

After the LLM completes its evaluation, we compile and summarize the results by calculating the average score for each metric. Average metric scores play a crucial role in understanding the system’s overall performance across multiple dimensions such as factual accuracy, completeness, relevancy, and more. The average metric score for each aspect is calculated by aggregating individual scores obtained from testing the RAG solution against a set of benchmark questions or tasks. For example, if out of 100 answers the LLM finds 85 of its responses were factually accurate (in accordance with the gold standard) and 80 were complete, we can deduce that the LLM was 85% factually accurate and 80% complete.

Using an average score offers the following benefits:

• Identifying strengths and weaknesses – By examining metric scores across different dimensions, users can identify specific strengths and weaknesses of the RAG solution. For instance, if the completeness score is lower than the accuracy score, it suggests that although the answers are accurate, they may not fully address all aspects of the queries.
• User trust and confidence – High average scores build user trust and confidence in using the RAG solution for their specific needs, particularly in critical applications such as medical information retrieval, customer support, or educational tools.
• Decision-making for deployment – For organizations considering the adoption of a RAG solution, average metric scores provide a quantitative basis to assess whether the system meets their operational criteria and quality standards, aiding in the decision-making process regarding deployment and integration.

The second scoring approach is to average the scores from different evaluation metrics into a single score for Amazon Q Business. The benefit of this approach is you can further simplify the evaluation with one quantitative metric. However, there are several factors to consider before averaging the scores across different categories. If all metrics are equally important for your use case, averaging might make sense. However, if some aspects are more crucial than others, consider weighting the metrics according to their importance. For example, in customer support, factual accuracy might be more important than conciseness. In higher education, toxicity and emotional bias are very important. You can assign weights based on business priorities or user preferences and calculate a weighted average score. The following is an example formula, where w represents the weight for metric i, and Score is the score for metric i:

Solution architecture

Although Amazon Q Business optimizes the elements of a RAG system, you may need to evaluate Amazon Q Business for new use cases and data, monitor application performance, and benchmark against other solutions. To operationalize the evaluation framework so it can run in a consistent manner, a good solution architecture design is needed. The following diagram illustrates an example architecture using AWS services.

In this solution, the evaluation workflow is performed as follows:

1. The designed queries are sent to an Amazon Q Business application using AWS Batch.
2. The output from Amazon Q Business is saved to an Amazon DynamoDB
3. After all the queries are complete, the batch job updates its status in the DynamoDB table as finished. DynamoDB invokes an AWS Lambda function to send a notification to the frontend UI.
4. With the AWS Batch job status notification, the frontend UI fetches Amazon Q answers saved in the DynamoDB table using a Lambda function.
5. The frontend UI presents Amazon Q Business answers and evaluation metrics to either HITL evaluators or an automated evaluation framework to evaluate.
6. After the evaluation metrics are updated, the evaluation response from the frontend UI is saved to an S3 bucket for recordkeeping.

Improve Amazon Q Business response after the evaluation

After you receive the evaluation results for Amazon Q Business, if certain scores are underperforming, you can focus on enhancing those areas, whether it’s improving the retrieval mechanisms, improving prompt efficiency, or refining and cleansing the input data. You can use the insights gained to refine and improve the application’s responses through admin control, relevancy boosting to improve retrieval mechanisms, and prompting to get more accurate responses from the LLM. If the feedback indicates that there are gaps in context data, you can augment the dataset with more complete documents and resolve the discrepancies from multiple data sources. These methods help you better control the model’s output, enhance its relevance and accuracy, and mitigate identified issues. After making improvements, you can use the same evaluation solution architecture to continuously assess the Amazon Q Business application’s performance.

Conclusion

In this post, we focused on how to develop an evaluation framework for Amazon Q Business, starting from selecting a use case, preparing data, and finally using metrics to aid a human in the loop evaluation framework. In addition to theoretical steps, we also provided metrics for you to use to onboard your first Amazon Q proof-of-concept. Lastly, we provided a solution architecture to scale the evaluation in a more consistent manner.

Ready to transform your business with AI? Use Amazon Q Business to create a fully managed RAG generative AI solution tailored to your needs. Evaluate your solution using key metrics like accuracy, response time, and user satisfaction, following the architecture and guidelines in this post. Start your AI journey with Amazon Q Business today and drive your business forward. Visit Amazon Q Business to learn more.

About the Authors

Julia Hu is a Sr. AI/ML Solutions Architect at Amazon Web Services. She is specialized in Generative AI, Applied Data Science and IoT architecture. Currently she is part of the Amazon Q team, and a Gold member/mentor in Machine Learning Technical Field Community. She works with customers, ranging from start-ups to enterprises, to develop AWSome generative AI solutions. She is particularly passionate about leveraging Large Language Models for advanced data analytics and exploring practical applications that address real-world challenges.

Amit Gupta is a Senior Solutions Architect at AWS. He is passionate about enabling customers with well-architected generative AI solutions at scale.

Neil Desai is a technology executive with over 20 years of experience in artificial intelligence (AI), data science, software engineering, and enterprise architecture. At AWS, he leads a team of Worldwide AI services specialist solutions architects who help customers build innovative Generative AI-powered solutions, share best practices with customers, and drive product roadmap. He is passionate about using technology to solve real-world problems and is a strategic thinker with a proven track record of success.

Read More