Posts in category: Amazon AWS

Artificial intelligence (AI) continues to transform how we do business and serve our customers. AWS offers a range of pre-trained AI services that provide ready-to-use intelligence for your applications. In this post, we explore the new AI service capabilities and how they are enhanced using foundation models (FMs).

We focus on the following major updates in this post across key AI services:

• Amazon Transcribe now offers FM-powered language support across over 100 languages to unlock rich insights.
• Amazon Transcribe Call Analytics now offers a new generative AI-powered summarization capability (in preview) that automates post-call summarization to improve contact center agent and manager productivity.
• Amazon Personalize now uses an FM to generate more compelling content and product recommendations
• Amazon Lex now uses large language models (LLMs) to provide accurate and conversational responses to FAQs (in preview), going beyond task-oriented dialogue

Amazon Transcribe expands language support and supercharges customer service productivity using FMs

In order to build global and inclusive speech-enabled applications that cater to users from diverse linguistic backgrounds, customers seek a truly global AI service that can understand and transcribe a wide array of languages with high accuracy. To help you scale globally, Amazon Transcribe now offers a speech FM-powered automatic speech recognition (ASR) system that expands support to over 100 languages.

FM-powered Amazon Transcribe delivers significant accuracy improvement between 20% and 50% across most languages. Apart from accuracy improvements, the new ASR system delivers several differentiating features across all supported languages (over 100) related to ease of use, customization, user safety, and privacy. Some examples include features such as automatic punctuation, custom vocabulary, automatic language identification, speaker diarization, word-level confidence scores, and custom vocabulary filters. Enabled by the high accuracy of Amazon Transcribe across different accents and noise conditions, its support for a large number of languages, and its breadth of value-added feature sets, thousands of enterprises will be empowered to unlock rich insights from their audio content, as well as increase the accessibility and discoverability of their audio and video content across various domains. All existing and new customers using Amazon Transcribe can experience the performance improvements out of the box, without any API changes.

Carbyne is a software company that develops cloud-based, mission-critical contact center solutions for emergency call responders. Carbyne’s mission is to help emergency responders save lives, and language cannot come in the way of their goals.

“AI-powered Carbyne Live Audio Translation is directly aimed at helping improve emergency response for the 68 million Americans who speak a language other than English at home, in addition to the up to 79 million foreign visitors to the country annually. By leveraging Amazon Transcribe’s new multilingual foundation model powered ASR, Carbyne will be even better equipped to democratize life-saving emergency services, because Every. Person. Counts.”

– Alex Dizengof, Co-Founder and CTO of Carbyne.

In a contact center, agents spend precious time after each call manually summarizing notes, which can impact their productivity and increase call wait times. Managers who have limited time to investigate calls and agent performance spend a significant amount of time listening to call recordings or reading entire transcripts while investigating caller issues. Amazon Transcribe Call Analytics now offers generative call summarization, a generative AI-powered capability that can automatically condense the entire interaction into a concise summary. For example, the following is a sample summary of a 10-minute phone call: “Customer reported that they didn’t receive their order even after 10 days from expected delivery date. The agent offered the customer a free replacement and $10 credit for future purchases. The agent will follow up with the customer in 2 days to confirm the receipt of the replacement order.”

This capability allows agents to spend more time talking to callers waiting in the queue rather than engaging in after-call work, thereby improving customer experience. Managers can review the call summary to quickly understand the context of an interaction without reading the whole transcript.

With AWS post call analytics solution, Principal can currently conduct large-scale historical analytics to understand where customer experiences can be improved, generate actionable insights, and prioritize where to act. We look forward to exploring the post call summarization feature using generative AI in Amazon Transcribe Call Analytics in order to enable our agents to focus their time and resources engaging with customers, rather than manual after contact work

– Miguel Antonio Sanchez, Regional Chief Data Officer, Principal Financial Group.

The following screenshots illustrate how to enable generative call summarization on the Amazon Transcribe console, and an example of a summarized transcript.

Amazon Personalize enables hyper-personalization with FMs

Customers across industries such as retail and media and entertainment are increasingly looking to make content and recommended products more tailored to user interest in order to drive higher engagement. For instance, on streaming platforms, users see the standard “Because you watched” recommendations, and on ecommerce websites, “frequently bought together” is used as a generic tagline. To offer more personalized browsing experiences with titles such as “Rise and Shine” and “Love, laughter, and hijinks,” companies need to allocate resources to generate compelling taglines manually. This is tedious and time consuming.

To help address this challenge, Amazon Personalize now offers the Content Generator—a new FM-powered capability that uses natural language to craft simple and engaging text that describes the thematic connections between recommended items. This enables companies to automatically generate engaging titles or email subject lines, to invite customers to click on videos or purchase items.

In addition, Amazon Personalize now offers Personalize on LangChain to power the journey of customers who want to build their own FM-based applications. With this integration, you can invoke Amazon Personalize, retrieve recommendations for a campaign or recommender, and seamlessly feed it into your FM-powered applications within the LangChain ecosystem.

“We are integrating generative AI with Amazon Personalize in order to deliver hyper-personalized experiences to our users. Amazon Personalize has helped us achieve high levels of automation in content customization. For instance, FOX Sports experienced a 400% increase in viewership content starts post-event when applied. Now, we are augmenting generative AI with Amazon Bedrock to our pipeline in order to help our content editors generate themed collections. We look forward to exploring features such as Amazon Personalize Content Generator and Personalize on Langchain to further personalize those collections for our users.”

– Daryl Bowden, Executive Vice President, Technology, Fox Corporation.

Amazon Lex offers FM-powered capabilities to build bots faster and improve containment

Driven by rising consumer demand for automated self-service, companies are prioritizing investments in conversational AI to optimize customer experience. To that end, AWS recently previewed Conversational FAQ (CFAQ), a new capability from Amazon Lex that answers frequently asked customer questions intelligently and at scale. Powered by FMs from Amazon Bedrock and approved knowledge sources, CFAQ enables companies to provide accurate, automated responses to common customer inquiries in a natural and engaging way. With this innovation, brands can deliver seamless self-service experiences that strengthen customer satisfaction and loyalty.

CFAQ simplifies bot development by eliminating the need to manually create intents, sample utterances, slots, and prompts to handle a wide range of frequently asked questions. It does so with a new intent type called QnAIntent that securely connects to knowledge sources like Amazon Bedrock, Amazon OpenSearch Service, and Amazon Kendra knowledge bases to retrieve the most relevant information to answer a question. Developers maintain control over response content, with the option to summarize retrieved information or use the authorized text as is. This allows highly regulated industries like financial services and healthcare to use CFAQ, enabling you to ensure responses use only compliant language. By streamlining access to relevant knowledge, CFAQ reduces the effort to build bots that handle common customer questions naturally and accurately.

Conclusion

AWS is constantly innovating on behalf of our customers. The latest set of advancements in AI services allow us to deliver more impactful capabilities that help organizations work smarter and provide personalized and intuitive experiences. To learn more about these launches, refer to the following:

• Amazon Transcribe announces a new speech foundation model-powered ASR system that expands support to over 100 languages
• Amazon Transcribe Call Analytics adds new generative AI-powered call summaries
• Drive hyper-personalized customer experiences with Amazon Personalize and generative AI
• Elevate your self-service assistants with new generative AI features in Amazon Lex

About the author

Bratin Saha is the Vice President of Artificial Intelligence and Machine Learning at AWS.

Read More

In this post, we talk about how generative AI is changing the conversational AI industry by providing new customer and bot builder experiences, and the new features in Amazon Lex that take advantage of these advances.

As the demand for conversational AI continues to grow, developers are seeking ways to enhance their chatbots with human-like interactions and advanced capabilities such as FAQ handling. Recent breakthroughs in generative AI are leading to significant improvements in natural language understanding that make conversational systems more intelligent. By training large neural network models on datasets with trillions of tokens, AI researchers have developed techniques that allow bots to understand more complex questions, provide nuanced and more natural human-sounding responses, and handle a wide range of topics. With these new generative AI innovations, you can create virtual assistants that feel more natural, intuitive, and helpful during text- or voice-based self-service interactions. The rapid progress in generative AI is bringing automated chatbots and virtual assistants significantly closer to the goal of having truly intelligent, free-flowing conversations. With further advances in deep learning and neural network techniques, conversational systems are poised to become even more flexible, relatable, and human-like. This new generation of AI-powered assistants can provide seamless self-service experiences across a multitude of use cases.

How Amazon Bedrock is changing the landscape of conversational AI

Amazon Bedrock is a user-friendly way to build and scale generative AI applications with foundational models (FMs). Amazon Bedrock offers an array of FMs from leading providers, so AWS customers have flexibility and choice to use the best models for their specific use case.

In today’s fast-paced world, we expect quick and efficient customer service from every business. However, providing excellent customer service can be significantly challenging when the volume of inquiries outpaces the human resources employed to address them. Businesses can overcome this challenge efficiently while also providing personalized customer service by taking advantage of advancements in generative AI powered by large language models (LLMs).

Over the years, AWS has invested in democratizing access to—and amplifying the understanding of—AI, machine learning (ML), and generative AI. LLMs can be highly useful in contact centers by providing automated responses to frequently asked questions, analyzing customer sentiment and intents to route calls appropriately, generating summaries of conversations to help agents, and even automatically generating emails or chat responses to common customer inquiries. By handling repetitive tasks and gaining insights from conversations, LLMs allow contact center agents to focus on delivering higher value through personalized service and resolving complex issues.

Improving the customer experience with conversational FAQs

Generative AI has tremendous potential to provide quick, reliable answers to commonly asked customer questions in a conversational manner. With access to authorized knowledge sources and LLMs, your existing Amazon Lex bot can provide helpful, natural, and accurate responses to FAQs, going beyond task-oriented dialogue. Our Retrieval Augmented Generation (RAG) approach allows Amazon Lex to harness both the breadth of knowledge available in repositories as well as the fluency of LLMs. You can simply ask your question in free-form, conversational language, and receive a natural, tailored response within seconds. The new conversational FAQ feature in Amazon Lex allows bot developers and conversation designers to focus on defining business logic rather than designing exhaustive FAQ-based conversation flows within a bot.

We are introducing a built-in QnAIntent that uses an LLM to query an authorized knowledge source and provide a meaningful and contextual response. In addition, developers can configure the QnAIntent to point to specific knowledge base sections, ensuring only specific portions of the knowledge content is queried at runtime to fulfill user requests. This capability fulfills the need for highly regulated industries, such as financial services and healthcare, to only provide responses in compliant language. The conversational FAQ feature in Amazon Lex allows organizations to improve containment rates while avoiding the high costs of missed queries and human representative transfers.

Building an Amazon Lex bot using the descriptive bot builder

Building conversational bots from scratch is a time-consuming process that requires deep knowledge of how users interact with bots in order to anticipate potential requests and code appropriate responses. Today, conversation designers and developers spend many days writing code to help run all possible user actions (intents), the various ways users phrase their requests (utterances), and the information needed from the user to complete those actions (slots).

The new descriptive bot building feature in Amazon Lex uses generative AI to accelerate the bot building process. Instead of writing code, conversation designers and bot developers can now describe in plain English what they want the bot to accomplish (for example, “Take reservations for my hotel using name and contact info, travel dates, room type, and payment info”). Using only this simple prompt, Amazon Lex will automatically generate intents, training utterances, slots, prompts, and a conversational flow to bring the described bot to life. By providing a baseline bot design, this feature immensely reduces the time and complexity of building conversational chatbots, allowing the builder to reprioritize effort on fine-tuning the conversational experience.

By tapping into the power of generative AI with LLMs, Amazon Lex enables developers and non-technical users to build bots simply by describing their goal. Rather than meticulously coding intents, utterances, slots, and so on, developers can provide a natural language prompt and Amazon Lex will automatically generate a basic bot flow ready for further refinement. This capability is initially only available in English, but developers can further customize the AI-generated bot as needed before deployment, saving many hours of manual development work.

Improving the user experience with assisted slot resolution

As consumers become more familiar with chatbots and interactive voice response (IVR) systems, they expect higher levels of intelligence baked into self-service experiences. Disambiguating responses that are more conversational is imperative to success as users expect more natural, human-like experiences. With rising consumer confidence in chatbot capabilities, there is also an expectation of elevated performance from natural language understanding (NLU). In the likely scenario that a semantically simple or complex utterance is not resolved properly to a slot, user confidence can dwindle. In such instances, an LLM can dynamically assist the existing Amazon Lex NLU model and ensure accurate slot resolution even when the user utterance is beyond the bounds of the slot model. In Amazon Lex, the assisted slot resolution feature provides the bot developer yet another tool for which to increase containment.

During runtime, when NLU fails to resolve a slot during a conversational turn, Amazon Lex will call the LLM selected by the bot developer to assist with resolving the slot. If the LLM is able to provide a value upon slot retry, the user can continue with the conversation as normal. For example, if upon slot retry, a bot asks “What city does the policy holder reside in?” and the user responds “I live in Springfield,” the LLM will be able to resolve the value to “Springfield.” The supported slot types for this feature include AMAZON.City, AMAZON.Country, AMAZON.Number, AMAZON.Date, AMAZON.AlphaNumeric (without regex) and AMAZON.PhoneNumber, and AMAZON.Confirmation. This feature is only available in English at the time of writing.

Improving the builder experience with training utterance generation

One of the pain points that bot builders and conversational designers often encounter is anticipating the variation and diversity of responses when invoking an intent or soliciting slot information. When a bot developer creates a new intent, sample utterances must be provided to train the ML model on the types of responses it can and should accept. It can often be difficult to anticipate the permutations on verbiage and syntax used by customers. With utterance generation, Amazon Lex uses foundational models such as Amazon Titan to generate training utterances with just one click, without the need for any prompt engineering.

Utterance generation uses the intent name, existing utterances, and optionally the intent description to generate new utterances with an LLM. Bot developers and conversational designers can edit or delete the generated utterances before accepting them. This feature works with both new and existing intents.

Conclusion

Recent advancements in generative AI have undoubtedly made automated consumer experiences better. With Amazon Lex, we are committed to infusing generative AI into every aspect of the builder and user experience. The features mentioned in this post are just the beginning—and we can’t wait to show you what is to come.

To learn more, refer to Amazon Lex Documentation, and try these features out on the Amazon Lex console.

About the authors

Anuradha Durfee is a Senior Product Manager on the Amazon Lex team and has more than 7 years of experience in conversational AI. She is fascinated by voice user interfaces and making technology more accessible through intuitive design.

Sandeep Srinivasan is a Senior Product Manager on the Amazon Lex team. As a keen observer of human behavior, he is passionate about customer experience. He spends his waking hours at the intersection of people, technology, and the future.

Read More

Amazon Transcribe is a fully managed automatic speech recognition (ASR) service that makes it straightforward for you to add speech-to-text capabilities to your applications. Today, we are happy to announce a next-generation multi-billion parameter speech foundation model-powered system that expands automatic speech recognition to over 100 languages. In this post, we discuss some of the benefits of this system, how companies are using it, and how to get started. We also provide an example of the transcription output below.

Transcribe’s speech foundation model is trained using best-in-class, self-supervised algorithms to learn the inherent universal patterns of human speech across languages and accents. It is trained on millions of hours of unlabeled audio data from over 100 languages. The training recipes are optimized through smart data sampling to balance the training data between languages, ensuring that traditionally under-represented languages also reach high accuracy levels.

Carbyne is a software company that develops cloud-based, mission-critical contact center solutions for emergency call responders. Carbyne’s mission is to help emergency responders save lives, and language can’t get in the way of their goals. Here is how they use Amazon Transcribe to pursue their mission:

“AI-powered Carbyne Live Audio Translation is directly aimed at helping improve emergency response for the 68 million Americans who speak a language other than English at home, in addition to the up to 79 million foreign visitors to the country annually. By leveraging Amazon Transcribe’s new multilingual foundation model powered ASR, Carbyne will be even better equipped to democratize life-saving emergency services, because Every. Person. Counts.”

– Alex Dizengof, Co-Founder and CTO of Carbyne.

By leveraging speech foundation model, Amazon Transcribe delivers significant accuracy improvement between 20% and 50% across most languages. On telephony speech, which is a challenging and data-scarce domain, accuracy improvement is between 30% and 70%. In addition to substantial accuracy improvement, this large ASR model also delivers improvements in readability with more accurate punctuation and capitalization. With the advent of generative AI, thousands of enterprises are using Amazon Transcribe to unlock rich insights from their audio content. With significantly improved accuracy and support for over 100 languages, Amazon Transcribe will positively impact all such use cases. All existing and new customers using Amazon Transcribe in batch mode can access speech foundation model-powered speech recognition without needing any change to either the API endpoint or input parameters.

The new ASR system delivers several key features across all the 100+ languages related to ease of use, customization, user safety, and privacy. These include features such as automatic punctuation, custom vocabulary, automatic language identification, speaker diarization, word-level confidence scores, and custom vocabulary filter. The system’s expanded support for different accents, noise environments, and acoustic conditions enables you to produce more accurate outputs and thereby helps you effectively embed voice technologies in your applications.

Enabled by the high accuracy of Amazon Transcribe across different accents and noise conditions, its support for a large number of languages, and its breadth of value-added feature sets, thousands of enterprises will be empowered to unlock rich insights from their audio content, as well as increase the accessibility and discoverability of their audio and video content across various domains. For instance, contact centers transcribe and analyze customer calls to identify insights and subsequently improve customer experience and agent productivity. Content producers and media distributors automatically generate subtitles using Amazon Transcribe to improve content accessibility.

Get started with Amazon Transcribe

You can use the AWS Command Line Interface (AWS CLI), AWS Management Console, and various AWS SDKs for batch transcriptions and continue to use the same StartTranscriptionJob API to get performance benefits from the enhanced ASR model without needing to make any code or parameter changes on your end. For more information about using the AWS CLI and the console, refer to Transcribing with the AWS CLI and Transcribing with the AWS Management Console, respectively.

The first step is to upload your media files into an Amazon Simple Storage Service (Amazon S3) bucket, an object storage service built to store and retrieve any amount of data from anywhere. Amazon S3 offers industry-leading durability, availability, performance, security, and virtually unlimited scalability at very low cost. You can choose to save your transcript in your own S3 bucket, or have Amazon Transcribe use a secure default bucket. To learn more about using S3 buckets, see Creating, configuring, and working with Amazon S3 buckets.

Transcription output

Amazon Transcribe uses JSON representation for its output. It provides the transcription result in two different formats: text format and itemized format. Nothing changes with respect to the API endpoint or input parameters.

The text format provides the transcript as a block of text, whereas itemized format provides the transcript in the form of timely ordered transcribed items, along with additional metadata per item. Both formats exist in parallel in the output file.

Depending on the features you select when creating the transcription job, Amazon Transcribe creates additional and enriched views of the transcription result. See the following example code:

{
   "jobName": "2x-speakers_2x-channels",
    "accountId": "************",
    "results": {
        "transcripts": [
{
                "transcript": "Hi, welcome."
            }
        ],
        "speaker_labels": [
            {
                "channel_label": "ch_0",
                "speakers": 2,
                "segments": [
                ]
            },
            {
                "channel_label": "ch_1",
                "speakers": 2,
                "segments": [
                ]
            }
        ],
        "channel_labels": {
            "channels": [
            ],
            "number_of_channels": 2
        },
        "items": [
            
        ],
        "segments": [
        ]
    },
    "status": "COMPLETED"
}

The views are as follows:

• Transcripts – Represented by the transcripts element, it contains only the text format of the transcript. In multi-speaker, multi-channel scenarios, concatenation of all transcripts is provided as a single block.
• Speakers – Represented by the speaker_labels element, it contains the text and itemized formats of the transcript grouped by speaker. It’s available only when the multi-speakers feature is enabled.
• Channels – Represented by the channel_labels element, it contains the text and itemized formats of the transcript, grouped by channel. It’s available only when the multi-channels feature is enabled.
• Items – Represented by the items element, it contains only the itemized format of the transcript. In multi-speaker, multi-channel scenarios, items are enriched with additional properties, indicating speaker and channel.
• Segments – Represented by the segments element, it contains the text and itemized formats of the transcript, grouped by alternative transcription. It’s available only when the alternative results feature is enabled.

Conclusion

At AWS, we are constantly innovating on behalf of our customers. By extending the language support in Amazon Transcribe to over 100 languages, we enable our customers to serve users from diverse linguistic backgrounds. This not only enhances accessibility, but also opens up new avenues for communication and information exchange on a global scale. To learn more about the features discussed in this post, check out features page and what’s new post.

About the authors

Sumit Kumar is a Principal Product Manager, Technical at AWS AI Language Services team. He has 10 years of product management experience across a variety of domains and is passionate about AI/ML. Outside of work, Sumit loves to travel and enjoys playing cricket and Lawn-Tennis.

Vivek Singh is a Senior Manager, Product Management at AWS AI Language Services team. He leads the Amazon Transcribe product team. Prior to joining AWS, he held product management roles across various other Amazon organizations such as consumer payments and retail. Vivek lives in Seattle, WA and enjoys running, and hiking.

Read More

Today, we are excited to announce three launches that will help you enhance personalized customer experiences using Amazon Personalize and generative AI. Whether you’re looking for a managed solution or build your own, you can use these new capabilities to power your journey.

Amazon Personalize is a fully managed machine learning (ML) service that makes it easy for developers to deliver personalized experiences to their users. It enables you to improve customer engagement by powering personalized product and content recommendations in websites, applications, and targeted marketing campaigns, with no ML expertise required. Using recipes (algorithms prepared for specific uses cases) provided by Amazon Personalize, you can offer diverse personalization experiences like “recommend for you”, “frequently bought together”, guidance on next best actions, and targeted marketing campaigns with user segmentation.

Generative AI is quickly transforming how enterprises do business. Gartner predicts that “by 2026, more than 80% of enterprises will have used generative AI APIs or models, or deployed generative AI-enabled applications in production environments, up from less than 5% in 2023.” While generative AI can quickly create content, it alone is not enough to provide higher degree of personalization to adapt to the ever-changing and nuanced preferences of individual users. Many companies are actively seeking solutions to enhance user experience using Amazon Personalize and generative AI.

FOX Corporation (FOX) produces and distributes news, sports, and entertainment content.

“We are integrating generative AI with Amazon Personalize in order to deliver hyper-personalized experiences to our users. Amazon Personalize has helped us achieve high levels of automation in content customization. For instance, FOX Sports experienced a 400% increase in viewership content starts post-event when applied. Now, we are augmenting generative AI with Amazon Bedrock to our pipeline in order to help our content editors generate themed collections. We look forward to exploring features such as Amazon Personalize Content Generator and Personalize on LangChain to further personalize those collections for our users.”

– Daryl Bowden, Executive Vice President of Technology Platforms.

Announcing Amazon Personalize Content Generator to make recommendations more compelling

Amazon Personalize has launched Content Generator, a new generative AI-powered capability that helps companies make recommendations more compelling by identifying thematic connections between the recommended items. This capability can elevate the recommendation experience beyond standard phrases like “People who bought this also bought…” to more engaging taglines such as “Rise and Shine” for a breakfast food collection, enticing users to click and purchase.

To explore the impact of Amazon Personalize Content Generator in detail, let’s look at two examples.

Use case 1: Carousel titles for movie collections

A micro-genre is a specialized subcategory within a broader genre of film, music, or other forms of media. Streaming platforms use micro-genres to enhance user experience by allowing viewers or listeners to discover content that aligns with their specific tastes and interests. By recommending media content with micro-genres, streaming platforms cater to diverse preferences, ultimately increasing user engagement and satisfaction.

Now you can use Amazon Personalize Content Generator to create carousel titles for micro-genre collections. First, import your datasets of users’ interactions and items into Amazon Personalize for training. You upload a list of itemId values as your seed items. Next, create a batch inference job selecting Themed recommendations with Content Generator on the Amazon Personalize console or setting batch-inference-job-mode to THEME_GENERATION in the API configuration.

As the batch inference output, you will get a set of similar items and a theme for each seed item. We also provide items-theme relevance scores that you can use to set a threshold to show only items that are strongly related to the theme. The following screenshot shows an example of the output:

{"input":{"itemId":"40"},"output":{
"recommendedItems":["36","50","44","22","21","29","3","1","2","39"],
"theme":"Movies with a strong female lead",
"itemsThemeRelevanceScores":[0.19994527,0.183059963,0.17478035,0.1618133,0.1574806,0.15468733,0.1499242,0.14353688,0.13531424,0.10291852]}}

{"input":{"itemId":"43"},"output":{
"recommendedItems":["50","21","36","3","17","2","39","1","10","5"],
"theme":"Romantic movies for a cozy night in",
"itemsThemeRelevanceScores":[0.184988,0.1795761,0.11143453,0.0989443,0.08258403,0.07952615,0.07115086,0.0621634,-0.138913,-0.188913]}}
...

Subsequently you can replace the generic phrase “More like X” with the output theme from Amazon Personalize Content Generator to make the recommendations more compelling.

Use case 2: Subject lines for marketing emails

Email marketing, although cost-effective, often struggles with low open rates and high unsubscribe rates. The decision to open an email critically depends on how attractive the subject line is, because it’s the first thing recipients see along with the sender’s name. However, scripting appealing subject lines can often be tedious and time-consuming.

Now with Amazon Personalize Content Generator, you can create compelling subject lines or headlines in the email body more efficiently, further personalizing your email campaigns. You follow the same process of data ingestion, training, and creating a batch inference job as in the previous use case. The following is an example of a marketing email that incorporates output from Amazon Personalize using Content Generator, including a set of recommended items and a generated subject line:

These examples showcase how Amazon Personalize Content Generator can assist you in creating a more engaging browsing experience or a more effective marketing campaign. For more detailed instructions, refer to Themed batch recommendations.

Announcing LangChain integration to seamlessly integrate Amazon Personalize with the LangChain framework

LangChain is a powerful open-source framework that allows for integration with large language models (LLMs). LLMs are typically versatile but may struggle with domain-specific tasks where deeper context and nuanced responses are needed. LangChain empowers developers in such scenarios to build modules (agents/chains) for their specific generative AI tasks. They can also introduce context and memory into LLMs by connecting and chaining LLM prompts to solve for varying use cases.

We are excited to launch LangChain integration. With this new capability, builders can use the Amazon Personalize custom chain on LangChain to seamlessly integrate Amazon Personalize with generative AI solutions. Adding a personalized touch to generative AI solutions helps you create more tailored and relevant interactions with end-users. The following code snippet demonstrates how you can invoke Amazon Personalize, retrieve recommendations for a campaign or recommender, and seamlessly feed it into your generative AI applications within the LangChain ecosystem. You can also use this for sequential chains.

from langchain.utilities import AmazonPersonalize
from langchain.chains import AmazonPersonalizeChain
from langchain.llms.bedrock import Bedrock

recommender_arn="<insert_arn>"
client=AmazonPersonalize(recommender_arn=recommender_arn, credentials_profile_name="default",region_name="us-west-2")

bedrock_llm = Bedrock(model_id="anthropic.claude-v2", region_name="us-west-2")

# Create personalize chain
chain = AmazonPersonalizeChain.from_llm( llm=bedrock_llm, client=client)
response = chain({'user_id': '1'})

You can use this capability to craft personalized marketing copies, generate concise summaries for recommended content, recommend products or content in chatbots, and build next-generation customer experiences with your creativity.

Amazon Personalize now enables you to return metadata in inference response to improve generative AI workflow

Amazon Personalize now improves your generative AI workflow by enabling return item metadata as part of the inference output. Getting recommendations along with metadata makes it more convenient to provide additional context to LLMs. This additional context, such as genre and product description, can help the models gain a deeper understanding of item attributes to generate more relevant content.

Amazon Personalize supports this capability for both custom recipes and domain optimized recommenders. When creating a campaign or a recommender, you can enable the option to return metadata with recommendation results, or adjust the setting by updating the campaign or recommender. You can select up to 10 metadata fields and 50 recommendation results to return metadata during an inference call, either through the Amazon Personalize API or the Amazon Personalize console.

The following is an example in the API:

## Create campaign with enabled metadata
example_name = 'metadata_response_enabled_campaign'
create_campaign_response = personalize.create_campaign(
    name = example_name,
    solutionVersionArn = example_solution_version_arn,
    minProvisionedTPS = 1,
    campaignConfig = {"enableMetadataWithRecommendations": True}
)

## GetRecommendations with metadata columns
metadataMap = {"ITEMS": ["genres", "num"]}
response = personalize_runtime.get_recommendations(campaignArn=example_campaign_arn,
     userId="0001", itemId="0002", metadataColumns=metadataMap, numResults=2)
     
## Example response with metadata
 itemList': 
 [
     {
      'itemId': '356',
      'metadata': {'genres': 'Comedy', 'num': '0.6103248'}
     },
     {
      'itemId': '260',
      'metadata': {'genres': 'Action|Adventure', 'num': '0.074548'}},
     }
 ]

Conclusion

At AWS, we are constantly innovating on behalf of our customers. By introducing these new launches powered by Amazon Personalize and Amazon Bedrock, we will enrich every aspect of the builder and user experience, elevating efficiency and end-user satisfaction. To learn more about the capabilities discussed in this post, check out Amazon Personalize features and the Amazon Personalize Developer Guide.

About the Authors

Jingwen Hu is a Senior Technical Product Manager working with AWS AI/ML on the Amazon Personalize team. In her spare time, she enjoys traveling and exploring local food.

Pranav Agarwal is a Senior Software Engineer with AWS AI/ML and works on architecting software systems and building AI-powered recommender systems at scale. Outside of work, he enjoys reading, running, and ice-skating.

Rishabh Agrawal is a Senior Software Engineer working on AI services at AWS. In his spare time, he enjoys hiking, traveling, and reading.

Ashish Lal is a Senior Product Marketing Manager who leads product marketing for AI services at AWS. He has 9 years of marketing experience and has led the product marketing effort for intelligent document processing. He got his master’s in Business Administration at the University of Washington.

Read More

Amazon Personalize is excited to announce the new Next Best Action (aws-next-best-action) recipe to help you determine the best actions to suggest to your individual users that will enable you to increase brand loyalty and conversion.

Amazon Personalize is a fully managed machine learning (ML) service that makes it effortless for developers to deliver highly personalized user experiences in real time. It enables you to improve customer engagement by powering personalized product and content recommendations in websites, applications, and targeted marketing campaigns. You can get started without any prior ML experience, using APIs to easily build sophisticated personalization capabilities in a few clicks. All your data is encrypted to be private and secure.

In this post, we show you how to use the Next Best Action recipe to personalize action recommendations based on each user’s past interactions, needs, and behavior.

Solution overview

With the rapid growth of digital channels and technology advances that make hyper-personalization more accessible, brands struggle to determine what actions will maximize engagement for each individual user. Brands either show the same actions to all users or rely on traditional user segmentation approaches to recommend actions to each user cohort. However, these approaches are no longer sufficient, because every user expects a unique experience and tends to abandon brands that don’t understand their needs. Furthermore, brands are unable to update the action recommendations in real time due to the manual nature of the process.

With Next Best Action, you can determine the actions that have the highest likelihood of engaging each individual user based on their preferences, needs, and history. Next Best Action takes the in-session interests of each user into account and provides action recommendations in real time. You can recommend actions such as enrolling in loyalty programs, signing up for a newsletter or magazine, exploring a new category, downloading an app, and other actions that encourage conversion. This will enable you to improve each user’s experience by providing them with recommendations on actions across their user journey that will help promote long-term brand engagement and revenue. It will also help improve your return on marketing investment by recommending the action that each user has a high likelihood of taking.

AWS Partners like Credera are excited by the personalization possibilities that the Amazon Personalize Next Best Action will unlock for their customers.

“Amazon Personalize is a world-class machine learning solution that enables companies to create meaningful customer experiences across a wide array of use cases without extensive rework or up-front implementation cost that is typically required of these types of solutions. We are really excited about the addition of the Next Best Action capability that will enable customers to provide personalized action recommendations, significantly improving their digital experiences and driving additional business value. Specifically, we expect anyone working within the retail or content space to see an improved experience for their customers and higher conversions as a direct result of using Amazon Personalize. We are extremely thrilled to be a launch partner with AWS on this release and looking forward to empowering businesses to drive ML-based personalized solutions with Next Best Action.”

– Jason Goth, Partner and Chief Technology Officer, Credera.

Example use cases

To explore the impact of this new feature in greater detail, let’s review an example by taking three users: A (User_id 11999), B (User_id 17141), and C (User_id 8103), who are in different stages of their user journey while making purchases on a website. We then see how Next Best Action suggests the optimal actions for each user based on their past interactions and preferences.

First, we look at the action interactions dataset to understand how users have interacted with actions in the past. The following example shows the three users and their different shopping patterns. User A is a frequent buyer and has shopped mostly in the “Beauty & Grooming” and “Jewelry” categories in the past. User B is a casual buyer who has made a few purchases in the “Electronics” category in the past, and User C is a new user on the website who has made their first purchase in the “Clothing” category.

Traditionally, brands either show the same actions to all users or employ user segmentation strategies to recommend actions to their user base. The following table is an example of a brand showing the same set of actions to all users. These actions may or may not be relevant to the users, reducing their engagement with the brand.

Now let’s use Next Best Action to recommend actions for each user. After you define the actions eligible for recommendations, the aws-next-best-action recipe returns a ranked list of actions, personalized for each user, based on user propensity (the probability of a user taking a particular action, ranging between 0.0–1.0) and value of that action, if provided. For the purpose of this post, we only consider user propensity.

In the following example, we see that for User A (frequent buyer), Subscribe to Loyalty Program is the top recommended action with a propensity score of 1.00, which means that this user is most likely to enroll in the loyalty program because they have made numerous purchases. Therefore, recommending the action Subscribe to Loyalty Program to User A has a high probability of increasing User A’s engagement.

Similarly, User B (casual buyer persona) has a higher probability to continue purchasing in “Electronics” category and also buying new products in a similar category, “Smart Homes”. Therefore, Next Best Action recommends you to prioritize actions, Purchase in “Electronics” category and Purchase in “Smart Homes” category. This means that if you prompt User B to buy products in these two categories, it can lead to greater engagement. We also notice the action to Subscribe to Loyalty Program is recommended to User B but with a lower propensity score of 0.71 as compared to User A, whose propensity score is 1.0. This is because users that have a deeper history and are further along their shopping journey benefit more from Loyalty programs due of the added benefits and are highly likely to interact more.

Finally, we see that Next Best Action for User C is purchasing in “Handbags & Shoes” category, which is similar to their previous action of Purchase in “Clothing” category. We also see that the propensity score to Download the mobile app is relatively lower (0.48) than another action, Purchase in “Handbags & Shoes” category, which has a higher propensity score of 0.60. This means that if you recommend User C to purchase products in a complementary category (“Handbags & Shoes”) over downloading the mobile app, they are more likely to stick with your brand and continue shopping in the future.

For more details on how to implement the Next Best Action (aws-next-best-action) recipe, refer to documentation.

Conclusion

The new Next Best Action recipe in Amazon Personalize helps you recommend the right actions to the right user in real time based on their individual behavior and needs. This will enable you to maximize user engagement and lead to greater conversion rates.

For more information about Amazon Personalize, see the Amazon Personalize Developer Guide.

About the Authors

Shreeya Sharma is a Sr. Technical Product Manager working with AWS AI/ML on Amazon Personalize. She has a background in computer science engineering, technology consulting, and data analytics. In her spare time, she enjoys traveling, performing theatre, and trying out new adventures.

Pranesh Anubhav is a Senior Software Engineer for Amazon Personalize. He is passionate about designing machine learning systems to serve customers at scale. Outside of his work, he loves playing soccer and is an avid follower of Real Madrid.

Aniket Deshmukh is an Applied Scientist in AWS AI labs supporting Amazon Personalize. Aniket works in the general area of recommendation systems, contextual bandits, and multi-modal deep learning.

Read More

This post is co-written with Marc Neumann, Amor Steinberg and Marinus Krommenhoek from BMW Group.

The BMW Group – headquartered in Munich, Germany – is driven by 149,000 employees worldwide and manufactures in over 30 production and assembly facilities across 15 countries. Today, the BMW Group is the world’s leading manufacturer of premium automobiles and motorcycles, and provider of premium financial and mobility services. The BMW Group sets trends in production technology and sustainability as an innovation leader with an intelligent material mix, a technological shift towards digitalization, and resource-efficient production.

In an increasingly digital and rapidly changing world, BMW Group’s business and product development strategies rely heavily on data-driven decision-making. With that, the need for data scientists and machine learning (ML) engineers has grown significantly. These skilled professionals are tasked with building and deploying models that improve the quality and efficiency of BMW’s business processes and enable informed leadership decisions.

Data scientists and ML engineers require capable tooling and sufficient compute for their work. Therefore, BMW established a centralized ML/deep learning infrastructure on premises several years ago and continuously upgraded it. To pave the way for the growth of AI, BMW Group needed to make a leap regarding scalability and elasticity while reducing operational overhead, software licensing, and hardware management.

In this post, we will talk about how BMW Group, in collaboration with AWS Professional Services, built its Jupyter Managed (JuMa) service to address these challenges. JuMa is a service of BMW Group’s AI platform for its data analysts, ML engineers, and data scientists that provides a user-friendly workspace with an integrated development environment (IDE). It is powered by Amazon SageMaker Studio and provides JupyterLab for Python and Posit Workbench for R. This offering enables BMW ML engineers to perform code-centric data analytics and ML, increases developer productivity by providing self-service capability and infrastructure automation, and tightly integrates with BMW’s centralized IT tooling landscape.

JuMa is now available to all data scientists, ML engineers, and data analysts at BMW Group. The service streamlines ML development and production workflows (MLOps) across BMW by providing a cost-efficient and scalable development environment that facilitates seamless collaboration between data science and engineering teams worldwide. This results in faster experimentation and shorter idea validation cycles. Moreover, the JuMa infrastructure, which is based on AWS serverless and managed services, helps reduce operational overhead for DevOps teams and allows them to focus on enabling use cases and accelerating AI innovation at BMW Group.

Challenges of growing an on-premises AI platform

Prior to introducing the JuMa service, BMW teams worldwide were using two on-premises platforms that provided teams JupyterHub and RStudio environments. These platforms were too limited regarding CPU, GPU, and memory to allow the scalability of AI at BMW Group. Scaling these platforms with managing more on-premises hardware, more software licenses, and support fees would require significant up-front investments and high efforts for its maintenance. To add to this, limited self-service capabilities were available, requiring high operational effort for its DevOps teams. More importantly, the use of these platforms was misaligned with BMW Group’s IT cloud-first strategy. For example, teams using these platforms missed an easy migration of their AI/ML prototypes to the industrialization of the solution running on AWS. In contrast, the data science and analytics teams already using AWS directly for experimentation needed to also take care of building and operating their AWS infrastructure while ensuring compliance with BMW Group’s internal policies, local laws, and regulations. This included a range of configuration and governance activities from ordering AWS accounts, limiting internet access, using allowed listed packages to keeping their Docker images up to date.

Overview of solution

JuMa is a fully managed multi-tenant, security hardened AI platform service built on AWS with SageMaker Studio at the core. By relying on AWS serverless and managed services as the main building blocks of the infrastructure, the JuMa DevOps team doesn’t need to worry about patching servers, upgrading storage, or managing any other infrastructure components. The service handles all those processes automatically, providing a powerful technical platform that is generally up to date and ready to use.

JuMa users can effortlessly order a workspace via a self-service portal to create a secure and isolated development and experimentation environment for their teams. After a JuMa workspace is provisioned, the users can launch JupyterLab or Posit workbench environments in SageMaker Studio with just a few clicks and start the development immediately, using the tools and frameworks they are most familiar with. JuMa is tightly integrated with a range of BMW Central IT services, including identity and access management, roles and rights management, BMW Cloud Data Hub (BMW’s data lake on AWS) and on-premises databases. The latter helps AI/ML teams seamlessly access required data, given they are authorized to do so, without needing to build data pipelines. Furthermore, the notebooks can be integrated into the corporate Git repositories to collaborate using version control.

The solution abstracts away all technical complexities associated with AWS account management, configuration, and customization for AI/ML teams, allowing them to fully focus on AI innovation. The platform ensures that the workspace configuration meets BMW’s security and compliance requirements out of the box.

The following diagram describes the high-level context view of the architecture.

User journey

BMW AI/ML team members can order their JuMa workspace using BMW’s standard catalog service. After approval by the line manager, the ordered JuMa workspace is provisioned by the platform fully automatedly. The workspace provisioning workflow includes the following steps (as numbered in the architecture diagram).

1. A data scientist team orders a new JuMa workspace in BMW’s Catalog. JuMa automatically provisions a new AWS account for the workspace. This ensures full isolation between the workspaces following the federated model account structure mentioned in SageMaker Studio Administration Best Practices.
2. JuMa configures a workspace (which is a Sagemaker domain) that only allows predefined Amazon SageMaker features required for experimentation and development, specific custom kernels, and lifecycle configurations. It also sets up the required subnets and security groups that ensure the notebooks run in a secure environment.
3. After the workspaces are provisioned, the authorized users log in to the JuMa portal and access the SageMaker Studio IDE within their workspace using a SageMaker pre-signed URL. Users can choose between opening a SageMaker Studio private space or a shared space. Shared spaces encourage collaboration between different members of a team that can work in parallel on the same notebooks, whereas private spaces allow for a development environment for solitary workloads.
4. Using the BMW data portal, users can request access to on-premises databases or data stored in BMW’s Cloud Data Hub, making it available in their workspace for development and experimentation, from data preparation and analysis to model training and validation.

After an AI model is developed and validated in JuMa, AI teams can use the MLOPs service of the BMW AI platform to deploy it to production quickly and effortlessly. This service provides users with a production-grade ML infrastructure and pipelines on AWS using SageMaker, which can be set up in minutes with just a few clicks. Users simply need to host their model on the provisioned infrastructure and customize the pipeline to meet their specific use case needs. In this way, the AI platform covers the entire AI lifecycle at BMW Group.

JuMa features

Following best practice architecting on AWS, the JuMa service was designed and implemented according to the AWS Well-Architected Framework. Architectural decisions of each Well-Architected pillar are described in detail in the following sections.

Security and compliance

To assure full isolation between the tenants, each workspace receives its own AWS account, where the authorized users can jointly collaborate on analytics tasks as well as on developing and experimenting with AI/ML models. The JuMa portal itself enforces isolation at runtime using policy-based isolation with AWS Identity and Access Management (IAM) and the JuMa user’s context. For more information about this strategy, refer to Run-time, policy-based isolation with IAM.

Data scientists can only access their domain through the BMW network via pre-signed URLs generated by the portal. Direct internet access is disabled within their domain. Their Sagemaker domain privileges are built using Amazon SageMaker Role Manager personas to ensure least privilege access to AWS services needed for the development such as SageMaker, Amazon Athena, Amazon Simple Storage Service (Amazon S3), and AWS Glue. This role implements ML guardrails (such as those described in Governance and control), including enforcement of ML training to occur in either Amazon Virtual Private Cloud (Amazon VPC) or without internet and allowing only the use of JuMa’s custom vetted and up-to-date SageMaker images.

Because JuMa is designed for development, experimentation, and ad-hoc analysis, it implements retention policies to remove data after 30 days. To access data whenever needed and store it for long term, JuMa seamlessly integrates with the BMW Cloud Data Hub and BMW on-premises databases.

Finally, JuMa supports multiple Regions to comply to special local legal situations which, for example, require it to process data locally to enable BMW’s data sovereignty.

Operational excellence

Both the JuMa platform backend and workspaces are implemented with AWS serverless and managed services. Using those services helps minimize the effort of the BMW platform team maintaining and operating the end-to-end solution, striving to be a no-ops service. Both the workspace and portal are monitored using Amazon CloudWatch logs, metrics, and alarms to check key performance indicators (KPIs) and proactively notify the platform team of any issues. Additionally, the AWS X-Ray distributed tracing system is used to trace requests throughout multiple components and annotate CloudWatch logs with workspace-relevant context.

All changes to the JuMa infrastructure are managed and implemented through automation using infrastructure as code (IaC). This helps reduce manual efforts and human errors, increase consistency, and ensure reproducible and version-controlled changes across both JuMa platform backend workspaces. Specifically, all workspaces are provisioned and updated through an onboarding process built on top of AWS Step Functions, AWS CodeBuild, and Terraform. Therefore, no manual configuration is required to onboard new workspaces to the JuMa platform.

Cost optimization

By using AWS serverless services, JuMa ensures on-demand scalability, pre-approved instance sizes, and a pay-as-you-go model for the resources used during the development and experimentation activities per the AI/ML teams’ needs. To further optimize costs, the JuMa platform monitors and identifies idle resources within SageMaker Studio and shuts them down automatically to prevent expenses for non-utilized resources.

Sustainability

JuMa replaces BMW’s two on-premises platforms for analytics and deep learning workloads that consume a considerable amount of electricity and produce CO2 emissions even when not in use. By migrating AI/ML workloads from on premises to AWS, BMW will slash its environmental impact by decommissioning the on-premises platforms.

Furthermore, the mechanism for auto shutdown of idle resources, data retention polices, and the workspace usage reports to its owners implemented in JuMa help further minimize the environmental footprint of running AI/ML workloads on AWS.

Performance efficiency

By using SageMaker Studio, BMW teams benefit from an easy adoption of the latest SageMaker features that can help accelerate their experimentation. For example, they can use Amazon SageMaker JumpStart capabilities to use the latest pre-trained, open source models. Additionally, it helps reduce AI/ML team efforts moving from experimentation to solution industrialization, because the development environment provides the same AWS core services but restricted to development capabilities.

Reliability

SageMaker Studio domains are deployed in a VPC-only mode to manage internet access and only allow access to intended AWS services. The network is deployed in two Availability Zones to protect against a single point of failure, achieving greater resiliency and availability of the platform to its users.

Changes to JuMa workspaces are automatically deployed and tested to development and integration environments, using IaC and CI/CD pipelines, before upgrading customer environments.

Finally, data stored in Amazon Elastic File System (Amazon EFS) for SageMaker Studio domains is kept after volumes are deleted for backup purposes.

Conclusion

In this post, we described how BMW Group in collaboration with AWS ProServe developed a fully managed AI platform service on AWS using SageMaker Studio and other AWS serverless and managed services.

With JuMa, BMW’s AI/ML teams are empowered to unlock new business value by accelerating experimentation as well as time-to-market for disruptive AI solutions. Furthermore, by migrating from its on-premises platform, BMW can reduce the overall operational efforts and costs while also increasing sustainability and the overall security posture.

To learn more about running your AI/ML experimentation and development workloads on AWS, visit Amazon SageMaker Studio.

About the Authors

Marc Neumann is the head of the central AI Platform at BMP Group. He is responsible for developing and implementing strategies to use AI technology for business value creation across the BMW Group. His primary goal is to ensure that the use of AI is sustainable and scalable, meaning it can be consistently applied across the organization to drive long-term growth and innovation. Through his leadership, Neumann aims to position the BMW Group as a leader in AI-driven innovation and value creation in the automotive industry and beyond.

Amor Steinberg is a Machine Learning Engineer at BMW Group and the service lead of Jupyter Managed, a new service that aims to provide a code-centric analytics and machine learning workbench for engineers and data scientists at the BMW Group. His past experience as a DevOps Engineer at financial institutions enabled him to gather a unique understanding of the challenges that faces banks in the European Union and keep the balance between striving for technological innovation, complying with laws and regulations, and maximizing security for customers.

Marinus Krommenhoek is a Senior Cloud Solution Architect and a Software Developer at BMW Group. He is enthusiastic about modernizing the IT landscape with state-of-the-art services that add high value and are easy to maintain and operate. Marinus is a big advocate of microservices, serverless architectures, and agile working. He has a record of working with distributed teams across the globe within large enterprises.

Nicolas Jacob Baer is a Principal Cloud Application Architect at AWS ProServe with a strong focus on data engineering and machine learning, based in Switzerland. He works closely with enterprise customers to design data platforms and build advanced analytics and ML use cases.

Joaquin Rinaudo is a Principal Security Architect at AWS ProServe. He is passionate about building solutions that help developers improve their software quality. Prior to AWS, he worked across multiple domains in the security industry, from mobile security to cloud and compliance-related topics. In his free time, Joaquin enjoys spending time with family and reading science-fiction novels.

Shukhrat Khodjaev is a Senior Global Engagement Manager at AWS ProServe. He specializes in delivering impactful big data and AI/ML solutions that enable AWS customers to maximize their business value through data utilization.

Read More

In today’s ever-evolving world of ecommerce, the influence of a compelling product description cannot be overstated. It can be the decisive factor that turns a potential visitor into a paying customer or sends them clicking off to a competitor’s site. The manual creation of these descriptions across a vast array of products is a labor-intensive process, and it can slow down the velocity of new innovation. This is where Amazon Bedrock with its generative AI capabilities steps in to reshape the game. In this post, we dive into how Amazon Bedrock is transforming the product description generation process, empowering e-retailers to efficiently scale their businesses while conserving valuable time and resources.

Unlocking the power of generative AI in retail

Generative AI has captured the attention of boards and CEOs worldwide, prompting them to ask, “How can we leverage generative AI for our business?” One of the most promising applications of generative AI in ecommerce is using it to craft product descriptions. Retailers and brands have invested significant resources in testing and evaluating the most effective descriptions, and generative AI excels in this area.

Creating engaging and informative product descriptions for a vast catalog is a monumental task, especially for global ecommerce platforms. Manual translation and adaptation of product descriptions for each market consumes time and resources. This results in generic or incomplete descriptions, leading to reduced sales and customer satisfaction.

The power of Amazon Bedrock: AI-generated product descriptions

Amazon Bedrock is a fully managed service that simplifies generative AI development, offering high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon through a single API. It provides a comprehensive set of capabilities for building generative AI applications while ensuring privacy and security are maintained. With Amazon Bedrock, you can experiment with various FMs and customize them privately using techniques like fine-tuning and Retrieval Augmented Generation (RAG). The platform enables you to create managed agents for complex business tasks without the need for coding, such as booking travel, processing insurance claims, creating ad campaigns, and managing inventory.

For example, ecommerce platforms can initially generate basic product descriptions that include size, color, and price. However, Amazon Bedrock’s flexibility allows these descriptions to be fine-tuned to incorporate customer reviews, integrate brand-specific language, and highlight specific product features, resulting in tailored descriptions that resonate with the target audience. Moreover, Amazon Bedrock offers access to foundation models from Amazon and leading AI startups through an intuitive API, making the entire process seamless and efficient.

Using AI can have the following impact on the product description process:

• Faster approvals – Vendors experience a streamlined process, moving from product listing to approval in under an hour, eliminating frustrating delays
• Improved product listing velocity – When automated, your ecommerce marketplace sees a surge in product listings, offering consumers access to the latest merchandise nearly instantaneously
• Future-proofing – By embracing cutting-edge AI, you secure your position as a forward-looking platform ready to meet evolving market demands
• Innovation – This solution liberates teams from mundane tasks, allowing them to focus on higher-value work and fostering a culture of innovation

Solution overview

Before we dive into the technical details, let’s see the high-level preview of what this solution offers. This solution will allow you to create and manage product descriptions for your ecommerce platform. It empowers your platform to:

• Generate descriptions from text – With the power of generative AI, Amazon Bedrock can convert plain text descriptions into vivid, informative, and captivating product descriptions.
• Craft images – Beyond text, it can also craft images that align perfectly with the product descriptions, enhancing the visual appeal of your listings.
• Enhance existing content – Do you have existing product descriptions that need a fresh perspective? Amazon Bedrock can take your current content and make it even more compelling and engaging.

This solution is available in the AWS Solutions Library. We’ve provided detailed instructions in the accompanying README file. The README file contains all the information you need to get started, from requirements to deployment guidelines.

The system architecture comprises several core components:

• UI portal – This is the user interface (UI) designed for vendors to upload product images.
• Amazon Rekognition – Amazon Rekognition is an image analysis service that detects objects, text, and labels in images.
• Amazon Bedrock – Foundation models in Amazon Bedrock use the labels detected by Amazon Rekognition to generate product descriptions.
• AWS Lambda – AWS Lambda provides serverless compute for processing.
• Product database – The central repository stores vendor products, images, labels, and generated descriptions. This could be any database of your choice. Note that in this solution, all of the storage is in the UI.
• Admin portal – This portal provides oversight of the system and product listings, ensuring smooth operation. This is not part of the solution; we’ve added it for understanding.

The following diagram illustrates the flow of data and interactions within the system

The workflow includes the following steps:

1. The client initiates a request to the Amazon API Gateway REST API.
2. Amazon API Gateway passes the request to AWS Lambda through a proxy integration.
3. When operating on product image inputs, AWS Lambda calls Amazon Rekognition to detect objects in the image.
4. AWS Lambda calls LLMs hosted by Amazon Bedrock, such as the Amazon Titan language models, to generate product descriptions.
5. The response is passed back from AWS Lambda to Amazon API Gateway.
6. Finally, HTTP response from Amazon API Gateway is returned to the client.

Example use case

Imagine a vendor uploads a product image of shoes, and Amazon Rekognition identifies key attributes like “white shoes,” “sneaker,” and “durable.” The Amazon Bedrock Titan AI takes this information and generates a product description like, “Here is a draft product description for a canvas running shoe based on the product photo: Introducing the Canvas Runner, the perfect lightweight sneaker for your active lifestyle. This running shoe features a breathable canvas upper with leather accents for a stylish, classic look. The lace-up design provides a secure fit, while the padded tongue and collar add comfort. Inside, a removable cushioned insole supports and comforts your feet. The EVA midsole absorbs shock with each step, reducing fatigue. Flex grooves in the rubber outsole ensure flexibility and traction. With its simple, retro-inspired style, the Canvas Runner seamlessly transitions from workouts to everyday wear. Whether you’re running errands or running miles, this versatile sneaker will keep you moving in comfort and style.”

Design details

Let’s explore the components in more detail:

• User interface:
  • Front end – The front end of the vendor portal allows vendors to upload product images and displays product listings.
  • API calls – The portal communicates with the backend through APIs to process images and generate descriptions.
• Amazon Rekognition:
  • Image analysis – Triggered by API calls, Amazon Rekognition analyzes images and detects objects, text, and labels.
  • Label output – It outputs label data derived from the analysis.
• Amazon Bedrock:
  • NLP text generation – Amazon Bedrock uses the Amazon Titan natural language processing (NLP) model to generate textual descriptions.
  • Label integration – It takes the labels detected by Amazon Rekognition as input to generate product descriptions.
  • Style matching – Amazon Bedrock provides fine-tuning capabilities for Amazon Titan models to ensure that the generated descriptions match the style of the platform.
• AWS Lambda:
  • Processing – Lambda handles the API calls to services.
• Product database:
  • Flexible database – The product database is chosen based on customer preferences and requirements. Note this is not provided as part of the solution.

Additional capabilities

This solution goes beyond just generating product descriptions. It offers two more incredible options:

• Image and description generation from text – With the power of generative AI, Amazon Bedrock can take text descriptions and create corresponding images along with detailed product descriptions. Consider the potential:
  • Instantly visualizing products from text.
  • Automating image creation for large catalogs.
  • Enhancing customer experience with rich visuals.
  • Reducing content creation time and costs.
• Description enhancement – If you already have existing product descriptions, Amazon Bedrock can enhance them. Simply supply the text and the prompt, and Amazon Bedrock will skillfully enhance and enrich the content, rendering it highly captivating and engaging for your customers.

Conclusion

In the fiercely competitive world of ecommerce, staying at the forefront of innovation is imperative. Amazon Bedrock offers a transformative capability for e-retailers looking to enhance their product content, optimize their listing process, and drive sales. With the power of AI-generated product descriptions, businesses can create compelling, informative, and culturally relevant content that resonates deeply with customers. The future of ecommerce has arrived, and it’s driven by machine learning with Amazon Bedrock.

Are you ready to unlock the full potential of AI-powered product descriptions? Take the next step in revolutionizing your ecommerce platform. Visit the AWS Solutions Library and explore how Amazon Bedrock can transform your product descriptions, streamline your processes, and boost your sales. It’s time to supercharge your ecommerce with Amazon Bedrock!

About the Authors

Dhaval Shah is a Senior Solutions Architect at AWS, specializing in Machine Learning. With a strong focus on digital native businesses, he empowers customers to leverage AWS and drive their business growth. As an ML enthusiast, Dhaval is driven by his passion for creating impactful solutions that bring positive change. In his leisure time, he indulges in his love for travel and cherishes quality moments with his family.

Doug Tiffan is the Head of World Wide Solution Strategy for Fashion & Apparel at AWS. In his role, Doug works with Fashion & Apparel executives to understand their goals and align with them on the best solutions. Doug has over 30 years of experience in retail, holding several merchandising and technology leadership roles. Doug holds a BBA from Texas A&M University and is based in Houston, Texas.

Nikhil Sharma is a Solutions Architecture Leader at Amazon Web Services (AWS) where he and his team of Solutions Architects help AWS customers solve critical business challenges using AWS cloud technologies and services.

Kevin Bell is a Sr. Solutions Architect at AWS based in Seattle. He has been building things in the cloud for about 10 years. You can find him online as @bellkev on GitHub.

Nipun Chagari is a Principal Solutions Architect based in the Bay Area, CA. Nipun is passionate about helping customers adopt Serverless technology to modernize applications and achieve their business objectives. His recent focus has been on assisting organizations in adopting modern technologies to enable digital transformation. Apart from work, Nipun finds joy in playing volleyball, cooking and traveling with his family.

Marshall Bunch is a Solutions Architect at AWS helping North American customers design secure, scalable and cost-effective workloads in the cloud. His passion lies in solving age-old business problems where data and the newest technologies enable novel solutions. Beyond his professional pursuits, Marshall enjoys hiking and camping in Colorado’s beautiful Rocky Mountains.

Altaaf Dawoodjee is a Solutions Architect Leader that supports AdTech customers in the Digital Native Business (DNB) segment at Amazon Web Service (AWS). He has over 20 years of experience in Technology and has deep expertise in Analytics. He is passionate about helping drive successful business outcomes for his customers leveraging the AWS cloud.

Scott Bell is a dynamic leader and innovator with 25+ years of technology management experience. He is passionate about leading and developing teams in providing technology to meet the challenges of global users and businesses. He has extensive experience in leading technology teams which provide global technology solutions supporting 35+ languages. He is also passionate about the way the AI and Generative AI transform businesses and the way they support customer’s current unmet needs.

Sachin Shetti is a Principal Customer Solution Manager at AWS. He is passionate about helping enterprises succeed and realize significant benefits from cloud adoption, driving everything from basic migration to large-scale cloud transformation across people, processes, and technology. Prior to joining AWS, Sachin worked as a software developer for over 12 years and held multiple senior leadership positions leading technology delivery and transformation in healthcare, financial services, retail, and insurance. He has an Executive MBA and a Bachelor’s degree in Mechanical Engineering.

Read More

Amazon SageMaker Canvas is a rich, no-code Machine Learning (ML) and Generative AI workspace that has allowed customers all over the world to more easily adopt ML technologies to solve old and new challenges thanks to its visual, no-code interface. It does so by covering the ML workflow end-to-end: whether you’re looking for powerful data preparation and AutoML, managed endpoint deployment, simplified MLOps capabilities, and ready-to-use models powered by AWS AI services and Generative AI, SageMaker Canvas can help you to achieve your goals.

As companies of all sizes adopt SageMaker Canvas, customers asked for ways to optimize cost. As defined in the AWS Well-Architected Framework, a cost-optimized workload fully uses all resources, meets your functional requirements, and achieves an outcome at the lowest possible price point.

Today, we’re introducing a new way to further optimize costs for SageMaker Canvas applications. SageMaker Canvas now collects Amazon CloudWatch metrics that provide insight into app usage and idleness. Customers can use this information to shut down automatically idle SageMaker Canvas applications to avoiding incurring unintended costs.

In this post, we’ll show you how to automatically shut down idle SageMaker Canvas apps to control costs by using a simple serverless architecture. Templates used in this post are available in GitHub.

Understanding and tracking costs

Education is always the first step into understanding and controlling costs for any workload, either on-premises or in the cloud. Let’s start by reviewing the SageMaker Canvas pricing model. In a nutshell, SageMaker Canvas has a pay-as-you-go pricing model, based on two dimensions:

• Workspace instance: ­ formerly known as session time, is the cost associated with running the SageMaker Canvas app
• AWS service charges: ­ costs associated with training the models, deploying the endpoints, generating inferences (resources to spin up SageMaker Canvas).

Customers always have full control over the resources that are launched by SageMaker Canvas and can keep track of costs associated with the SageMaker Canvas app by using the AWS Billing and Cost Management service. For more information, refer to Manage billing and cost in SageMaker Canvas.

To limit the cost associated with the workspace instances, as a best practice, you must log out, do not close the browser tab. To log out, choose the Log out button on the left panel of the SageMaker Canvas app.

Automatically shutting down SageMaker Canvas applications

For IT Administrators that are looking to provide automated controls for shutting down SageMaker Canvas applications and keeping costs under control, there are two approaches:

1. Shutdown applications on a schedule (every day at 19:00 or every Friday at 18:00)
2. Shutdown automatically idle applications (when the application hasn’t been used for two hours)

Shutdown applications on a schedule

Scheduled shutdown of SageMaker Canvas applications can be achieved with very little effort by using a cron expression (with Amazon EventBridge Cron Rule), a compute component (an AWS Lambda function) that calls the Amazon SageMaker API DeleteApp. This approach has been discussed in the Provision and manage ML environments with Amazon SageMaker Canvas using AWS CDK and AWS Service Catalog post, and implemented in the associated GitHub repository.

One of the advantages of the above architecture is that it is very simple to duplicate it to achieve scheduled creation of the SageMaker Canvas app. By using a combination of scheduled creation and scheduled deletion, a cloud administrator can make sure that the SageMaker Canvas application is ready to be used whenever users start their business day (e.g. 9AM on a work day), and that the app also automatically shuts down at the end of the business day (e.g. 7PM on a work day, always shut down during weekends). All that is needed to do is change the line of code calling the DeleteApp API into CreateApp, as well as updating the cron expression to reflect the desired app creation time.

While this approach is very easy to implement and test, a drawback of the suggested architecture is that it does not take into account whether an application is currently being used or not, shutting it down regardless of its current activity status. According to different situations, this might cause friction with active users, which might suddenly see their session terminated.

You can retrieve the template associated to this architecture from the following GitHub repository:

Shutdown automatically idle applications

Starting today, Amazon SageMaker Canvas emits CloudWatch metrics that provide insight into app usage and idleness. This allows an administrator to define a solution that reads the idleness metric, compares it against a threshold, and defines a specific logic for automatic shutdown. A more detailed overview of the idleness metric emitted by SageMaker Canvas is shown in the following paragraph.

To achieve automatic shutdown of SageMaker Canvas applications based on the idleness metrics, we provide an AWS CloudFormation template. This template consists of three main components:

1. An Amazon CloudWatch Alarm, which runs a query to check the MAX value of the TimeSinceLastActive metric. If this value is greater than a threshold provided as input to the CloudFormation template, it triggers the rest of the automation. This query can be run on a single user profile, on a single domain, or across all domains. According to the level of control that you wish to have, you can use:
   1. the all-domains-all-users template, which checks this across all users and all domains in the region where the template is deployed
   2. the one-domain-all-users template, which checks this across all users in one domain in the region where the template is deployed
   3. the one-domain-one-user template, which checks this for one user profile, in one domain, in the region where the template is deployed
2. The alarm state change creates an event on the default event bus in Amazon EventBridge, which has an Amazon EventBridge Rule set up to trigger an AWS Lambda function
3. The AWS Lambda function identifies which SageMaker Canvas app has been running in idle for more than the specified threshold, and deletes it with the DeleteApp API.

You can retrieve the AWS CloudFormation templates associated to this architecture from the following GitHub repository:

How SageMaker Canvas idleness metric work

SageMaker Canvas emits a TimeSinceLastActive metric in the /aws/sagemaker/Canvas/AppActivity namespace, which shows the number of seconds that the app has been idle with no user activity. We can use this new metric to trigger an automatic shutdown of the SageMaker Canvas app when it has been idle for a defined period. SageMaker Canvas exposes the TimeSinceLastActive with the following schema:

{
    "Namespace": "/aws/sagemaker/Canvas/AppActivity",
    "Dimensions": [
        [
            "DomainId",
            "UserProfileName"
        ]
    ],
    "Metrics": [
        {
            "Name": "TimeSinceLastActive",
            "Unit": "Seconds",
            "Value": 12345
        }
    ]
}

The key components of this metric are as follows:

• Dimensions, in particular DomainID and UserProfileName, that allow an administrator to pinpoint which applications are idle across all domains and users
• Value of the metric, which indicates the number of seconds since the last activity in the SageMaker Canvas applications. SageMaker Canvas considers the following as activity:
  • Any action taken in the SageMaker Canvas application (clicking a button, transforming a dataset, generating an in-app inference, deploying a model);
  • Using a ready-to-use model or interacting with the Generative AI models using chat interface;
  • A batch inference scheduled to run at a specific time; for more information, refer to  Manage automations.

This metric can be read via Amazon CloudWatch API such as get_metric_data. For example, using the AWS SDK for Python (boto3):

import boto3, datetime

cw = boto3.client('cloudwatch')
metric_data_results = cw.get_metric_data(
    MetricDataQueries=[
        {
            "Id": "q1",
            "Expression": 'SELECT MAX(TimeSinceLastActive) FROM "/aws/sagemaker/Canvas/AppActivity" GROUP BY DomainId, UserProfileName',
            "Period": 900
        }
    ],
    StartTime=datetime.datetime(2023, 1, 1),
    EndTime=datetime.datetime.now(),
    ScanBy='TimestampAscending'
)

The Python query extracts the MAX value of TimeSinceLastActive from the namespace associated to SageMaker Canvas after grouping these values by DomainID and UserProfileName.

Deploying and testing the auto-shutdown solution

To deploy the auto-shutdown stack, do the following:

1. Download the AWS CloudFormation template that refers to the solution you want to implement from the above GitHub repository. Choose whether you want to implement a solution for all SageMaker Domains, for a single SageMaker Domain, or for a single user;
2. Update template parameters:
   1. The idle timeout – time (in seconds) that the SageMaker Canvas app is allowed to stay in idle before it gets shutdown; default value is 2 hours
   2. The alarm period – aggregation time (in seconds) used by CloudWatch Alarm to compute the idle timeout; default value is 20 minutes
   3. (optional) SageMaker Domain ID and user profile name
3. Deploy the CloudFormation stack to create the resources

Once deployed (should take less than two minutes), the AWS Lambda function and Amazon CloudWatch alarm are configured to automatically shut down the Canvas app when idle. To test the auto-shutdown script, do the following:

1. Make sure that the SageMaker Canvas app is running within the right domain and with the right user profile (if you have configured them).
2. Stop using the SageMaker Canvas app and wait for the idle timeout period (default, 2 hours)
3. Check that the app is stopped after being idle for the threshold time by checking that the CloudWatch alarm has been triggered and, after triggering the automation, it has gone back to the normal state.

In our test, we have set the idle timeout period to two hours (7200 seconds). In the following graph plotted by Amazon CloudWatch Metrics, you can see that the SageMaker Canvas app has been emitting the TimeSinceLastActive metric until the threshold was met (1), which triggered the alarm. Once the alarm was triggered, the AWS Lambda function was executed, which deleted the app and brought the metric back below the threshold (2).

Conclusion

In this post, we implemented an automated shutdown solution for idle SageMaker Canvas apps using AWS Lambda and CloudWatch Alarm and the newly emitted metric of idleness from SageMaker Canvas. Thanks to this solution, customers not only can optimize costs for their ML workloads but can also avoid unintended charges for applications that they forgot were running in their SageMaker Domain.

We’re looking forward to seeing what new use cases and workloads customers can solve with the peace of mind brought by this solution. For more examples of how SageMaker Canvas can help you achieve your business goals, refer to the following posts:

• Predict customer churn with no-code machine learning using Amazon SageMaker Canvas
• Publish predictive dashboards in Amazon QuickSight using ML predictions from Amazon SageMaker Canvas
• Democratize computer vision defect detection for manufacturing quality using no-code machine learning with Amazon SageMaker Canvas
• Use no-code machine learning to derive insights from product reviews using Amazon SageMaker Canvas sentiment analysis and text analysis models
• Empower your business users to extract insights from company documents using Amazon SageMaker Canvas Generative AI

To learn how you can run production-level workloads with Amazon SageMaker Canvas, refer to the following posts:

• Build, Share, Deploy: how business analysts and data scientists achieve faster time-to-market using no-code ML and Amazon SageMaker Canvas
• Retrain ML models and automate batch predictions in Amazon SageMaker Canvas using updated datasets
• Operationalize ML models built in Amazon SageMaker Canvas to production using the Amazon SageMaker Model Registry
• Provision and manage ML environments with Amazon SageMaker Canvas using AWS CDK and AWS Service Catalog

About the authors

Davide Gallitelli is a Senior Specialist Solutions Architect for AI/ML. He is based in Brussels and works closely with customers all around the globe that are looking to adopt Low-Code/No-Code Machine Learning technologies, and Generative AI. He has been a developer since he was very young, starting to code at the age of 7. He started learning AI/ML at university, and has fallen in love with it since then.

Huong Nguyen is a Sr. Product Manager at AWS. She is leading the data ecosystem integration for SageMaker, with 14 years of experience building customer-centric and data-driven products for both enterprise and consumer spaces.

Gunjan Garg is a Principal Engineer at Amazon SageMaker team in AWS, providing technical leadership for the product. She has worked in several roles in the AI/ML org for last 5 years and is currently focused on Amazon SageMaker Canvas.

Ziyao Huang is a Software Development Engineer with Amazon SageMaker Data Wrangler. He is passionate about building great product that makes ML easy for the customers. Outside of work, Ziyao likes to read, and hang out with his friends.

Read More

This post was co-written with Greg Benson, Chief Scientist; Aaron Kesler, Sr. Product Manager; and Rich Dill, Enterprise Solutions Architect from SnapLogic.

Many customers are building generative AI apps on Amazon Bedrock and Amazon CodeWhisperer to create code artifacts based on natural language. This use case highlights how large language models (LLMs) are able to become a translator between human languages (English, Spanish, Arabic, and more) and machine interpretable languages (Python, Java, Scala, SQL, and so on) along with sophisticated internal reasoning. This emergent ability in LLMs has compelled software developers to use LLMs as an automation and UX enhancement tool that transforms natural language to a domain-specific language (DSL): system instructions, API requests, code artifacts, and more. In this post, we show you how SnapLogic, an AWS customer, used Amazon Bedrock to power their SnapGPT product through automated creation of these complex DSL artifacts from human language.

When customers create DSL objects from LLMs, the resulting DSL is either an exact replica or a derivative of an existing interface data and schema that forms the contract between the UI and the business logic in the backing service. This pattern is particularly trending with independent software vendors (ISVs) and software as a service (SaaS) ISVs due to their unique way of representing configurations through code and the desire to simplify the user experience for their customers. Example use cases include:

• Converting natural language to tool-specific visuals and calculation expressions, as seen with the recently announced generative BI capabilities in Amazon QuickSight
• Generating the correct API call based on individual user intent and context across best-of-breed productivity SaaS apps with the upcoming productivity features in AWS AppFabric
• Creating elaborate integration pipelines with simple human prompts, as you will read more about in this post

The most straightforward way to build and scale text-to-pipeline applications with LLMs on AWS is using Amazon Bedrock. Amazon Bedrock is the easiest way to build and scale generative AI applications with foundation models (FMs). It is a fully managed service that offers access to a choice of high-performing foundation FMs from leading AI via a single API, along with a broad set of capabilities you need to build generative AI applications with privacy and security. Anthropic, an AI safety and research lab that builds reliable, interpretable, and steerable AI systems, is one of the leading AI companies that offers access to their state-of-the art LLM, Claude, on Amazon Bedrock. Claude is an LLM that excels at a wide range of tasks, from thoughtful dialogue, content creation, complex reasoning, creativity, and coding. Anthropic offers both Claude and Claude Instant models, all of which are available through Amazon Bedrock. Claude has quickly gained popularity in these text-to-pipeline applications because of its improved reasoning ability, which allows it to excel in ambiguous technical problem solving. Claude 2 on Amazon Bedrock supports a 100,000-token context window, which is equivalent to about 200 pages of English text. This is a particularly important feature that you can rely on when building text-to-pipeline applications that require complex reasoning, detailed instructions, and comprehensive examples.

SnapLogic background

SnapLogic is an AWS customer on a mission to bring enterprise automation to the world. The SnapLogic Intelligent Integration Platform (IIP) enables organizations to realize enterprise-wide automation by connecting their entire ecosystem of applications, databases, big data, machines and devices, APIs, and more with pre-built, intelligent connectors called Snaps. SnapLogic recently released a feature called SnapGPT, which provides a text interface where you can type the desired integration pipeline you want to create in simple human language. SnapGPT uses Anthropic’s Claude model through Amazon Bedrock to automate the creation of these integration pipelines as code, which are then used through SnapLogic’s flagship integration solution. However, SnapLogic’s journey to SnapGPT has been a culmination of many years operating in the AI space.

SnapLogic’s AI journey

In the realm of integration platforms, SnapLogic has consistently been at the forefront, harnessing the transformative power of artificial intelligence. Over the years, the company’s commitment to innovating with AI has become evident, especially when we trace the journey from Iris to AutoLink.

The humble beginnings with Iris

In 2017, SnapLogic unveiled Iris, an industry-first AI-powered integration assistant. Iris was designed to use machine learning (ML) algorithms to predict the next steps in building a data pipeline. By analyzing millions of metadata elements and data flows, Iris could make intelligent suggestions to users, democratizing data integration and allowing even those without a deep technical background to create complex workflows.

AutoLink: Building momentum

Building on the success and learnings from Iris, SnapLogic introduced AutoLink, a feature aimed at further simplifying the data mapping process. The tedious task of manually mapping fields between source and target systems became a breeze with AutoLink. Using AI, AutoLink automatically identified and suggested potential matches. Integrations that once took hours could be run in mere minutes.

The generative leap with SnapGPT

SnapLogic’s latest foray in AI brings us SnapGPT, which aims to revolutionize integration even further. With SnapGPT, SnapLogic introduces the world’s first generative integration solution. This is not just about simplifying existing processes, but entirely reimagining how integrations are designed. The power of generative AI can create entire integration pipelines from scratch, optimizing the workflow based on the desired outcome and data characteristics.

SnapGPT is extremely impactful to SnapLogic’s customers because they are able to drastically decrease the amount of time required to generate their first SnapLogic pipeline. Traditionally, SnapLogic customers would need to spend days or weeks configuring integration pipelines from scratch. Now, these customers are able to simply ask SnapGPT to, for example, “create a pipeline which will move all of my active SFDC customers to WorkDay.” A working first draft of a pipeline is automatically created for this customer, drastically cutting down the development time required for creation of the base of their integration pipeline. This allows the end customer to spend more time focusing on what has true business impact to them instead of working on configurations of an integration pipeline. The following example shows how a SnapLogic customer can enter a description into the SnapGPT feature to quickly generate a pipeline, using natural language.

AWS and SnapLogic have collaborated closely throughout this product build and have learned a lot along the way. The rest of this post will focus on the technical learnings AWS and SnapLogic have had around using LLMs for text-to-pipeline applications.

Solution overview

To solve this text-to-pipeline problem, AWS and SnapLogic designed a comprehensive solution shown in the following architecture.

A request to SnapGPT goes through the following workflow:

1. A user submits a description to the application.
2. SnapLogic uses a Retrieval Augmented Generation (RAG) approach to retrieve relevant examples of SnapLogic pipelines that are similar to the user’s request.
3. These extracted relevant examples are combined with the user input and go through some text preprocessing before they’re sent to Claude on Amazon Bedrock.
4. Claude produces a JSON artifact that represents a SnapLogic pipeline.
5. The JSON artifact is directly integrated to the core SnapLogic integration platform.
6. The SnapLogic pipeline is rendered to the user in a visual friendly manner.

Through various experimentation between AWS and SnapLogic, we have found the prompt engineering step of the solution diagram to be extremely important to generating high-quality outputs for these text-to-pipeline outputs. The next section goes further into some specific techniques used with Claude in this space.

Prompt experimentation

Throughout the development phase of SnapGPT, AWS and SnapLogic found that rapid iteration on prompts being sent to Claude was a critical development task to improving the accuracy and relevancy of text-to-pipeline outputs in SnapLogic’s outputs. By using Amazon SageMaker Studio interactive notebooks, the AWS and SnapLogic team were able to quickly work through different versions of prompts by using the Boto3 SDK connection to Amazon Bedrock. Notebook-based development allowed the teams to quickly create client-side connections to Amazon Bedrock, include text-based descriptions alongside Python code for sending prompts to Amazon Bedrock, and hold joint prompt engineering sessions where iterations were made quickly between multiple personas.

Anthropic Claude prompt engineering methods

In this section, we describe some of the iterative techniques we used to create a high-performing prompt based on an illustrative user request: “Make a pipeline which uses the ExampleCompany database which retrieves all active customers.” Note that this example is not the schema that SnapGPT is powered by, and is only used to illustrate a text-to-pipeline application.

To baseline our prompt engineering, we use the following original prompt:

Make a pipeline which uses the ExampleCompany database which retrieves all active customers

The expected output is as follows:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status = 'active'"
}

Improvement #1: Using the Human and Assistant annotations

Claude’s training procedure teaches the FM to understand dialogue between a human and an assistant in its prompt structure. Claude users can take advantage of this structure by ending their prompt in Assistant:, which will trigger Claude to start generating the response to a query based on what the human has said. Note that because Claude will continue to generate text until it is told to stop, make sure you use a stop sequence of nnHuman: in the API request to Amazon Bedrock when using Claude. This will ensure you only receive the response from the assistant and not the predicted next response from the human in the dialogue chain. These nnHuman and nnAssistant indicators not only help in limiting Claude’s response to relevant information, but also help in providing clear demarcation between primary human inputs and assistant responses from other interactions. This especially helps in a few-shot prompt setting where you can have multiple queries and responses with queries tagged with nnHuman and previous responses tagged with nnAssistant. Let’s look at the improved prompt and associated illustrative API call to Amazon Bedrock via the Boto3 SDK.

We use the following prompt:

Human: Make a pipeline which uses the ExampleCompany database which retrieves all active customers

Assistant:

The output is as follows:

Here is a MongoDB pipeline to retrieve all active customers from the ExampleCompany database:

pipeline = [
    {
        '$match': {
            'database': 'ExampleCompany', 
            'status': 'active'
        }
    }
]
db.customers.aggregate(pipeline)

The key steps are:

– `$match` to filter the documents to only those in the ExampleCompany database and with status equal to “active”.

– `db.customers.aggregate()` runs the pipeline on the customers collection and returns the filtered results.

So this will return all customer documents from the ExampleCompany database where the status field is set to “active”.

Notice how in the output, Claude makes up an answer to our question without any context. The answer is potentially helpful, but it’s not specific to the problem we’re trying to solve because it is not provided with enough context. It’s also important to note that Anthropic requires this human and assistant format when using the Claude models; if you don’t include this step, Amazon Bedrock will return an error.

Improvement #2: Include hyper-specific context for Claude before it’s told what to do

Claude performs best when it’s given good context about what it needs to accomplish as well as how it is supposed to accomplish that task. Here are a few examples of things to include as context for Claude before you tell it the task:

• Describe the job function that you want Claude to fulfill. By doing this, you are making Claude think in the role of someone working in that job, or more like someone just being hired in that specific job role. This will help Claude formulate the answer from that specific job role’s perspective, reducing hallucination and randomness in the answer.
• Include technical details regarding the output format. Foundation models are generally good at following actions, so asking Claude to generate the output in a specific format or schema with one or a few examples will help Claude provide the answer in the right format, reducing the need for filtering and postprocessing of the output. The example in the following prompt shows a fictitious schema that matches the expected output.

Let’s combine these suggestions to improve upon our original prompt:

Human: Your job is to act as an expert on ETL pipelines. Specifically, your job is to create a JSON representation of an ETL pipeline which will solve the user request provided to you.

The JSON output should follow the following format:

{
  "database": "the name of the database you will be using here",
  "query": "place a sql representation for solving the task at hand here"
}

Here is your task: make a pipeline which uses the ExampleCompany database which retrieves all active customers

Assistant:

The output is as follows:

Here is a JSON representation of an ETL pipeline to retrieve all active customers from the ExampleCompany database:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM customers WHERE status = 'active'"
}

This pipeline uses the ExampleCompany database and a SQL query to select all records from the customers table where the status is ‘active’.

With the addition of context, the model is now able to get most of the way to our answer. It now knows how to structure the output and it understands that it needs to write a SQL statement to solve the task. However, you will notice that the model doesn’t correctly use the ec_prod.customers nomenclature we are expecting. Room for improvement! Also note, in practice, you will need to include significantly more context than what we provided in this example for high-quality results in text-to-pipeline use cases.

Improvement #3: Enclosing examples in XML tags

Few-shot learning is a popular way to increase the accuracy with FMs where you have relevant examples to guide the FM in their task. This is particularly important when working in text-to-pipeline applications because these hyper-specific pipeline outputs are usually not included in any public dataset because they are specific to an individual customer’s data representation. When using Claude, you can take advantage of its innate knowledge of arbitrary XML tags in order to provide these examples in a few-shot format. In the following improved prompt, notice the addition of examples enclosed in <example> XML tags. Also, you can use abbreviations for human and assistant with H and A to provide context to Claude without making it think these were specific pieces of the dialogue that have already occurred.

We use the following prompt:

Human: Your job is to act as an expert on ETL pipelines. Specifically, your job is to create a JSON representation of an ETL pipeline which will solve the user request provided to you.

The JSON output should follow the following format:

{
  "database": "the name of the database you will be using here"
  "query": "place a sql representation for solving the task at hand here"
}

Here are some examples of user requests and associated output JSON artifacts which correctly solve the task provided. Make sure to use the same nomenclature which is provided in the examples if you are using the same databases.

<example-1>

H: query the ExampleCompany database for any active orders for customer #1234

A:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.orders WHERE customer_id = 1234 AND status  = 'active'"
}

The reasoning for this query is that the customer #1234 is filtered and we are searching the “orders” table for any existing records after the “active” filter.

</example-1>

<example-2>

H: remove all customers in the ExampleCompany database which have not been active in the last month

A:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status != 'active'"
}

The reasoning for this query is that the “ec_prod.customers” table is the only table in ExampleCompany database which contains customer records.

</example-2>

Here is your task: make a pipeline which uses the ExampleCompany database which retrieves all active customers.

Assistant:

The output is as follows:

Here is the JSON representation for an ETL pipeline to retrieve all active customers from the ExampleCompany database:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status = 'active'"
}

The reasoning is:

The ExampleCompany database is specified as the database to use

The ec_prod.customers table contains the customer data

The status field indicates if a customer is active so the query filters for only active customers

This should retrieve all active customer records from the ExampleCompany database. Let me know if you need any clarification or have additional requirements for the ETL pipeline!

The JSON output now has the correct database nomenclature, which Claude was able to use because of the examples that we provided. Coupling human and assistant indicators with XML tags makes it easier for Claude to understand what is the task and what was the answer in similar examples with very clear separation between multiple few-shot examples. The easier it is for Claude to understand, the better and more relevant the answer will be, further reducing the chance for the model to hallucinate and provide random irrelevant answers.

Improvement #4: Triggering Claude to begin JSON generation with XML tags

A small challenge with text-to-pipeline applications using FMs is the need to exactly parse an output from resulting text so it can be interpreted as code in a downstream application. One way to solve this with Claude is to take advantage of its XML tag understanding and combine this with a custom stop sequence. In the following prompt, we have instructed Claude to enclose the output in <json></json> XML tags. Then, we have added the <json> tag to the end of the prompt. This ensures that the first text that comes out of Claude will be the start of the JSON output. If you don’t do this, Claude often responds with some conversational text, then the true code response. By instructing Claude to immediately start generating the output, you can easily stop generation when you see the closing </json> tag. This is shown in the updated Boto3 API call. The benefits of this technique are twofold. First, you are able to exactly parse the code response from Claude. Second, you are able to reduce cost because Claude only generates code outputs and no extra text. This reduces cost on Amazon Bedrock because you are charged for each token that is produced as output from all FMs.

We use the following prompt:

Human: Your job is to act as an expert on ETL pipelines. Specifically, your job is to create a JSON representation of an ETL pipeline which will solve the user request provided to you.

The JSON output should follow the following format:

{
    "database": "the name of the database you will be using here"
    "query": "place a sql representation for solving the task at hand here"
}

Here are some examples of user requests and associated output JSON artifacts which correctly solve the task provided. Make sure to use the same nomenclature which is provided in the examples if you are using the same databases.

<example-1>

H: query the ExampleCompany database for any active orders for customer #1234

A:

<json>

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.orders WHERE customer_id = 1234 AND status  = 'active'"
}

</json>

The reasoning for this query is that the customer #1234 is filtered and we are searching the “orders” table for any existing records after the “active” filter.

</example-1>

<example-2>

H: remove all customers in the ExampleCompany database which have not been active in the last month

A:

<json>

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status != 'active'"
}

</json>

The reasoning for this query is that the “ec_prod.customers” table is the only table in ExampleCompany database which contains customer records.

</example-2>

Always remember to enclose your JSON outputs in <json></json> tags.

Here is your task: make a pipeline which uses the ExampleCompany database which retrieves all active customers.

Assistant:

<json>

We use the following code:

body = json.dumps({"prompt": prompt, "stop_sequences": ['nnHuman:', '</json>']})
response = bedrock.invoke_model(
   body=body,
   modelId='anthropic.claude-v2'
)

The output is as follows:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status = 'active'"
}

Now we have arrived at the expected output with only the JSON object returned! By using this method, we are able to generate an immediately usable technical artifact as well as reduce the cost of generation by reducing output tokens.

Conclusion

To get started today with SnapGPT, request a free trial of SnapLogic or request a demo of the product. If you would like to use these concepts for building applications today, we recommend experimenting hands-on with the prompt engineering section in this post, using the same flow on a different DSL generation use case that suits your business, and diving deeper into the RAG features that are available through Amazon Bedrock.

SnapLogic and AWS have been able to partner effectively to build an advanced translator between human language and the complex schema of SnapLogic integration pipelines powered by Amazon Bedrock. Throughout this journey, we have seen how the output generated with Claude can be improved in text-to-pipeline applications using specific prompt engineering techniques. AWS and SnapLogic are excited to continue this partnership in Generative AI and look forward to future collaboration and innovation in this fast-moving space.

About the Authors

Greg Benson is a Professor of Computer Science at the University of San Francisco and Chief Scientist at SnapLogic. He joined the USF Department of Computer Science in 1998 and has taught undergraduate and graduate courses including operating systems, computer architecture, programming languages, distributed systems, and introductory programming. Greg has published research in the areas of operating systems, parallel computing, and distributed systems. Since joining SnapLogic in 2010, Greg has helped design and implement several key platform features including cluster processing, big data processing, the cloud architecture, and machine learning. He currently is working on Generative AI for data integration.

Aaron Kesler is the Senior Product Manager for AI products and services at SnapLogic, Aaron applies over ten years of product management expertise to pioneer AI/ML product development and evangelize services across the organization. He is the author of the upcoming book “What’s Your Problem?” aimed at guiding new product managers through the product management career. His entrepreneurial journey began with his college startup, STAK, which was later acquired by Carvertise with Aaron contributing significantly to their recognition as Tech Startup of the Year 2015 in Delaware. Beyond his professional pursuits, Aaron finds joy in golfing with his father, exploring new cultures and foods on his travels, and practicing the ukulele.

Rich Dill is a Principal Solutions Architect with experience cutting broadly across multiple areas of specialization. A track record of success spanning multi-platform enterprise software and SaaS. Well known for turning customer advocacy (serving as the voice of the customer) into revenue-generating new features and products. Proven ability to drive cutting-edge products to market and projects to completion on schedule and under budget in fast-paced onshore and offshore environments. A simple way to describe me: the mind of a scientist, the heart of an explorer and the soul of an artist.

Clay Elmore is an AI/ML Specialist Solutions Architect at AWS. After spending many hours in a materials research lab, his background in chemical engineering was quickly left behind to pursue his interest in machine learning. He has worked on ML applications in many different industries ranging from energy trading to hospitality marketing. Clay’s current work at AWS centers around helping customers bring software development practices to ML and generative AI workloads, allowing customers to build repeatable, scalable solutions in these complex environments. In his spare time, Clay enjoys skiing, solving Rubik’s cubes, reading, and cooking.

Sina Sojoodi is a technology executive, systems engineer, product leader, ex-founder and startup advisor. He joined AWS in March 2021 as a Principal Solutions Architect. Sina is currently the US-West ISV area lead Solutions Architect. He works with SaaS and B2B software companies to build and grow their businesses on AWS. Previous to his role at Amazon, Sina was a technology executive at VMware, and Pivotal Software (IPO in 2018, VMware M&A in 2020) and served multiple leadership roles including founding engineer at Xtreme Labs (Pivotal acquisition in 2013). Sina has dedicated the past 15 years of his work experience to building software platforms and practices for enterprises, software businesses and the public sector. He is an industry leader with a passion for innovation. Sina holds a BA from the University of Waterloo where he studied Electrical Engineering and Psychology.

Sandeep Rohilla is a Senior Solutions Architect at AWS, supporting ISV customers in the US West region. He focuses on helping customers architect solutions leveraging containers and generative AI on the AWS cloud. Sandeep is passionate about understanding customers’ business problems and helping them achieve their goals through technology. He joined AWS after working more than a decade as a solutions architect, bringing his 17 years of experience to bear. Sandeep holds an MSc. in Software Engineering from the University of the West of England in Bristol, UK.

Dr. Farooq Sabir is a Senior Artificial Intelligence and Machine Learning Specialist Solutions Architect at AWS. He holds PhD and MS degrees in Electrical Engineering from the University of Texas at Austin and an MS in Computer Science from Georgia Institute of Technology. He has over 15 years of work experience and also likes to teach and mentor college students. At AWS, he helps customers formulate and solve their business problems in data science, machine learning, computer vision, artificial intelligence, numerical optimization, and related domains. Based in Dallas, Texas, he and his family love to travel and go on long road trips.

Read More

This is a guest post by A.K Roy from Qualcomm AI.

Amazon Elastic Compute Cloud (Amazon EC2) DL2q instances, powered by Qualcomm AI 100 Standard accelerators, can be used to cost-efficiently deploy deep learning (DL) workloads in the cloud. They can also be used to develop and validate performance and accuracy of DL workloads that will be deployed on Qualcomm devices. DL2q instances are the first instances to bring Qualcomm’s artificial intelligent (AI) technology to the cloud.

With eight Qualcomm AI 100 Standard accelerators and 128 GiB of total accelerator memory, customers can also use DL2q instances to run popular generative AI applications, such as content generation, text summarization, and virtual assistants, as well as classic AI applications for natural language processing and computer vision. Additionally, Qualcomm AI 100 accelerators feature the same AI technology used across smartphones, autonomous driving, personal computers, and extended reality headsets, so DL2q instances can be used to develop and validate these AI workloads before deployment.

New DL2q instance highlights

Each DL2q instance incorporates eight Qualcomm Cloud AI100 accelerators, with an aggregated performance of over 2.8 PetaOps of Int8 inference performance and 1.4 PetaFlops of FP16 inference performance. The instance has an aggregate 112 of AI cores, accelerator memory capacity of 128 GB and memory bandwidth of 1.1 TB per second.

Each DL2q instance has 96 vCPUs, a system memory capacity of 768 GB and supports a networking bandwidth of 100 Gbps as well as Amazon Elastic Block Store (Amazon EBS) storage of 19 Gbps.

Qualcomm Cloud AI100 accelerator innovation

The Cloud AI100 accelerator system-on-chip (SoC) is a purpose-built, scalable multi-core architecture, supporting a wide range of deep-learning use-cases spanning from the datacenter to the edge. The SoC employs scalar, vector, and tensor compute cores with an industry-leading on-die SRAM capacity of 126 MB. The cores are interconnected with a high-bandwidth low-latency network-on-chip (NoC) mesh.

The AI100 accelerator supports a broad and comprehensive range of models and use-cases. The table below highlights the range of the model support.

* Most automotive networks are composite networks consisting of a fusion of individual networks.

The large on-die SRAM on the DL2q accelerator enables efficient implementation of advanced performance techniques such as MX6 micro-exponent precision for storing the weights and MX9 micro-exponent precision for accelerator-to-accelerator communication. The micro-exponent technology is described in the following Open Compute Project (OCP) industry announcement: AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Standardize Next-Generation Narrow Precision Data Formats for AI » Open Compute Project.

The instance user can use the following strategy to maximize the performance-per-cost:

• Store weights using the MX6 micro-exponent precision in the on-accelerator DDR memory. Using the MX6 precision maximizes the utilization of the available memory capacity and the memory-bandwidth to deliver best-in-class throughput and latency.
• Compute in FP16 to deliver the required use case accuracy, while using the superior on-chip SRAM and spare TOPs on the card, to implement high-performance low-latency MX6 to FP16 kernels.
• Use an optimized batching strategy and a higher batch-size by using the large on-chip SRAM available to maximize the reuse of weights, while retaining the activations on-chip to the maximum possible.

DL2q AI Stack and toolchain

The DL2q instance is accompanied by the Qualcomm AI Stack that delivers a consistent developer experience across Qualcomm AI in the cloud and other Qualcomm products. The same Qualcomm AI stack and base AI technology runs on the DL2q instances and Qualcomm edge devices, providing customers a consistent developer experience, with a unified API across their cloud, automotive, personal computer, extended reality, and smartphone development environments.

The toolchain enables the instance user to quickly onboard a previously trained model, compile and optimize the model for the instance capabilities, and subsequently deploy the compiled models for production inference use-cases in three steps shown in the following figure.

To learn more about tuning the performance of a model, see the Cloud AI 100 Key Performance Parameters Documentation.

Get started with DL2q instances

In this example, you compile and deploy a pre-trained BERT model from Hugging Face on an EC2 DL2q instance using a pre-built available DL2q AMI, in four steps.

You can use either a pre-built Qualcomm DLAMI on the instance or start with an Amazon Linux2 AMI and build your own DL2q AMI with the Cloud AI 100 Platform and Apps SDK available in this Amazon Simple Storage Service (Amazon S3) bucket: s3://ec2-linux-qualcomm-ai100-sdks/latest/.

The steps that follow use the pre-built DL2q AMI, Qualcomm Base AL2 DLAMI.

Use SSH to access your DL2q instance with the Qualcomm Base AL2 DLAMI AMI and follow steps 1 thru 4.

Step 1. Set up the environment and install required packages

1. Install Python 3.8.

   sudo amazon-linux-extras install python3.8

2. Set up the Python 3.8 virtual environment.

   python3.8 -m venv /home/ec2-user/userA/pyenv

3. Activate the Python 3.8 virtual environment.

   source /home/ec2-user/userA/pyenv/bin/activate

4. Install the required packages, shown in the requirements.txt document available at the Qualcomm public Github site.

   pip3 install -r requirements.txt

5. Import the necessary libraries.

   import transformers 
   from transformers import AutoTokenizer, AutoModelForMaskedLM
   import sys
   import qaic
   import os
   import torch
   import onnx
   from onnxsim import simplify
   import argparse
   import numpy as np

Step 2. Import the model

1. Import and tokenize the model.

   model_card = 'bert-base-cased'
   model = AutoModelForMaskedLM.from_pretrained(model_card)
   tokenizer = AutoTokenizer.from_pretrained(model_card)

2. Define a sample input and extract the inputIds and attentionMask.

   sentence = "The dog [MASK] on the mat."
   encodings = tokenizer(sentence, max_length=128, truncation=True, padding="max_length", return_tensors='pt')
   inputIds = encodings["input_ids"]
   attentionMask = encodings["attention_mask"]

3. Convert the model to ONNX, which can then be passed to the compiler.

   # Set dynamic dims and axes.
   dynamic_dims = {0: 'batch', 1 : 'sequence'}
   dynamic_axes = {
       "input_ids" : dynamic_dims,
       "attention_mask" : dynamic_dims,
       "logits" : dynamic_dims
   }
   input_names = ["input_ids", "attention_mask"]
   inputList = [inputIds, attentionMask]
   
   torch.onnx.export(
       model,
       args=tuple(inputList),
       f=f"{gen_models_path}/{model_base_name}.onnx",
       verbose=False,
       input_names=input_names,
       output_names=["logits"],
       dynamic_axes=dynamic_axes,
       opset_version=11,
   )

4. You’ll run the model in FP16 precision. So, you need to check if the model contains any constants beyond the FP16 range. Pass the model to the fix_onnx_fp16 function to generate the new ONNX file with the fixes required.

   from onnx import numpy_helper
           
   def fix_onnx_fp16(
       gen_models_path: str,
       model_base_name: str,
   ) -> str:
       finfo = np.finfo(np.float16)
       fp16_max = finfo.max
       fp16_min = finfo.min
       model = onnx.load(f"{gen_models_path}/{model_base_name}.onnx")
       fp16_fix = False
       for tensor in onnx.external_data_helper._get_all_tensors(model):
           nptensor = numpy_helper.to_array(tensor, gen_models_path)
           if nptensor.dtype == np.float32 and (
               np.any(nptensor > fp16_max) or np.any(nptensor < fp16_min)
           ):
               # print(f'tensor value : {nptensor} above {fp16_max} or below {fp16_min}')
               nptensor = np.clip(nptensor, fp16_min, fp16_max)
               new_tensor = numpy_helper.from_array(nptensor, tensor.name)
               tensor.CopyFrom(new_tensor)
               fp16_fix = True
               
       if fp16_fix:
           # Save FP16 model
           print("Found constants out of FP16 range, clipped to FP16 range")
           model_base_name += "_fix_outofrange_fp16"
           onnx.save(model, f=f"{gen_models_path}/{model_base_name}.onnx")
           print(f"Saving modified onnx file at {gen_models_path}/{model_base_name}.onnx")
       return model_base_name
   
   fp16_model_name = fix_onnx_fp16(gen_models_path=gen_models_path, model_base_name=model_base_name)

Step 3. Compile the model

The qaic-exec command line interface (CLI) compiler tool is used to compile the model. The input to this compiler is the ONNX file generated in step 2. The compiler produces a binary file (called QPC, for Qualcomm program container) in the path defined by -aic-binary-dir argument.

In the compile command below, you use four AI compute cores and a batch size of one to compile the model.

/opt/qti-aic/exec/qaic-exec 
-m=bert-base-cased/generatedModels/bert-base-cased_fix_outofrange_fp16.onnx 
-aic-num-cores=4 
-convert-to-fp16 
-onnx-define-symbol=batch,1 -onnx-define-symbol=sequence,128 
-aic-binary-dir=bert-base-cased/generatedModels/bert-base-cased_fix_outofrange_fp16_qpc 
-aic-hw -aic-hw-version=2.0 
-compile-only

The QPC is generated in the bert-base-cased/generatedModels/bert-base-cased_fix_outofrange_fp16_qpc folder.

Step 4. Run the model

Set up a session to run the inference on a Cloud AI100 Qualcomm accelerator in the DL2q instance.

The Qualcomm qaic Python library is a set of APIs that provides support for running inference on the Cloud AI100 accelerator.

1. Use the Session API call to create an instance of session. The Session API call is the entry point to using the qaic Python library.

   qpcPath = 'bert-base-cased/generatedModels/bert-base-cased_fix_outofrange_fp16_qpc'
   
   bert_sess = qaic.Session(model_path= qpcPath+'/programqpc.bin', num_activations=1)  
   bert_sess.setup() # Loads the network to the device. 
   
   # Here we are reading out all the input and output shapes/types
   input_shape, input_type = bert_sess.model_input_shape_dict['input_ids']
   attn_shape, attn_type = bert_sess.model_input_shape_dict['attention_mask']
   output_shape, output_type = bert_sess.model_output_shape_dict['logits']
   
   #create the input dictionary for given input sentence
   input_dict = {"input_ids": inputIds.numpy().astype(input_type), "attention_mask" : attentionMask.numpy().astype(attn_type)}
   
   #run inference on Cloud AI 100
   output = bert_sess.run(input_dict)

2. Restructure the data from output buffer with output_shape and output_type.

   token_logits = np.frombuffer(output['logits'], dtype=output_type).reshape(output_shape)

3. Decode the output produced.

   mask_token_logits = torch.from_numpy(token_logits[0, mask_token_index, :]).unsqueeze(0)
   top_5_results = torch.topk(mask_token_logits, 5, dim=1)
   print("Model output (top5) from Qualcomm Cloud AI 100:")
   for i in range(5):
       idx = top_5_results.indices[0].tolist()[i]
       val = top_5_results.values[0].tolist()[i]
       word = tokenizer.decode([idx])
       print(f"{i+1} :(word={word}, index={idx}, logit={round(val,2)})")

Here are the outputs for the input sentence “The dog [MASK] on the mat.”

1 :(word=sat, index=2068, logit=11.46)
2 :(word=landed, index=4860, logit=11.11)
3 :(word=spat, index=15732, logit=10.95)
4 :(word=settled, index=3035, logit=10.84)
5 :(word=was, index=1108, logit=10.75)

That’s it. With just a few steps, you compiled and ran a PyTorch model on an Amazon EC2 DL2q instance. To learn more about onboarding and compiling models on the DL2q instance, see the Cloud AI100 Tutorial Documentation.

To learn more about which DL model architectures are a good fit for AWS DL2q instances and the current model support matrix, see the Qualcomm Cloud AI100 documentation.

Available now

You can launch DL2q instances today in the US West (Oregon) and Europe (Frankfurt) AWS Regions as On-demand, Reserved, and Spot Instances, or as part of a Savings Plan. As usual with Amazon EC2, you pay only for what you use. For more information, see Amazon EC2 pricing.

DL2q instances can be deployed using AWS Deep Learning AMIs (DLAMI), and container images are available through managed services such as Amazon SageMaker, Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Container Service (Amazon ECS), and AWS ParallelCluster.

To learn more, visit the Amazon EC2 DL2q instance page, and send feedback to AWS re:Post for EC2 or through your usual AWS Support contacts.

About the authors

A.K Roy is a Director of Product Management at Qualcomm, for Cloud and Datacenter AI products and solutions. He has over 20 years of experience in product strategy and development, with the current focus of best-in-class performance and performance/$ end-to-end solutions for AI inference in the Cloud, for the broad range of use-cases, including GenAI, LLMs, Auto and Hybrid AI.

Jianying Lang is a Principal Solutions Architect at AWS Worldwide Specialist Organization (WWSO). She has over 15 years of working experience in HPC and AI field. At AWS, she focuses on helping customers deploy, optimize, and scale their AI/ML workloads on accelerated computing instances. She is passionate about combining the techniques in HPC and AI fields. Jianying holds a PhD degree in Computational Physics from the University of Colorado at Boulder.

Read More