New method leverages vision-language models to formalize a comparison that had previously required human judgment.Read More
Deploy Amazon SageMaker pipelines using AWS Controllers for Kubernetes
Kubernetes is a popular orchestration platform for managing containers. Its scalability and load-balancing capabilities make it ideal for handling the variable workloads typical of machine learning (ML) applications. DevOps engineers often use Kubernetes to manage and scale ML applications, but before an ML model is available, it must be trained and evaluated and, if the quality of the obtained model is satisfactory, uploaded to a model registry.
Amazon SageMaker provides capabilities to remove the undifferentiated heavy lifting of building and deploying ML models. SageMaker simplifies the process of managing dependencies, container images, auto scaling, and monitoring. Specifically for the model building stage, Amazon SageMaker Pipelines automates the process by managing the infrastructure and resources needed to process data, train models, and run evaluation tests.
A challenge for DevOps engineers is the additional complexity that comes from using Kubernetes to manage the deployment stage while resorting to other tools (such as the AWS SDK or AWS CloudFormation) to manage the model building pipeline. One alternative to simplify this process is to use AWS Controllers for Kubernetes (ACK) to manage and deploy a SageMaker training pipeline. ACK allows you to take advantage of managed model building pipelines without needing to define resources outside of the Kubernetes cluster.
In this post, we introduce an example to help DevOps engineers manage the entire ML lifecycle—including training and inference—using the same toolkit.
Solution overview
We consider a use case in which an ML engineer configures a SageMaker model building pipeline using a Jupyter notebook. This configuration takes the form of a Directed Acyclic Graph (DAG) represented as a JSON pipeline definition. The JSON document can be stored and versioned in an Amazon Simple Storage Service (Amazon S3) bucket. If encryption is required, it can be implemented using an AWS Key Management Service (AWS KMS) managed key for Amazon S3. A DevOps engineer with access to fetch this definition file from Amazon S3 can load the pipeline definition into an ACK service controller for SageMaker, which is running as part of an Amazon Elastic Kubernetes Service (Amazon EKS) cluster. The DevOps engineer can then use the Kubernetes APIs provided by ACK to submit the pipeline definition and initiate one or more pipeline runs in SageMaker. This entire workflow is shown in the following solution diagram.
Prerequisites
To follow along, you should have the following prerequisites:
- An EKS cluster where the ML pipeline will be created.
- A user with access to an AWS Identity and Access Management (IAM) role that has IAM permissions (
iam:CreateRole,iam:AttachRolePolicy, andiam:PutRolePolicy) to allow creating roles and attaching policies to roles. - The following command line tools on the local machine or cloud-based development environment used to access the Kubernetes cluster:
- The AWS Command Line Interface (AWS CLI) installed for interacting with AWS services.
- kubectl for working with Kubernetes clusters.
- eksctl for working with EKS clusters.
- yq for YAML processing. (For Linux environments, use the wget plain binary installation.)
- Helm 3.7+ for installing and managing Kubernetes applications.
- curl for transmitting data with URLs.
Install the SageMaker ACK service controller
The SageMaker ACK service controller makes it straightforward for DevOps engineers to use Kubernetes as their control plane to create and manage ML pipelines. To install the controller in your EKS cluster, complete the following steps:
- Configure IAM permissions to make sure the controller has access to the appropriate AWS resources.
- Install the controller using a SageMaker Helm Chart to make it available on the client machine.
The following tutorial provides step-by-step instructions with the required commands to install the ACK service controller for SageMaker.
Generate a pipeline JSON definition
In most companies, ML engineers are responsible for creating the ML pipeline in their organization. They often work with DevOps engineers to operate those pipelines. In SageMaker, ML engineers can use the SageMaker Python SDK to generate a pipeline definition in JSON format. A SageMaker pipeline definition must follow the provided schema, which includes base images, dependencies, steps, and instance types and sizes that are needed to fully define the pipeline. This definition then gets retrieved by the DevOps engineer for deploying and maintaining the infrastructure needed for the pipeline.
The following is a sample pipeline definition with one training step:
With SageMaker, ML model artifacts and other system artifacts are encrypted in transit and at rest. SageMaker encrypts these by default using the AWS managed key for Amazon S3. You can optionally specify a custom key using the KmsKeyId property of the OutputDataConfig argument. For more information on how SageMaker protects data, see Data Protection in Amazon SageMaker.
Furthermore, we recommend securing access to the pipeline artifacts, such as model outputs and training data, to a specific set of IAM roles created for data scientists and ML engineers. This can be achieved by attaching an appropriate bucket policy. For more information on best practices for securing data in Amazon S3, see Top 10 security best practices for securing data in Amazon S3.
Create and submit a pipeline YAML specification
In the Kubernetes world, objects are the persistent entities in the Kubernetes cluster used to represent the state of your cluster. When you create an object in Kubernetes, you must provide the object specification that describes its desired state, as well as some basic information about the object (such as a name). Then, using tools such as kubectl, you provide the information in a manifest file in YAML (or JSON) format to communicate with the Kubernetes API.
Refer to the following Kubernetes YAML specification for a SageMaker pipeline. DevOps engineers need to modify the .spec.pipelineDefinition key in the file and add the ML engineer-provided pipeline JSON definition. They then prepare and submit a separate pipeline execution YAML specification to run the pipeline in SageMaker. There are two ways to submit a pipeline YAML specification:
- Pass the pipeline definition inline as a JSON object to the pipeline YAML specification.
- Convert the JSON pipeline definition into String format using the command line utility jq. For example, you can use the following command to convert the pipeline definition to a JSON-encoded string:
In this post, we use the first option and prepare the YAML specification (my-pipeline.yaml) as follows:
Submit the pipeline to SageMaker
To submit your prepared pipeline specification, apply the specification to your Kubernetes cluster as follows:
Create and submit a pipeline execution YAML specification
Refer to the following Kubernetes YAML specification for a SageMaker pipeline. Prepare the pipeline execution YAML specification (pipeline-execution.yaml) as follows:
To start a run of the pipeline, use the following code:
Review and troubleshoot the pipeline run
To list all pipelines created using the ACK controller, use the following command:
To list all pipeline runs, use the following command:
To get more details about the pipeline after it’s submitted, like checking the status, errors, or parameters of the pipeline, use the following command:
To troubleshoot a pipeline run by reviewing more details about the run, use the following command:
Clean up
Use the following command to delete any pipelines you created:
Use the following command to cancel any pipeline runs you started:
Conclusion
In this post, we presented an example of how ML engineers familiar with Jupyter notebooks and SageMaker environments can efficiently work with DevOps engineers familiar with Kubernetes and related tools to design and maintain an ML pipeline with the right infrastructure for their organization. This enables DevOps engineers to manage all the steps of the ML lifecycle with the same set of tools and environment they are used to, which enables organizations to innovate faster and more efficiently.
Explore the GitHub repository for ACK and the SageMaker controller to start managing your ML operations with Kubernetes.
About the Authors
Pratik Yeole is a Senior Solutions Architect working with global customers, helping customers build value-driven solutions on AWS. He has expertise in MLOps and containers domains. Outside of work, he enjoys time with friends, family, music, and cricket.
Felipe Lopez is a Senior AI/ML Specialist Solutions Architect at AWS. Prior to joining AWS, Felipe worked with GE Digital and SLB, where he focused on modeling and optimization products for industrial applications.
Effectively manage foundation models for generative AI applications with Amazon SageMaker Model Registry
Generative artificial intelligence (AI) foundation models (FMs) are gaining popularity with businesses due to their versatility and potential to address a variety of use cases. The true value of FMs is realized when they are adapted for domain specific data. Managing these models across the business and model lifecycle can introduce complexity. As FMs are adapted to different domains and data, operationalizing these pipelines becomes critical.
Amazon SageMaker, a fully managed service to build, train, and deploy machine learning (ML) models, has seen increased adoption to customize and deploy FMs that power generative AI applications. SageMaker provides rich features to build automated workflows for deploying models at scale. One of the key features that enables operational excellence around model management is the Model Registry. Model Registry helps catalog and manage model versions and facilitates collaboration and governance. When a model is trained and evaluated for performance, it can be stored in the Model Registry for model management.
Amazon SageMaker has released new features in Model Registry that make it easy to version and catalog FMs. Customers can use SageMaker to train or tune FMs, including Amazon SageMaker JumpStart and Amazon Bedrock models, and also manage these models within Model Registry. As customers begin to scale generative AI applications across various use cases such as fine-tuning for domain-specific tasks, the number of models can quickly grow. To keep track of models, versions, and associated metadata, SageMaker Model Registry can be used as an inventory of models.
In this post, we explore the new features of Model Registry that streamline FM management: you can now register unzipped model artifacts and pass an End User License Agreement (EULA) acceptance flag without needing users to intervene.
Overview
Model Registry has worked well for traditional models, which are smaller in size. For FMs, there were challenges because of their size and requirements for user intervention for EULA acceptance. With the new features in Model Registry, it’s become easier to register a fine-tuned FM within Model Registry, which then can be deployed for actual use.
A typical model development lifecycle is an iterative process. We conduct many experimentation cycles to achieve expected performance from the model. Once trained, these models can be registered in the Model Registry where they are cataloged as versions. The models can be organized in groups, the versions can be compared for their quality metrics, and models can have an associated approval status indicating if its deployable.
Once the model is manually approved, a continuous integration and continuous deployment (CI/CD) pipeline can be triggered to deploy these models to production. Optionally, Model Registry can be used as a repository of models that are approved for use by an enterprise. Various teams can then deploy these approved models from Model Registry and build applications around it.
An example workflow could follow these steps and is shown in the following diagram:
- Select a SageMaker JumpStart model and register it in Model Registry
- Alternatively, you can fine-tune a SageMaker JumpStart model
- Evaluate the model with SageMaker model evaluation. SageMaker allows for human evaluation if desired.
- Create a model group in the Model Registry. For each run, create a model version. Add your model group into one or more Model Registry Collections, which can be used to group registered models that are related to each other. For example, you could have a collection of large language models (LLMs) and another collection of diffusion models.
- Deploy the models as SageMaker Inference endpoints that can be consumed by generative AI applications.
Figure 1: Model Registry workflow for foundation models
To better support generative AI applications, Model Registry released two new features: ModelDataSource, and source model URI. The following sections will explore these features and how to use them.
ModelDataSource speeds up deployment and provides access to EULA dependent models
Until now, model artifacts had to be stored along with the inference code when a model gets registered in Model Registry in a compressed format. This posed challenges for generative AI applications where FMs are of very large size with billions of parameters. The large size of FMs when stored as zipped models was causing increased latency with SageMaker endpoint startup time because decompressing these models at run time took very long. The model_data_source parameter can now accept the location of the unzipped model artifacts in Amazon Simple Storage Service (Amazon S3) making the registration process simple. This also eliminates the need for endpoints to unzip the model weights, leading to reduced latency during endpoint startup times.
Additionally, public JumpStart models and certain FMs from independent service providers, such as LLAMA2, require that their EULA must be accepted prior to using the models. Thus, when public models from SageMaker JumpStart were tuned, they could not be stored in the Model Registry because a user needed to accept the license agreement. Model Registry added a new feature: EULA acceptance flag support within the model_data_source parameter, allowing the registration of such models. Now customers can catalog, version, associate metadata such as training metrics, and more in Model Registry for a wider variety of FMs.
Register unzipped models stored in Amazon S3 using the AWS SDK.
Register models requiring a EULA.
Source model URI provides simplified registration and proprietary model support
Model Registry now supports automatic population of inference specification files for some recognized model IDs, including select AWS Marketplace models, hosted models, or versioned model packages in Model Registry. Because of SourceModelURI’s support for automatic population, you can register proprietary JumpStart models from providers such as AI21 labs, Cohere, and LightOn without needing the inference specification file, allowing your organization to use a broader set of FMs in Model Registry.
Previously, to register a trained model in the SageMaker Model Registry, you had to provide the complete inference specification required for deployment, including an Amazon Elastic Container Registry (Amazon ECR) image and the trained model file. With the launch of source_uri support, SageMaker has made it easy for users to register any model by providing a source model URI, which is a free form field that stores model ID or location to a proprietary JumpStart and Bedrock model ID, S3 location, and MLflow model ID. Rather than having to supply the details required for deploying to SageMaker hosting at the time of registrations, you can add the artifacts later on. After registration, to deploy a model, you can package the model an inference specification and update Model Registry accordingly.
For example, you can register a model in Model Registry with a model Amazon Resource Name (ARN) SourceURI.
Later, you can update the registered model with the inference specification, making it deployable on SageMaker.
Register an Amazon JumpStart proprietary FM.
Conclusion
As organizations continue to adopt generative AI in different parts of their business, having robust model management and versioning becomes paramount. With Model Registry, you can achieve version control, tracking, collaboration, lifecycle management, and governance of FMs.
In this post, we explored how Model Registry can now more effectively support managing generative AI models across the model lifecycle, empowering you to better govern and adopt generative AI to achieve transformational outcomes.
To learn more about Model Registry, see Register and Deploy Models with Model Registry. To get started, visit the SageMaker console.
About the Authors
Chaitra Mathur serves as a Principal Solutions Architect at AWS, where her role involves advising clients on building robust, scalable, and secure solutions on AWS. With a keen interest in data and ML, she assists clients in leveraging AWS AI/ML and generative AI services to address their ML requirements effectively. Throughout her career, she has shared her expertise at numerous conferences and has authored several blog posts in the ML area.
Kait Healy is a Solutions Architect II at AWS. She specializes in working with startups and enterprise automotive customers, where she has experience building AI/ML solutions at scale to drive key business outcomes.
Saumitra Vikaram is a Senior Software Engineer at AWS. He is focused on AI/ML technology, ML model management, ML governance, and MLOps to improve overall organizational efficiency and productivity.
Siamak Nariman is a Senior Product Manager at AWS. He is focused on AI/ML technology, ML model management, and ML governance to improve overall organizational efficiency and productivity. He has extensive experience automating processes and deploying various technologies
Build an ecommerce product recommendation chatbot with Amazon Bedrock Agents
Many ecommerce applications want to provide their users with a human-like chatbot that guides them to choose the best product as a gift for their loved ones or friends. To enhance the customer experience, the chatbot need to engage in a natural, conversational manner to understand the user’s preferences and requirements, such as the recipient’s gender, the occasion for the gift, and the desired product category. Based on the discussion with the user, the chatbot should be able to query the ecommerce product catalog, filter the results, and recommend the most suitable products.
Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading artificial intelligence (AI) companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.
Amazon Bedrock Agents is a feature that enables generative AI applications to run multistep tasks across company systems and data sources. In this post, we show you how to build an ecommerce product recommendation chatbot using Amazon Bedrock Agents and FMs available in Amazon Bedrock.
Solution overview
Traditional rule-based chatbots often struggle to handle the nuances and complexities of open-ended conversations, leading to frustrating experiences for users. Furthermore, manually coding all the possible conversation flows and product filtering logic is time-consuming and error-prone, especially as the product catalog grows.
To address this challenge, you need a solution that uses the latest advancements in generative AI to create a natural conversational experience. The solution should seamlessly integrate with your existing product catalog API and dynamically adapt the conversation flow based on the user’s responses, reducing the need for extensive coding.
With Amazon Bedrock Agents, you can build intelligent chatbots that can converse naturally with users, understand their preferences, and efficiently retrieve and recommend the most relevant products from the catalog. Amazon Bedrock Agents simplifies the process of building and deploying generative AI models, enabling businesses to create engaging and personalized conversational experiences without the need for extensive machine learning (ML) expertise.
For our use case, we create a recommender chatbot using Amazon Bedrock Agents that prompts users to describe who they want to buy the gift for and the relevant occasion. The agent queries the product information stored in an Amazon DynamoDB table, using an API implemented as an AWS Lambda function. The agent adapts the API inputs to filter products based on its discussion with the user, for example gender, occasion, and category. After obtaining the user’s gift preferences by asking clarifying questions, the agent responds with the most relevant products that are available in the DynamoDB table based on user preferences.
The following diagram illustrates the solution architecture.
As shown in the preceding diagram, the ecommerce application first uses the agent to drive the conversation with users and generate product recommendations. The agent uses an API backed by Lambda to get product information. Lastly, the Lambda function looks up product data from DynamoDB.
Prerequisites
You need to have an AWS account with a user or role that has at minimum the following AWS Identity and Access Management (IAM) policies and permissions:
- AWS managed policies:
AmazonBedrockFullAccessAWSMarketplaceManageSubscriptionsAWSLambda_ReadOnlyAccessAmazonDynamoDBReadOnlyAccess
- IAM actions:
iam:CreateRoleiam:CreatePolicyiam:AttachRolePolicy
Deploy the solution resources with AWS CloudFormation
Before you create your agent, you need to set up the product database and API. We use an AWS CloudFormation template to create a DynamoDB table to store product information and a Lambda function to serve as the API for retrieving product details.
At the time of writing this post, you can use any of the following AWS Regions to deploy the solution: US East (N. Virginia), US West (Oregon), Asia Pacific (Mumbai, Sydney), Europe (Frankfurt, Paris), Canada (Central), or South America (São Paulo). Visit Supported regions and models for Amazon Bedrock Agents for updates.
To deploy the template, choose Launch Stack:
This template creates a DynamoDB table named Products with the following attributes: product_name (partition key), category, gender, and occasion. It also defines a global secondary index (GSI) for each of these attributes to enable efficient querying.
Additionally, the template sets up a Lambda function named GetProductDetailsFunction that acts as an API for retrieving product details, This Lambda function accepts query parameters such as category, gender, and occasion. It constructs a filter expression based on the provided parameters and scans the DynamoDB table to retrieve matching products. If no parameters are provided, it retrieves all the products in the table and returns the first 100 products.
The template also creates another Lambda function called PopulateProductsTableFunction that generates sample data to store in the Products table. The CloudFormation template includes a custom resource that will run the PopulateProductsTableFunction function one time as part of the template deployment, to add 100 sample product entries in the products DynamoDB table, with various combinations of product names, descriptions, categories, genders, and occasions.
You can optionally update the sample product entries or replace it with your own product data. To do so, open the DynamoDB console, choose Explore items, and select the Products table. Choose Scan and choose Run to view and edit the current items or choose Create item to add a new item. If your data has different attributes than the sample product entries, you need to adjust the code of the Lambda function GetProductDetailsFunction, the OpenAPI schema, and the instructions for the agent that are used in the following section.
Create the agent
Now that you have the infrastructure in place, you can create the agent. The first step is to request model access.
- On the Amazon Bedrock console, choose Model access in the navigation pane.
- Choose Enable specific models.
- Select the model you need access to (for this post, we select Claude 3 Sonnet).
Wait for the model access status to change to Access granted.
Now you can create your agent. We use a CloudFormation template to create the agent and the action group that will invoke the Lambda function.
- To deploy the template, choose Launch Stack:
Now you can check the details of the agent that was created by the stack.
- On the Amazon Bedrock console, choose Agents under Builder tools in the navigation pane.
- Choose the agent
product-recommendation-agent, then choose Edit in Agent Builder. - The Instructions for the Agent section includes a set of instructions that guides the agent in how to communicate with the user and use the API. You can adjust the instructions based on different use cases and business scenarios as well as the available APIs.
The agent’s primary goal is to engage in a conversation with the user to gather information about the recipient’s gender, the occasion for the gift, and the desired category. Based on this information, the agent will query the Lambda function to retrieve and recommend suitable products.
Your next step is to check the action group that enables the agent to invoke the Lambda function.
- In the Action groups section, choose the
Get-Product-Recommendationsaction group.
You can see the GetProductDetailsFunction Lambda function is selected in the Action group invocation section.
In the Action group schema section, you can see the OpenAPI schema, which enables the agent to understand the description, inputs, outputs, and the actions of the API that it can use during the conversation with the user.
Now you can use the Test Agent pane to have conversations with the chatbot.
Test the chatbot
The following screenshots show example conversations, with the chatbot recommending products after calling the API.
In the sample conversation, the chatbot asks relevant questions to determine the gift recipient’s gender, the occasion, and the desired category. After it has gathered enough information, it queries the API and presents a list of recommended products matching the user’s preferences.
You can see the rationale for each response by choosing Show trace. The following screenshots show how the agent decided to use different API filters based on the discussion.
You can see in the rationale field how the agent made its decision for each interaction. This trace data can help you understand the reasons behind a recommendation. Logging this information can be beneficial for future refinements of your agent’s recommendations.
Clean up
Complete the following steps to clean up your resources:
- On the AWS CloudFormation console, delete the stack
AgentStack. - Then delete the stack
Productstableandapi.
Conclusion
This post showed you how to use Amazon Bedrock Agents to create a conversational chatbot that can assist users in finding the perfect gift. The chatbot intelligently gathers user preferences, queries a backend API to retrieve relevant product details, and presents its recommendations to the user. This approach demonstrates the power of Agents for Amazon Bedrock in building engaging and context-aware conversational experiences.
We recommend you follow best practices while using Amazon Bedrock Agents. For instance, using AWS CloudFormation to create and configure the agent allows you to minimize human error and recreate the agent across different environments and Regions. Also, automating your agent testing using a set of golden questions and their expected answers enables you to test the quality of the instructions for the agent and compare the outputs of the different models on Amazon Bedrock in relation to your use case.
Visit Amazon Bedrock Agents to learn more about features and details.
About the Author
Mahmoud Salaheldin is a Senior Solutions Architect in AWS, working with customers in the Middle East, North Africa, and Turkey, where he helps enterprises, digital-centered businesses, and independent software vendors innovate new products that can enhance their customer experience and increase their business efficiency. He is a generative AI ambassador as well as a containers community member. He lives in Dubai, United Arab Emirates, and enjoys riding motorcycles and traveling.
How Thomson Reuters Labs achieved AI/ML innovation at pace with AWS MLOps services
This post is co-written by Danilo Tommasina and Andrei Voinov from Thomson Reuters.
Thomson Reuters (TR) is one of the world’s most trusted information organizations for businesses and professionals. TR provides companies with the intelligence, technology, and human expertise they need to find trusted answers, enabling them to make better decisions more quickly. TR’s customers span the financial, risk, legal, tax, accounting, and media markets.
Thomson Reuters Labs (TR Labs) is the dedicated applied research division within TR. TR Labs is focused on the research, development, and application of artificial intelligence (AI) and emerging trends in technologies that can be infused into existing TR products or new offerings. TR Labs works collaboratively with various product teams to experiment, prototype, test, and deliver AI-powered innovation in pursuit of smarter and more valuable tools for our customers. The TR Labs team includes over 150 applied scientists, machine learning specialists, and machine learning engineers.
In this post, we explore how TR Labs was able to develop an efficient, flexible, and powerful MLOps process by adopting a standardized MLOps framework that uses AWS SageMaker, SageMaker Experiments, SageMaker Model Registry, and SageMaker Pipelines. The goal being to accelerate how quickly teams can experiment and innovate using AI and machine learning (ML)—whether using natural language processing (NLP), generative AI, or other techniques. We discuss how this has helped decrease the time to market for fresh ideas and helped build a cost-efficient machine learning lifecycle. Lastly, we will go through the MLOps toolchain that TR Labs built to standardize the MLOps process for developers, scientists, and engineers.
The challenge
Machine learning operations (MLOps) is the intersection of people, for gaining business value from machine learning. An MLOps practice is essential for an organization with large teams of ML engineers and data scientists. Correctly using AI/ML tools to increase productivity directly influences efficiency and cost of development. TR Labs was founded in 1992 with a vision to be a world-leading AI/ML research and development practice, forming the core innovation team that works alongside the tax, legal and news divisions of TR to ensure that their offerings remain at the cutting edge of their markets.
The TR Labs team started off as a small team in its early days, with a team directive to spearhead ML innovation to help the company in various domains including but not limited to text summarization, document categorization, and various other NLP tasks. The team made remarkable progress from an early stage with AI/ML models being integrated into TR’s products and internal editorial systems to help with efficiency and productivity.
However, as the company grew, so did the team’s size and task complexity. The team had grown to over 100 people, and they were facing new challenges. Model development and training processes were becoming increasingly complex and challenging to manage. The team had different members working on different use cases, and therefore, models. Each researcher also had their own way of developing the models. This led to a situation where there was little standardization in the process for model development. Each researcher needed to configure all the underlying resources manually, and a large amount of boilerplate code was created in parallel by different teams. A significant portion of time was spent on tasks that could be performed more efficiently.
The TR Labs leadership recognized that the existing MLOps process wasn’t scalable and needed to be standardized. It lacked sufficient automation and assistance for those new to the platform. The idea was to take well architected practices for ML model development and operations and create a customized workflow specific to Labs that uses Amazon Web Services (AWS). The vision was to harmonize and simplify the model development process and accelerate the pace of innovation. They also aimed to set the path to quickly mature research and development solutions into an operational state that would support a high degree of automation for monitoring and retraining.
In this post, we will focus on the MLOps process parts involved in the research and model development phases.
The overview section will take you through the innovative solution that TR Labs created and how it helped lower the barrier to entry while increasing the adoption of AI/ML services for new ML users on AWS while decreasing time to market for new projects.
Solution overview
The existing ML workflow required a TR user to start from scratch every time they started a new project. Research teams would have to familiarize themselves with the TR Labs standards and deploy and configure the entire MLOps toolchain manually with little automation in place. Inconsistent practices within the research community meant extra work was needed to align with production grade deployments. Many research projects had to be refactored when handing code over to MLOps engineers, who often had to reverse engineer to achieve a similar level of functionality to make the code ready to deploy to production. The team had to create an environment where researchers and engineers worked on one shared codebase and use the same toolchain, reducing the friction between experimentation and production stages. A shared codebase is also a key element for long term maintenance—changes to the existing system should be integrated directly in the production level code and not reverse engineered and re-merged out of a research repository into the production codebase. This is an anti-pattern that leads to large costs and risks over time.
Regardless of the chosen model architecture, or even if the chosen model is a third-party provider for large language models (LLMs) without any fine tuning, a robust ML system requires validation on a relevant dataset. There are multiple testing methods, such as zero-shot learning, a machine learning technique that allows a model to classify objects from previously unseen classes, without receiving any specific training for those classes, with a transition to later introduce fine tuning to improve the model’s performance. How many iterations are necessary to obtain the expected initial quality and maintain or even improve the level over time depends on the use case and the model type being developed. However, when thinking about long-term systems, teams go through tens or even hundreds of repetitions. These repetitions will contain several recurring steps such as pre-processing, training, and post processing, which are similar, if not the same, no matter which approach is taken. Repeating the process manually without following a harmonized approach is also an anti-pattern.
This process inefficiency presented an opportunity to create a coherent set of MLOps tools that would enforce TR Labs standards for how to configure and deploy SageMaker services and expose these MLOps capabilities to a user by providing standard configuration and boilerplate code. The initiative was named TR MLTools and joined several MLOps libraries developed in TR Labs under one umbrella. Under this umbrella, the team provided a command line interface (CLI) tool that would support a standard project structure and deliver boilerplate code abstracting the underlying infrastructure deployment process and promoting a standardized TR ML workflow.
MLTools and MLTools CLI were designed to be flexible and extendable while incorporating a TR Labs-opinionated view on how to run MLOps in line with TR enterprise cloud platform standards.
MLTools CLI
MLTools CLI is a Python package and a command-line tool that promotes the standardization of TR Labs ML experiments workflow (ML model development, training, and so on) by providing code and configuration templates directly into the users’ code repository. At its core, MLTools CLI aims to connect all ML experiment-related entities (Python scripts, Jupyter notebooks, configuration files, data, pipeline definitions, and so on) and provide an easy way to bootstrap new experiments, conduct trials, and run user-defined scripts, testing them locally and remotely running them at scale as SageMaker jobs.
MLTools CLI is added as a development dependency to a new or existing Python project, where code for the planned ML experiments will be developed and tracked, for example in GitHub. As part of an initial configuration step, this source-code project is associated with specific AI Platform Machine Learning Workspaces. The users can then start using the MLTools CLI for running their ML experiments using SageMaker capabilities like Processing and Training jobs, Experiments, Pipelines, and so on.
Note: AI Platform Workspaces is an internal service, developed in TR, that provides secure access to Amazon Simple Storage Service (Amazon S3)-hosted data and AWS resources like SageMaker or SageMaker Studio Notebook instances for our ML researchers. You can find more information about the AI Platform Workspaces in this AWS blog: How Thomson Reuters built an AI platform using Amazon SageMaker to accelerate delivery of ML projects.
MLTools CLI acts effectively as a frontend or as a delivery channel for the set of capabilities (libraries, tools, and templates) that TR collectively refers to as MLTools. The following diagram shows a typical TR Labs ML experiments workflow, with a focus on the role of MLTools and MLTools CLI:
MLTools CLI offers various templates that can be generated using a command-line, including the following:
- Separate directory structure for new ML experiments and experiment trials.
- Script templates for launching SageMaker processing, training, and batch transform jobs.
- Complete experiment pipeline template based on SageMaker Pipeline, with user scripts as steps.
- Docker image templates for packaging user scripts. For example, for delivery to production.
MLTools CLI also provides the following features to support effective ML experiments:
- User scripts can be run directly as SageMaker jobs without the need to build Docker images.
- Each experiment runs in a sandboxed Poetry environment and can have its own code package and dependency tree.
- The main, project-level code package is shared and can be used by all project experiments and user scripts code, allowing re-use of common code with no copy-paste.
- Context-aware API resolves and loads experiment and trial metadata based on the current working directory.
- Created AWS resources are automatically tagged with the experiment metadata.
- Utilities to query these experiment-related AWS resources are available.
ML experiment workflow
After MLTools CLI is installed and configured on a laptop or notebook instance, a user can begin ML experimentation work. The first step is to create a new experiment using the MLTools CLI create-experiment command:
An experiment template is generated in a sub-directory of the user’s project. The generated experiment folder has a standard structure, including the initial experiment’s configuration, a sandboxed Poetry package, and sample Jupyter notebooks to help quickly bootstrap new ML experiments:
The user can then create script templates for the planned ML experiment steps:
Generated script templates are placed under the experiment directory:
Script names should be short and unique within their parent experiment, because they’re used to generate standardized AWS resource names. Script templates are supplemented by a job configuration for a specific type of job, as specified by the user. Templates and configurations for SageMaker processing, training, and batch transform jobs are currently supported by MLTools—these offerings will be expanded in the future. A requirements.txt file is also included where users can add any dependencies required by the script code to be automatically installed by SageMaker at runtime. The script’s parent experiment and project packages are added to the requirements.txt by default, so the user can import and run code from the whole project hierarchy.
The user would then proceed to add or adapt code in the generated script templates. Experiment scripts are ordinary Python scripts that contain common boilerplate code to give users a head start. They can be run locally while adapting and debugging the code. After the code is working, the same scripts can be launched directly as SageMaker jobs. The required SageMaker job configuration is defined separately in a <script_name>_job.py file, and job configuration details are largely abstracted from the notebook experiment code. As a result, an experiment script can be launched as a SageMaker job with a few lines of code:
Let’s explore the previous code snippet in detail.
First, the MLTools experiment context is loaded based on the current working directory using the load_experiment() factory method. The experiment context concept is a central point of the MLTools API. It provides access to the experiment’s user configuration, the experiment’s scripts, and the job configuration. All project experiments are also integrated with the project-linked AI Platform workspace and therefore have access to the resources and metadata of this workspace. For example, the experiments can access the workspace AWS Identity and Access Management (IAM) role, S3 bucket and its default Amazon Elastic Container Registry (Amazon ECR) repository.
From the experiment, a job context can be loaded, providing one of the experiment’s script names—load_job("train") in this instance. During this operation, the job configuration is loaded from the script’s <script_name>_job.py module. Also, if the script code depends on the experiment or the project packages, they’re automatically built (as Python wheels) and pre-packaged together with the script code, ready to be uploaded to S3.
Next, the training script is launched as a SageMaker training job. In the background, the MLTools factory code ensures that the respective SageMaker estimator or processor instances are created with the default configuration and conform to the rules and best practices accepted in TR. This includes naming conventions, virtual private cloud (VPC) and security configurations, and tagging. Note that SageMaker local mode is fully supported (set in the example by local=True) while its specific configuration details are abstracted from the code. Although the externalized job configuration provides all the defaults, these can be overwritten by the user. In the previous example, custom hyperparameters are provided.
SageMaker jobs that were launched as part of an experiment can be listed directly from the notebook using the experiment’s list_training_jobs() and list_processing_jobs() utilities. SageMaker ExperimentAnalytics data is also available for analysis and can be retrieved by calling the experiment’s experiment_analytics() method.
Integration with SageMaker Experiments
For every new MLTools experiment, a corresponding entity is automatically created in SageMaker Experiments. Experiment names in SageMaker are standardized and made unique by adding a prefix that includes the associated workspace ID and the root commit hash of the user repository. For any job launched from within an MLTools experiment context (that is by using job.run() as shown in the preceding code snippet), a SageMaker Experiments Run instance is created and the job is automatically launched within the SageMaker Experiments Run context. This means all MLTools job runs are automatically tracked in SageMaker Experiments, ensuring that all job run metadata is recorded. This also means that users can then browse their experiments and runs directly in the experiments browser in SageMaker Studio, create visualizations for analysis, and compare model metrics, among other tasks.
As shown in the following diagram, the MLTools experiment workflow is fully integrated with SageMaker Experiments:
Integration with SageMaker Pipelines
Some of the important factors that make ML experiments scalable are their reproducibility and their operationalization level. To support this, MLTools CLI provides users with a capability to add a template with boilerplate code to link the steps of their ML experiment into a deployable workflow (pipeline) that can be automated and delivers reproducible results. The MLTools experiment pipeline implementation is based on AWS SageMaker Pipelines. The same experiment scripts that might have been run and tested as standalone SageMaker jobs can naturally form the experiment pipeline steps.
MLTools currently offers the following standard experiment pipeline template:
We made a deliberate design decision to offer a simple, linear, single-model experiment pipeline template with well-defined standard steps. Oftentimes our project work on multi-model solutions involved an ensemble of ML models that might be ultimately trained on the same set of training data. In such cases, pipelines with more complex flows, or even integrated multi-model experiment pipelines, can be perceived as more efficient. Nevertheless, from a reproducibility and standardization standpoint, a decision to develop a customized experiment pipeline would need to be justified and is generally better suited for the later stages of ML operations where efficient model deployment might be a factor.
On the other hand, using the standard MLTools experiment pipeline template, users can create and start running their experiment pipelines in the early stages of their ML experiments. The underlying pipeline template implementation allows users to easily configure and deploy partial pipelines where only some of the defined steps are implemented. For example, a user can start with a pipeline that only has a single step implemented, such as a DataPreparation step, then add ModelTraining and ModelEvaluation steps and so on. This approach aligns well with the iterative nature of ML experiments and allows for gradually creating a complete experiment pipeline as the ML experiment itself matures.
As shown in the following diagram, MLTools allows users to deploy and run their complete experiment pipelines based on SageMaker Pipelines integrated with SageMaker Model Registry and SageMaker Studio.
Results and future improvements
TR Labs’s successful creation of the MLTools toolchain helps to standardize the MLOps framework throughout the organization and provides several benefits—the first of these is faster model development times. With a consistent process, team members can now work more efficiently by using project templates that deliver a modular setup, facilitating all phases of the ML development process. The structure delivers out-of-the-box integration with TR’s AWS-based AI Platform and the ability to switch between phases of the development including research and data analysis, running experiments at scale, and delivering end-to-end ML pipeline automation. This allows the team to focus on the critical aspects of model development while technicalities are handled and provisioned in advance.
The toolchain is designed to support a close collaboration between researchers and engineers who can work on different aspects of an ML delivery while sharing a codebase that follows software development best practices.
By following a standardized MLOps process, the TR Labs team can also quickly identify issues and model performance drifts more efficiently. It becomes easier to pinpoint where errors are occurring and how to fix them. This can help to reduce downtime and improve the overall efficiency of the development and maintenance processes. The standardized process also ensures that researchers working in model development are using the same environment as ML engineers. This leads to a more efficient transition from ideation and development to deploying the output as models in production and entering the maintenance phase.
Standardizing the MLOps platform has also led to cost savings through efficiencies. With a defined process, the team can reduce the time and resources required to develop and deploy models. This leads to cost savings in the long run, making the development, and particularly the long-term maintenance processes, more cost-effective.
A difficulty the team observed was in measuring how much the toolchain improved time to market and reduced costs. Thoroughly evaluating this would require a dedicated study where independent teams would work on the same use cases with and without the toolchain and comparing the results. However, there are subjective components and possibly different approaches that you can take to resolve this question. Such an approach would be very costly and still contain a high degree of imprecision.
The TR Labs team found an alternate solution for how to measure success. At a yearly interval we run an assessment with the userbase of the toolchain. The assessment covers a variety of aspects ranging over the entire AI/ML lifecycle. Toolchain users are asked to provide subjective assessments on how much of their development time is considered “wasted” on infrastructure issues, configuration issues, or manual tasks that are repetitive. Other questions cover the level of satisfaction with the current toolchain and the perceived improvement in productivity comparing current and past work without the toolchain or earlier versions of the toolchain. The resulting values are averaged over the entire userbase, which includes a mix of job roles ranging from engineers to data scientists to researchers.
The reduction of time spent on inefficiencies, the increase in perceived productivity, and user satisfaction can be used to compute the approximate monetary savings, improvement in code quality, and reduction in time to market. These combined factors contribute to user satisfaction and improvement in the retention of talent within the ML community at TR.
As a measure of success, the TR Labs team was able to achieve reductions in accumulated time spent on inefficiencies and found that this ranges between 3 to 5 days per month per person. Measuring the impact over a period of 12 months, TR has seen improvements of up to 40 percent in perceived productivity in several areas of the lifecycle and a measurable increase in user satisfaction. These numbers are based on what the users of the toolchain reported in the self-assessments.
Conclusion
A standardized MLOps framework can lead to the reduction of bugs, faster model development times, faster troubleshooting of issues, faster reaction to model performance drifts, and cost savings gained through a more efficient end-to-end machine learning process that facilitates experimentation and model creating at scale. By adopting a standardized MLOps framework that uses AWS SageMaker, SageMaker Experiments, SageMaker Model Registry, and SageMaker Pipelines, TR Labs was able to ensure that their machine learning models were developed and deployed efficiently and effectively. This has resulted in a faster time to market and accelerated business value through development.
To learn more about how AWS can help you with your AI/ML and MLOps journey, see What is Amazon SageMaker.
About the Authors
Andrei Voinov is a Lead Software Engineer at Thomson Reuters (TR). He is currently leading a team of engineers in TR Labs with the mandate to develop and support capabilities that help researchers and engineers in TR to efficiently transition ML projects from inception, through research, integration, and delivery into production. He brings over 25 years of experience with software engineering in various sectors and extended knowledge both in the cloud and ML spaces.
Danilo Tommasina is a Distinguished Engineer at Thomson Reuters (TR). With over 20 years of experience working in technology roles ranging from Software Engineer, over Director of Engineering and now as Distinguished Engineer. As a passionate generalist, proficient in multiple programming languages, cloud technologies, DevOps practices and with engineering knowledge in the ML space, he contributed to the scaling of TR Labs’ engineering organization. He is also a big fan of automation including but not limited to MLOps processes and Infrastructure as Code principles.
Simone Zucchet is a Manager of Solutions Architecture at AWS. With close to a decade of experience as a Cloud Architect, Simone enjoys working on innovative projects that help transform the way organizations approach business problems. He helps support large enterprise customers at AWS and is part of the Machine Learning TFC. Outside of his professional life, he enjoys working on cars and photography.
Jeremy Bartosiewicz is a Senior Solutions Architect at AWS. With over 15 years of experience working in technology in multiple roles. Coming from a consulting background, Jeremy enjoys working on a multitude of projects that help organizations grow using cloud solutions. He helps support large enterprise customers at AWS and is part of the Advertising and Machine Learning TFCs.
Build a generative AI image description application with Anthropic’s Claude 3.5 Sonnet on Amazon Bedrock and AWS CDK
Generating image descriptions is a common requirement for applications across many industries. One common use case is tagging images with descriptive metadata to improve discoverability within an organization’s content repositories. Ecommerce platforms also use automatically generated image descriptions to provide customers with additional product details. Descriptive image captions also improve accessibility for users with visual impairments.
With advances in generative artificial intelligence (AI) and multimodal models, producing image descriptions is now more straightforward. Amazon Bedrock provides access to the Anthropic’s Claude 3 family of models, which incorporates new computer vision capabilities enabling Anthropic’s Claude to comprehend and analyze images. This unlocks new possibilities for multimodal interaction. However, building an end-to-end application often requires substantial infrastructure and slows development.
The Generative AI CDK Constructs coupled with Amazon Bedrock offer a powerful combination to expedite application development. This integration provides reusable infrastructure patterns and APIs, enabling seamless access to cutting-edge foundation models (FMs) from Amazon and leading startups. Amazon Bedrock is a fully managed service that offers a choice of high-performing FMs from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI. Generative AI CDK Constructs can accelerate application development by providing reusable infrastructure patterns, allowing you to focus your time and effort on the unique aspects of your application.
In this post, we delve into the process of building and deploying a sample application capable of generating multilingual descriptions for multiple images with a Streamlit UI, AWS Lambda powered with the Amazon Bedrock SDK, and AWS AppSync driven by the open source Generative AI CDK Constructs.
Multimodal models
Multimodal AI systems are an advanced type of AI that can process and analyze data from multiple modalities at once, including text, images, audio, and video. Unlike traditional AI models trained on a single data type, multimodal AI integrates diverse data sources to develop a more comprehensive understanding of complex information.
Anthropic’s Claude 3 on Amazon Bedrock is a leading multimodal model with computer vision capabilities to analyze images and generate descriptive text outputs. Anthropic’s Claude 3 excels at interpreting complex visual assets like charts, graphs, diagrams, reports, and more. The model combines its computer vision with language processing to provide nuanced text summaries of key information extracted from images. This allows Anthropic’s Claude 3 to develop a deeper understanding of visual data than traditional single-modality AI.
In March 2024, Amazon Bedrock provided access to the Anthropic’s Claude 3 family. The three models in the family are Anthropic’s Claude 3 Haiku, the fastest and most compact model for near-instant responsiveness, Anthropic’s Claude 3 Sonnet, the ideal balanced model between skills and speed, and Anthropic’s Claude 3 Opus, the most intelligent offering for top-level performance on highly complex tasks. In June 2024, Amazon Bedrock announced support for Anthropic’s Claude 3.5 as well. The sample application in this post supports Claude 3.5 Sonnet and all the three Claude 3 models.
Generative AI CDK Constructs
Generative AI CDK Constructs, an extension to the AWS Cloud Development Kit (AWS CDK), is an open source development framework for defining cloud infrastructure as code (IaC) and deploying it through AWS CloudFormation.
Constructs are the fundamental building blocks of AWS CDK applications. The AWS Construct Library categorizes constructs into three levels: Level 1 (the lowest-level construct with no abstraction), Level 2 (mapping directly to single AWS CloudFormation resources), and Level 3 (patterns with the highest level of abstraction).
The Generative AI CDK Constructs Library provides modular building blocks to seamlessly integrate AWS services and resources into solutions using generative AI capabilities. By using Amazon Bedrock to access FMs and combining with serverless AWS services such as Lambda and AWS AppSync, these AWS CDK constructs streamline the process of assembling cloud infrastructure for generative AI. You can rapidly configure and deploy solutions to generate content using intuitive abstractions. This approach boosts productivity and reduces time-to-market for delivering innovative applications powered by the latest advances in generative AI on the AWS Cloud.
Solution overview
The sample application in this post uses the aws-summarization-appsync-stepfn construct from the Generative AI CDK Constructs Library. The aws-summarization-appsync-stepfn construct provides a serverless architecture that uses AWS AppSync, AWS Step Functions, and Amazon EventBridge to deliver an asynchronous image summarization service. This construct offers a scalable and event-driven solution for processing and generating descriptions for image assets.
AWS AppSync acts as the entry point, exposing a GraphQL API that enables clients to initiate image summarization and description requests. The API utilizes subscription mutations, allowing for asynchronous runs of the requests. This decoupling promotes best practices for event-driven, loosely coupled systems.
EventBridge serves as the event bus, facilitating the communication between AWS AppSync and Step Functions. When a client submits a request through the GraphQL API, an event is emitted to EventBridge, invoking a run of the Step Functions workflow.
Step Functions orchestrates the run of three Lambda functions, each responsible for a specific task in the image summarization process:
- Input validator – This Lambda function performs input validation, making sure the provided requests adhere to the expected format. It also handles the upload of the input image assets to an Amazon Simple Storage Service (Amazon S3) bucket designated for raw assets.
- Document reader – This Lambda function retrieves the raw image assets from the input asset bucket, performs image moderation checks using Amazon Rekognition, and uploads the processed assets to an S3 bucket designated for transformed files. This separation of raw and processed assets facilitates auditing and versioning.
- Generate summary – This Lambda function generates a textual summary or description for the processed image assets, using machine learning (ML) models or other image analysis techniques.
The Step Functions workflow orchestrator employs a Map state, enabling parallel runs of multiple image assets. This concurrent processing capability provides optimal resource utilization and minimizes latency, delivering a highly scalable and efficient image summarization solution.
User authentication and authorization are handled by Amazon Cognito, providing secure access management and identity services for the application’s users. This makes sure only authenticated and authorized users can access and interact with the image summarization service. The solution incorporates observability features through integration with Amazon CloudWatch and AWS X-Ray.
The UI for the application is implemented using the Streamlit open source framework, providing a modern and responsive experience for interacting with the image summarization service. You can access the source code for the project in the public GitHub repository.
The following diagram shows the architecture to deliver this use case.
The workflow to generate image descriptions includes the following steps:
- The user uploads the input image to an S3 bucket designated for input assets.
- The upload invokes the image summarization mutation API exposed by AWS AppSync. This will initiate the serverless workflow.
- AWS AppSync publishes an event to EventBridge to invoke the next step in the workflow.
- EventBridge routes the event to a Step Functions state machine.
- The Step Functions state machine invokes a Lambda function that validates the input request parameters.
- Upon successful validation, the Step Functions state machine invokes a document reader Lambda function. This function runs an image moderation check using Amazon Rekognition. If no unsafe or explicit content is detected, it pushes the image to a transformed assets S3 bucket.
- A summary generator Lambda function is invoked, which reads the transformed image. It uses the Amazon Bedrock library to invoke the Anthropic’s Claude 3 Sonnet model, passing the image bytes as input.
- Anthropic’s Claude 3 Sonnet generates a textual description for the input image.
- The summary generator publishes the generated description through an AWS AppSync subscription. The Streamlit UI application listens for events from this subscription and displays the generated description to the user once received.
The following figure illustrates the workflow of the Step Functions state machine.
Prerequisites
To implement this solution, you should have the following prerequisites:
- An AWS account
- The AWS Command Line Interface (AWS CLI) installed and your credentials configured:
- js version v18.12.1
- Python version 3.9 of higher
- AWS CDK version 2.68.0
- jq version jq-1.6
- Docker desktop installed
Build and deploy the solution
Complete the following steps to set up the solution:
- Clone the GitHub repository.
If using HTTPS, use the following code:If using SSH, use the following code:
- Change the directory to the sample solution:
- Update the stage variable to a unique value:
- Open
image-description-stack.ts - Install all dependencies:
- Bootstrap AWS CDK resources on the AWS account. Replace ACCOUNT_ID and REGION with your own values:
- Deploy the solution:
The preceding command deploys the stack in your account. The deployment will take approximately 5 minutes to complete.
- Configure
client_app: - Within the
/client_appdirectory, create a new file named.envwith the following content. Replace the property values with the values retrieved from the stack outputs.
COGNITO_CLIENT_SECRET is a secret value that can be retrieved from the Amazon Cognito console. Navigate to the user pool created by the stack. Under App integration, navigate to App clients and analytics, and choose App client name. Under App client information, choose Show client secret and copy the value of the client secret.
- Run
client_app:
When the client application is up and running, it will open the browser 8501 port (http://localhost:8501/Home).
Make sure your virtual environment is free from SSL certificate issues. If any SSL certificate issues are present, reinstall the CA certificates and OpenSSL package using the following command:
Test the solution
To test the solution, we upload some sample images and generate descriptions in different applications. Complete the following steps:
- In the Streamlit UI, choose Log In and register the user for the first time
- After the user is registered and logged in, choose Image Description in the navigation pane.
- Upload multiple images and select the preferred model configuration ( Anthropic’s Claude 3.5 Sonnet or Anthropic’s Claude 3), then choose Submit.
The uploaded image and the generated description are shown in the center pane.
- Set the language as French in the left pane and upload a new image, then choose Submit.
The image description is generated in French.
Clean up
To avoid incurring unintended charges, delete the resources you created:
- Remove all data from the S3 buckets.
- Run the
CDK destroy - Delete the S3 buckets.
Conclusion
In this post, we discussed how to integrate Amazon Bedrock with Generative AI CDK Constructs. This solution enables the rapid development and deployment of cloud infrastructure tailored for an image description application by using the power of generative AI, specifically Anthropic’s Claude 3. The Generative AI CDK Constructs abstract the intricate complexities of infrastructure, thereby accelerating development timelines.
The Generative AI CDK Constructs Library offers a comprehensive suite of constructs, empowering developers to augment and enhance generative AI capabilities within their applications, unlocking a myriad of possibilities for innovation. Try out the Generative AI CDK Constructs Library for your own use cases, and share your feedback and questions in the comments.
About the Authors
Dinesh Sajwan is a Senior Solutions Architect with the Prototyping Acceleration team at Amazon Web Services. He helps customers to drive innovation and accelerate their adoption of cutting-edge technologies, enabling them to stay ahead of the curve in an ever-evolving technological landscape. Beyond his professional endeavors, Dinesh enjoys a quiet life with his wife and three children.
Justin Lewis leads the Emerging Technology Accelerator at AWS. Justin and his team help customers build with emerging technologies like generative AI by providing open source software examples to inspire their own innovation. He lives in the San Francisco Bay Area with his wife and son.
Alain Krok is a Senior Solutions Architect with a passion for emerging technologies. His past experience includes designing and implementing IIoT solutions for the oil and gas industry and working on robotics projects. He enjoys pushing the limits and indulging in extreme sports when he is not designing software.
Michael Tran is a Sr. Solutions Architect with Prototyping Acceleration team at Amazon Web Services. He provides technical guidance and helps customers innovate by showing the art of the possible on AWS. He specializes in building prototypes in the AI/ML space. You can contact him @Mike_Trann on Twitter.
Use LangChain with PySpark to process documents at massive scale with Amazon SageMaker Studio and Amazon EMR Serverless
Harnessing the power of big data has become increasingly critical for businesses looking to gain a competitive edge. From deriving insights to powering generative artificial intelligence (AI)-driven applications, the ability to efficiently process and analyze large datasets is a vital capability. However, managing the complex infrastructure required for big data workloads has traditionally been a significant challenge, often requiring specialized expertise. That’s where the new Amazon EMR Serverless application integration in Amazon SageMaker Studio can help.
With the introduction of EMR Serverless support for Apache Livy endpoints, SageMaker Studio users can now seamlessly integrate their Jupyter notebooks running sparkmagic kernels with the powerful data processing capabilities of EMR Serverless. This allows SageMaker Studio users to perform petabyte-scale interactive data preparation, exploration, and machine learning (ML) directly within their familiar Studio notebooks, without the need to manage the underlying compute infrastructure. By using the Livy REST APIs, SageMaker Studio users can also extend their interactive analytics workflows beyond just notebook-based scenarios, enabling a more comprehensive and streamlined data science experience within the Amazon SageMaker ecosystem.
In this post, we demonstrate how to leverage the new EMR Serverless integration with SageMaker Studio to streamline your data processing and machine learning workflows.
Benefits of integrating EMR Serverless with SageMaker Studio
The EMR Serverless application integration in SageMaker Studio offers several key benefits that can transform the way your organization approaches big data:
- Simplified infrastructure management – By abstracting away the complexities of setting up and managing Spark clusters, the EMR Serverless integration allows you to quickly spin up the compute resources needed for your big data workloads, without the work of provisioning and configuring the underlying infrastructure.
- Seamless integration with SageMaker – As a built-in feature of the SageMaker platform, the EMR Serverless integration provides a unified and intuitive experience for data scientists and engineers. You can access and utilize this functionality directly within the SageMaker Studio environment, allowing for a more streamlined and efficient development workflow.
- Cost optimization – The serverless nature of the integration means you only pay for the compute resources you use, rather than having to provision and maintain a persistent cluster. This can lead to significant cost savings, especially for workloads with variable or intermittent usage patterns.
- Scalability and performance – The EMR Serverless integration automatically scales the compute resources up or down based on your workload’s demands, making sure you always have the necessary processing power to handle your big data tasks. This flexibility helps optimize performance and minimize the risk of bottlenecks or resource constraints.
- Reduced operational overhead – The EMR Serverless integration with AWS streamlines big data processing by managing the underlying infrastructure, freeing up your team’s time and resources. This feature in SageMaker Studio empowers data scientists, engineers, and analysts to focus on developing data-driven applications, simplifying infrastructure management, reducing costs, and enhancing scalability. By unlocking the potential of your data, this powerful integration drives tangible business results.
Solution overview
SageMaker Studio is a fully integrated development environment (IDE) for ML that enables data scientists and developers to build, train, debug, deploy, and monitor models within a single web-based interface. SageMaker Studio runs inside an AWS managed virtual private cloud (VPC), with network access for SageMaker Studio domains, in this setup configured as VPC-only. SageMaker Studio automatically creates an elastic network interface within your VPC’s private subnet, which connects to the required AWS services through VPC endpoints. This same interface is also used for provisioning EMR clusters. The following diagram illustrates this solution.
An ML platform administrator can manage permissioning for the EMR Serverless integration in SageMaker Studio. The administrator can configure the appropriate privileges by updating the runtime role with an inline policy, allowing SageMaker Studio users to interactively create, update, list, start, stop, and delete EMR Serverless clusters. SageMaker Studio users are presented with built-in forms within the SageMaker Studio UI that don’t require additional configuration to interact with both EMR Serverless and Amazon Elastic Compute Cloud (Amazon EC2) based clusters.
Apache Spark and its Python API, PySpark, empower users to process massive datasets effortlessly by using distributed computing across multiple nodes. These powerful frameworks simplify the complexities of parallel processing, enabling you to write code in a familiar syntax while the underlying engine manages data partitioning, task distribution, and fault tolerance. With scalability as a core strength, Spark and PySpark allow you to handle datasets of virtually any size, eliminating the constraints of a single machine.
Empowering knowledge retrieval and generation with scalable Retrieval Augmented Generation (RAG) architecture is increasingly important in today’s era of ever-growing information. Effectively using data to provide contextual and informative responses has become a crucial challenge. This is where RAG systems excel, combining the strengths of information retrieval and text generation to deliver comprehensive and accurate results. In this post, we explore how to build a scalable and efficient RAG system using the new EMR Serverless integration, Spark’s distributed processing, and an Amazon OpenSearch Service vector database powered by the LangChain orchestration framework. This solution enables you to process massive volumes of textual data, generate relevant embeddings, and store them in a powerful vector database for seamless retrieval and generation.
Authentication mechanism
When integrating EMR Serverless in SageMaker Studio, you can use runtime roles. Runtime roles are AWS Identity and Access Management (IAM) roles that you can specify when submitting a job or query to an EMR Serverless application. These runtime roles provide the necessary permissions for your workloads to access AWS resources, such as Amazon Simple Storage Service (Amazon S3) buckets. When integrating EMR Serverless in SageMaker Studio, you can configure the IAM role to be used by SageMaker Studio. By using EMR runtime roles, you can make sure your workloads have the minimum set of permissions required to access the necessary resources, following the principle of least privilege. This enhances the overall security of your data processing pipelines and helps you maintain better control over the access to your AWS resources.
Cost attribution of EMR Serverless clusters
EMR Serverless clusters created within SageMaker Studio are automatically tagged with system default tags, specifically the domain-arn and user-profile-arn tags. These system-generated tags simplify cost allocation and attribution of Amazon EMR resources. See the following code:
To learn more about enterprise-level cost allocation for ML environments, refer to Set up enterprise-level cost allocation for ML environments and workloads using resource tagging in Amazon SageMaker.
Prerequisites
Before you get started, complete the prerequisite steps in this section.
Create a SageMaker Studio domain
This post walks you through the integration between SageMaker Studio and EMR Serverless using an interactive SageMaker Studio notebook. We assume you already have a SageMaker Studio domain provisioned with a UserProfile and an ExecutionRole. If you don’t have a SageMaker Studio domain available, refer to Quick setup to Amazon SageMaker to provision one.
Create an EMR Serverless job runtime role
EMR Serverless allows you to specify IAM role permissions that an EMR Serverless job run can assume when calling other services on your behalf. This includes access to Amazon S3 for data sources and targets, as well as other AWS resources like Amazon Redshift clusters and Amazon DynamoDB tables. To learn more about creating a role, refer to Create a job runtime role.
The sample following IAM inline policy attached to a runtime role allows EMR Serverless to assume a runtime role that provides access to an S3 bucket and AWS Glue. You can modify the role to include any additional services that EMR Serverless needs to access at runtime. Additionally, make sure you scope down the resources in the runtime policies to adhere to the principle of least privilege.
Lastly, make sure your role has a trust relationship with EMR Serverless:
Optionally, you can create a runtime role and policy using infrastructure as code (IaC), such as with AWS CloudFormation or Terraform, or using the AWS Command Line Interface (AWS CLI).
Update the SageMaker role to allow EMR Serverless access
This one-time task enables SageMaker Studio users to create, update, list, start, stop, and delete EMR Serverless clusters. We begin by creating an inline policy that grants the necessary permissions for these actions on EMR Serverless clusters, then attach the policy to the Studio domain or user profile role:
Update the domain with EMR Serverless runtime roles
SageMaker Studio supports access to EMR Serverless clusters in two ways: in the same account as the SageMaker Studio domain or across accounts.
To interact with EMR Serverless clusters created in the same account as the SageMaker Studio domain, create a file named same-account-update-domain.json:
Then run an update-domain command to allow all users inside a domain to allow users to use the runtime role:
For EMR Serverless clusters created in a different account, create a file named cross-account-update-domain.json:
Then run an update-domain command to allow all users inside a domain to allow users to use the runtime role:
Update the user profile with EMR Serverless runtime roles
Optionally, this update can be applied more granularly at the user profile level instead of the domain level. Similar to domain update, to interact with EMR Serverless clusters created in the same account as the SageMaker Studio domain, create a file named same-account-update-user-profile.json:
Then run an update-user-profile command to allow this user profile use this run time role:
For EMR Serverless clusters created in a different account, create a file named cross-account-update-user-profile.json:
Then run an update-user-profile command to allow all users inside a domain to allow users to use the runtime role:
Grant access to the Amazon ECR repository
The recommended way to customize environments within EMR Serverless clusters is by using custom Docker images.
Make sure you have an Amazon ECR repository in the same AWS Region where you launch EMR Serverless applications. To create an ECR private repository, refer to Creating an Amazon ECR private repository to store images.
To grant users access to your ECR repository, add the following policies to the users and roles that create or update EMR Serverless applications using images from this repository:
Customize the runtime environment in EMR Serverless clusters
Customizing cluster runtimes in advance is crucial for a seamless experience. As mentioned earlier, we use custom-built Docker images from an ECR repository to optimize our cluster environment, including the necessary packages and binaries. The simplest way to build these images is by using the SageMaker Studio built-in Docker functionality, as discussed in Accelerate ML workflows with Amazon SageMaker Studio Local Mode and Docker support. In this post, we build a Docker image that includes the Python 3.11 runtime and essential packages for a typical RAG workflow, such as langchain, sagemaker, opensearch-py, PyPDF2, and more.
Complete the following steps:
- Start by launching a SageMaker Studio JupyterLab notebook.
- Install Docker in your JupyterLab environment. For instructions, refer to Accelerate ML workflows with Amazon SageMaker Studio Local Mode and Docker support.
- Open a new terminal within your JupyterLab environment and verify the Docker installation by running the following:
- Create a Docker file (refer to Using custom images with EMR Serverless) and publish the image to an ECR repository:
- From your JupyterLab terminal, run the following command to log in to the ECR repository:
- Run the following set of Docker commands to build, tag, and push the Docker image to the ECR repository:
Use the EMR Serverless integration with SageMaker Studio
In this section, we demonstrate the integration of EMR Serverless into SageMaker Studio and how you can effortlessly interact with your clusters, whether they are in the same account or across different accounts. To access SageMaker Studio, complete the following steps:
- On the SageMaker console, open SageMaker Studio.
- Depending on your organization’s setup, you can log in to Studio either through the IAM console or using AWS IAM Identity Center.
The new Studio experience is a serverless web UI, which makes sure any updates occur seamlessly and asynchronously, without interrupting your development experience.
- Under Data in the navigation pane, choose EMR Clusters.
You can navigate to two different tabs: EMR Serverless Applications or EMR Clusters (on Amazon EC2). For this post, we focus on EMR Serverless.
Create an EMR Serverless cluster
To create a new EMR Serverless cluster, complete the following steps:
- On the EMR Serverless Applications tab, choose Create.
- In the Network connections section, you can optionally select Connect to your VPC and nest your EMR Serverless cluster within a VPC and private subnet.
- To customize your cluster runtime, choose a compatible custom image from your ECR repository and make sure your user profile role has the necessary permissions to pull from this repository.
Interact with EMR Serverless clusters
EMR Serverless clusters can automatically scale down to zero when not in use, eliminating costs associated with idling resources. This feature makes EMR Serverless clusters highly flexible and cost-effective. You can list, view, create, start, stop, and delete all your EMR Serverless clusters directly within SageMaker Studio.
You can also interactively attach an existing cluster to a notebook by choosing Attach to new notebook.
Build a RAG document processing engine using PySpark
In this section, we use the SageMaker Studio cluster integration to parallelize data processing at a massive scale. A typical RAG framework consists of two main components:
- Offline document embedding generation – This process involves extracting data (text, images, tables, and metadata) from various sources and generating embeddings using a large language embeddings model. These embeddings are then stored in a vector database, such as OpenSearch Service.
- Online text generation with context – During this process, a user’s query is searched against the vector database, and the documents most similar to the query are retrieved. The retrieved documents, along with the user’s query, are combined into an augmented prompt and sent to a large language model (LLM), such as Meta Llama 3 or Anthropic Claude on Amazon Bedrock, for text generation.
In the following sections, we focus on the offline document embedding generation process and explore how to use PySpark on EMR Serverless using an interactive SageMaker Studio JupyterLab notebook to efficiently parallel process PDF documents.
Deploy an embeddings model
For this use case, we use the Hugging Face All MiniLM L6 v2 embeddings model from Amazon SageMaker JumpStart. To quickly deploy this embedding model, complete the following steps:
- In SageMaker Studio, choose JumpStart in the navigation pane.
- Search for and choose All MiniLM L6 v2.
- On the model card, choose Deploy.
Your model will be ready within a few minutes. Alternatively, you can choose any other embedding models from SageMaker JumpStart by filtering Task type to Text embedding.
Interactively build an offline document embedding generator
In this section, we use code from the following GitHub repo and interactively build a document processing engine using LangChain and PySpark. Complete the following steps:
- Create a SageMaker Studio JupyterLab development environment. For more details, see Boost productivity on Amazon SageMaker Studio: Introducing JupyterLab Spaces and generative AI tools.
- Choose an appropriate instance type and EBS storage volume for your development environment.
You can change the instance type at any time by stopping and restarting the space.
- Clone the sample code from the following GitHub repository and use the notebook available under
use-cases/pyspark-langchain-rag-processor/Offline_RAG_Processor_on_SageMaker_Studio_using_EMR-Serverless.ipynb - In SageMaker Studio, under Data in the navigation pane, choose EMR Clusters.
- On the EMR Serverless Applications tab, choose Create to create a cluster.
- Select your cluster and choose Attach to new notebook.
- Attach this cluster to a JupyterLab notebook running inside a space.
Alternatively, you can attach your cluster to any notebook within your JupyterLab space by choosing Cluster and selecting the EMR Serverless cluster you want to attach to the notebook.
Make sure you choose the SparkMagic PySpark kernel when interactively running PySpark workloads.
A successful cluster connection to a notebook should result in a useable Spark session and links to the Spark UI and driver logs.
When a notebook cell is run within a SparkMagic PySpark kernel, the operations are, by default, run inside a Spark cluster. However, if you decorate the cell with %%local, it allows the code to be run on the local compute where the JupyterLab notebook is hosted. We begin by reading a list of PDF documents from Amazon S3 directly into the cluster memory, as illustrated in the following diagram.
- Use the following code to read the documents:
Next, you can visualize the size of each document to understand the volume of data you’re processing.
- You can generate charts and visualize your data within your PySpark notebook cell using static visualization tools like matplotlib and seaborn. See the following code:
Every PDF document contains multiple pages to process, and this task can be run in parallel using Spark. Each document is split page by page, with each page referencing the global in-memory PDFs. We achieve parallelism at the page level by creating a list of pages and processing each one in parallel. The following diagram provides a visual representation of this process.
The extracted text from each page of multiple documents is converted into a LangChain-friendly Document class.
- The
CustomDocumentclass, shown in the following code, is a custom implementation of theDocumentclass that allows you to convert custom text blobs into a format recognized by LangChain. After conversion, the documents are split into chunks and prepared for embedding. - Next, you can use LangChain’s built-in
OpenSearchVectorSearchto create text embeddings. However, we use a customEmbeddingsGeneratorclass that parallelizes (using PySpark) the embeddings generation process using a load-balanced SageMaker hosted embeddings model endpoint:
The custom EmbeddingsGenerator class can generate embeddings for approximately 2,500 pages (12,000 chunks) of documents in under 180 seconds using just two concurrent load-balanced SageMaker embedding model endpoints and 10 PySpark worker nodes. This process can be further accelerated by increasing the number of load-balanced embedding endpoints and worker nodes in the cluster.
Conclusion
The integration of EMR Serverless with SageMaker Studio represents a significant leap forward in simplifying and enhancing big data processing and ML workflows. By eliminating the complexities of infrastructure management, enabling seamless scalability, and optimizing costs, this powerful combination empowers organizations to use petabyte-scale data processing without the overhead typically associated with managing Spark clusters. The streamlined experience within SageMaker Studio enables data scientists and engineers to focus on what truly matters—driving insights and innovation from their data. Whether you’re processing massive datasets, building RAG systems, or exploring other advanced analytics, this integration opens up new possibilities for efficiency and scale, all within the familiar and user-friendly environment of SageMaker Studio.
As data continues to grow in volume and complexity, adopting tools like EMR Serverless and SageMaker Studio will be key to maintaining a competitive edge in the ever-evolving landscape of data-driven decision-making. We encourage you to try this feature today by setting up SageMaker Studio using the SageMaker quick setup guide. To learn more about the EMR Serverless integration with SageMaker Studio, refer to Prepare data using EMR Serverless. You can explore more generative AI samples and use cases in the GitHub repository.
About the authors
Raj Ramasubbu is a Senior Analytics Specialist Solutions Architect focused on big data and analytics and AI/ML with Amazon Web Services. He helps customers architect and build highly scalable, performant, and secure cloud-based solutions on AWS. Raj provided technical expertise and leadership in building data engineering, big data analytics, business intelligence, and data science solutions for over 18 years prior to joining AWS. He helped customers in various industry verticals like healthcare, medical devices, life science, retail, asset management, car insurance, residential REIT, agriculture, title insurance, supply chain, document management, and real estate.
Pranav Murthy is an AI/ML Specialist Solutions Architect at AWS. He focuses on helping customers build, train, deploy and migrate machine learning (ML) workloads to SageMaker. He previously worked in the semiconductor industry developing large computer vision (CV) and natural language processing (NLP) models to improve semiconductor processes using state of the art ML techniques. In his free time, he enjoys playing chess and traveling. You can find Pranav on LinkedIn.
Naufal Mir is an Senior GenAI/ML Specialist Solutions Architect at AWS. He focuses on helping customers build, train, deploy and migrate machine learning (ML) workloads to SageMaker. He previously worked at financial services institutes developing and operating systems at scale. He enjoys ultra endurance running and cycling.
Kunal Jha is a Senior Product Manager at AWS. He is focused on building Amazon SageMaker Studio as the best-in-class choice for end-to-end ML development. In his spare time, Kunal enjoys skiing and exploring the Pacific Northwest. You can find him on LinkedIn.
Ashwin Krishna is a Senior SDE working for SageMaker Studio at Amazon Web Services. He is focused on building interactive ML solutions for AWS enterprise customers to achieve their business needs. He is a big supporter of Arsenal football club and spends spare time playing and watching soccer.
Harini Narayanan is a software engineer at AWS, where she’s excited to build cutting-edge data preparation technology for machine learning at SageMaker Studio. With a keen interest in sustainability, interior design, and a love for all things green, Harini brings a thoughtful approach to innovation, blending technology with her diverse passions.
Best practices for prompt engineering with Meta Llama 3 for Text-to-SQL use cases
With the rapid growth of generative artificial intelligence (AI), many AWS customers are looking to take advantage of publicly available foundation models (FMs) and technologies. This includes Meta Llama 3, Meta’s publicly available large language model (LLM). The partnership between Meta and Amazon signifies collective generative AI innovation, and Meta and Amazon are working together to push the boundaries of what’s possible.
In this post, we provide an overview of the Meta Llama 3 models available on AWS at the time of writing, and share best practices on developing Text-to-SQL use cases using Meta Llama 3 models. All the code used in this post is publicly available in the accompanying Github repository.
Background of Meta Llama 3
Meta Llama 3, the successor to Meta Llama 2, maintains the same 70-billion-parameter capacity but achieves superior performance through enhanced training techniques rather than sheer model size. This approach underscores Meta’s strategy of optimizing data utilization and methodologies to push AI capabilities further. The release includes new models based on Meta Llama 2’s architecture, available in 8-billion- and 70-billion-parameter variants, each offering base and instruct versions. This segmentation allows Meta to deliver versatile solutions suitable for different hardware and application needs.
A significant upgrade in Meta Llama 3 is the adoption of a tokenizer with a 128,256-token vocabulary, enhancing text encoding efficiency for multilingual tasks. The 8-billion-parameter model integrates grouped-query attention (GQA) for improved processing of longer data sequences, enhancing real-world application performance. Training involved a dataset of over 15 trillion tokens across two GPU clusters, significantly more than Meta Llama 2. Meta Llama 3 Instruct, optimized for dialogue applications, underwent fine-tuning with over 10 million human-annotated samples using advanced techniques like proximal policy optimization and supervised fine-tuning. Meta Llama 3 models are licensed permissively, allowing redistribution, fine-tuning, and derivative work creation, now requiring explicit attribution. This licensing update reflects Meta’s commitment to fostering innovation and collaboration in AI development with transparency and accountability.
Prompt engineering best practices for Meta Llama 3
The following are best practices for prompt engineering for Meta Llama 3:
- Base model usage – Base models offer the following:
- Prompt-less flexibility – Base models in Meta Llama 3 excel in continuing sequences and handling zero-shot or few-shot tasks without requiring specific prompt formats. They serve as versatile tools suitable for a wide range of applications and provide a solid foundation for further fine-tuning.
- Instruct versions – Instruct versions offer the following:
- Structured dialogue – Instruct versions of Meta Llama 3 use a structured prompt format designed for dialogue systems. This format maintains coherent interactions by guiding system responses based on user inputs and predefined prompts.
- Text-to-SQL parsing – For tasks like Text-to-SQL parsing, note the following:
- Effective prompt design – Engineers should design prompts that accurately reflect user queries to SQL conversion needs. Meta Llama 3’s capabilities enhance accuracy and efficiency in understanding and generating SQL queries from natural language inputs.
- Development best practices – Keep in mind the following:
- Iterative refinement – Continuous refinement of prompt structures based on real-world data improves model performance and consistency across different applications.
- Validation and testing – Thorough testing and validation make sure that prompt-engineered models perform reliably and accurately across diverse scenarios, enhancing overall application effectiveness.
By implementing these practices, engineers can optimize the use of Meta Llama 3 models for various tasks, from generic inference to specialized natural language processing (NLP) applications like Text-to-SQL parsing, using the model’s capabilities effectively.
Solution overview
The demand for using LLMs to improve Text-to-SQL queries is growing more important because it enables non-technical users to access and query databases using natural language. This democratizes access to generative AI and improves efficiency in writing complex queries without needing to learn SQL or understand complex database schemas. For example, if you’re a financial customer and you have a MySQL database of customer data spanning multiple tables, you could use Meta Llama 3 models to build SQL queries from natural language. Additional use cases include:
- Improved accuracy – LLMs can generate SQL queries that more accurately capture the intent behind natural language queries, thanks to their advanced language understanding capabilities. This reduces the need to rephrase or refine your queries.
- Handling complexity – LLMs can handle complex queries involving multiple tables (which we demonstrate in this post), joins, filters, and aggregations, which would be challenging for rule-based or traditional Text-to-SQL systems. This expands the range of queries that can be handled using natural language.
- Incorporating context – LLMs can use contextual information like database schemas, table descriptions, and relationships to generate more accurate and relevant SQL queries. This helps bridge the gap between ambiguous natural language and precise SQL syntax.
- Scalability – After they’re trained, LLMs can generalize to new databases and schemas without extensive retraining or rule-writing, making them more scalable than traditional approaches.
For the solution, we follow a Retrieval Augmented Generation (RAG) pattern to generate SQL from a natural language query using the Meta Llama 3 70B model on Amazon SageMaker JumpStart, a hub that provides access to pre-trained models and solutions. SageMaker JumpStart provides a seamless and hassle-free way to deploy and experiment with the latest state-of-the-art LLMs like Meta Llama 3, without the need for complex infrastructure setup or deployment code. With just a few clicks, you can have Meta Llama 3 models up and running in a secure AWS environment under your virtual private cloud (VPC) controls, maintaining data security. SageMaker JumpStart offers access to a range of Meta Llama 3 model sizes (8B and 70B parameters). This flexibility allows you to choose the appropriate model size based on your specific requirements. You can also incrementally train and tune these models before deployment.
The solution also includes an embeddings model hosted on SageMaker JumpStart and publicly available vector databases like ChromaDB to store the embeddings.
ChromaDB and other vector engines
In the realm of Text-to-SQL applications, ChromaDB is a powerful, publicly available, embedded vector database designed to streamline the storage, retrieval, and manipulation of high-dimensional vector data. Seamlessly integrating with machine learning (ML) and NLP workflows, ChromaDB offers a robust solution for applications such as semantic search, recommendation systems, and similarity-based analysis. ChromaDB offers several notable features:
- Efficient vector storage – ChromaDB uses advanced indexing techniques to efficiently store and retrieve high-dimensional vector data, enabling fast similarity searches and nearest neighbor queries.
- Flexible data modeling – You can define custom collections and metadata schemas tailored to your specific use cases, allowing for flexible data modeling.
- Seamless integration – ChromaDB can be seamlessly embedded into existing applications and workflows, providing a lightweight and performant solution for vector data management.
Why choose ChromaDB for Text-to-SQL use cases?
- Efficient vector storage for text embeddings – ChromaDB’s efficient storage and retrieval of high-dimensional vector embeddings are crucial for Text-to-SQL tasks. It enables fast similarity searches and nearest neighbor queries on text embeddings, facilitating accurate mapping of natural language queries to SQL statements.
- Seamless integration with LLMs – ChromaDB can be quickly integrated with LLMs, enabling RAG architectures. This allows LLMs to use relevant context, such as providing only the relevant table schemas necessary to fulfill the query.
- Customizable and community support – ChromaDB offers flexibility and customization with an active community of developers and users who contribute to its development, provide support, and share best practices. This provides a collaborative and supportive landscape for Text-to-SQL applications.
- Cost-effective – ChromaDB eliminates the need for expensive licensing fees, making it a cost-effective choice for organizations of all sizes.
By using vector database engines like ChromaDB, you gain more flexibility for your specific use cases and can build robust and performant Text-to-SQL systems for generative AI applications.
Solution architecture
The solution uses the AWS services and features illustrated in the following architecture diagram.
The process flow includes the following steps:
- A user sends a text query specifying the data they want returned from the databases.
- Database schemas, table structures, and their associated metadata are processed through an embeddings model hosted on SageMaker JumpStart to generate embeddings.
- These embeddings, along with additional contextual information about table relationships, are stored in ChromaDB to enable semantic search, allowing the system to quickly retrieve relevant schema and table context when processing user queries.
- The query is sent to ChromaDB to be converted to vector embeddings using a text embeddings model hosted on SageMaker JumpStart. The generated embeddings are used to perform a semantic search on the ChromaDB.
- Following the RAG pattern, ChromaDB outputs the relevant table schemas and table context that pertain to the query. Only relevant context is sent to the Meta Llama 3 70B model. The augmented prompt is created using this information from ChromaDB as well as the user query.
- The augmented prompt is sent to the Meta Llama3 70B model hosted on SageMaker JumpStart to generate the SQL query.
- After the SQL query is generated, you can run the SQL query against Amazon Relational Database Service (Amazon RDS) for MySQL, a fully managed cloud database service that allows you to quickly operate and scale your relational databases like MySQL.
- From there, the output is sent back to the Meta Llama 3 70B model hosted on SageMaker JumpStart to provide a response the user.
- Response sent back to the user.
Depending on where your data lives, you can implement this pattern with other relational database management systems such as PostgreSQL or alternative database types, depending on your existing data infrastructure and specific requirements.
Prerequisites
Complete the following prerequisite steps:
- Have an AWS account.
- Install the AWS Command Line Interface (AWS CLI) and have the Amazon SDK for Python (Boto3) set up.
- Request model access on the Amazon Bedrock console for access to the Meta Llama 3 models.
- Have access to use Jupyter notebooks (whether locally or on Amazon SageMaker Studio).
- Install packages and dependencies for LangChain, the Amazon Bedrock SDK (Boto3), and ChromaDB.
Deploy the Text-to-SQL environment to your AWS account
To deploy your resources, use the provided AWS CloudFormation template, which is a tool for deploying infrastructure as code. Supported AWS Regions are US East (N. Virginia) and US West (Oregon). Complete the following steps to launch the stack:
- On the AWS CloudFormation console, create a new stack.
- For Template source, choose Upload a template file then upload the yaml for deploying the Text-to-SQL environment.
- Choose Next.
- Name the stack
text2sql. - Keep the remaining settings as default and choose Submit.
The template stack should take 10 minutes to deploy. When it’s done, the stack status will show as CREATE_COMPLETE.
- When the stack is complete, navigate to the stack Outputs
- Choose the
SagemakerNotebookURLlink to open the SageMaker notebook in a separate tab.
- In the SageMaker notebook, navigate to the
Meta-Llama-on-AWS/blob/text2sql-blog/RAG-recipesdirectory and openllama3-chromadb-text2sql.ipynb.
- If the notebook prompts you to set the kernel, choose the
conda_pytorch_p310kernel, then choose Set kernel.
Implement the solution
You can use the following Jupyter notebook, which includes all the code snippets provided in this section, to build the solution. In this solution, you can choose which service (SageMaker Jumpstart or Amazon Bedrock) to use as the hosting model service using ask_for_service() in the notebook. Amazon Bedrock is a fully managed service that offers a choice of high-performing FMs. We give you the choice between solutions so that your teams can evaluate if SageMaker JumpStart is preferred or if your teams want to reduce operational overhead with the user-friendly Amazon Bedrock API. You have the choice to use SageMaker JumpStart to host the embeddings model of your choice or Amazon Bedrock to host the Amazon Titan Embeddings model (amazon.titan-embed-text-v2:0).
Now that the notebook is ready to use, follow the instructions in the notebook. With these steps, you create an RDS for MySQL connector, ingest the dataset into an RDS database, ingest the table schemas into ChromaDB, and generate Text-to-SQL queries to run your prompts and analyze data residing in Amazon RDS.
- Create a SageMaker endpoint with the BGE Large En v1.5 Embedding model from Hugging Face:
- Create a collection in ChromaDB for the RAG framework:
- Build the document with the table schema and sample questions to enhance the retriever’s accuracy:
- Add documents to ChromaDB:
- Build the prompt (
final_question) by combining the user input in natural language (user_query), the relevant metadata from the vector store (vector_search_match), and instructions (details): - Submit a question to ChromaDB and retrieve the table schema SQL
- Invoke Meta Llama 3 on SageMaker and prompt it to generate the SQL query. The function
get_llm_sql_analysiswill run and pass the SQL query results to Meta Llama 3 to provide a comprehensive analysis of the data:
Although Meta Llama 3 doesn’t natively support function calling, you can simulate an agentic workflow. In this approach, a query is first generated, then run, and the results are sent back to Meta Llama 3 for interpretation.
Run queries
For our first query, we provide the input “How many unique airplane producers are represented in the database?” The following is the table schema retrieved from ChromaDB:
The following is the generated query:
The following is the data analysis generated from the previous SQL query:
For our second query, we ask “Find the airplane IDs and producers for airplanes that have flown to New York.” The following are the table schemas retrieved from ChromaDB:
The following is our generated query:
The following is the data analysis generated from the previous SQL query:
Clean up
To avoid incurring continued AWS usage charges, delete all the resources you created as part of this post. Make sure you delete the SageMaker endpoints you created within the application before you delete the CloudFormation stack.
Conclusion
In this post, we explored a solution that uses the vector engine ChromaDB and Meta Llama 3, a publicly available FM hosted on SageMaker JumpStart, for a Text-to-SQL use case. We shared a brief history of Meta Llama 3, best practices for prompt engineering with Meta Llama 3 models, and an architecture pattern using few-shot prompting and RAG to extract the relevant schemas stored as vectors in ChromaDB. Finally, we provided a solution with code samples that gives you flexibility to choose SageMaker Jumpstart or Amazon Bedrock for a more managed experience to host Meta Llama 3 70B, Meta Llama3 8B, and embeddings models.
The use of publicly available FMs and services alongside AWS services helps drive more flexibility and provides more control over the tools being used. We recommend following the SageMaker JumpStart GitHub repo for getting started guides and examples. The solution code is also available in the following Github repo.
We look forward to your feedback and ideas on how you apply these calculations for your business needs.
About the Authors
Marco Punio is a Sr. Specialist Solutions Architect focused on generative AI strategy, applied AI solutions, and conducting research to help customers hyperscale on AWS. Marco is based in Seattle, WA, and enjoys writing, reading, exercising, and building applications in his free time.
Armando Diaz is a Solutions Architect at AWS. He focuses on generative AI, AI/ML, and Data Analytics. At AWS, Armando helps customers integrating cutting-edge generative AI capabilities into their systems, fostering innovation and competitive advantage. When he’s not at work, he enjoys spending time with his wife and family, hiking, and traveling the world.
Breanne Warner is an Enterprise Solutions Architect at Amazon Web Services supporting healthcare and life science (HCLS) customers. She is passionate about supporting customers to leverage generative AI and evangelizing model adoption. Breanne is also on the Women@Amazon board as co-director of Allyship with the goal of fostering inclusive and diverse culture at Amazon. Breanne holds a Bachelor of Science in Computer Engineering.
Varun Mehta is a Solutions Architect at AWS. He is passionate about helping customers build enterprise-scale Well-Architected solutions on the AWS Cloud. He works with strategic customers who are using AI/ML to solve complex business problems. Outside of work, he loves to spend time with his wife and kids.
Chase Pinkerton is a Startups Solutions Architect at Amazon Web Services. He holds a Bachelor’s in Computer Science with a minor in Economics from Tufts University. He’s passionate about helping startups grow and scale their businesses. When not working, he enjoys road cycling, hiking, playing volleyball, and photography.
Kevin Lu is a Technical Business Developer intern at Amazon Web Services on the Generative AI team. His work focuses primarily on machine learning research as well as generative AI solutions. He is currently an undergraduate at the University of Pennsylvania, studying computer science and math. Outside of work, he enjoys spending time with friends and family, golfing, and trying new food.
Implementing advanced prompt engineering with Amazon Bedrock
Despite the ability of generative artificial intelligence (AI) to mimic human behavior, it often requires detailed instructions to generate high-quality and relevant content. Prompt engineering is the process of crafting these inputs, called prompts, that guide foundation models (FMs) and large language models (LLMs) to produce desired outputs. Prompt templates can also be used as a structure to construct prompts. By carefully formulating these prompts and templates, developers can harness the power of FMs, fostering natural and contextually appropriate exchanges that enhance the overall user experience. The prompt engineering process is also a delicate balance between creativity and a deep understanding of the model’s capabilities and limitations. Crafting prompts that elicit clear and desired responses from these FMs is both an art and a science.
This post provides valuable insights and practical examples to help balance and optimize the prompt engineering workflow. We specifically focus on advanced prompt techniques and best practices for the models provided in Amazon Bedrock, a fully managed service that offers a choice of high-performing FMs from leading AI companies such as Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API. With these prompting techniques, developers and researchers can harness the full capabilities of Amazon Bedrock, providing clear and concise communication while mitigating potential risks or undesirable outputs.
Overview of advanced prompt engineering
Prompt engineering is an effective way to harness the power of FMs. You can pass instructions within the context window of the FM, allowing you to pass specific context into the prompt. By interacting with an FM through a series of questions, statements, or detailed instructions, you can adjust FM output behavior based on the specific context of the output you want to achieve.
By crafting well-designed prompts, you can also enhance the model’s safety, making sure it generates outputs that align with your desired goals and ethical standards. Furthermore, prompt engineering allows you to augment the model’s capabilities with domain-specific knowledge and external tools without the need for resource-intensive processes like fine-tuning or retraining the model’s parameters. Whether seeking to enhance customer engagement, streamline content generation, or develop innovative AI-powered solutions, harnessing the abilities of prompt engineering can give generative AI applications a competitive edge.
To learn more about the basics of prompt engineering, refer to What is Prompt Engineering?
COSTAR prompting framework
COSTAR is a structured methodology that guides you through crafting effective prompts for FMs. By following its step-by-step approach, you can design prompts tailored to generate the types of responses you need from the FM. The elegance of COSTAR lies in its versatility—it provides a robust foundation for prompt engineering, regardless of the specific technique or approach you employ. Whether you’re using few-shot learning, chain-of-thought prompting, or another method (covered later in this post), the COSTAR framework equips you with a systematic way to formulate prompts that unlock the full potential of FMs.
COSTAR stands for the following:
- Context – Providing background information helps the FM understand the specific scenario and provide relevant responses
- Objective – Clearly defining the task directs the FM’s focus to meet that specific goal
- Style – Specifying the desired writing style, such as emulating a famous personality or professional expert, guides the FM to align its response with your needs
- Tone – Setting the tone makes sure the response resonates with the required sentiment, whether it be formal, humorous, or empathetic
- Audience – Identifying the intended audience tailors the FM’s response to be appropriate and understandable for specific groups, such as experts or beginners
- Response – Providing the response format, like a list or JSON, makes sure the FM outputs in the required structure for downstream tasks
By breaking down the prompt creation process into distinct stages, COSTAR empowers you to methodically refine and optimize your prompts, making sure every aspect is carefully considered and aligned with your specific goals. This level of rigor and deliberation ultimately translates into more accurate, coherent, and valuable outputs from the FM.
Chain-of-thought prompting
Chain-of-thought (CoT) prompting is an approach that improves the reasoning abilities of FMs by breaking down complex questions or tasks into smaller, more manageable steps. It mimics how humans reason and solve problems by systematically breaking down the decision-making process. With traditional prompting, a language model attempts to provide a final answer directly based on the prompt. However, in many cases, this may lead to suboptimal or incorrect responses, especially for tasks that require multistep reasoning or logical deductions.
CoT prompting addresses this issue by guiding the language model to explicitly lay out its step-by-step thought process, known as a reasoning chain, before arriving at the final answer. This approach makes the model’s reasoning process more transparent and interpretable. This technique has been shown to significantly improve performance on tasks that require multistep reasoning, logical deductions, or complex problem-solving. Overall, CoT prompting is a powerful technique that uses the strengths of FMs while mitigating their weaknesses in complex reasoning tasks, ultimately leading to more reliable and well-reasoned outputs.
Let’s look at some examples of CoT prompting with its different variants.
CoT with zero-shot prompting
The first example is a zero-shot CoT prompt. Zero-shot prompting is a technique that doesn’t include a desired output example in the initial prompt.
The following example uses Anthropic’s Claude in Amazon Bedrock. XML tags are used to provide further context in the prompt. Although Anthropic Claude can understand the prompt in a variety of formats, it was trained using XML tags. In this case, there are typically better quality and latency results if we use this tagging structure to add further instructions in the prompt. For more information on how to provide additional context or instructions, refer to the relevant documentation for the FM you are using.
You can use Amazon Bedrock to send Anthropic Claude Text Completions API or Anthropic Claude Messages API inference requests, as seen in the following examples. See the full documentation at Anthropic Claude models.
We enter the following prompt:
As you can see in the example, the FM provided reasoning using the <thinking></thinking> tags to produce the final answer. This additional context allows us to perform further experimentation by tweaking the prompt instructions.
CoT with few-shot prompting
Few-shot prompting is a technique that includes a desired output example in the initial prompt. The following example includes a simple CoT sample response to help the model answer the follow-up question. Few-shot prompting examples can be defined in a prompt catalog or template, which is discussed later in this post.
The following is our standard few-shot prompt (not CoT prompting):
We get the following response:
Although this response is correct, we may want to know the number of goldfish and rainbow fish that are left. Therefore, we need to be more specific in how we want to structure the output. We can do this by adding a thought process we want the FM to mirror in our example answer.
The following is our CoT prompt (few-shot):
We get the following correct response:
Self-consistency prompting
To further improve your CoT prompting abilities, you can generate multiple responses that are aggregated and select the most common output. This is known as self-consistency prompting. Self-consistency prompting requires sampling multiple, diverse reasoning paths through few-shot CoT. It then uses the generations to select the most consistent answer. Self-consistency with CoT is proven to outperform standard CoT because selecting from multiple responses usually leads to a more consistent solution.
If there is uncertainty in the response or if the results disagree significantly, either a human or an overarching FM (see the prompt chaining section in this post) can review each outcome and select the most logical choice.
For further details on self-consistency prompting with Amazon Bedrock, see Enhance performance of generative language models with self-consistency prompting on Amazon Bedrock.
Tree of Thoughts prompting
Tree of Thoughts (ToT) prompting is a technique used to improve FM reasoning capabilities by breaking down larger problem statements into a treelike format, where each problem is divided into smaller subproblems. Think of this as a tree structure: the tree begins with a solid trunk (representing the main topic) and then separates into smaller branches (smaller questions or topics).
This approach allows the FMs to self-evaluate. The model is prompted to reason through each subtopic and combine the solutions to arrive at the final answer. The ToT outputs are then combined with search algorithms, such as breadth-first search (BFS) and depth-first search (DFS), which allows you to traverse forward and backward through each topic in the tree. According to Tree of Thoughts: Deliberate Problem Solving with Large Language Models, ToT substantially outperforms other prompting methods.
One method of using ToT is to ask the LMM to evaluate whether each thought in the tree is logical, possible, or impossible if you’re solving a complex problem. You can also apply ToT prompting in other use cases. For example, if you ask an FM, “What are the effects of climate change?” you can use ToT to help break this topic down into subtopics such as “list the environmental effects” and “list the social effects.”
The following example uses the ToT prompting technique to allow Claude 3 Sonnet to solve where the ball is hidden. The FM can take the ToT output (subproblems 1–5) and formulate a final answer.
We use the following prompt:
We get the following response:
Using the ToT prompting technique, the FM has broken down the problem of, “Where is the ball?” into a set of subproblems that are simpler to answer. We typically see more logical results with this prompting approach compared to a zero-shot direct question such as, “Where is the ball?”
Differences between CoT and ToT
The following table summarizes the key differences between ToT and CoT prompting.
| CoT | ToT | |
| Structure | CoT prompting follows a linear chain of reasoning steps. | ToT prompting has a hierarchical, treelike structure with branching subproblems. |
| Depth | CoT can use the self-consistency method for increased understanding. | ToT prompting encourages the FM to reason more deeply by breaking down subproblems into smaller ones, allowing for more granular reasoning. |
| Complexity | CoT is a simpler approach, requiring less effort than ToT. | ToT prompting is better suited for handling more complex problems that require reasoning at multiple levels or considering multiple interrelated factors. |
| Visualization | CoT is simple to visualize because it follows a linear trajectory. If using self-consistency, it may require multiple reruns. | The treelike structure of ToT prompting can be visually represented in a tree structure, making it straightforward to understand and analyze the reasoning process. |
The following diagram visualizes the discussed techniques.
Prompt chaining
Building on the discussed prompting techniques, we now explore prompt chaining methods, which are useful in handling more advanced problems. In prompt chaining, the output of an FM is passed as input to another FM in a predefined sequence of N models, with prompt engineering between each step. This allows you to break down complex tasks and questions into subtopics, each as a different input prompt to a model. You can use ToT, CoT, and other prompting techniques with prompt chaining.
Amazon Bedrock Prompt Flows can orchestrate the end-to-end prompt chaining workflow, allowing users to input prompts in a logical sequence. These features are designed to accelerate the development, testing, and deployment of generative AI applications so developers and business users can create more efficient and effective solutions that are simple to maintain. You can use prompt management and flows graphically in the Amazon Bedrock console or Amazon Bedrock Studio or programmatically through the Amazon Bedrock AWS SDK APIs.
Other options for prompt chaining include using third-party LangChain libraries or LangGraph, which can manage the end-to-end orchestration. These are third-party frameworks designed to simplify the creation of applications using FMs.
The following diagram showcases how a prompt chaining flow can work:
The following example uses prompt chaining to perform a legal case review.
Prompt 1:
Response 1:
We then provide a follow-up prompt and question.
Prompt 2:
Response 2:
The following is a final prompt and question.
Prompt 3:
Response 3 (final output):
To get started with hands-on examples of prompt chaining, refer to the GitHub repo.
Prompt catalogs
A prompt catalog, also known as a prompt library, is a collection of prewritten prompts and prompt templates that you can use as a starting point for various natural language processing (NLP) tasks, such as text generation, question answering, or data analysis. By using a prompt catalog, you can save time and effort crafting prompts from scratch and instead focus on fine-tuning or adapting the existing prompts to your specific use cases. This approach also assists with consistency and re-usability, as the template can be shared across teams within an organization.
Prompt Management for Amazon Bedrock consists of a prompt builder, a prompt library (catalog), versioning, and testing methods for prompt templates. For more information on how to orchestrate the prompt flow by using Prompt Management for Amazon Bedrock, refer to Advanced prompts in Amazon Bedrock.
The following example uses a prompt template to structure the FM response.
Prompt template:
Sample prompt:
Model response:
For further examples of prompting templates, refer to the following resources:
- Prompt templates and examples for Amazon Bedrock text models
- The Anthropic Claude Prompt Library, a collection of Anthropic Claude optimized prompts for a breadth of tasks
- LangSmith Prompt Hub, where you can discover, share, and version prompts for LangChain and FMs in general
Prompt misuses
When building and designing a generative AI application, it’s crucial to understand FM vulnerabilities regarding prompt engineering. This section covers some of the most common types of prompt misuses so you can adopt security in the design from the beginning.
FMs available through Amazon Bedrock already provide built-in protections to prevent the generation of harmful responses. However, it’s best practice to add additional, personalized prompt security measures, such as with Guardrails for Amazon Bedrock. Refer to the prompt defense techniques section in this post to learn more about handling these use cases.
Prompt injection
Prompt injection attacks involve injecting malicious or unintended prompts into the system, potentially leading to the generation of harmful, biased, or unauthorized outputs from the FM. In this case, an unauthorized user crafts a prompt to trick the FM into running unintended actions or revealing sensitive information. For example, an unauthorized user could inject a prompt that instructs the FM to ignore or bypass security filters such as XML tags, allowing the generation of offensive or illegal content. For examples, refer to Hugging Face prompt-injections.
The following is an example attacker prompt:
Prompt leaking
Prompt leaking can be considered a form of prompt injection. Prompt leaking occurs when an unauthorized user aims to leak the details or instructions from the original prompt. This attack can expose behind-the-scenes prompt data or instructions in the response back to the user. For example:
Jailbreaking
Jailbreaking, in the context of prompt engineering security, refers to an unauthorized user attempting to bypass the ethical and safety constraints imposed on the FM. This can lead it to generate unintended responses. For example:
Alternating languages and special characters
Alternating languages in the input prompt increases the chance of confusing the FM with conflicting instructions or bypassing certain FM guardrails (see more on FM guardrails in the prompt defense techniques section). This also applies to the use of special characters in a prompt, such as , +, → or !—, which is an attempt to get the FM to forget its original instructions.
The following is an example of a prompt misuse. The text in the brackets represents a language other than English:
For more information on prompt misuses, refer to Common prompt injection attacks.
Prompt defense techniques
This section discusses how to help prevent these misuses of FM responses by putting security mechanisms in place.
Guardrails for Amazon Bedrock
FM guardrails help to uphold data privacy and provide safe and reliable model outputs by preventing the generation of harmful or biased content. Guardrails for Amazon Bedrock evaluates user inputs and FM responses based on use case–specific policies and provides an additional layer of safeguards regardless of the underlying FM. You can apply guardrails across FMs on Amazon Bedrock, including fine-tuned models. This additional layer of security detects harmful instructions in an incoming prompt and catches it before the event reaches the FM. You can customize your guardrails based on your internal AI policies.
For examples of the differences between responses with or without guardrails in place, refer this Comparison table. For more information, see How Guardrails for Amazon Bedrock works.
Use unique delimiters to wrap prompt instructions
As highlighted in some of the examples, prompt engineering techniques can use delimiters (such as XML tags) in their template. Some prompt injection attacks try to take advantage of this structure by wrapping malicious instructions in common delimiters, leading the model to believe that the instruction was part of its original template. By using a unique delimiter value (for example, <tagname-abcde12345>), you can make sure the FM will only consider instructions that are within these tags. For more information, refer to Best practices to avoid prompt injection attacks.
Detect threats by providing specific instructions
You can also include instructions that explain common threat patterns to teach the FM how to detect malicious events. The instructions focus on the user input query. They instruct the FM to identify the presence of key threat patterns and return “Prompt Attack Detected” if it discovers a pattern. These instructions serve as a shortcut for the FM to deal with common threats. This shortcut is mostly relevant when the template uses delimiters, such as the <thinking></thinking> and <answer></answer> tags.
For more information, see Prompt engineering best practices to avoid prompt injection attacks on modern LLMs.
Prompt engineering best practices
In this section, we summarize prompt engineering best practices.
Clearly define prompts using COSTAR framework
Craft prompts in a way that leaves minimal room for misinterpretation by using the discussed COSTAR framework. It’s important to explicitly state the type of response expected, such as a summary, analysis, or list. For example, if you ask for a novel summary, you need to clearly indicate that you want a concise overview of the plot, characters, and themes rather than a detailed analysis.
Sufficient prompt context
Make sure that there is sufficient context within the prompt and, if possible, include an example output response (few-shot technique) to guide the FM toward the desired format and structure. For instance, if you want a list of the most popular movies from the 1990s presented in a table format, you need to explicitly state the number of movies to list and specify that the output should be in a table. This level of detail helps the FM understand and meet your expectations.
Balance simplicity and complexity
Remember that prompt engineering is an art and a science. It’s important to balance simplicity and complexity in your prompts to avoid vague, unrelated, or unexpected responses. Overly simple prompts may lack the necessary context, whereas excessively complex prompts can confuse the FM. This is particularly important when dealing with complex topics or domain-specific language that may be less familiar to the LM. Use plain language and delimiters (such as XML tags if your FM supports them) and break down complex topics using the techniques discussed to enhance FM understanding.
Iterative experimentation
Prompt engineering is an iterative process that requires experimentation and refinement. You may need to try multiple prompts or different FMs to optimize for accuracy and relevance. Continuously test, analyze, and refine your prompts, reducing their size or complexity as needed. You can also experiment with adjusting the FM temperature setting. There are no fixed rules for how FMs generate output, so flexibility and adaptability are essential for achieving the desired results.
Prompt length
Models are better at using information that occurs at the very beginning or end of its prompt context. Performance can degrade when models must access and use information located in the middle of its prompt context. If the prompt input is very large or complex, it should be broken down using the discussed techniques. For more details, refer to Lost in the Middle: How Language Models Use Long Contexts.
Tying it all together
Let’s bring the overall techniques we’ve discussed together into a high-level architecture to showcase a full end-to-end prompting workflow. The overall workflow may look similar to the following diagram.
The workflow consists of the following steps:
- Prompting – The user decides which prompt engineering techniques they want to adopt. They then send the prompt request to the generative AI application and wait for a response. A prompt catalog can also be used during this step.
- Input guardrails (Amazon Bedrock) – A guardrail combines a single policy or multiple policies configured for prompts, including content filters, denied topics, sensitive information filters, and word filters. The prompt input is evaluated against the configured policies specified in the guardrail. If the input evaluation results in a guardrail intervention, a configured blocked message response is returned, and the FM inference is discarded.
- FM and LLM built-in guardrails – Most modern FM providers are trained with security protocols and have built-in guardrails to prevent inappropriate use. It is best practice to also create and establish an additional security layer using Guardrails for Amazon Bedrock.
- Output guardrails (Amazon Bedrock) – If the response results in a guardrail intervention or violation, it will be overridden with preconfigured blocked messaging or masking of the sensitive information. If the response’s evaluation succeeds, the response is returned to the application without modifications.
- Final output – The response is returned to the user.
Cleanup
Running the lab in the GitHub repo referenced in the conclusion is subject to Amazon Bedrock inference charges. For more information about pricing, see Amazon Bedrock Pricing.
Conclusion
Ready to get hands-on with these prompting techniques? As a next step, refer to our GitHub repo. This workshop contains examples of the prompting techniques discussed in this post using FMs in Amazon Bedrock as well as deep-dive explanations.
We encourage you to implement the discussed prompting techniques and best practices when developing a generative AI application. For more information about advanced prompting techniques, see Prompt engineering guidelines.
Happy prompting!
About the Authors
Jonah Craig is a Startup Solutions Architect based in Dublin, Ireland. He works with startup customers across the UK and Ireland and focuses on developing AI and machine learning (AI/ML) and generative AI solutions. Jonah has a master’s degree in computer science and regularly speaks on stage at AWS conferences, such as the annual AWS London Summit and the AWS Dublin Cloud Day. In his spare time, he enjoys creating music and releasing it on Spotify.
Manish Chugh is a Principal Solutions Architect at AWS based in San Francisco, CA. He specializes in machine learning and generative AI. He works with organizations ranging from large enterprises to early-stage startups on problems related to machine learning. His role involves helping these organizations architect scalable, secure, and cost-effective machine learning workloads on AWS. He regularly presents at AWS conferences and other partner events. Outside of work, he enjoys hiking on East Bay trails, road biking, and watching (and playing) cricket.
Doron Bleiberg is a Senior Startup Solutions Architect at AWS, based in Tel Aviv, Israel. In his role, Doron provides FinTech startups with technical guidance and support using AWS Cloud services. With the advent of generative AI, Doron has helped numerous startups build and deploy generative AI workloads in the AWS Cloud, such as financial chat assistants, automated support agents, and personalized recommendation systems.
Accelerate Generative AI Inference with NVIDIA NIM Microservices on Amazon SageMaker
This post is co-written with Eliuth Triana, Abhishek Sawarkar, Jiahong Liu, Kshitiz Gupta, JR Morgan and Deepika Padmanabhan from NVIDIA.
At the 2024 NVIDIA GTC conference, we announced support for NVIDIA NIM Inference Microservices in Amazon SageMaker Inference. This integration allows you to deploy industry-leading large language models (LLMs) on SageMaker and optimize their performance and cost. The optimized prebuilt containers enable the deployment of state-of-the-art LLMs in minutes instead of days, facilitating their seamless integration into enterprise-grade AI applications.
NIM is built on technologies like NVIDIA TensorRT, NVIDIA TensorRT-LLM, and vLLM. NIM is engineered to enable straightforward, secure, and performant AI inferencing on NVIDIA GPU-accelerated instances hosted by SageMaker. This allows developers to take advantage of the power of these advanced models using SageMaker APIs and just a few lines of code, accelerating the deployment of cutting-edge AI capabilities within their applications.
NIM, part of the NVIDIA AI Enterprise software platform listed on AWS Marketplace, is a set of inference microservices that bring the power of state-of-the-art LLMs to your applications, providing natural language processing (NLP) and understanding capabilities, whether you’re developing chatbots, summarizing documents, or implementing other NLP-powered applications. You can use pre-built NVIDIA containers to host popular LLMs that are optimized for specific NVIDIA GPUs for quick deployment. Companies like Amgen, A-Alpha Bio, Agilent, and Hippocratic AI are among those using NVIDIA AI on AWS to accelerate computational biology, genomics analysis, and conversational AI.
In this post, we provide a walkthrough of how customers can use generative artificial intelligence (AI) models and LLMs using NVIDIA NIM integration with SageMaker. We demonstrate how this integration works and how you can deploy these state-of-the-art models on SageMaker, optimizing their performance and cost.
You can use the optimized pre-built NIM containers to deploy LLMs and integrate them into your enterprise-grade AI applications built with SageMaker in minutes, rather than days. We also share a sample notebook that you can use to get started, showcasing the simple APIs and few lines of code required to harness the capabilities of these advanced models.
Solution overview
Getting started with NIM is straightforward. Within the NVIDIA API catalog, developers have access to a wide range of NIM optimized AI models that you can use to build and deploy your own AI applications. You can get started with prototyping directly in the catalog using the GUI (as shown in the following screenshot) or interact directly with the API for free.
To deploy NIM on SageMaker, you need to download NIM and subsequently deploy it. You can initiate this process by choosing Run Anywhere with NIM for the model of your choice, as shown in the following screenshot.
You can sign up for the free 90-day evaluation license on the API Catalog by signing up with your organization email address. This will grant you a personal NGC API key for pulling the assets from NGC and running on SageMaker. For pricing details on SageMaker, refer to Amazon SageMaker pricing.
Prerequisites
As a prerequisite, set up an Amazon SageMaker Studio environment:
- Make sure the existing SageMaker domain has Docker access enabled. If not, run the following command to update the domain:
- After Docker access is enabled for the domain, create a user profile by running the following command:
- Create a JupyterLab space for the user profile you created.
- After you create the JupyterLab space, run the following bash script to install the Docker CLI.
Set up your Jupyter notebook environment
For this series of steps, we use a SageMaker Studio JupyterLab notebook. You also need to attach an Amazon Elastic Block Store (Amazon EBS) volume of at least 300 MB in size, which you can do in the domain settings for SageMaker Studio. In this example, we use an ml.g5.4xlarge instance, powered by a NVIDIA A10G GPU.
We start by opening the example notebook provided on our JupyterLab instance, import the corresponding packages, and set up the SageMaker session, role, and account information:
Pull the NIM container from the public container to push it to your private container
The NIM container that comes with SageMaker integration built in is available in the Amazon ECR Public Gallery. To deploy it on your own SageMaker account securely, you can pull the Docker container from the public Amazon Elastic Container Registry (Amazon ECR) container maintained by NVIDIA and re-upload it to your own private container:
Set up the NVIDIA API key
NIMs can be accessed using the NVIDIA API catalog. You just need to register for an NVIDIA API key from the NGC catalog by choosing Generate Personal Key.
When creating an NGC API key, choose at least NGC Catalog on the Services Included dropdown menu. You can include more services if you plan to reuse this key for other purposes.
For the purposes of this post, we store it in an environment variable:
NGC_API_KEY = YOUR_KEY
This key is used to download pre-optimized model weights when running the NIM.
Create your SageMaker endpoint
We now have all the resources prepared to deploy to a SageMaker endpoint. Using your notebook after setting up your Boto3 environment, you first need to make sure you reference the container you pushed to Amazon ECR in an earlier step:
After the model definition is set up correctly, the next step is to define the endpoint configuration for deployment. In this example, we deploy the NIM on one ml.g5.4xlarge instance:
Lastly, create the SageMaker endpoint:
Run inference against the SageMaker endpoint with NIM
After the endpoint is deployed successfully, you can run requests against the NIM-powered SageMaker endpoint using the REST API to try out different questions and prompts to interact with the generative AI models:
That’s it! You now have an endpoint in service using NIM on SageMaker.
NIM licensing
NIM is part of the NVIDIA Enterprise License. NIM comes with a 90-day evaluation license to start with. To use NIMs on SageMaker beyond the 90-day license, connect with NVIDIA for AWS Marketplace private pricing. NIM is also available as a paid offering as part of the NVIDIA AI Enterprise software subscription available on AWS Marketplace
Conclusion
In this post, we showed you how to get started with NIM on SageMaker for pre-built models. Feel free to try it out following the example notebook.
We encourage you to explore NIM to adopt it to benefit your own use cases and applications.
About the Authors
Saurabh Trikande is a Senior Product Manager for Amazon SageMaker Inference. He is passionate about working with customers and is motivated by the goal of democratizing machine learning. He focuses on core challenges related to deploying complex ML applications, multi-tenant ML models, cost optimizations, and making deployment of deep learning models more accessible. In his spare time, Saurabh enjoys hiking, learning about innovative technologies, following TechCrunch, and spending time with his family.
James Park is a Solutions Architect at Amazon Web Services. He works with Amazon.com to design, build, and deploy technology solutions on AWS, and has a particular interest in AI and machine learning. In his spare time, he enjoys seeking out new cultures, new experiences, and staying up to date with the latest technology trends. You can find him on LinkedIn.
Qing Lan is a Software Development Engineer in AWS. He has been working on several challenging products in Amazon, including high performance ML inference solutions and high-performance logging systems. Qing’s team successfully launched the first billion-parameter model in Amazon Advertising with very low latency required. Qing has in-depth knowledge on infrastructure optimization and deep learning acceleration.
Raghu Ramesha is a Senior GenAI/ML Solutions Architect on the Amazon SageMaker Service team. He focuses on helping customers build, deploy, and migrate ML production workloads to SageMaker at scale. He specializes in machine learning, AI, and computer vision domains, and holds a master’s degree in computer science from UT Dallas. In his free time, he enjoys traveling and photography.
Eliuth Triana is a Developer Relations Manager at NVIDIA empowering Amazon’s AI MLOps, DevOps, Scientists and AWS technical experts to master the NVIDIA computing stack for accelerating and optimizing Generative AI Foundation models spanning from data curation, GPU training, model inference and production deployment on AWS GPU instances. In addition, Eliuth is a passionate mountain biker, skier, tennis and poker player.
Abhishek Sawarkar is a product manager in the NVIDIA AI Enterprise team working on integrating NVIDIA AI Software in Cloud MLOps platforms. He focuses on integrating the NVIDIA AI end-to-end stack within Cloud platforms & enhancing user experience on accelerated computing.
Jiahong Liu is a Solutions Architect on the Cloud Service Provider team at NVIDIA. He assists clients in adopting machine learning and AI solutions that leverage NVIDIA-accelerated computing to address their training and inference challenges. In his leisure time, he enjoys origami, DIY projects, and playing basketball.
Kshitiz Gupta is a Solutions Architect at NVIDIA. He enjoys educating cloud customers about the GPU AI technologies NVIDIA has to offer and assisting them with accelerating their machine learning and deep learning applications. Outside of work, he enjoys running, hiking, and wildlife watching.
JR Morgan is a Principal Technical Product Manager in NVIDIA’s Enterprise Product Group, thriving at the intersection of partner services, APIs, and open source. After work, he can be found on a Gixxer, at the beach, or spending time with his amazing family.
Deepika Padmanabhan is a Solutions Architect at NVIDIA. She enjoys building and deploying NVIDIA’s software solutions in the cloud. Outside work, she enjoys solving puzzles and playing video games like Age of Empires.