Monthly Archives: November 2023

This post is co-written with Marc Neumann, Amor Steinberg and Marinus Krommenhoek from BMW Group.

The BMW Group – headquartered in Munich, Germany – is driven by 149,000 employees worldwide and manufactures in over 30 production and assembly facilities across 15 countries. Today, the BMW Group is the world’s leading manufacturer of premium automobiles and motorcycles, and provider of premium financial and mobility services. The BMW Group sets trends in production technology and sustainability as an innovation leader with an intelligent material mix, a technological shift towards digitalization, and resource-efficient production.

In an increasingly digital and rapidly changing world, BMW Group’s business and product development strategies rely heavily on data-driven decision-making. With that, the need for data scientists and machine learning (ML) engineers has grown significantly. These skilled professionals are tasked with building and deploying models that improve the quality and efficiency of BMW’s business processes and enable informed leadership decisions.

Data scientists and ML engineers require capable tooling and sufficient compute for their work. Therefore, BMW established a centralized ML/deep learning infrastructure on premises several years ago and continuously upgraded it. To pave the way for the growth of AI, BMW Group needed to make a leap regarding scalability and elasticity while reducing operational overhead, software licensing, and hardware management.

In this post, we will talk about how BMW Group, in collaboration with AWS Professional Services, built its Jupyter Managed (JuMa) service to address these challenges. JuMa is a service of BMW Group’s AI platform for its data analysts, ML engineers, and data scientists that provides a user-friendly workspace with an integrated development environment (IDE). It is powered by Amazon SageMaker Studio and provides JupyterLab for Python and Posit Workbench for R. This offering enables BMW ML engineers to perform code-centric data analytics and ML, increases developer productivity by providing self-service capability and infrastructure automation, and tightly integrates with BMW’s centralized IT tooling landscape.

JuMa is now available to all data scientists, ML engineers, and data analysts at BMW Group. The service streamlines ML development and production workflows (MLOps) across BMW by providing a cost-efficient and scalable development environment that facilitates seamless collaboration between data science and engineering teams worldwide. This results in faster experimentation and shorter idea validation cycles. Moreover, the JuMa infrastructure, which is based on AWS serverless and managed services, helps reduce operational overhead for DevOps teams and allows them to focus on enabling use cases and accelerating AI innovation at BMW Group.

Challenges of growing an on-premises AI platform

Prior to introducing the JuMa service, BMW teams worldwide were using two on-premises platforms that provided teams JupyterHub and RStudio environments. These platforms were too limited regarding CPU, GPU, and memory to allow the scalability of AI at BMW Group. Scaling these platforms with managing more on-premises hardware, more software licenses, and support fees would require significant up-front investments and high efforts for its maintenance. To add to this, limited self-service capabilities were available, requiring high operational effort for its DevOps teams. More importantly, the use of these platforms was misaligned with BMW Group’s IT cloud-first strategy. For example, teams using these platforms missed an easy migration of their AI/ML prototypes to the industrialization of the solution running on AWS. In contrast, the data science and analytics teams already using AWS directly for experimentation needed to also take care of building and operating their AWS infrastructure while ensuring compliance with BMW Group’s internal policies, local laws, and regulations. This included a range of configuration and governance activities from ordering AWS accounts, limiting internet access, using allowed listed packages to keeping their Docker images up to date.

Overview of solution

JuMa is a fully managed multi-tenant, security hardened AI platform service built on AWS with SageMaker Studio at the core. By relying on AWS serverless and managed services as the main building blocks of the infrastructure, the JuMa DevOps team doesn’t need to worry about patching servers, upgrading storage, or managing any other infrastructure components. The service handles all those processes automatically, providing a powerful technical platform that is generally up to date and ready to use.

JuMa users can effortlessly order a workspace via a self-service portal to create a secure and isolated development and experimentation environment for their teams. After a JuMa workspace is provisioned, the users can launch JupyterLab or Posit workbench environments in SageMaker Studio with just a few clicks and start the development immediately, using the tools and frameworks they are most familiar with. JuMa is tightly integrated with a range of BMW Central IT services, including identity and access management, roles and rights management, BMW Cloud Data Hub (BMW’s data lake on AWS) and on-premises databases. The latter helps AI/ML teams seamlessly access required data, given they are authorized to do so, without needing to build data pipelines. Furthermore, the notebooks can be integrated into the corporate Git repositories to collaborate using version control.

The solution abstracts away all technical complexities associated with AWS account management, configuration, and customization for AI/ML teams, allowing them to fully focus on AI innovation. The platform ensures that the workspace configuration meets BMW’s security and compliance requirements out of the box.

The following diagram describes the high-level context view of the architecture.

User journey

BMW AI/ML team members can order their JuMa workspace using BMW’s standard catalog service. After approval by the line manager, the ordered JuMa workspace is provisioned by the platform fully automatedly. The workspace provisioning workflow includes the following steps (as numbered in the architecture diagram).

1. A data scientist team orders a new JuMa workspace in BMW’s Catalog. JuMa automatically provisions a new AWS account for the workspace. This ensures full isolation between the workspaces following the federated model account structure mentioned in SageMaker Studio Administration Best Practices.
2. JuMa configures a workspace (which is a Sagemaker domain) that only allows predefined Amazon SageMaker features required for experimentation and development, specific custom kernels, and lifecycle configurations. It also sets up the required subnets and security groups that ensure the notebooks run in a secure environment.
3. After the workspaces are provisioned, the authorized users log in to the JuMa portal and access the SageMaker Studio IDE within their workspace using a SageMaker pre-signed URL. Users can choose between opening a SageMaker Studio private space or a shared space. Shared spaces encourage collaboration between different members of a team that can work in parallel on the same notebooks, whereas private spaces allow for a development environment for solitary workloads.
4. Using the BMW data portal, users can request access to on-premises databases or data stored in BMW’s Cloud Data Hub, making it available in their workspace for development and experimentation, from data preparation and analysis to model training and validation.

After an AI model is developed and validated in JuMa, AI teams can use the MLOPs service of the BMW AI platform to deploy it to production quickly and effortlessly. This service provides users with a production-grade ML infrastructure and pipelines on AWS using SageMaker, which can be set up in minutes with just a few clicks. Users simply need to host their model on the provisioned infrastructure and customize the pipeline to meet their specific use case needs. In this way, the AI platform covers the entire AI lifecycle at BMW Group.

JuMa features

Following best practice architecting on AWS, the JuMa service was designed and implemented according to the AWS Well-Architected Framework. Architectural decisions of each Well-Architected pillar are described in detail in the following sections.

Security and compliance

To assure full isolation between the tenants, each workspace receives its own AWS account, where the authorized users can jointly collaborate on analytics tasks as well as on developing and experimenting with AI/ML models. The JuMa portal itself enforces isolation at runtime using policy-based isolation with AWS Identity and Access Management (IAM) and the JuMa user’s context. For more information about this strategy, refer to Run-time, policy-based isolation with IAM.

Data scientists can only access their domain through the BMW network via pre-signed URLs generated by the portal. Direct internet access is disabled within their domain. Their Sagemaker domain privileges are built using Amazon SageMaker Role Manager personas to ensure least privilege access to AWS services needed for the development such as SageMaker, Amazon Athena, Amazon Simple Storage Service (Amazon S3), and AWS Glue. This role implements ML guardrails (such as those described in Governance and control), including enforcement of ML training to occur in either Amazon Virtual Private Cloud (Amazon VPC) or without internet and allowing only the use of JuMa’s custom vetted and up-to-date SageMaker images.

Because JuMa is designed for development, experimentation, and ad-hoc analysis, it implements retention policies to remove data after 30 days. To access data whenever needed and store it for long term, JuMa seamlessly integrates with the BMW Cloud Data Hub and BMW on-premises databases.

Finally, JuMa supports multiple Regions to comply to special local legal situations which, for example, require it to process data locally to enable BMW’s data sovereignty.

Operational excellence

Both the JuMa platform backend and workspaces are implemented with AWS serverless and managed services. Using those services helps minimize the effort of the BMW platform team maintaining and operating the end-to-end solution, striving to be a no-ops service. Both the workspace and portal are monitored using Amazon CloudWatch logs, metrics, and alarms to check key performance indicators (KPIs) and proactively notify the platform team of any issues. Additionally, the AWS X-Ray distributed tracing system is used to trace requests throughout multiple components and annotate CloudWatch logs with workspace-relevant context.

All changes to the JuMa infrastructure are managed and implemented through automation using infrastructure as code (IaC). This helps reduce manual efforts and human errors, increase consistency, and ensure reproducible and version-controlled changes across both JuMa platform backend workspaces. Specifically, all workspaces are provisioned and updated through an onboarding process built on top of AWS Step Functions, AWS CodeBuild, and Terraform. Therefore, no manual configuration is required to onboard new workspaces to the JuMa platform.

Cost optimization

By using AWS serverless services, JuMa ensures on-demand scalability, pre-approved instance sizes, and a pay-as-you-go model for the resources used during the development and experimentation activities per the AI/ML teams’ needs. To further optimize costs, the JuMa platform monitors and identifies idle resources within SageMaker Studio and shuts them down automatically to prevent expenses for non-utilized resources.

Sustainability

JuMa replaces BMW’s two on-premises platforms for analytics and deep learning workloads that consume a considerable amount of electricity and produce CO2 emissions even when not in use. By migrating AI/ML workloads from on premises to AWS, BMW will slash its environmental impact by decommissioning the on-premises platforms.

Furthermore, the mechanism for auto shutdown of idle resources, data retention polices, and the workspace usage reports to its owners implemented in JuMa help further minimize the environmental footprint of running AI/ML workloads on AWS.

Performance efficiency

By using SageMaker Studio, BMW teams benefit from an easy adoption of the latest SageMaker features that can help accelerate their experimentation. For example, they can use Amazon SageMaker JumpStart capabilities to use the latest pre-trained, open source models. Additionally, it helps reduce AI/ML team efforts moving from experimentation to solution industrialization, because the development environment provides the same AWS core services but restricted to development capabilities.

Reliability

SageMaker Studio domains are deployed in a VPC-only mode to manage internet access and only allow access to intended AWS services. The network is deployed in two Availability Zones to protect against a single point of failure, achieving greater resiliency and availability of the platform to its users.

Changes to JuMa workspaces are automatically deployed and tested to development and integration environments, using IaC and CI/CD pipelines, before upgrading customer environments.

Finally, data stored in Amazon Elastic File System (Amazon EFS) for SageMaker Studio domains is kept after volumes are deleted for backup purposes.

Conclusion

In this post, we described how BMW Group in collaboration with AWS ProServe developed a fully managed AI platform service on AWS using SageMaker Studio and other AWS serverless and managed services.

With JuMa, BMW’s AI/ML teams are empowered to unlock new business value by accelerating experimentation as well as time-to-market for disruptive AI solutions. Furthermore, by migrating from its on-premises platform, BMW can reduce the overall operational efforts and costs while also increasing sustainability and the overall security posture.

To learn more about running your AI/ML experimentation and development workloads on AWS, visit Amazon SageMaker Studio.

About the Authors

Marc Neumann is the head of the central AI Platform at BMP Group. He is responsible for developing and implementing strategies to use AI technology for business value creation across the BMW Group. His primary goal is to ensure that the use of AI is sustainable and scalable, meaning it can be consistently applied across the organization to drive long-term growth and innovation. Through his leadership, Neumann aims to position the BMW Group as a leader in AI-driven innovation and value creation in the automotive industry and beyond.

Amor Steinberg is a Machine Learning Engineer at BMW Group and the service lead of Jupyter Managed, a new service that aims to provide a code-centric analytics and machine learning workbench for engineers and data scientists at the BMW Group. His past experience as a DevOps Engineer at financial institutions enabled him to gather a unique understanding of the challenges that faces banks in the European Union and keep the balance between striving for technological innovation, complying with laws and regulations, and maximizing security for customers.

Marinus Krommenhoek is a Senior Cloud Solution Architect and a Software Developer at BMW Group. He is enthusiastic about modernizing the IT landscape with state-of-the-art services that add high value and are easy to maintain and operate. Marinus is a big advocate of microservices, serverless architectures, and agile working. He has a record of working with distributed teams across the globe within large enterprises.

Nicolas Jacob Baer is a Principal Cloud Application Architect at AWS ProServe with a strong focus on data engineering and machine learning, based in Switzerland. He works closely with enterprise customers to design data platforms and build advanced analytics and ML use cases.

Joaquin Rinaudo is a Principal Security Architect at AWS ProServe. He is passionate about building solutions that help developers improve their software quality. Prior to AWS, he worked across multiple domains in the security industry, from mobile security to cloud and compliance-related topics. In his free time, Joaquin enjoys spending time with family and reading science-fiction novels.

Shukhrat Khodjaev is a Senior Global Engagement Manager at AWS ProServe. He specializes in delivering impactful big data and AI/ML solutions that enable AWS customers to maximize their business value through data utilization.

Read More

In today’s ever-evolving world of ecommerce, the influence of a compelling product description cannot be overstated. It can be the decisive factor that turns a potential visitor into a paying customer or sends them clicking off to a competitor’s site. The manual creation of these descriptions across a vast array of products is a labor-intensive process, and it can slow down the velocity of new innovation. This is where Amazon Bedrock with its generative AI capabilities steps in to reshape the game. In this post, we dive into how Amazon Bedrock is transforming the product description generation process, empowering e-retailers to efficiently scale their businesses while conserving valuable time and resources.

Unlocking the power of generative AI in retail

Generative AI has captured the attention of boards and CEOs worldwide, prompting them to ask, “How can we leverage generative AI for our business?” One of the most promising applications of generative AI in ecommerce is using it to craft product descriptions. Retailers and brands have invested significant resources in testing and evaluating the most effective descriptions, and generative AI excels in this area.

Creating engaging and informative product descriptions for a vast catalog is a monumental task, especially for global ecommerce platforms. Manual translation and adaptation of product descriptions for each market consumes time and resources. This results in generic or incomplete descriptions, leading to reduced sales and customer satisfaction.

The power of Amazon Bedrock: AI-generated product descriptions

Amazon Bedrock is a fully managed service that simplifies generative AI development, offering high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon through a single API. It provides a comprehensive set of capabilities for building generative AI applications while ensuring privacy and security are maintained. With Amazon Bedrock, you can experiment with various FMs and customize them privately using techniques like fine-tuning and Retrieval Augmented Generation (RAG). The platform enables you to create managed agents for complex business tasks without the need for coding, such as booking travel, processing insurance claims, creating ad campaigns, and managing inventory.

For example, ecommerce platforms can initially generate basic product descriptions that include size, color, and price. However, Amazon Bedrock’s flexibility allows these descriptions to be fine-tuned to incorporate customer reviews, integrate brand-specific language, and highlight specific product features, resulting in tailored descriptions that resonate with the target audience. Moreover, Amazon Bedrock offers access to foundation models from Amazon and leading AI startups through an intuitive API, making the entire process seamless and efficient.

Using AI can have the following impact on the product description process:

• Faster approvals – Vendors experience a streamlined process, moving from product listing to approval in under an hour, eliminating frustrating delays
• Improved product listing velocity – When automated, your ecommerce marketplace sees a surge in product listings, offering consumers access to the latest merchandise nearly instantaneously
• Future-proofing – By embracing cutting-edge AI, you secure your position as a forward-looking platform ready to meet evolving market demands
• Innovation – This solution liberates teams from mundane tasks, allowing them to focus on higher-value work and fostering a culture of innovation

Solution overview

Before we dive into the technical details, let’s see the high-level preview of what this solution offers. This solution will allow you to create and manage product descriptions for your ecommerce platform. It empowers your platform to:

• Generate descriptions from text – With the power of generative AI, Amazon Bedrock can convert plain text descriptions into vivid, informative, and captivating product descriptions.
• Craft images – Beyond text, it can also craft images that align perfectly with the product descriptions, enhancing the visual appeal of your listings.
• Enhance existing content – Do you have existing product descriptions that need a fresh perspective? Amazon Bedrock can take your current content and make it even more compelling and engaging.

This solution is available in the AWS Solutions Library. We’ve provided detailed instructions in the accompanying README file. The README file contains all the information you need to get started, from requirements to deployment guidelines.

The system architecture comprises several core components:

• UI portal – This is the user interface (UI) designed for vendors to upload product images.
• Amazon Rekognition – Amazon Rekognition is an image analysis service that detects objects, text, and labels in images.
• Amazon Bedrock – Foundation models in Amazon Bedrock use the labels detected by Amazon Rekognition to generate product descriptions.
• AWS Lambda – AWS Lambda provides serverless compute for processing.
• Product database – The central repository stores vendor products, images, labels, and generated descriptions. This could be any database of your choice. Note that in this solution, all of the storage is in the UI.
• Admin portal – This portal provides oversight of the system and product listings, ensuring smooth operation. This is not part of the solution; we’ve added it for understanding.

The following diagram illustrates the flow of data and interactions within the system

The workflow includes the following steps:

1. The client initiates a request to the Amazon API Gateway REST API.
2. Amazon API Gateway passes the request to AWS Lambda through a proxy integration.
3. When operating on product image inputs, AWS Lambda calls Amazon Rekognition to detect objects in the image.
4. AWS Lambda calls LLMs hosted by Amazon Bedrock, such as the Amazon Titan language models, to generate product descriptions.
5. The response is passed back from AWS Lambda to Amazon API Gateway.
6. Finally, HTTP response from Amazon API Gateway is returned to the client.

Example use case

Imagine a vendor uploads a product image of shoes, and Amazon Rekognition identifies key attributes like “white shoes,” “sneaker,” and “durable.” The Amazon Bedrock Titan AI takes this information and generates a product description like, “Here is a draft product description for a canvas running shoe based on the product photo: Introducing the Canvas Runner, the perfect lightweight sneaker for your active lifestyle. This running shoe features a breathable canvas upper with leather accents for a stylish, classic look. The lace-up design provides a secure fit, while the padded tongue and collar add comfort. Inside, a removable cushioned insole supports and comforts your feet. The EVA midsole absorbs shock with each step, reducing fatigue. Flex grooves in the rubber outsole ensure flexibility and traction. With its simple, retro-inspired style, the Canvas Runner seamlessly transitions from workouts to everyday wear. Whether you’re running errands or running miles, this versatile sneaker will keep you moving in comfort and style.”

Design details

Let’s explore the components in more detail:

• User interface:
  • Front end – The front end of the vendor portal allows vendors to upload product images and displays product listings.
  • API calls – The portal communicates with the backend through APIs to process images and generate descriptions.
• Amazon Rekognition:
  • Image analysis – Triggered by API calls, Amazon Rekognition analyzes images and detects objects, text, and labels.
  • Label output – It outputs label data derived from the analysis.
• Amazon Bedrock:
  • NLP text generation – Amazon Bedrock uses the Amazon Titan natural language processing (NLP) model to generate textual descriptions.
  • Label integration – It takes the labels detected by Amazon Rekognition as input to generate product descriptions.
  • Style matching – Amazon Bedrock provides fine-tuning capabilities for Amazon Titan models to ensure that the generated descriptions match the style of the platform.
• AWS Lambda:
  • Processing – Lambda handles the API calls to services.
• Product database:
  • Flexible database – The product database is chosen based on customer preferences and requirements. Note this is not provided as part of the solution.

Additional capabilities

This solution goes beyond just generating product descriptions. It offers two more incredible options:

• Image and description generation from text – With the power of generative AI, Amazon Bedrock can take text descriptions and create corresponding images along with detailed product descriptions. Consider the potential:
  • Instantly visualizing products from text.
  • Automating image creation for large catalogs.
  • Enhancing customer experience with rich visuals.
  • Reducing content creation time and costs.
• Description enhancement – If you already have existing product descriptions, Amazon Bedrock can enhance them. Simply supply the text and the prompt, and Amazon Bedrock will skillfully enhance and enrich the content, rendering it highly captivating and engaging for your customers.

Conclusion

In the fiercely competitive world of ecommerce, staying at the forefront of innovation is imperative. Amazon Bedrock offers a transformative capability for e-retailers looking to enhance their product content, optimize their listing process, and drive sales. With the power of AI-generated product descriptions, businesses can create compelling, informative, and culturally relevant content that resonates deeply with customers. The future of ecommerce has arrived, and it’s driven by machine learning with Amazon Bedrock.

Are you ready to unlock the full potential of AI-powered product descriptions? Take the next step in revolutionizing your ecommerce platform. Visit the AWS Solutions Library and explore how Amazon Bedrock can transform your product descriptions, streamline your processes, and boost your sales. It’s time to supercharge your ecommerce with Amazon Bedrock!

About the Authors

Dhaval Shah is a Senior Solutions Architect at AWS, specializing in Machine Learning. With a strong focus on digital native businesses, he empowers customers to leverage AWS and drive their business growth. As an ML enthusiast, Dhaval is driven by his passion for creating impactful solutions that bring positive change. In his leisure time, he indulges in his love for travel and cherishes quality moments with his family.

Doug Tiffan is the Head of World Wide Solution Strategy for Fashion & Apparel at AWS. In his role, Doug works with Fashion & Apparel executives to understand their goals and align with them on the best solutions. Doug has over 30 years of experience in retail, holding several merchandising and technology leadership roles. Doug holds a BBA from Texas A&M University and is based in Houston, Texas.

Nikhil Sharma is a Solutions Architecture Leader at Amazon Web Services (AWS) where he and his team of Solutions Architects help AWS customers solve critical business challenges using AWS cloud technologies and services.

Kevin Bell is a Sr. Solutions Architect at AWS based in Seattle. He has been building things in the cloud for about 10 years. You can find him online as @bellkev on GitHub.

Nipun Chagari is a Principal Solutions Architect based in the Bay Area, CA. Nipun is passionate about helping customers adopt Serverless technology to modernize applications and achieve their business objectives. His recent focus has been on assisting organizations in adopting modern technologies to enable digital transformation. Apart from work, Nipun finds joy in playing volleyball, cooking and traveling with his family.

Marshall Bunch is a Solutions Architect at AWS helping North American customers design secure, scalable and cost-effective workloads in the cloud. His passion lies in solving age-old business problems where data and the newest technologies enable novel solutions. Beyond his professional pursuits, Marshall enjoys hiking and camping in Colorado’s beautiful Rocky Mountains.

Altaaf Dawoodjee is a Solutions Architect Leader that supports AdTech customers in the Digital Native Business (DNB) segment at Amazon Web Service (AWS). He has over 20 years of experience in Technology and has deep expertise in Analytics. He is passionate about helping drive successful business outcomes for his customers leveraging the AWS cloud.

Scott Bell is a dynamic leader and innovator with 25+ years of technology management experience. He is passionate about leading and developing teams in providing technology to meet the challenges of global users and businesses. He has extensive experience in leading technology teams which provide global technology solutions supporting 35+ languages. He is also passionate about the way the AI and Generative AI transform businesses and the way they support customer’s current unmet needs.

Sachin Shetti is a Principal Customer Solution Manager at AWS. He is passionate about helping enterprises succeed and realize significant benefits from cloud adoption, driving everything from basic migration to large-scale cloud transformation across people, processes, and technology. Prior to joining AWS, Sachin worked as a software developer for over 12 years and held multiple senior leadership positions leading technology delivery and transformation in healthcare, financial services, retail, and insurance. He has an Executive MBA and a Bachelor’s degree in Mechanical Engineering.

Read More

Amazon SageMaker Canvas is a rich, no-code Machine Learning (ML) and Generative AI workspace that has allowed customers all over the world to more easily adopt ML technologies to solve old and new challenges thanks to its visual, no-code interface. It does so by covering the ML workflow end-to-end: whether you’re looking for powerful data preparation and AutoML, managed endpoint deployment, simplified MLOps capabilities, and ready-to-use models powered by AWS AI services and Generative AI, SageMaker Canvas can help you to achieve your goals.

As companies of all sizes adopt SageMaker Canvas, customers asked for ways to optimize cost. As defined in the AWS Well-Architected Framework, a cost-optimized workload fully uses all resources, meets your functional requirements, and achieves an outcome at the lowest possible price point.

Today, we’re introducing a new way to further optimize costs for SageMaker Canvas applications. SageMaker Canvas now collects Amazon CloudWatch metrics that provide insight into app usage and idleness. Customers can use this information to shut down automatically idle SageMaker Canvas applications to avoiding incurring unintended costs.

In this post, we’ll show you how to automatically shut down idle SageMaker Canvas apps to control costs by using a simple serverless architecture. Templates used in this post are available in GitHub.

Understanding and tracking costs

Education is always the first step into understanding and controlling costs for any workload, either on-premises or in the cloud. Let’s start by reviewing the SageMaker Canvas pricing model. In a nutshell, SageMaker Canvas has a pay-as-you-go pricing model, based on two dimensions:

• Workspace instance: ­ formerly known as session time, is the cost associated with running the SageMaker Canvas app
• AWS service charges: ­ costs associated with training the models, deploying the endpoints, generating inferences (resources to spin up SageMaker Canvas).

Customers always have full control over the resources that are launched by SageMaker Canvas and can keep track of costs associated with the SageMaker Canvas app by using the AWS Billing and Cost Management service. For more information, refer to Manage billing and cost in SageMaker Canvas.

To limit the cost associated with the workspace instances, as a best practice, you must log out, do not close the browser tab. To log out, choose the Log out button on the left panel of the SageMaker Canvas app.

Automatically shutting down SageMaker Canvas applications

For IT Administrators that are looking to provide automated controls for shutting down SageMaker Canvas applications and keeping costs under control, there are two approaches:

1. Shutdown applications on a schedule (every day at 19:00 or every Friday at 18:00)
2. Shutdown automatically idle applications (when the application hasn’t been used for two hours)

Shutdown applications on a schedule

Scheduled shutdown of SageMaker Canvas applications can be achieved with very little effort by using a cron expression (with Amazon EventBridge Cron Rule), a compute component (an AWS Lambda function) that calls the Amazon SageMaker API DeleteApp. This approach has been discussed in the Provision and manage ML environments with Amazon SageMaker Canvas using AWS CDK and AWS Service Catalog post, and implemented in the associated GitHub repository.

One of the advantages of the above architecture is that it is very simple to duplicate it to achieve scheduled creation of the SageMaker Canvas app. By using a combination of scheduled creation and scheduled deletion, a cloud administrator can make sure that the SageMaker Canvas application is ready to be used whenever users start their business day (e.g. 9AM on a work day), and that the app also automatically shuts down at the end of the business day (e.g. 7PM on a work day, always shut down during weekends). All that is needed to do is change the line of code calling the DeleteApp API into CreateApp, as well as updating the cron expression to reflect the desired app creation time.

While this approach is very easy to implement and test, a drawback of the suggested architecture is that it does not take into account whether an application is currently being used or not, shutting it down regardless of its current activity status. According to different situations, this might cause friction with active users, which might suddenly see their session terminated.

You can retrieve the template associated to this architecture from the following GitHub repository:

Shutdown automatically idle applications

Starting today, Amazon SageMaker Canvas emits CloudWatch metrics that provide insight into app usage and idleness. This allows an administrator to define a solution that reads the idleness metric, compares it against a threshold, and defines a specific logic for automatic shutdown. A more detailed overview of the idleness metric emitted by SageMaker Canvas is shown in the following paragraph.

To achieve automatic shutdown of SageMaker Canvas applications based on the idleness metrics, we provide an AWS CloudFormation template. This template consists of three main components:

1. An Amazon CloudWatch Alarm, which runs a query to check the MAX value of the TimeSinceLastActive metric. If this value is greater than a threshold provided as input to the CloudFormation template, it triggers the rest of the automation. This query can be run on a single user profile, on a single domain, or across all domains. According to the level of control that you wish to have, you can use:
   1. the all-domains-all-users template, which checks this across all users and all domains in the region where the template is deployed
   2. the one-domain-all-users template, which checks this across all users in one domain in the region where the template is deployed
   3. the one-domain-one-user template, which checks this for one user profile, in one domain, in the region where the template is deployed
2. The alarm state change creates an event on the default event bus in Amazon EventBridge, which has an Amazon EventBridge Rule set up to trigger an AWS Lambda function
3. The AWS Lambda function identifies which SageMaker Canvas app has been running in idle for more than the specified threshold, and deletes it with the DeleteApp API.

You can retrieve the AWS CloudFormation templates associated to this architecture from the following GitHub repository:

How SageMaker Canvas idleness metric work

SageMaker Canvas emits a TimeSinceLastActive metric in the /aws/sagemaker/Canvas/AppActivity namespace, which shows the number of seconds that the app has been idle with no user activity. We can use this new metric to trigger an automatic shutdown of the SageMaker Canvas app when it has been idle for a defined period. SageMaker Canvas exposes the TimeSinceLastActive with the following schema:

{
    "Namespace": "/aws/sagemaker/Canvas/AppActivity",
    "Dimensions": [
        [
            "DomainId",
            "UserProfileName"
        ]
    ],
    "Metrics": [
        {
            "Name": "TimeSinceLastActive",
            "Unit": "Seconds",
            "Value": 12345
        }
    ]
}

The key components of this metric are as follows:

• Dimensions, in particular DomainID and UserProfileName, that allow an administrator to pinpoint which applications are idle across all domains and users
• Value of the metric, which indicates the number of seconds since the last activity in the SageMaker Canvas applications. SageMaker Canvas considers the following as activity:
  • Any action taken in the SageMaker Canvas application (clicking a button, transforming a dataset, generating an in-app inference, deploying a model);
  • Using a ready-to-use model or interacting with the Generative AI models using chat interface;
  • A batch inference scheduled to run at a specific time; for more information, refer to  Manage automations.

This metric can be read via Amazon CloudWatch API such as get_metric_data. For example, using the AWS SDK for Python (boto3):

import boto3, datetime

cw = boto3.client('cloudwatch')
metric_data_results = cw.get_metric_data(
    MetricDataQueries=[
        {
            "Id": "q1",
            "Expression": 'SELECT MAX(TimeSinceLastActive) FROM "/aws/sagemaker/Canvas/AppActivity" GROUP BY DomainId, UserProfileName',
            "Period": 900
        }
    ],
    StartTime=datetime.datetime(2023, 1, 1),
    EndTime=datetime.datetime.now(),
    ScanBy='TimestampAscending'
)

The Python query extracts the MAX value of TimeSinceLastActive from the namespace associated to SageMaker Canvas after grouping these values by DomainID and UserProfileName.

Deploying and testing the auto-shutdown solution

To deploy the auto-shutdown stack, do the following:

1. Download the AWS CloudFormation template that refers to the solution you want to implement from the above GitHub repository. Choose whether you want to implement a solution for all SageMaker Domains, for a single SageMaker Domain, or for a single user;
2. Update template parameters:
   1. The idle timeout – time (in seconds) that the SageMaker Canvas app is allowed to stay in idle before it gets shutdown; default value is 2 hours
   2. The alarm period – aggregation time (in seconds) used by CloudWatch Alarm to compute the idle timeout; default value is 20 minutes
   3. (optional) SageMaker Domain ID and user profile name
3. Deploy the CloudFormation stack to create the resources

Once deployed (should take less than two minutes), the AWS Lambda function and Amazon CloudWatch alarm are configured to automatically shut down the Canvas app when idle. To test the auto-shutdown script, do the following:

1. Make sure that the SageMaker Canvas app is running within the right domain and with the right user profile (if you have configured them).
2. Stop using the SageMaker Canvas app and wait for the idle timeout period (default, 2 hours)
3. Check that the app is stopped after being idle for the threshold time by checking that the CloudWatch alarm has been triggered and, after triggering the automation, it has gone back to the normal state.

In our test, we have set the idle timeout period to two hours (7200 seconds). In the following graph plotted by Amazon CloudWatch Metrics, you can see that the SageMaker Canvas app has been emitting the TimeSinceLastActive metric until the threshold was met (1), which triggered the alarm. Once the alarm was triggered, the AWS Lambda function was executed, which deleted the app and brought the metric back below the threshold (2).

Conclusion

In this post, we implemented an automated shutdown solution for idle SageMaker Canvas apps using AWS Lambda and CloudWatch Alarm and the newly emitted metric of idleness from SageMaker Canvas. Thanks to this solution, customers not only can optimize costs for their ML workloads but can also avoid unintended charges for applications that they forgot were running in their SageMaker Domain.

We’re looking forward to seeing what new use cases and workloads customers can solve with the peace of mind brought by this solution. For more examples of how SageMaker Canvas can help you achieve your business goals, refer to the following posts:

• Predict customer churn with no-code machine learning using Amazon SageMaker Canvas
• Publish predictive dashboards in Amazon QuickSight using ML predictions from Amazon SageMaker Canvas
• Democratize computer vision defect detection for manufacturing quality using no-code machine learning with Amazon SageMaker Canvas
• Use no-code machine learning to derive insights from product reviews using Amazon SageMaker Canvas sentiment analysis and text analysis models
• Empower your business users to extract insights from company documents using Amazon SageMaker Canvas Generative AI

To learn how you can run production-level workloads with Amazon SageMaker Canvas, refer to the following posts:

• Build, Share, Deploy: how business analysts and data scientists achieve faster time-to-market using no-code ML and Amazon SageMaker Canvas
• Retrain ML models and automate batch predictions in Amazon SageMaker Canvas using updated datasets
• Operationalize ML models built in Amazon SageMaker Canvas to production using the Amazon SageMaker Model Registry
• Provision and manage ML environments with Amazon SageMaker Canvas using AWS CDK and AWS Service Catalog

About the authors

Davide Gallitelli is a Senior Specialist Solutions Architect for AI/ML. He is based in Brussels and works closely with customers all around the globe that are looking to adopt Low-Code/No-Code Machine Learning technologies, and Generative AI. He has been a developer since he was very young, starting to code at the age of 7. He started learning AI/ML at university, and has fallen in love with it since then.

Huong Nguyen is a Sr. Product Manager at AWS. She is leading the data ecosystem integration for SageMaker, with 14 years of experience building customer-centric and data-driven products for both enterprise and consumer spaces.

Gunjan Garg is a Principal Engineer at Amazon SageMaker team in AWS, providing technical leadership for the product. She has worked in several roles in the AI/ML org for last 5 years and is currently focused on Amazon SageMaker Canvas.

Ziyao Huang is a Software Development Engineer with Amazon SageMaker Data Wrangler. He is passionate about building great product that makes ML easy for the customers. Outside of work, Ziyao likes to read, and hang out with his friends.

Read More

This post was co-written with Greg Benson, Chief Scientist; Aaron Kesler, Sr. Product Manager; and Rich Dill, Enterprise Solutions Architect from SnapLogic.

Many customers are building generative AI apps on Amazon Bedrock and Amazon CodeWhisperer to create code artifacts based on natural language. This use case highlights how large language models (LLMs) are able to become a translator between human languages (English, Spanish, Arabic, and more) and machine interpretable languages (Python, Java, Scala, SQL, and so on) along with sophisticated internal reasoning. This emergent ability in LLMs has compelled software developers to use LLMs as an automation and UX enhancement tool that transforms natural language to a domain-specific language (DSL): system instructions, API requests, code artifacts, and more. In this post, we show you how SnapLogic, an AWS customer, used Amazon Bedrock to power their SnapGPT product through automated creation of these complex DSL artifacts from human language.

When customers create DSL objects from LLMs, the resulting DSL is either an exact replica or a derivative of an existing interface data and schema that forms the contract between the UI and the business logic in the backing service. This pattern is particularly trending with independent software vendors (ISVs) and software as a service (SaaS) ISVs due to their unique way of representing configurations through code and the desire to simplify the user experience for their customers. Example use cases include:

• Converting natural language to tool-specific visuals and calculation expressions, as seen with the recently announced generative BI capabilities in Amazon QuickSight
• Generating the correct API call based on individual user intent and context across best-of-breed productivity SaaS apps with the upcoming productivity features in AWS AppFabric
• Creating elaborate integration pipelines with simple human prompts, as you will read more about in this post

The most straightforward way to build and scale text-to-pipeline applications with LLMs on AWS is using Amazon Bedrock. Amazon Bedrock is the easiest way to build and scale generative AI applications with foundation models (FMs). It is a fully managed service that offers access to a choice of high-performing foundation FMs from leading AI via a single API, along with a broad set of capabilities you need to build generative AI applications with privacy and security. Anthropic, an AI safety and research lab that builds reliable, interpretable, and steerable AI systems, is one of the leading AI companies that offers access to their state-of-the art LLM, Claude, on Amazon Bedrock. Claude is an LLM that excels at a wide range of tasks, from thoughtful dialogue, content creation, complex reasoning, creativity, and coding. Anthropic offers both Claude and Claude Instant models, all of which are available through Amazon Bedrock. Claude has quickly gained popularity in these text-to-pipeline applications because of its improved reasoning ability, which allows it to excel in ambiguous technical problem solving. Claude 2 on Amazon Bedrock supports a 100,000-token context window, which is equivalent to about 200 pages of English text. This is a particularly important feature that you can rely on when building text-to-pipeline applications that require complex reasoning, detailed instructions, and comprehensive examples.

SnapLogic background

SnapLogic is an AWS customer on a mission to bring enterprise automation to the world. The SnapLogic Intelligent Integration Platform (IIP) enables organizations to realize enterprise-wide automation by connecting their entire ecosystem of applications, databases, big data, machines and devices, APIs, and more with pre-built, intelligent connectors called Snaps. SnapLogic recently released a feature called SnapGPT, which provides a text interface where you can type the desired integration pipeline you want to create in simple human language. SnapGPT uses Anthropic’s Claude model through Amazon Bedrock to automate the creation of these integration pipelines as code, which are then used through SnapLogic’s flagship integration solution. However, SnapLogic’s journey to SnapGPT has been a culmination of many years operating in the AI space.

SnapLogic’s AI journey

In the realm of integration platforms, SnapLogic has consistently been at the forefront, harnessing the transformative power of artificial intelligence. Over the years, the company’s commitment to innovating with AI has become evident, especially when we trace the journey from Iris to AutoLink.

The humble beginnings with Iris

In 2017, SnapLogic unveiled Iris, an industry-first AI-powered integration assistant. Iris was designed to use machine learning (ML) algorithms to predict the next steps in building a data pipeline. By analyzing millions of metadata elements and data flows, Iris could make intelligent suggestions to users, democratizing data integration and allowing even those without a deep technical background to create complex workflows.

AutoLink: Building momentum

Building on the success and learnings from Iris, SnapLogic introduced AutoLink, a feature aimed at further simplifying the data mapping process. The tedious task of manually mapping fields between source and target systems became a breeze with AutoLink. Using AI, AutoLink automatically identified and suggested potential matches. Integrations that once took hours could be run in mere minutes.

The generative leap with SnapGPT

SnapLogic’s latest foray in AI brings us SnapGPT, which aims to revolutionize integration even further. With SnapGPT, SnapLogic introduces the world’s first generative integration solution. This is not just about simplifying existing processes, but entirely reimagining how integrations are designed. The power of generative AI can create entire integration pipelines from scratch, optimizing the workflow based on the desired outcome and data characteristics.

SnapGPT is extremely impactful to SnapLogic’s customers because they are able to drastically decrease the amount of time required to generate their first SnapLogic pipeline. Traditionally, SnapLogic customers would need to spend days or weeks configuring integration pipelines from scratch. Now, these customers are able to simply ask SnapGPT to, for example, “create a pipeline which will move all of my active SFDC customers to WorkDay.” A working first draft of a pipeline is automatically created for this customer, drastically cutting down the development time required for creation of the base of their integration pipeline. This allows the end customer to spend more time focusing on what has true business impact to them instead of working on configurations of an integration pipeline. The following example shows how a SnapLogic customer can enter a description into the SnapGPT feature to quickly generate a pipeline, using natural language.

AWS and SnapLogic have collaborated closely throughout this product build and have learned a lot along the way. The rest of this post will focus on the technical learnings AWS and SnapLogic have had around using LLMs for text-to-pipeline applications.

Solution overview

To solve this text-to-pipeline problem, AWS and SnapLogic designed a comprehensive solution shown in the following architecture.

A request to SnapGPT goes through the following workflow:

1. A user submits a description to the application.
2. SnapLogic uses a Retrieval Augmented Generation (RAG) approach to retrieve relevant examples of SnapLogic pipelines that are similar to the user’s request.
3. These extracted relevant examples are combined with the user input and go through some text preprocessing before they’re sent to Claude on Amazon Bedrock.
4. Claude produces a JSON artifact that represents a SnapLogic pipeline.
5. The JSON artifact is directly integrated to the core SnapLogic integration platform.
6. The SnapLogic pipeline is rendered to the user in a visual friendly manner.

Through various experimentation between AWS and SnapLogic, we have found the prompt engineering step of the solution diagram to be extremely important to generating high-quality outputs for these text-to-pipeline outputs. The next section goes further into some specific techniques used with Claude in this space.

Prompt experimentation

Throughout the development phase of SnapGPT, AWS and SnapLogic found that rapid iteration on prompts being sent to Claude was a critical development task to improving the accuracy and relevancy of text-to-pipeline outputs in SnapLogic’s outputs. By using Amazon SageMaker Studio interactive notebooks, the AWS and SnapLogic team were able to quickly work through different versions of prompts by using the Boto3 SDK connection to Amazon Bedrock. Notebook-based development allowed the teams to quickly create client-side connections to Amazon Bedrock, include text-based descriptions alongside Python code for sending prompts to Amazon Bedrock, and hold joint prompt engineering sessions where iterations were made quickly between multiple personas.

Anthropic Claude prompt engineering methods

In this section, we describe some of the iterative techniques we used to create a high-performing prompt based on an illustrative user request: “Make a pipeline which uses the ExampleCompany database which retrieves all active customers.” Note that this example is not the schema that SnapGPT is powered by, and is only used to illustrate a text-to-pipeline application.

To baseline our prompt engineering, we use the following original prompt:

Make a pipeline which uses the ExampleCompany database which retrieves all active customers

The expected output is as follows:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status = 'active'"
}

Improvement #1: Using the Human and Assistant annotations

Claude’s training procedure teaches the FM to understand dialogue between a human and an assistant in its prompt structure. Claude users can take advantage of this structure by ending their prompt in Assistant:, which will trigger Claude to start generating the response to a query based on what the human has said. Note that because Claude will continue to generate text until it is told to stop, make sure you use a stop sequence of nnHuman: in the API request to Amazon Bedrock when using Claude. This will ensure you only receive the response from the assistant and not the predicted next response from the human in the dialogue chain. These nnHuman and nnAssistant indicators not only help in limiting Claude’s response to relevant information, but also help in providing clear demarcation between primary human inputs and assistant responses from other interactions. This especially helps in a few-shot prompt setting where you can have multiple queries and responses with queries tagged with nnHuman and previous responses tagged with nnAssistant. Let’s look at the improved prompt and associated illustrative API call to Amazon Bedrock via the Boto3 SDK.

We use the following prompt:

Human: Make a pipeline which uses the ExampleCompany database which retrieves all active customers

Assistant:

The output is as follows:

Here is a MongoDB pipeline to retrieve all active customers from the ExampleCompany database:

pipeline = [
    {
        '$match': {
            'database': 'ExampleCompany', 
            'status': 'active'
        }
    }
]
db.customers.aggregate(pipeline)

The key steps are:

– `$match` to filter the documents to only those in the ExampleCompany database and with status equal to “active”.

– `db.customers.aggregate()` runs the pipeline on the customers collection and returns the filtered results.

So this will return all customer documents from the ExampleCompany database where the status field is set to “active”.

Notice how in the output, Claude makes up an answer to our question without any context. The answer is potentially helpful, but it’s not specific to the problem we’re trying to solve because it is not provided with enough context. It’s also important to note that Anthropic requires this human and assistant format when using the Claude models; if you don’t include this step, Amazon Bedrock will return an error.

Improvement #2: Include hyper-specific context for Claude before it’s told what to do

Claude performs best when it’s given good context about what it needs to accomplish as well as how it is supposed to accomplish that task. Here are a few examples of things to include as context for Claude before you tell it the task:

• Describe the job function that you want Claude to fulfill. By doing this, you are making Claude think in the role of someone working in that job, or more like someone just being hired in that specific job role. This will help Claude formulate the answer from that specific job role’s perspective, reducing hallucination and randomness in the answer.
• Include technical details regarding the output format. Foundation models are generally good at following actions, so asking Claude to generate the output in a specific format or schema with one or a few examples will help Claude provide the answer in the right format, reducing the need for filtering and postprocessing of the output. The example in the following prompt shows a fictitious schema that matches the expected output.

Let’s combine these suggestions to improve upon our original prompt:

Human: Your job is to act as an expert on ETL pipelines. Specifically, your job is to create a JSON representation of an ETL pipeline which will solve the user request provided to you.

The JSON output should follow the following format:

{
  "database": "the name of the database you will be using here",
  "query": "place a sql representation for solving the task at hand here"
}

Here is your task: make a pipeline which uses the ExampleCompany database which retrieves all active customers

Assistant:

The output is as follows:

Here is a JSON representation of an ETL pipeline to retrieve all active customers from the ExampleCompany database:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM customers WHERE status = 'active'"
}

This pipeline uses the ExampleCompany database and a SQL query to select all records from the customers table where the status is ‘active’.

With the addition of context, the model is now able to get most of the way to our answer. It now knows how to structure the output and it understands that it needs to write a SQL statement to solve the task. However, you will notice that the model doesn’t correctly use the ec_prod.customers nomenclature we are expecting. Room for improvement! Also note, in practice, you will need to include significantly more context than what we provided in this example for high-quality results in text-to-pipeline use cases.

Improvement #3: Enclosing examples in XML tags

Few-shot learning is a popular way to increase the accuracy with FMs where you have relevant examples to guide the FM in their task. This is particularly important when working in text-to-pipeline applications because these hyper-specific pipeline outputs are usually not included in any public dataset because they are specific to an individual customer’s data representation. When using Claude, you can take advantage of its innate knowledge of arbitrary XML tags in order to provide these examples in a few-shot format. In the following improved prompt, notice the addition of examples enclosed in <example> XML tags. Also, you can use abbreviations for human and assistant with H and A to provide context to Claude without making it think these were specific pieces of the dialogue that have already occurred.

We use the following prompt:

Human: Your job is to act as an expert on ETL pipelines. Specifically, your job is to create a JSON representation of an ETL pipeline which will solve the user request provided to you.

The JSON output should follow the following format:

{
  "database": "the name of the database you will be using here"
  "query": "place a sql representation for solving the task at hand here"
}

Here are some examples of user requests and associated output JSON artifacts which correctly solve the task provided. Make sure to use the same nomenclature which is provided in the examples if you are using the same databases.

<example-1>

H: query the ExampleCompany database for any active orders for customer #1234

A:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.orders WHERE customer_id = 1234 AND status  = 'active'"
}

The reasoning for this query is that the customer #1234 is filtered and we are searching the “orders” table for any existing records after the “active” filter.

</example-1>

<example-2>

H: remove all customers in the ExampleCompany database which have not been active in the last month

A:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status != 'active'"
}

The reasoning for this query is that the “ec_prod.customers” table is the only table in ExampleCompany database which contains customer records.

</example-2>

Here is your task: make a pipeline which uses the ExampleCompany database which retrieves all active customers.

Assistant:

The output is as follows:

Here is the JSON representation for an ETL pipeline to retrieve all active customers from the ExampleCompany database:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status = 'active'"
}

The reasoning is:

The ExampleCompany database is specified as the database to use

The ec_prod.customers table contains the customer data

The status field indicates if a customer is active so the query filters for only active customers

This should retrieve all active customer records from the ExampleCompany database. Let me know if you need any clarification or have additional requirements for the ETL pipeline!

The JSON output now has the correct database nomenclature, which Claude was able to use because of the examples that we provided. Coupling human and assistant indicators with XML tags makes it easier for Claude to understand what is the task and what was the answer in similar examples with very clear separation between multiple few-shot examples. The easier it is for Claude to understand, the better and more relevant the answer will be, further reducing the chance for the model to hallucinate and provide random irrelevant answers.

Improvement #4: Triggering Claude to begin JSON generation with XML tags

A small challenge with text-to-pipeline applications using FMs is the need to exactly parse an output from resulting text so it can be interpreted as code in a downstream application. One way to solve this with Claude is to take advantage of its XML tag understanding and combine this with a custom stop sequence. In the following prompt, we have instructed Claude to enclose the output in <json></json> XML tags. Then, we have added the <json> tag to the end of the prompt. This ensures that the first text that comes out of Claude will be the start of the JSON output. If you don’t do this, Claude often responds with some conversational text, then the true code response. By instructing Claude to immediately start generating the output, you can easily stop generation when you see the closing </json> tag. This is shown in the updated Boto3 API call. The benefits of this technique are twofold. First, you are able to exactly parse the code response from Claude. Second, you are able to reduce cost because Claude only generates code outputs and no extra text. This reduces cost on Amazon Bedrock because you are charged for each token that is produced as output from all FMs.

We use the following prompt:

Human: Your job is to act as an expert on ETL pipelines. Specifically, your job is to create a JSON representation of an ETL pipeline which will solve the user request provided to you.

The JSON output should follow the following format:

{
    "database": "the name of the database you will be using here"
    "query": "place a sql representation for solving the task at hand here"
}

Here are some examples of user requests and associated output JSON artifacts which correctly solve the task provided. Make sure to use the same nomenclature which is provided in the examples if you are using the same databases.

<example-1>

H: query the ExampleCompany database for any active orders for customer #1234

A:

<json>

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.orders WHERE customer_id = 1234 AND status  = 'active'"
}

</json>

The reasoning for this query is that the customer #1234 is filtered and we are searching the “orders” table for any existing records after the “active” filter.

</example-1>

<example-2>

H: remove all customers in the ExampleCompany database which have not been active in the last month

A:

<json>

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status != 'active'"
}

</json>

The reasoning for this query is that the “ec_prod.customers” table is the only table in ExampleCompany database which contains customer records.

</example-2>

Always remember to enclose your JSON outputs in <json></json> tags.

Here is your task: make a pipeline which uses the ExampleCompany database which retrieves all active customers.

Assistant:

<json>

We use the following code:

body = json.dumps({"prompt": prompt, "stop_sequences": ['nnHuman:', '</json>']})
response = bedrock.invoke_model(
   body=body,
   modelId='anthropic.claude-v2'
)

The output is as follows:

{
  "database": "ExampleCompany",
  "query": "SELECT * FROM ec_prod.customers WHERE status = 'active'"
}

Now we have arrived at the expected output with only the JSON object returned! By using this method, we are able to generate an immediately usable technical artifact as well as reduce the cost of generation by reducing output tokens.

Conclusion

To get started today with SnapGPT, request a free trial of SnapLogic or request a demo of the product. If you would like to use these concepts for building applications today, we recommend experimenting hands-on with the prompt engineering section in this post, using the same flow on a different DSL generation use case that suits your business, and diving deeper into the RAG features that are available through Amazon Bedrock.

SnapLogic and AWS have been able to partner effectively to build an advanced translator between human language and the complex schema of SnapLogic integration pipelines powered by Amazon Bedrock. Throughout this journey, we have seen how the output generated with Claude can be improved in text-to-pipeline applications using specific prompt engineering techniques. AWS and SnapLogic are excited to continue this partnership in Generative AI and look forward to future collaboration and innovation in this fast-moving space.

About the Authors

Greg Benson is a Professor of Computer Science at the University of San Francisco and Chief Scientist at SnapLogic. He joined the USF Department of Computer Science in 1998 and has taught undergraduate and graduate courses including operating systems, computer architecture, programming languages, distributed systems, and introductory programming. Greg has published research in the areas of operating systems, parallel computing, and distributed systems. Since joining SnapLogic in 2010, Greg has helped design and implement several key platform features including cluster processing, big data processing, the cloud architecture, and machine learning. He currently is working on Generative AI for data integration.

Aaron Kesler is the Senior Product Manager for AI products and services at SnapLogic, Aaron applies over ten years of product management expertise to pioneer AI/ML product development and evangelize services across the organization. He is the author of the upcoming book “What’s Your Problem?” aimed at guiding new product managers through the product management career. His entrepreneurial journey began with his college startup, STAK, which was later acquired by Carvertise with Aaron contributing significantly to their recognition as Tech Startup of the Year 2015 in Delaware. Beyond his professional pursuits, Aaron finds joy in golfing with his father, exploring new cultures and foods on his travels, and practicing the ukulele.

Rich Dill is a Principal Solutions Architect with experience cutting broadly across multiple areas of specialization. A track record of success spanning multi-platform enterprise software and SaaS. Well known for turning customer advocacy (serving as the voice of the customer) into revenue-generating new features and products. Proven ability to drive cutting-edge products to market and projects to completion on schedule and under budget in fast-paced onshore and offshore environments. A simple way to describe me: the mind of a scientist, the heart of an explorer and the soul of an artist.

Clay Elmore is an AI/ML Specialist Solutions Architect at AWS. After spending many hours in a materials research lab, his background in chemical engineering was quickly left behind to pursue his interest in machine learning. He has worked on ML applications in many different industries ranging from energy trading to hospitality marketing. Clay’s current work at AWS centers around helping customers bring software development practices to ML and generative AI workloads, allowing customers to build repeatable, scalable solutions in these complex environments. In his spare time, Clay enjoys skiing, solving Rubik’s cubes, reading, and cooking.

Sina Sojoodi is a technology executive, systems engineer, product leader, ex-founder and startup advisor. He joined AWS in March 2021 as a Principal Solutions Architect. Sina is currently the US-West ISV area lead Solutions Architect. He works with SaaS and B2B software companies to build and grow their businesses on AWS. Previous to his role at Amazon, Sina was a technology executive at VMware, and Pivotal Software (IPO in 2018, VMware M&A in 2020) and served multiple leadership roles including founding engineer at Xtreme Labs (Pivotal acquisition in 2013). Sina has dedicated the past 15 years of his work experience to building software platforms and practices for enterprises, software businesses and the public sector. He is an industry leader with a passion for innovation. Sina holds a BA from the University of Waterloo where he studied Electrical Engineering and Psychology.

Sandeep Rohilla is a Senior Solutions Architect at AWS, supporting ISV customers in the US West region. He focuses on helping customers architect solutions leveraging containers and generative AI on the AWS cloud. Sandeep is passionate about understanding customers’ business problems and helping them achieve their goals through technology. He joined AWS after working more than a decade as a solutions architect, bringing his 17 years of experience to bear. Sandeep holds an MSc. in Software Engineering from the University of the West of England in Bristol, UK.

Dr. Farooq Sabir is a Senior Artificial Intelligence and Machine Learning Specialist Solutions Architect at AWS. He holds PhD and MS degrees in Electrical Engineering from the University of Texas at Austin and an MS in Computer Science from Georgia Institute of Technology. He has over 15 years of work experience and also likes to teach and mentor college students. At AWS, he helps customers formulate and solve their business problems in data science, machine learning, computer vision, artificial intelligence, numerical optimization, and related domains. Based in Dallas, Texas, he and his family love to travel and go on long road trips.

Read More

This is a guest post by A.K Roy from Qualcomm AI.

Amazon Elastic Compute Cloud (Amazon EC2) DL2q instances, powered by Qualcomm AI 100 Standard accelerators, can be used to cost-efficiently deploy deep learning (DL) workloads in the cloud. They can also be used to develop and validate performance and accuracy of DL workloads that will be deployed on Qualcomm devices. DL2q instances are the first instances to bring Qualcomm’s artificial intelligent (AI) technology to the cloud.

With eight Qualcomm AI 100 Standard accelerators and 128 GiB of total accelerator memory, customers can also use DL2q instances to run popular generative AI applications, such as content generation, text summarization, and virtual assistants, as well as classic AI applications for natural language processing and computer vision. Additionally, Qualcomm AI 100 accelerators feature the same AI technology used across smartphones, autonomous driving, personal computers, and extended reality headsets, so DL2q instances can be used to develop and validate these AI workloads before deployment.

New DL2q instance highlights

Each DL2q instance incorporates eight Qualcomm Cloud AI100 accelerators, with an aggregated performance of over 2.8 PetaOps of Int8 inference performance and 1.4 PetaFlops of FP16 inference performance. The instance has an aggregate 112 of AI cores, accelerator memory capacity of 128 GB and memory bandwidth of 1.1 TB per second.

Each DL2q instance has 96 vCPUs, a system memory capacity of 768 GB and supports a networking bandwidth of 100 Gbps as well as Amazon Elastic Block Store (Amazon EBS) storage of 19 Gbps.

Qualcomm Cloud AI100 accelerator innovation

The Cloud AI100 accelerator system-on-chip (SoC) is a purpose-built, scalable multi-core architecture, supporting a wide range of deep-learning use-cases spanning from the datacenter to the edge. The SoC employs scalar, vector, and tensor compute cores with an industry-leading on-die SRAM capacity of 126 MB. The cores are interconnected with a high-bandwidth low-latency network-on-chip (NoC) mesh.

The AI100 accelerator supports a broad and comprehensive range of models and use-cases. The table below highlights the range of the model support.

* Most automotive networks are composite networks consisting of a fusion of individual networks.

The large on-die SRAM on the DL2q accelerator enables efficient implementation of advanced performance techniques such as MX6 micro-exponent precision for storing the weights and MX9 micro-exponent precision for accelerator-to-accelerator communication. The micro-exponent technology is described in the following Open Compute Project (OCP) industry announcement: AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Standardize Next-Generation Narrow Precision Data Formats for AI » Open Compute Project.

The instance user can use the following strategy to maximize the performance-per-cost:

• Store weights using the MX6 micro-exponent precision in the on-accelerator DDR memory. Using the MX6 precision maximizes the utilization of the available memory capacity and the memory-bandwidth to deliver best-in-class throughput and latency.
• Compute in FP16 to deliver the required use case accuracy, while using the superior on-chip SRAM and spare TOPs on the card, to implement high-performance low-latency MX6 to FP16 kernels.
• Use an optimized batching strategy and a higher batch-size by using the large on-chip SRAM available to maximize the reuse of weights, while retaining the activations on-chip to the maximum possible.

DL2q AI Stack and toolchain

The DL2q instance is accompanied by the Qualcomm AI Stack that delivers a consistent developer experience across Qualcomm AI in the cloud and other Qualcomm products. The same Qualcomm AI stack and base AI technology runs on the DL2q instances and Qualcomm edge devices, providing customers a consistent developer experience, with a unified API across their cloud, automotive, personal computer, extended reality, and smartphone development environments.

The toolchain enables the instance user to quickly onboard a previously trained model, compile and optimize the model for the instance capabilities, and subsequently deploy the compiled models for production inference use-cases in three steps shown in the following figure.

To learn more about tuning the performance of a model, see the Cloud AI 100 Key Performance Parameters Documentation.

Get started with DL2q instances

In this example, you compile and deploy a pre-trained BERT model from Hugging Face on an EC2 DL2q instance using a pre-built available DL2q AMI, in four steps.

You can use either a pre-built Qualcomm DLAMI on the instance or start with an Amazon Linux2 AMI and build your own DL2q AMI with the Cloud AI 100 Platform and Apps SDK available in this Amazon Simple Storage Service (Amazon S3) bucket: s3://ec2-linux-qualcomm-ai100-sdks/latest/.

The steps that follow use the pre-built DL2q AMI, Qualcomm Base AL2 DLAMI.

Use SSH to access your DL2q instance with the Qualcomm Base AL2 DLAMI AMI and follow steps 1 thru 4.

Step 1. Set up the environment and install required packages

1. Install Python 3.8.

   sudo amazon-linux-extras install python3.8

2. Set up the Python 3.8 virtual environment.

   python3.8 -m venv /home/ec2-user/userA/pyenv

3. Activate the Python 3.8 virtual environment.

   source /home/ec2-user/userA/pyenv/bin/activate

4. Install the required packages, shown in the requirements.txt document available at the Qualcomm public Github site.

   pip3 install -r requirements.txt

5. Import the necessary libraries.

   import transformers 
   from transformers import AutoTokenizer, AutoModelForMaskedLM
   import sys
   import qaic
   import os
   import torch
   import onnx
   from onnxsim import simplify
   import argparse
   import numpy as np

Step 2. Import the model

1. Import and tokenize the model.

   model_card = 'bert-base-cased'
   model = AutoModelForMaskedLM.from_pretrained(model_card)
   tokenizer = AutoTokenizer.from_pretrained(model_card)

2. Define a sample input and extract the inputIds and attentionMask.

   sentence = "The dog [MASK] on the mat."
   encodings = tokenizer(sentence, max_length=128, truncation=True, padding="max_length", return_tensors='pt')
   inputIds = encodings["input_ids"]
   attentionMask = encodings["attention_mask"]

3. Convert the model to ONNX, which can then be passed to the compiler.

   # Set dynamic dims and axes.
   dynamic_dims = {0: 'batch', 1 : 'sequence'}
   dynamic_axes = {
       "input_ids" : dynamic_dims,
       "attention_mask" : dynamic_dims,
       "logits" : dynamic_dims
   }
   input_names = ["input_ids", "attention_mask"]
   inputList = [inputIds, attentionMask]
   
   torch.onnx.export(
       model,
       args=tuple(inputList),
       f=f"{gen_models_path}/{model_base_name}.onnx",
       verbose=False,
       input_names=input_names,
       output_names=["logits"],
       dynamic_axes=dynamic_axes,
       opset_version=11,
   )

4. You’ll run the model in FP16 precision. So, you need to check if the model contains any constants beyond the FP16 range. Pass the model to the fix_onnx_fp16 function to generate the new ONNX file with the fixes required.

   from onnx import numpy_helper
           
   def fix_onnx_fp16(
       gen_models_path: str,
       model_base_name: str,
   ) -> str:
       finfo = np.finfo(np.float16)
       fp16_max = finfo.max
       fp16_min = finfo.min
       model = onnx.load(f"{gen_models_path}/{model_base_name}.onnx")
       fp16_fix = False
       for tensor in onnx.external_data_helper._get_all_tensors(model):
           nptensor = numpy_helper.to_array(tensor, gen_models_path)
           if nptensor.dtype == np.float32 and (
               np.any(nptensor > fp16_max) or np.any(nptensor < fp16_min)
           ):
               # print(f'tensor value : {nptensor} above {fp16_max} or below {fp16_min}')
               nptensor = np.clip(nptensor, fp16_min, fp16_max)
               new_tensor = numpy_helper.from_array(nptensor, tensor.name)
               tensor.CopyFrom(new_tensor)
               fp16_fix = True
               
       if fp16_fix:
           # Save FP16 model
           print("Found constants out of FP16 range, clipped to FP16 range")
           model_base_name += "_fix_outofrange_fp16"
           onnx.save(model, f=f"{gen_models_path}/{model_base_name}.onnx")
           print(f"Saving modified onnx file at {gen_models_path}/{model_base_name}.onnx")
       return model_base_name
   
   fp16_model_name = fix_onnx_fp16(gen_models_path=gen_models_path, model_base_name=model_base_name)

Step 3. Compile the model

The qaic-exec command line interface (CLI) compiler tool is used to compile the model. The input to this compiler is the ONNX file generated in step 2. The compiler produces a binary file (called QPC, for Qualcomm program container) in the path defined by -aic-binary-dir argument.

In the compile command below, you use four AI compute cores and a batch size of one to compile the model.

/opt/qti-aic/exec/qaic-exec 
-m=bert-base-cased/generatedModels/bert-base-cased_fix_outofrange_fp16.onnx 
-aic-num-cores=4 
-convert-to-fp16 
-onnx-define-symbol=batch,1 -onnx-define-symbol=sequence,128 
-aic-binary-dir=bert-base-cased/generatedModels/bert-base-cased_fix_outofrange_fp16_qpc 
-aic-hw -aic-hw-version=2.0 
-compile-only

The QPC is generated in the bert-base-cased/generatedModels/bert-base-cased_fix_outofrange_fp16_qpc folder.

Step 4. Run the model

Set up a session to run the inference on a Cloud AI100 Qualcomm accelerator in the DL2q instance.

The Qualcomm qaic Python library is a set of APIs that provides support for running inference on the Cloud AI100 accelerator.

1. Use the Session API call to create an instance of session. The Session API call is the entry point to using the qaic Python library.

   qpcPath = 'bert-base-cased/generatedModels/bert-base-cased_fix_outofrange_fp16_qpc'
   
   bert_sess = qaic.Session(model_path= qpcPath+'/programqpc.bin', num_activations=1)  
   bert_sess.setup() # Loads the network to the device. 
   
   # Here we are reading out all the input and output shapes/types
   input_shape, input_type = bert_sess.model_input_shape_dict['input_ids']
   attn_shape, attn_type = bert_sess.model_input_shape_dict['attention_mask']
   output_shape, output_type = bert_sess.model_output_shape_dict['logits']
   
   #create the input dictionary for given input sentence
   input_dict = {"input_ids": inputIds.numpy().astype(input_type), "attention_mask" : attentionMask.numpy().astype(attn_type)}
   
   #run inference on Cloud AI 100
   output = bert_sess.run(input_dict)

2. Restructure the data from output buffer with output_shape and output_type.

   token_logits = np.frombuffer(output['logits'], dtype=output_type).reshape(output_shape)

3. Decode the output produced.

   mask_token_logits = torch.from_numpy(token_logits[0, mask_token_index, :]).unsqueeze(0)
   top_5_results = torch.topk(mask_token_logits, 5, dim=1)
   print("Model output (top5) from Qualcomm Cloud AI 100:")
   for i in range(5):
       idx = top_5_results.indices[0].tolist()[i]
       val = top_5_results.values[0].tolist()[i]
       word = tokenizer.decode([idx])
       print(f"{i+1} :(word={word}, index={idx}, logit={round(val,2)})")

Here are the outputs for the input sentence “The dog [MASK] on the mat.”

1 :(word=sat, index=2068, logit=11.46)
2 :(word=landed, index=4860, logit=11.11)
3 :(word=spat, index=15732, logit=10.95)
4 :(word=settled, index=3035, logit=10.84)
5 :(word=was, index=1108, logit=10.75)

That’s it. With just a few steps, you compiled and ran a PyTorch model on an Amazon EC2 DL2q instance. To learn more about onboarding and compiling models on the DL2q instance, see the Cloud AI100 Tutorial Documentation.

To learn more about which DL model architectures are a good fit for AWS DL2q instances and the current model support matrix, see the Qualcomm Cloud AI100 documentation.

Available now

You can launch DL2q instances today in the US West (Oregon) and Europe (Frankfurt) AWS Regions as On-demand, Reserved, and Spot Instances, or as part of a Savings Plan. As usual with Amazon EC2, you pay only for what you use. For more information, see Amazon EC2 pricing.

DL2q instances can be deployed using AWS Deep Learning AMIs (DLAMI), and container images are available through managed services such as Amazon SageMaker, Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Container Service (Amazon ECS), and AWS ParallelCluster.

To learn more, visit the Amazon EC2 DL2q instance page, and send feedback to AWS re:Post for EC2 or through your usual AWS Support contacts.

About the authors

A.K Roy is a Director of Product Management at Qualcomm, for Cloud and Datacenter AI products and solutions. He has over 20 years of experience in product strategy and development, with the current focus of best-in-class performance and performance/$ end-to-end solutions for AI inference in the Cloud, for the broad range of use-cases, including GenAI, LLMs, Auto and Hybrid AI.

Jianying Lang is a Principal Solutions Architect at AWS Worldwide Specialist Organization (WWSO). She has over 15 years of working experience in HPC and AI field. At AWS, she focuses on helping customers deploy, optimize, and scale their AI/ML workloads on accelerated computing instances. She is passionate about combining the techniques in HPC and AI fields. Jianying holds a PhD degree in Computational Physics from the University of Colorado at Boulder.

Read More

Yes, the AWS re:Invent season is upon us and as always, the place to be is Las Vegas! You marked your calendars, you booked your hotel, and you even purchased the airfare. Now all you need is some guidance on generative AI and machine learning (ML) sessions to attend at this twelfth edition of re:Invent. And although generative AI has appeared in previous events, this year we’re taking it to the next level. In addition to several exciting announcements during keynotes, most of the sessions in our track will feature generative AI in one form or another, so we can truly call our track “Generative AI and ML.” In this post, we give you a sense of how the track is organized and highlight a few sessions we think you’ll like. And although our track focuses on generative AI, many other tracks have related sessions. Use the “Generative AI” tag as you are browsing the session catalog to find them.

The technical sessions in our track are divided into five areas. First, we’ll have a few foundational sessions related to various aspects of Amazon Bedrock—a fully managed generative AI service we launched earlier this year. These will help you understand the building blocks of your generative AI applications. Second, we’ll have sessions covering the common generative AI use cases and applications. Here you’ll also have a chance to discover novel use cases and techniques. Third, a number of sessions will be of interest to ML practitioners who build, deploy, and operationalize both traditional and generative AI models. This year, learn about LLMOps, not just MLOps! Then, as we started doing last re:Invent, we’ll be offering several sessions on how to build AI responsibly. The greater the power of latest transformer-based models, the greater the responsibility of all ML practitioners to do this right. Be sure to check out the session on the just launched PartyRock, an educational tool for providing any builder with low-friction access to learn through experimentation in a foundation model playground built on Amazon Bedrock. And last but not least (and always fun!) are the sessions dedicated to AWS DeepRacer!

Generative AI is at the heart of the AWS Village this year. Interact with several demos that feature new applications, including a competition that involves using generative AI tech to pilot a drone around an obstacle course. Talk with AWS experts in 14 different industries and explore industry-specific generative AI use cases, including demos from advertising and marketing, aerospace and satellite, manufacturing, and more. The Emerging Tech Zone within the Expo features innovative startups that were selected into the AWS Generative AI Accelerator and the NVIDIA Inception 100 programs.

If you’re new to re:Invent, you can attend sessions of the following types:

• Keynotes – Join in person or virtually and learn about all the exciting announcements.
• Innovation Talks – Learn about the latest cloud technology from AWS technology leaders and discover how these advancements can help you push your business forward. These sessions will be livestreamed, recorded, and published to YouTube.
• Breakout sessions – These 60-minute sessions are expected to have broad appeal, are delivered to larger audiences, and will be recorded. If you miss them, you can watch them on demand after re:Invent.
• Chalk talks – Enjoy 60 minutes of content delivered to smaller audiences with an interactive whiteboarding session. Chalk talks are where discussions happen, and these offer you the greatest opportunity to ask questions or share your opinion.
• Workshops – In these hands-on learning opportunities, in the course of 2 hours, you’ll be able to build a solution to a problem, and understand the inner workings of the resulting infrastructure and cross-service interaction. Bring your laptop and be ready to learn!
• Builders’ sessions – These highly interactive 60-minute mini-workshops are conducted in small groups of fewer than 10 attendees. Some of these appeal to beginners, and others are on specialized topics.
• NEW! Code talks – In this new session type for re:Invent 2023, code talks are similar to our popular chalk talk format, but instead of focusing on an architecture solution with whiteboarding, the speakers lead an interactive discussion featuring live coding or code samples. These 60-minute sessions focus on the actual code that goes into building a solution. Attendees are encouraged to ask questions and follow along.

If you have reserved your seat at any of the sessions, great! If not, we always set aside some spots for walk-ins, so make a plan and come to the session early.

To help you plan your agenda for this year’s re:Invent, here are some highlights of the generative AI and ML track. So buckle up, and start registering for your favorite sessions.

Visit the session catalog to learn about all our generative AI and ML sessions.

Keynotes

Adam Selipsky, Chief Executive Officer, Amazon Web Services – Keynote

Tuesday November 28 | 8:30 AM – 10:30 AM (PST) | The Venetian

Join Adam Selipsky, CEO of Amazon Web Services, as he shares his perspective on cloud transformation. He highlights innovations in data, infrastructure, and artificial intelligence and machine learning that are helping AWS customers achieve their goals faster, mine untapped potential, and create a better future.

Swami Sivasubramanian, Vice President of AWS Data and Machine Learning – Keynote

Wednesday November 29 | 8:30 AM – 10:30 AM (PST) | The Venetian

A powerful relationship between humans, data, and AI is unfolding right before us. Generative AI is augmenting our productivity and creativity in new ways, while also being fueled by massive amounts of enterprise data and human intelligence. Join Swami Sivasubramanian, Vice President of Data and AI at AWS, to discover how you can use your company data to build differentiated generative AI applications and accelerate productivity for employees across your organization. Also hear from customer speakers with real-world examples of how they’ve used their data to support their generative AI use cases and create new experiences for their customers.

Innovation Talks

Dr. Bratin Saha, VP of AWS AI and ML Services | AIM245-INT | Innovate faster with generative AI

Wednesday November 29 | 1:00 PM – 2:00 PM (PST) | Venetian | Level 5 | Palazzo Ballroom B

With the emergence of generative AI, we are at a tipping point in the widespread adoption of machine learning. Join Dr. Bratin Saha, VP of AWS AI and ML Services, to hear how customers across industries are transforming their business with the latest breakthroughs in AI and ML, including generative AI. Discover the latest AWS innovations, hear from top customers, and explore where AI/ML is headed.

Francessca Vasquez, Vice President of Professional Services | ARC217-INT | From hype to impact: Building a generative AI architecture

Wednesday November 29 | 11:30 AM – 12:30 PM (PST) | Venetian | Level 5 | Palazzo Ballroom B

Generative AI represents a paradigm shift for how companies operate today. Generative AI is empowering developers to reimagine customer experiences and applications while transforming virtually every industry. Organizations are rapidly innovating to create the right architecture for scaling generative AI securely, economically, and responsibly to deliver business value. In this talk, learn how leaders are modernizing their data foundation, selecting industry-leading foundation models, and deploying purpose-built accelerators to unlock the possibilities of generative AI.

Shaown Nandi, AWS Director of Technology for Industries and Strategic Accounts | AIM248-INT | Unlocking the industry potential of generative AI

Wednesday November 29 | 4:00 PM – 5:00 PM (PST) | Venetian | Level 5 | Palazzo Ballroom B

Generative AI has captured the imagination of many industries and is poised to bring in the next wave of technological advancements. In this innovation talk, hear how the largest industries, from healthcare and financial services to automotive and media and entertainment, are using generative AI to drive outcomes for their customers. Join Shaown Nandi, AWS Director of Technology for Industries and Strategic Accounts, and industry leaders to hear how generative AI is accelerating content creation and helping organizations reimagine customer experiences.

Mai-Lan Tomsen Bukovec, Vice President, Technology | AIM250-INT | Putting your data to work with generative AI

Thursday November 30 | 12:30 PM – 1:30 PM (PST) | Venetian | Level 5 | Palazzo Ballroom B

How can you turn your data lake into a business advantage with generative AI? In this talk, explore strategies for putting your proprietary datasets to work when building unique, differentiated generative AI solutions. Learn how to utilize your datasets using Amazon SageMaker and Amazon Bedrock as well as popular frameworks like PyTorch with AWS compute, storage, and analytics. Hear best practices for using unstructured (video, image, PDF), semi-structured (Parquet), and table-formatted (Iceberg) data for training, fine-tuning, checkpointing, and prompt engineering. Also hear different architectural patterns that customers use today to harness their business data for customized generative AI solutions.

Breakout sessions

AIM218 (LVL 200) | Build your first generative AI application with Amazon Bedrock

Monday November 27 | 2:30 PM – 3:30 PM (PST)

We are truly at an exciting inflection point in the widespread adoption of ML with the growth of generative AI applications. In this session, learn how to build your first generative AI application with key services such as Amazon Bedrock. Get hints and tips for getting started fast, and see example reference architectures for common use cases built with AWS AI and ML such as self-service customer support, text analysis, report generation, post-call analysis, and forecasting trends.

Reserve your seat now!

AIM225 (LVL 200) | Drive personalized CX using generative AI and Amazon Personalize

Tuesday November 28 | 5:00 PM – 6:00 PM (PST)

Delivering the best experience is critical to capture and retain customers today. With generative AI, it is possible to hyper-personalize targeted recommendations for shopping and streaming. While standard taglines like “People who bought this also bought . . .” or “Because you watched . . .” entice some, they don’t fully address individual interests. Companies must find ways to dynamically generate compelling, highly customized content. Amazon Personalize delivers capabilities powered by ML and generative AI to help brands create meaningful experiences. Join this session to hear from powerhouse AWS media customer FOX and learn how hyper-personalized experiences can be used to build engagement and drive revenue.

Reserve your seat now!

AIM327 (LVL 300) | Scaling FM inference to hundreds of models with Amazon SageMaker

Wednesday November 29 | 4:30 PM – 5:30 PM (PST)

Companies need robust and cost-effective solutions to deploy foundation models at scale. Additionally, SaaS providers need scalable and cost-effective ways to serve hundreds of models to their customers. This session explores how to use Amazon SageMaker to roll out hundreds of FMs cost-effectively at scale. Get a detailed overview of deployment strategies to support large-scale generative AI inferencing for SaaS, and learn how to architect solutions that maximize scaling capabilities for performance and cost.

Reserve your seat now!

AIM333 (LVL 300) | Explore text-generation FMs for top use cases with Amazon Bedrock

Tuesday November 28| 2:00 PM – 3:00 PM (PST)

Foundation models can be used for natural language processing tasks such as summarization, text generation, classification, open-ended Q&A, and information extraction. With Amazon Bedrock, you can choose powerful FMs from AI21 Labs, Anthropic, and Cohere to find the right FM for your use case such as the Jurassic-2, Claude, and Command families of text-generation FMs. Join this session to learn which FM is best suited for your use case.

Reserve your seat now!

AIM332 (LVL 300) | Explore image generation and search with FMs on Amazon Bedrock

Thursday November 30 | 11:00 AM – 12:00 PM (PST)

Foundation models understand multiple forms of input, such as images and texts. Join this session to learn how to build transformational experiences using images in Amazon Bedrock.

Reserve your seat now!

AIM377 (LVL 300) | Prompt engineering best practices for LLMs on Amazon Bedrock

Monday November 27 | 9:00 AM – 10:00 AM (PST)

Prompt engineering is the process of guiding large language models to produce desired outputs. In this session, get an overview of prompt engineering best practices and learn how to choose the most appropriate formats, phrases, words, and symbols to get the most out of generative AI solutions while improving accuracy and performance. This session uses the Claude 2 LLM as an example of how prompt engineering helps to solve complex customer use cases. Also learn how prompts can be integrated with your architecture and how to use API parameters for tuning the model parameters using Amazon Bedrock.

Reserve your seat now!

Chalk talks

AIM341 (LVL 300) | Deliver customized search capabilities using Amazon Bedrock

Wednesday November 29 | 5:30 PM – 6:30 PM (PST)

Vector embeddings are numerical representations of your text, image, audio, and video data that can be used to understand the relationship between sentences or words to find more relevant and contextual information in response to a user query. Embeddings can be stored in a database and are used to enable streamlined and more accurate searches. You can use an embeddings model in Amazon Bedrock to create vectors of your organization’s data, which can then be used to enable semantic search. Join this hands-on chalk talk to learn how.

Reserve your seat now!

AIM340-R (LVL 300) | Customize your FMs securely to deliver differentiated experiences

Wednesday November 29 | 6:00 PM – 7:00 PM (PST)

Foundation model customizations help you build differentiated generative AI applications using your own data. It’s easy to securely customize models in Amazon Bedrock. You can point Amazon Bedrock at a few labeled examples in Amazon S3, and the service can fine-tune the FM for a particular task without having to annotate large volumes of data; none of your data is used to train the original base FMs. Join this chalk talk for a deep dive on FM customizations through an interactive demo.

Reserve your seat now!

This session will be repeated Thursday, November 30 11:00 AM – 12:00 PM (PST), and Friday, December 1 8:30 AM – 9:30 AM PST.

AIM342 (LVL 300) | Advancing responsible AI: Assessing and mitigating risk

Wednesday November 29 | 4:30 PM – 5:30 PM (PST)

Risk assessment is an essential part of developing AI solutions responsibly, especially with emerging industry standards and laws regarding AI risk, such as ISO 42001 and the EU AI Act. This chalk talk provides an introduction to best practices for risk assessment related to fairness, robustness, explainability, privacy and security, transparency, and governance. Explore examples to estimate the severity and likelihood of potential events that could be harmful. Learn about Amazon SageMaker tooling for model governance, bias, explainability, and monitoring, and about transparency in the form of service cards as potential risk mitigation strategies.

Reserve your seat now!

AIM347-R (LVL 300) | Next-generation ML builder experience

Thursday November 30 | 4:00 PM – 5:00 PM (PST)

Amazon SageMaker offers different integrated development environments (IDEs) that are purpose-built for machine learning. In this chalk talk, learn how to select and use your preferred environment to perform end-to-end ML development steps, from preparing data to building, training, and deploying your ML models. Discover how you can quickly upload data, create new notebooks, train and tune models, move back and forth between steps to adjust experiments, collaborate seamlessly within your organization, and deploy models to production all in one place.

Reserve your seat now!

This session will be repeated Friday, December 1 9:00 AM – 10:00 AM (PST), and Friday, December 1 11:30 AM – 12:00 PM (PST).

AIM352-R (LVL 300) | Securely build generative AI apps and control data with Amazon Bedrock

Monday November 27 | 11:30 AM – 12:30 PM (PST)

Generative AI applications have captured widespread attention; however, they have also introduced new security challenges, especially around the handling of customer data. Organizations want to ensure that their data remains safe and secure while working with foundation models and don’t want to worry about their data being used to train an FM. Amazon Bedrock provides comprehensive data protection and privacy. In this chalk talk, explore architectures, data flows, and security-related aspects of model fine-tuning, as well as prompting and inference, while you learn about Amazon Bedrock’s security capabilities.

Reserve your seat now!

This session will be repeated Wednesday, November 29 6:00 PM – 7:00 PM (PST), and Thursday, November 30 4:00 PM – 5:00 PM (PST).

AIM404 (LVL 400) | Train and deploy FMs on Amazon EC2 and Amazon SageMaker, feat. Flip AI

Wednesday November 29 | 2:30 PM – 3:30 PM (PST)

Organizations that are running machine learning systems and generative AI applications on their local laptops/servers want to take advantage of the scalability and performance of the AWS Cloud. In this chalk talk, hear about compute and ML services from self-managed Amazon EC2 to fully managed Amazon SageMaker that you can use to build, train, and deploy foundation models. See a demo of how you can fine-tune a Stable Diffusion model on Amazon EC2 and then deploy it on SageMaker using the AWS Deep Learning AMIs (DLAMI) and AWS Deep Learning Containers. Also, hear how Flip AI built their own models using these AWS services.

Reserve your seat now!

Workshops

AIM302 (LVL 300) | Use generative AI to extract insights from contact center recordings

Monday November 27 | 8:30 AM – 10:30 AM (PST)

Learn how to derive insights from contact center recordings and other media using Amazon Transcribe and generative AI. In this workshop, learn how to combine automatic call recording, transcription, post-call analysis, sentiment analysis, issue detection, and call summarization from your own telephony recordings (Cisco, Genesys, Talkdesk, Avaya, and more) using AWS Contact Center Intelligence (CCI) solutions and generative AI. See demos on how to build analytics dashboards and integrations between LLMs and Amazon QuickSight to visualize your key metrics. You must bring your laptop to participate.

Reserve your seat now!

AIM307 (LVL 300) | Retrieval Augmented Generation with Amazon Bedrock

Wednesday November 29 | 8:30 AM – 10:30 AM (PST)

Large language models are often limited by the data they were trained on and don’t always provide up-to-date responses—or worse, they make things up. To overcome this limitation, you can supplement prompts with up-to-date information using embeddings stored in vector databases, a process known as Retrieval Augmented Generation (RAG). With supplemental information in the prompt providing more context, the LLM can respond more accurately and is less likely to hallucinate. In this workshop, learn how to use vector databases with Amazon Bedrock, a service that makes foundation models from Amazon and leading AI companies available via a single API. You must bring your laptop to participate.

Reserve your seat now!

AIM304 (LVL 300) | How to generate text responsibly using foundation models on AWS

Wednesday November 29 | 5:30 PM – 7:30 PM (PST)

Foundation models such as Claude are commonly used to create new pieces of original content, such as short stories, essays, social media posts, and webpage copy, and also to summarize text from articles, blog posts, books, and more. In this workshop, learn how you can generate text in minutes using foundation models available through Amazon Bedrock in a responsible way. You must bring your laptop to participate.

Reserve your seat now!

Code talks

AIM364-R (LVL 300) | Boost ML development with Amazon SageMaker Studio notebooks

Tuesday November 28 | 4:00 PM – 5:00 PM (PST)

Amazon SageMaker Studio notebooks are collaborative notebooks that you can launch quickly and that can help you integrate with purpose-built ML tools in SageMaker and other AWS services for complete ML development. In this code talk, learn how to prepare data at scale using built-in data preparation assistance, co-edit the same notebook in real time, and automate conversion of notebook code to production-ready jobs. This talk also introduces the new generative AI-powered features that can help you maximize productivity, write higher-quality code, and improve security.

Reserve your seat now!

This session will be repeated Wednesday, November 29 12:00 PM – 1:00 PM (PST).

Builders’ sessions

AIM219-R (LVL 200) | Learn and experiment with LLMs in Amazon SageMaker Studio Lab

Monday November 27 | 10:00 AM – 11:00 AM (PST)

Machine learning can sound complicated, but Amazon SageMaker Studio Lab makes it easier for anyone to get started at no cost. In this hands-on builders’ session, be guided through the basics of experimenting with large language models in Amazon SageMaker Studio Lab. No prior machine learning experience is required. You must bring your laptop to participate.

This session will be repeated Monday, November 27 4:00 PM – 5:00 PM (PST), Tuesday, November 28 3:30 PM – 4:30 PM (PST), Wednesday, November 29, 12:00 PM – 1:00 PM (PST), and Thursday, November 30 11:30 AM – 12:30 PM (PST).

Reserve your seat now!

AWS DeepRacer

Get ready to race with AWS DeepRacer at re:Invent 2023!

Developers, fasten your seatbelts—AWS DeepRacer is bringing ML to everyone at re:Invent! Whether you’re looking to get started with ML or improve your skills, AWS DeepRacer offers an exciting way to get hands-on with ML.

Watch the world’s top 72 racers of the AWS DeepRacer 2023 League battle it out Monday through Wednesday at our Championship Stadium at the Venetian Expo. It will all come down to the finale on Wednesday (November 29) at 2:30 PM (PST) as the eight finalists compete for the cup and $44,000 in prize money. You can also get behind the wheel yourself on November 30, when the track opens for the 2024 Open Racing. Post the fastest time and you’ll win a ticket back to Vegas for the 2024 Championship!

Dive into 10 not-to-miss workshops where you’ll learn to train reinforcement learning models, solve business problems with generative AI, and more. Want to learn tips and tricks from the best racers in the world? Be sure to check out our DPR301 workshop featuring five of our top AWS DeepRacer League Champions who will be sharing their approaches for training their AWS DeepRacer models and answering questions during an open Q&A.

Don’t forget to check out the rest of the AWS DeepRacer workshops before they fill up to reserve your spot! Whether you take a workshop, take a spin in our gamified virtual racing simulator, catch the global competition, or test your own ML model on the track, AWS DeepRacer brings the thrill of high-speed racing to hands-on machine learning at re:Invent. Let the countdown begin. We can’t wait to see you in Las Vegas!

See you at re:Invent!

Make sure to check out the re:Invent content catalog and the generative AI at re:Invent guide for more gen AI and ML content at re:Invent. We’ll see you there!

About the authors

Denis V. Batalov is a 17-year Amazon veteran and a PhD in Machine Learning, Denis worked on such exciting projects as Search Inside the Book, Amazon Mobile apps and Kindle Direct Publishing. Since 2013 he has helped AWS customers adopt AI/ML technology as a Solutions Architect. Currently, Denis is a Worldwide Tech Leader for AI/ML responsible for the functioning of AWS ML Specialist Solutions Architects globally. Denis is a frequent public speaker, you can follow him on Twitter @dbatalov.

Paxton Hall is a Marketing Program Manager for the AWS AI/ML Community on the AI/ML Education team at AWS. He has worked in retail and experiential marketing for the past 7 years, focused on developing communities and marketing campaigns. Out of the office, he’s passionate about public lands access and conservation, and enjoys backcountry skiing, climbing, biking, and hiking throughout Washington’s Cascade mountains.

Read More