Monthly Archives: October 2024

Large language models (LLMs) are very large deep-learning models that are pre-trained on vast amounts of data. LLMs are incredibly flexible. One model can perform completely different tasks such as answering questions, summarizing documents, translating languages, and completing sentences. LLMs have the potential to revolutionize content creation and the way people use search engines and virtual assistants. Retrieval Augmented Generation (RAG) is the process of optimizing the output of an LLM, so it references an authoritative knowledge base outside of its training data sources before generating a response. While LLMs are trained on vast volumes of data and use billions of parameters to generate original output, RAG extends the already powerful capabilities of LLMs to specific domains or an organization’s internal knowledge base—without having to retrain the LLMs. RAG is a fast and cost-effective approach to improve LLM output so that it remains relevant, accurate, and useful in a specific context. RAG introduces an information retrieval component that uses the user input to first pull information from a new data source. This new data from outside of the LLM’s original training data set is called external data. The data might exist in various formats such as files, database records, or long-form text. An AI technique called embedding language models converts this external data into numerical representations and stores it in a vector database. This process creates a knowledge library that generative AI models can understand.

RAG introduces additional data engineering requirements:

• Scalable retrieval indexes must ingest massive text corpora covering requisite knowledge domains.
• Data must be preprocessed to enable semantic search during inference. This includes normalization, vectorization, and index optimization.
• These indexes continuously accumulate documents. Data pipelines must seamlessly integrate new data at scale.
• Diverse data amplifies the need for customizable cleaning and transformation logic to handle the quirks of different sources.

In this post, we will explore building a reusable RAG data pipeline on LangChain—an open source framework for building applications based on LLMs—and integrating it with AWS Glue and Amazon OpenSearch Serverless. The end solution is a reference architecture for scalable RAG indexing and deployment. We provide sample notebooks covering ingestion, transformation, vectorization, and index management, enabling teams to consume disparate data into high-performing RAG applications.

Data preprocessing for RAG

Data pre-processing is crucial for responsible retrieval from your external data with RAG. Clean, high-quality data leads to more accurate results with RAG, while privacy and ethics considerations necessitate careful data filtering. This lays the foundation for LLMs with RAG to reach their full potential in downstream applications.

To facilitate effective retrieval from external data, a common practice is to first clean up and sanitize the documents. You can use Amazon Comprehend or AWS Glue sensitive data detection capability to identify sensitive data and then use Spark to clean up and sanitize the data. The next step is to split the documents into manageable chunks. The chunks are then converted to embeddings and written to a vector index, while maintaining a mapping to the original document. This process is shown in the figure that follows. These embeddings are used to determine semantic similarity between queries and text from the data sources

Solution overview

In this solution, we use LangChain integrated with AWS Glue for Apache Spark and Amazon OpenSearch Serverless. To make this solution scalable and customizable, we use Apache Spark’s distributed capabilities and PySpark’s flexible scripting capabilities. We use OpenSearch Serverless as a sample vector store and use the Llama 3.1 model.

The benefits of this solution are:

• You can flexibly achieve data cleaning, sanitizing, and data quality management in addition to chunking and embedding.
• You can build and manage an incremental data pipeline to update embeddings on Vectorstore at scale.
• You can choose a wide variety of embedding models.
• You can choose a wide variety of data sources including databases, data warehouses, and SaaS applications supported in AWS Glue.

This solution covers the following areas:

• Processing unstructured data such as HTML, Markdown, and text files using Apache Spark. This includes distributed data cleaning, sanitizing, chunking, and embedding vectors for downstream consumption.
• Bringing it all together into a Spark pipeline that incrementally processes sources and publishes vectors to an OpenSearch Serverless
• Querying the indexed content using the LLM model of your choice to provide natural language answers.

Prerequisites

To continue this tutorial, you must create the following AWS resources in advance:

• An Amazon Simple Storage Service (Amazon S3) bucket for storing data
• An AWS Identity and Access Management (IAM) role for your AWS Glue notebook as instructed in Set up IAM permissions for AWS Glue Studio. It requires IAM permission for OpenSearch Service Serverless. Here’s an example policy:

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Sid": "OpenSearchServerless",
              "Effect": "Allow",
              "Action": [
                  "aoss:APIAccessAll",
                  "aoss:CreateAccessPolicy",
                  "aoss:CreateCollection",
                  "aoss:CreateSecurityPolicy",
                  "aoss:DeleteAccessPolicy",
                  "aoss:DeleteCollection",
                  "aoss:DeleteSecurityPolicy",
                  "aoss:ListCollections"
              ],
              "Resource": "*"
          }
      ]
  }

Complete the following steps to launch an AWS Glue Studio notebook:

1. Download the Jupyter Notebook file.
2. On the AWS Glue console, chooseNotebooks in the navigation pane.
3. Under Create job, select Notebook.
4. For Options, choose Upload Notebook.
5. Choose Create notebook. The notebook will start up in a minute.

6. Run the first two cells to configure an AWS Glue interactive session.

Now you have configured the required settings for your AWS Glue notebook.

Vector store setup

First, create a vector store. A vector store provides efficient vector similarity search by providing specialized indexes. RAG complements LLMs with an external knowledge base that’s typically built using a vector database hydrated with vector-encoded knowledge articles.

In this example, you will use Amazon OpenSearch Serverless for its simplicity and scalability to support a vector search at low latency and up to billions of vectors. Learn more in Amazon OpenSearch Service’s vector database capabilities explained.

Complete the following steps to set up OpenSearch Serverless:

1. For the cell under Vectorestore Setup, replace <your-iam-role-arn> with your IAM role Amazon Resource Name (ARN), replace <region> with your AWS Region, and run the cell.
2. Run the next cell to create the OpenSearch Serverless collection, security policies, and access policies.

You have provisioned OpenSearch Serverless successfully. Now you’re ready to inject documents into the vector store.

Document preparation

In this example, you will use a sample HTML file as the HTML input. It’s an article with specialized content that LLMs cannot answer without using RAG.

1. Run the cell under Sample document download to download the HTML file, create a new S3 bucket, and upload the HTML file to the bucket.

2. Run the cell under Document preparation. It loads the HTML file into Spark DataFrame df_html.

3. Run the two cells under Parse and clean up HTMLto define functions parse_html and format_md. We use Beautiful Soup to parse HTML, and convert it to Markdown using markdownify in order to use MarkdownTextSplitter for chunking. These functions will be used inside a Spark Python user-defined function (UDF) in later cells.

4. Run the cell under Chunking HTML. The example uses LangChain’s MarkdownTextSplitter to split the text along markdown-formatted headings into manageable chunks. Adjusting chunk size and overlap is crucial to help prevent the interruption of contextual meaning, which can affect the accuracy of subsequent vector store searches. The example uses a chunk size of 1,000 and a chunk overlap of 100 to preserve information continuity, but these settings can be fine-tuned to suit different use cases.

5. Run the three cells under Embedding. The first two cells configure LLMs and deploy them through Amazon SageMaker In the third cell, the function process_batchinjects the documents into the vector store through OpenSearch implementation inside LangChain, which inputs the embeddings model and the documents to create the entire vector store.

6. Run the two cells under Pre-process HTML document. The first cell defines the Spark UDF, and the second cell triggers the Spark action to run the UDF per record containing the entire HTML content.

You have successfully ingested an embedding into the OpenSearch Serverless collection.

Question answering

In this section, we are going to demonstrate the question-answering capability using the embedding ingested in the previous section.

1. Run the two cells under Question Answering to create the OpenSearchVectorSearch client, the LLM using Llama 3.1, and define RetrievalQA where you can customize how the documents fetched should be added to the prompt using the chain_type Optionally, you can choose other foundation models (FMs). For such cases, refer to the model card to adjust the chunking length.

2. Run the next cell to do a similarity search using the query “What is Task Decomposition?” against the vector store providing the most relevant information. It takes a few seconds to make documents available in the index. If you get an empty output in the next cell, wait 1-3 minutes and retry.

Now that you have the relevant documents, it’s time to use the LLM to generate an answer based on the embeddings.

3. Run the next cell to invoke the LLM to generate an answer based on the embeddings.

As you expect, the LLM answered with a detailed explanation about task decomposition. For production workloads, balancing latency and cost efficiency is crucial in semantic searches through vector stores. It’s important to select the most suitable k-NN algorithm and parameters for your specific needs, as detailed in this post. Additionally, consider using product quantization (PQ) to reduce the dimensionality of embeddings stored in the vector database. This approach can be advantageous for latency-sensitive tasks, though it might involve some trade-offs in accuracy. For additional details, see Choose the k-NN algorithm for your billion-scale use case with OpenSearch.

Clean up

Now to the final step, cleaning up the resources:

1. Run the cell under Clean up to delete S3, OpenSearch Serverless, and SageMaker resources.

2. Delete the AWS Glue notebook job.

Conclusion

This post explored a reusable RAG data pipeline using LangChain, AWS Glue, Apache Spark, Amazon SageMaker JumpStart, and Amazon OpenSearch Serverless. The solution provides a reference architecture for ingesting, transforming, vectorizing, and managing indexes for RAG at scale by using Apache Spark’s distributed capabilities and PySpark’s flexible scripting capabilities. This enables you to preprocess your external data in the phases including cleaning, sanitization, chunking documents, generating vector embeddings for each chunk, and loading into a vector store.

About the Authors

Noritaka Sekiyama is a Principal Big Data Architect on the AWS Glue team. He is responsible for building software artifacts to help customers. In his spare time, he enjoys cycling with his road bike.

Akito Takeki is a Cloud Support Engineer at Amazon Web Services. He specializes in Amazon Bedrock and Amazon SageMaker. In his spare time, he enjoys travelling and spending time with his family.

Ray Wang is a Senior Solutions Architect at Amazon Web Services. Ray is dedicated to building modern solutions on the Cloud, especially in NoSQL, big data, and machine learning. As a hungry go-getter, he passed all 12 AWS certificates to make his technical field not only deep but wide. He loves to read and watch sci-fi movies in his spare time.

Vishal Kajjam is a Software Development Engineer on the AWS Glue team. He is passionate about distributed computing and using ML/AI for designing and building end-to-end solutions to address customers’ Data Integration needs. In his spare time, he enjoys spending time with family and friends.

Savio Dsouza is a Software Development Manager on the AWS Glue team. His team works on generative AI applications for the Data Integration domain and distributed systems for efficiently managing data lakes on AWS and optimizing Apache Spark for performance and reliability.

Kinshuk Pahare is a Principal Product Manager on AWS Glue. He leads a team of Product Managers who focus on AWS Glue platform, developer experience, data processing engines, and generative AI. He had been with AWS for 4.5 years. Before that he did product management at Proofpoint and Cisco.

Read More

The AWS Generative AI Innovation Center (GenAIIC) is a team of AWS science and strategy experts who have deep knowledge of generative AI. They help AWS customers jumpstart their generative AI journey by building proofs of concept that use generative AI to bring business value. Since the inception of AWS GenAIIC in May 2023, we have witnessed high customer demand for chatbots that can extract information and generate insights from massive and often heterogeneous knowledge bases. Such use cases, which augment a large language model’s (LLM) knowledge with external data sources, are known as Retrieval-Augmented Generation (RAG).

This two-part series shares the insights gained by AWS GenAIIC from direct experience building RAG solutions across a wide range of industries. You can use this as a practical guide to building better RAG solutions.

In this first post, we focus on the basics of RAG architecture and how to optimize text-only RAG. The second post outlines how to work with multiple data formats such as structured data (tables, databases) and images.

Anatomy of RAG

RAG is an efficient way to provide an FM with additional knowledge by using external data sources and is depicted in the following diagram:

• Retrieval: Based on a user’s question (1), relevant information is retrieved from a knowledge base (2) (for example, an OpenSearch index).
• Augmentation: The retrieved information is added to the FM prompt (3.a) to augment its knowledge, along with the user query (3.b).
• Generation: The FM generates an answer (4) by using the information provided in the prompt.

The following is a general diagram of a RAG workflow. From left to right are the retrieval, the augmentation, and the generation. In practice, the knowledge base is often a vector store.

A deeper dive in the retriever

In a RAG architecture, the FM will base its answer on the information provided by the retriever. Therefore, a RAG is only as good as its retriever, and many of the tips that we share in our practical guide are about how to optimize the retriever. But what is a retriever exactly? Broadly speaking, a retriever is a module that takes a query as input and outputs relevant documents from one or more knowledge sources relevant to that query.

Document ingestion

In a RAG architecture, documents are often stored in a vector store. As shown in the following diagram, vector stores are populated by chunking the documents into manageable pieces (1) (if a document is short enough, chunking might not be required) and transforming each chunk of the document into a high-dimensional vector using a vector embedding (2), such as the Amazon Titan embeddings model. These embeddings have the characteristic that two chunks of texts that are semantically close have vector representations that are also close in that embedding (in the sense of the cosine or Euclidean distance).

The following diagram illustrates the ingestion of text documents in the vector store using an embedding model. Note that the vectors are stored alongside the corresponding text chunk (3), so that at retrieval time, when you identify the chunks closest to the query, you can return the text chunk to be passed to the FM prompt.

Semantic search

Vector stores allow for efficient semantic search: as shown in the following diagram, given a user query (1), we vectorize it (2) (using the same embedding as the one that was used to build the vector store) and then look for the nearest vectors in the vector store (3), which will correspond to the document chunks that are semantically closest to the initial query (4). Although vector stores and semantic search have become the default in RAG architectures, more traditional keyword-based search is still valuable, especially when searching for domain-specific words (such as technical jargon) or names. Hybrid search is a way to use both semantic search and keywords to rank a document, and we will give more details on this technique in the section on advanced RAG techniques.

The following diagram illustrates the retrieval of text documents that are semantically close to the user query. You must use the same embedding model at ingestion time and at search time.

Implementation on AWS

A RAG chatbot can be set up in a matter of minutes using Amazon Bedrock Knowledge Bases. The knowledge base can be linked to an Amazon Simple Storage Service (Amazon S3) bucket and will automatically chunk and index the documents it contains in an OpenSearch index, which will act as the vector store. The retrieve_and_generate API does both the retrieval and a call to an FM (Amazon Titan or Anthropic’s Claude family of models on Amazon Bedrock), for a fully managed solution. The retrieve API only implements the retrieval component and allows for a more custom approach downstream, such as document post processing before calling the FM separately.

In this blog post, we will provide tips and code to optimize a fully custom RAG solution with the following components:

• An OpenSearch Serverless vector search collection as the vector store
• Custom chunking and ingestion functions to ingest the documents in the OpenSearch index
• A custom retrieval function that takes a user query as an input and outputs the relevant documents from the OpenSearch index
• FM calls to your model of choice on Amazon Bedrock to generate the final answer.

In this post, we focus on a custom solution to help readers understand the inner workings of RAG. Most of the tips we provide can be adapted to work with Amazon Bedrock Knowledge Bases, and we will point this out in the relevant sections.

Overview of RAG use cases

While working with customers on their generative AI journey, we encountered a variety of use cases that fit within the RAG paradigm. In traditional RAG use cases, the chatbot relies on a database of text documents (.doc, .pdf, or .txt). In part 2 of this post, we will discuss how to extend this capability to images and structured data. For now, we’ll focus on a typical RAG workflow: the input is a user question, and the output is the answer to that question, derived from the relevant text chunks or documents retrieved from the database. Use cases include the following:

• Customer service– This can include the following:
  • Internal– Live agents use an internal chatbot to help them answer customer questions.
  • External– Customers directly chat with a generative AI chatbot.
  • Hybrid– The model generates smart replies for live agents that they can edit before sending to customers.
• Employee training and resources– In this use case, chatbots can use employee training manuals, HR resources, and IT service documents to help employees onboard faster or find the information they need to troubleshoot internal issues.
• Industrial maintenance– Maintenance manuals for complex machines can have several hundred pages. Building a RAG solution around these manuals helps maintenance technicians find relevant information faster. Note that maintenance manuals often have images and schemas, which could put them in a multimodal bucket.
• Product information search– Field specialists need to identify relevant products for a given use case, or conversely find the right technical information about a given product.
• Retrieving and summarizing financial news– Analysts need the most up-to-date information on markets and the economy and rely on large databases of news or commentary articles. A RAG solution is a way to efficiently retrieve and summarize the relevant information on a given topic.

In the following sections, we will give tips that you can use to optimize each aspect of the RAG pipeline (ingestion, retrieval, and answer generation) depending on the underlying use case and data format. To verify that the modifications improve the solution, you first need to be able to assess the performance of the RAG solution.

Evaluating a RAG solution

Contrary to traditional machine learning (ML) models, for which evaluation metrics are well defined and straightforward to compute, evaluating a RAG framework is still an open problem. First, collecting ground truth (information known to be correct) for the retrieval component and the generation component is time consuming and requires human intervention. Secondly, even with several question-and-answer pairs available, it’s difficult to automatically evaluate if the RAG answer is close enough to the human answer.

In our experience, when a RAG system performs poorly, we found the retrieval part to almost always be the culprit. Large pre-trained models such as Anthropic’s Claude model will generate high-quality answers if provided with the right information, and we notice two main failure modes:

• The relevant information isn’t present in the retrieved documents: In this case, the FM can try to make up an answer or use its own knowledge to answer. Adding guardrails against such behavior is essential.
• Relevant information is buried within an excessive amount of irrelevant data: When the scope of the retriever is too broad, the FM can get confused and start mixing up multiple data sources, resulting in a wrong answer. More advanced models such as Anthropic’s Claude Sonnet 3.5 and Opus are reported to be more robust against such behavior, but this is still a risk to be aware of.

To evaluate the quality of the retriever, you can use the following traditional retrieval metrics:

• Top-k accuracy: Measures whether at least one relevant document is found within the top k retrieved documents.
• Mean Reciprocal Rank (MRR)– This metric considers the ranking of the retrieved documents. It’s calculated as the average of the reciprocal ranks (RR) for each query. The RR is the inverse of the rank position of the first relevant document. For example, if the first relevant document is in third position, the RR is 1/3. A higher MRR indicates that the retriever can rank the most relevant documents higher.
• Recall– This metric measures the ability of the retriever to retrieve relevant documents from the corpus. It’s calculated as the number of relevant documents that are successfully retrieved over the total number of relevant documents. Higher recall indicates that the retriever can find most of the relevant information.
• Precision– This metric measures the ability of the retriever to retrieve only relevant documents and avoid irrelevant ones. It’s calculated by the number of relevant documents successfully retrieved over the total number of documents retrieved. Higher precision indicates that the retriever isn’t retrieving too many irrelevant documents.

Note that if the documents are chunked, the metrics must be computed at the chunk level. This means the ground truth to evaluate a retriever is pairs of question and list of relevant document chunks. In many cases, there is only one chunk that contains the answer to the question, so the ground truth becomes question and relevant document chunk.

To evaluate the quality of the generated response, two main options are:

• Evaluation by subject matter experts: this provides the highest reliability in terms of evaluation but can’t scale to a large number of questions and slows down iterations on the RAG solution.
• Evaluation by FM (also called LLM-as-a-judge):
  • With a human-created starting point: Provide the FM with a set of ground truth question-and-answer pairs and ask the FM to evaluate the quality of the generated answer by comparing it to the ground truth one.
  • With an FM-generated ground truth: Use an FM to generate question-and-answer pairs for given chunks, and then use this as a ground truth, before resorting to an FM to compare RAG answers to that ground truth.

We recommend that you use an FM for evaluations to iterate faster on improving the RAG solution, but to use subject-matter experts (or at least human evaluation) to provide a final assessment of the generated answers before deploying the solution.

A growing number of libraries offer automated evaluation frameworks that rely on additional FMs to create a ground truth and evaluate the relevance of the retrieved documents as well as the quality of the response:

• Ragas– This framework offers FM-based metrics previously described, such as context recall, context precision, answer faithfulness, and answer relevancy. It needs to be adapted to Anthropic’s Claude models because of its heavy dependence on specific prompts.
• LlamaIndex– This framework provides multiple modules to independently evaluate the retrieval and generation components of a RAG system. It also integrates with other tools such as Ragas and DeepEval. It contains modules to create ground truth (query-and-context pairs and question-and-answer pairs) using an FM, which alleviates the use of time-consuming human collection of ground truth.
• RefChecker– This is an Amazon Science library focused on fine-grained hallucination detection.

Troubleshooting RAG

Evaluation metrics give an overall picture of the performance of retrieval and generation, but they don’t help diagnose issues. Diving deeper into poor responses can help you understand what’s causing them and what you can do to alleviate the issue. You can diagnose the issue by looking at evaluation metrics and also by having a human evaluator take a closer look at both the LLM answer and the retrieved documents.

The following is a brief overview of issues and potential fixes. We will describe each of the techniques in more detail, including real-world use cases and code examples, in the next section.

• The relevant chunk wasn’t retrieved (retriever has low top k accuracy and low recall or spotted by human evaluation):
  • Try increasing the number of documents retrieved by the nearest neighbor search and re-ranking the results to cut back on the number of chunks after retrieval.
  • Try hybrid search. Using keywords in combination with semantic search (known as hybrid search) might help, especially if the queries contain names or domain-specific jargon.
  • Try query rewriting. Having an FM detect the intent or rewrite the query can help create a query that’s better suited for the retriever. For instance, a user query such as “What information do you have in the knowledge base about the economic outlook in China?” contains a lot of context that isn’t relevant to the search and would be more efficient if rewritten as “economic outlook in China” for search purposes.

• Too many chunks were retrieved (retriever has low precision or spotted by human evaluation):
  • Try using keyword matching to restrict the search results. For example, if you’re looking for information about a specific entity or property in your knowledge base, only retrieve documents that explicitly mention them.
  • Try metadata filtering in your OpenSearch index. For example, if you’re looking for information in news articles, try using the date field to filter only the most recent results.
  • Try using query rewriting to get the right metadata filtering. This advanced technique uses the FM to rewrite the user query as a more structured query, allowing you to make the most of OpenSearch filters. For example, if you’re looking for the specifications of a specific product in your database, the FM can extract the product name from the query, and you can then use the product name field to filter out the product name.
  • Try using reranking to cut down on the number of chunks passed to the FM.
• A relevant chunk was retrieved, but it’s missing some context (can only be assessed by human evaluation):
  • Try changing the chunking strategy. Keep in mind that small chunks are good for precise questions, while large chunks are better for questions that require a broad context:
    • Try increasing the chunk size and overlap as a first step.
    • Try using section-based chunking. If you have structured documents, use sections delimiters to cut your documents into chunks to have more coherent chunks. Be aware that you might lose some of the more fine-grained context if your chunks are larger.
  • Try small-to-large retrievers. If you want to keep the fine-grained details of small chunks but make sure you retrieve all the relevant context, small-to-large retrievers will retrieve your chunk along with the previous and next ones.
• If none of the above help:
  • Consider training a custom embedding.
• The retriever isn’t at fault, the problem is with FM generation (evaluated by a human or LLM):
  • Try prompt engineering to mitigate hallucinations.
  • Try prompting the FM to use quotes in its answers, to allow for manual fact checking.
  • Try using another FM to evaluate or correct the answer.

A practical guide to improving the retriever

Note that not all the techniques that follow need to be implemented together to optimize your retriever—some might even have opposite effects. Use the preceding troubleshooting guide to get a shortlist of what might work, then look at the examples in the corresponding sections that follow to assess if the method can be beneficial to your retriever.

Hybrid search

Example use case: A large manufacturer built a RAG chatbot to retrieve product specifications. These documents contain technical terms and product names. Consider the following example queries:

query_1 = "What is the viscosity of product XYZ?"
query_2 = "How viscous is XYZ?"

The queries are equivalent and need to be answered with the same document. The keyword component will make sure that you’re boosting documents mentioning the name of the product, XYZ while the semantic component will make sure that documents containing viscosity get a high score, even when the query contains the word viscous.

Combining vector search with keyword search can effectively handle domain-specific terms, abbreviations, and product names that embedding models might struggle with. Practically, this can be achieved in OpenSearch by combining a k-nearest neighbors (k-NN) query with keyword matching. The weights for the semantic search compared to keyword search can be adjusted. See the following example code:

vector_embedding = compute_embedding(query)
size = 10
semantic_weight = 10
keyword_weight = 1
search_query = {"size":size, "query": { "bool": { "should":[] , "must":[] } } }
    # semantic search
    search_query['query']['bool']['should'].append(
            {"function_score": 
             { "query": 
              {"knn": 
               {"vector_field": 
                {"vector": vector_embedding, 
                "k": 10 # The number of nearest neighbors to retrieve
                }}}, 
              "weight": semantic_weight } })
              
    # keyword search
    search_query['query']['bool']['should'].append({
             "function_score": 
            { "query": 
             {"match": 
             # This will increase the score of chunks that match the words in the query
              {"chunk_text":  query} 
              },
             "weight": keyword_weight } })

Amazon Bedrock Knowledge Bases also supports hybrid search, but you can’t adjust the weights for semantic compared to keyword search.

Adding metadata information to text chunks

Example use case: Using the same example of a RAG chatbot for product specifications, consider product specifications that are several pages long and where the product name is only present in the header of the document. When ingesting the document into the knowledge base, it’s chunked into smaller pieces for the embedding model, and the product name only appears in the first chunk, which contains the header. See the following example:

# Note: the following document was generated by Anthropic’s Claude Sonnet 
# and does not contain information about a real product

document_name = "Chemical Properties for Product XYZ"

chunk_1 = """
Product Description:
XYZ is a multi-purpose cleaning solution designed for industrial and commercial use. 
It is a concentrated liquid formulation containing anionic and non-ionic surfactants, 
solvents, and alkaline builders.

Chemical Composition:
- Water (CAS No. 7732-18-5): 60-80%
- 2-Butoxyethanol (CAS No. 111-76-2): 5-10%
- Sodium Hydroxide (CAS No. 1310-73-2): 2-5%
- Ethoxylated Alcohols (CAS No. 68439-46-3): 1-3%
- Sodium Metasilicate (CAS No. 6834-92-0): 1-3%
- Fragrance (Proprietary Mixture): <1%
"""

# chunk 2 below doesn't contain any mention of "XYZ"
chunk_2 = """
Physical Properties:
- Appearance: Clear, yellow liquid
- Odor: Mild, citrus fragrance
- pH (concentrate): 12.5 - 13.5
- Specific Gravity: 1.05 - 1.10
- Solubility in Water: Complete
- VOC Content: <10%

Shelf-life:
When stored in its original, unopened container at temperatures between 15°C and 25°C,
 the product has a shelf life of 24 months from the date of manufacture.
Once opened, the shelf life is reduced due to potential contamination and exposure to
 air. It is recommended to use the product within 6 months after opening the container.
"""

# append the document name to the chunks to improve context,
# now chunk 2 will contain the product name

chunk_1 = document_name + chunk_1
chunk_2 = document_name + chunk_2

document_title = "Resolving network issues"

chunk_1 = """
[....]

# Troubleshooting internet access:

1. Check your physical connections:
   - Ensure that the Ethernet cable (if using a wired connection) is securely 
   plugged into both your computer and the modem/router.
   - If using a wireless connection, check that your device's Wi-Fi is turned 
   on and connected to the correct network.

2. Restart your devices:
   - Reboot your computer, laptop, or mobile device.
   - Power cycle your modem and router by unplugging them from the power source, 
   waiting for a minute, and then plugging them back in.

"""

chunk_2 = """
3. Check for network outages:
   - Contact your internet service provider (ISP) to inquire about any known 
   outages or service disruptions in your area.
   - Visit your ISP's website or check their social media channels for updates on 
   service status.
  
4. Check for interference:
   - If using a wireless connection, try moving your device closer to the router or access point.
   - Identify and eliminate potential sources of interference, such as microwaves, cordless phones, or other wireless devices operating on the same frequency.

# Router configuration

[....]
"""

document_name = doc['document_name'] 
current_chunk = doc['current_chunk']

query = {
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "document_name": document_name
                    }
                }
            ],
            "should": [
                {"term": {"chunk_number": current_chunk - 1}},
                {"term": {"chunk_number": current_chunk + 1}}
            ],
            "minimum_should_match": 1
        }
    }
}

query = """" 
Can you list all the ingredients in the nuts and seeds granola?
Put the allergens in all caps. 
"""

import json

query_rewriting_prompt = """
Rewrite the query as a json with the following keys:
- rewritten_query: a better version of the user's query that will be used to compute 
an embedding and do semantic search
- keywords: a list of keywords that correspond to the query, to be used in a 
search engine, it should not contain the product name.
- product_name: if the query is a about a specific product, give the name here,
 otherwise say None.

<example>
H: what are the ingedients in the savory trail mix?
A: {{
  "rewritten_query": "ingredients savory trail mix",
  "keywords": ["ingredients"],
  "product_name": "savory trail mix"
}}
</example>

<query>
{query}
</query>

Only output the json, nothing else.
"""

def rewrite_query(query):
    response = call_FM(query_rewriting_prompt.format(query=query))
    print(response)
    json_query = json.loads(response)
    return json_query
    
rewrite_query(query)

{ 
"rewritten_query":"ingredients nuts and seeds granola allergens",
"keywords": ["ingredients", "allergens"], 
"product_name": "nuts and seeds granola" 
}

oss = get_opensearch_serverless_client()
request = {
"product_info": product_info, # full text for the product information
"vector_field_product":embed_query_titan(product_info), # embedding for product information
"product_name": product_name,
"date": date, # optional field, can allow to sort by most recent
"_op_type": "index",
"source": file_key # this is the s3 location, you can replace this with a URL
}
oss.index(index = index_name, body = request)

query = """
Can you list all the ingredients in the nuts and seeds granola?
Put the allergens in bold and all caps.
"""
# using the rewrite_query function from the previous section
json_query = rewrite_query(query) 

# get the product name and keywords from the json query
product_name = json_query["product_name"] 
keywords = json_query["keywords"]

# compute the vector embedding of the rewritten query
vector_embedding = compute_embedding(json_query["rewritten_query"])

#initialize search query dictionary
search_query = {"size":10, "query": { "bool": { "should":[] , "must":[] } } }
# add must with match_phrase clause to filter on product name
search_query['query']['bool']['should'].append(
    {"match_phrase": {
            "product_name": product_name # Extracted product name must match product name field 
        }
        }

# semantic search
search_query['query']['bool']['should'].append(
        {"function_score": 
            { "query": 
            {"knn": 
            {"vector_field_product": 
            {"vector": vector_embedding, 
            "k": 10 # The number of nearest neighbors to retrieve
            }}}, 
            "weight": semantic_weight } })
            
# keyword search
search_query['query']['bool']['should'].append(
{"function_score": 
        { "query": 
            {"match": 
            # This will increase the score of chunks that match the words in the query
            {"product_info":  query} 
            },
            "weight": keyword_weight } })

# Note: The scouting reports for Player A and Player B used in the example below
# were generated by Anthropic’s Claude Sonnet 3.
quotes_tags_prompt = """
You are an Ai assistant chatbot designed to answer questions about your team's baseballe scouting reports.

Here are some reports about players that you can use to answer the question:

<doc_0>
{doc_0}
</doc_0>

<doc_1>
{doc_1}
</doc_1>

<instructions>
- In <scratchpad> tags, you should put the document names and quotes from these documents 
 that help you answer this question. 
 You must format this as one or more jsons format with 'document_name' and 'quote' as keys. 
- Then put your answer in <answer> tags. 
 If you refer to documents in your answer, make sure you are using the corresponding tag 
 e.g. in doc_0, in doc_3 etc.
- If you can't answer the question because there is not enough information in the reports say 
 "I can't answer this based on the provided reports." 
</instructions>

<question>
{question}
</question>
"""

"""
<scratchpad>
[
{
"document_name": "doc_0",
"quote": "The main concern with Player A is his approach at the plate. He can be overly aggressive at times and will chase pitches out of the zone. Improving his plate discipline and pitch recognition will be key to him reaching his full offensive potential."
},
{
"document_name": "doc_1",
"quote": "The main area of focus for Player B's development will be improving his plate discipline and pitch recognition. He can be overly aggressive at times, leading to a high strikeout rate and a lower on-base percentage than his hit tool would suggest."
}
]
</scratchpad>
<answer>
The main weaknesses of the players are:
For Player A:
- Overly aggressive approach at the plate, chasing pitches out of the zone
- Needs to improve plate discipline and pitch recognition to reach full offensive potential
For Player B:
- Can be overly aggressive at the plate, leading to high strikeout rate and lower on-base percentage
- Needs to improve plate discipline and pitch recognition
</answer>
"""

# Note: The scouting reports for Player A used in the example below
# were generated by Anthropic’s Claude Sonnet 3.

quotes_in_text_prompt = """
You are an Ai assistant chatbot designed to answer questions about your team's baseballe scouting reports.

Here are some reports about players that you can use to answer the question:

<doc_0>
{doc_0}
</doc_0>

...

<doc_10>
{doc_10}
</doc_10>

<instructions>
- Put your answer in <answer> tags.
- Use as much information from different reports as possible.
- You should only use information in the documents to answer. If you don't have enough information in the reports to answer, say you can't answer based on the reports.
- You should ground your answer by quoting the relevant documents by using quotation marks.
- After the quotes, put an inline citation <example>Player A is "very athletic" (doc_2)</example>
</instructions>

<question>
{question}
</question>
"""

Below is the FM response to the question "What are the strengths of player A?":

"""
<scratchpad>
[
{
"document_name": "doc_0",
"quote": "The main concern with Player A is his approach at the plate. He can be overly aggressive at times and will chase pitches out of the zone. Improving his plate discipline and pitch recognition will be key to him reaching his full offensive potential."
},
{
"document_name": "doc_1",
"quote": "The main area of focus for Player B's development will be improving his plate discipline and pitch recognition. He can be overly aggressive at times, leading to a high strikeout rate and a lower on-base percentage than his hit tool would suggest."
}
]
</scratchpad>
<answer>
The main weaknesses of the players are:
For Player A:
- Overly aggressive approach at the plate, chasing pitches out of the zone
- Needs to improve plate discipline and pitch recognition to reach full offensive potential
For Player B:
- Can be overly aggressive at the plate, leading to high strikeout rate and lower on-base percentage
- Needs to improve plate discipline and pitch recognition
</answer>
"""

Read More

Machine learning (ML) helps organizations to increase revenue, drive business growth, and reduce costs by optimizing core business functions such as supply and demand forecasting, customer churn prediction, credit risk scoring, pricing, predicting late shipments, and many others.

Conventional ML development cycles take weeks to many months and requires sparse data science understanding and ML development skills. Business analysts’ ideas to use ML models often sit in prolonged backlogs because of data engineering and data science team’s bandwidth and data preparation activities.

In this post, we dive into a business use case for a banking institution. We will show you how a financial or business analyst at a bank can easily predict if a customer’s loan will be fully paid, charged off, or current using a machine learning model that is best for the business problem at hand. The analyst can easily pull in the data they need, use natural language to clean up and fill any missing data, and finally build and deploy a machine learning model that can accurately predict the loan status as an output, all without needing to become a machine learning expert to do so. The analyst will also be able to quickly create a business intelligence (BI) dashboard using the results from the ML model within minutes of receiving the predictions. Let’s learn about the services we will use to make this happen.

Amazon SageMaker Canvas is a web-based visual interface for building, testing, and deploying machine learning workflows. It allows data scientists and machine learning engineers to interact with their data and models and to visualize and share their work with others with just a few clicks.

SageMaker Canvas has also integrated with Data Wrangler, which helps with creating data flows and preparing and analyzing your data. Built into Data Wrangler, is the Chat for data prep option, which allows you to use natural language to explore, visualize, and transform your data in a conversational interface.

Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service that makes it cost-effective to efficiently analyze all your data using your existing business intelligence tools.

Amazon QuickSight powers data-driven organizations with unified (BI) at hyperscale. With QuickSight, all users can meet varying analytic needs from the same source of truth through modern interactive dashboards, paginated reports, embedded analytics, and natural language queries.

Solution overview

The solution architecture that follows illustrates:

1. A business analyst signing in to SageMaker Canvas.
2. The business analyst connects to the Amazon Redshift data warehouse and pulls the desired data into SageMaker Canvas to use.
3. We tell SageMaker Canvas to build a predictive analysis ML model.
4. After the model has been built, get batch prediction results.
5. Send the results to QuickSight for users to further analyze.
   

Prerequisites

Before you begin, make sure you have the following prerequisites in place:

• An AWS account and role with the AWS Identity and Access Management (IAM) privileges to deploy the following resources:
  • IAM roles.
  • A provisioned or serverless Amazon Redshift data warehouse. For this post we’ll use a provisioned Amazon Redshift cluster.
  • A SageMaker domain.
  • A QuickSight account (optional).
• Basic knowledge of a SQL query editor.

Set up the Amazon Redshift cluster

We’ve created a CloudFormation template to set up the Amazon Redshift cluster.

1. Deploy the Cloudformation template to your account.
   
2. Enter a stack name, then choose Next twice and keep the rest of parameters as default.
3. In the review page, scroll down to the Capabilities section, and select I acknowledge that AWS CloudFormation might create IAM resources.
4. Choose Create stack.

The stack will run for 10–15 minutes. After it’s finished, you can view the outputs of the parent and nested stacks as shown in the following figures:

Parent stack

Nested stack 

Sample data

You will use a publicly available dataset that AWS hosts and maintains in our own S3 bucket as a workshop for bank customers and their loans that includes customer demographic data and loan terms.

Implementation steps

Load data to the Amazon Redshift cluster

1. Connect to your Amazon Redshift cluster using Query Editor v2. To navigate to the Amazon Redshift Query v2 editor, please follow the steps Opening query editor v2.
2. Create a table in your Amazon Redshift cluster using the following SQL command:

   DROP table IF EXISTS public.loan_cust;
   
   CREATE TABLE public.loan_cust (
       loan_id bigint,
       cust_id bigint,
       loan_status character varying(256),
       loan_amount bigint,
       funded_amount_by_investors double precision,
       loan_term bigint,
       interest_rate double precision,
       installment double precision,
       grade character varying(256),
       sub_grade character varying(256),
       verification_status character varying(256),
       issued_on character varying(256),
       purpose character varying(256),
       dti double precision,
       inquiries_last_6_months bigint,
       open_credit_lines bigint,
       derogatory_public_records bigint,
       revolving_line_utilization_rate double precision,
       total_credit_lines bigint,
       city character varying(256),
       state character varying(256),
       gender character varying(256),
       ssn character varying(256),
       employment_length bigint,
       employer_title character varying(256),
       home_ownership character varying(256),
       annual_income double precision,
       age integer
   ) DISTSTYLE AUTO;

   

3. Load data into the loan_cust table using the following COPY command:

   COPY loan_cust  FROM 's3://redshift-demos/bootcampml/loan_cust.csv'
   iam_role default
   region 'us-east-1' 
   delimiter '|'
   csv
   IGNOREHEADER 1;

   

4. Query the table to see what the data looks like:

   SELECT * FROM loan_cust LIMIT 100;

   

Set up chat for data

1. To use the chat for data option in Sagemaker Canvas, you must enable it in Amazon Bedrock.
   1. Open the AWS Management Console, go to Amazon Bedrock, and choose Model access in the navigation pane.
      
   2. Choose Enable specific models, under Anthropic, select Claude and select Next.
      
   3. Review the selection and click Submit.
      
2. Navigate to Amazon SageMaker service from the AWS management console, select Canvas and click on Open Canvas.
   
3. Choose Datasets from the navigation pane, then choose the Import data dropdown, and select Tabular.
   

3. For Dataset name, enter redshift_loandata and choose Create.
   
4. On the next page, choose Data Source and select Redshift as the source. Under Redshift, select + Add Connection.
   
5. Enter the following details to establish your Amazon Redshift connection :
   1. Cluster Identifier: Copy the ProducerClusterName from the CloudFormation nested stack outputs.
   2. You can reference the preceding screen shot for Nested Stack, where you will find the cluster identifier output.
   3. Database name: Enter dev.
   4. Database user: Enter awsuser.
   5. Unload IAM Role ARN: Copy theRedshiftDataSharingRoleName from the nested stack outputs.
   6. Connection Name: Enter MyRedshiftCluster.
   7. Choose Add connection.

   

6. After the connection is created, expand the public schema, drag the loan_cust table into the editor, and choose Create dataset.
   
7. Choose the redshift_loandata dataset and choose Create a data flow.
   
8. Enter redshift_flow for the name and choose Create.
   
9. After the flow is created, choose Chat for data prep.
   
10. In the text box, enter summarize my data and choose the run arrow.
    
11. The output should look something like the following:
    

1. Now you can use natural language to prep the dataset. Enter Drop ssn and filter for ages over 17 and click on the run arrow. You will see it was able to handle both steps. You can also view the PySpark code that it ran. To add these steps as dataset transforms, choose Add to steps.
   
2. Rename the step to drop ssn and filter age > 17, choose Update, and then choose Create model.
   
3. Export data and create model: Enter loan_data_forecast_dataset for the Dateset name, for Model name, enter loan_data_forecast, for Problem type, select Predictive analysis, for Target column, select loan_status, and click Export and create model.
   
4. Verify the correct Target column and Model type is selected and click on Quick build.
   
5. Now the model is being created. It usually takes 14–20 minutes depending on the size of your data set.
6. After the model has completed training, you will be routed to the Analyze tab. There, you can see the average prediction accuracy and the column impact on prediction outcome. Note that your numbers might differ from the ones you see in the following figure, because of the stochastic nature of the ML process.
   

Use the model to make predictions

1. Now let’s use the model to make predictions for the future status of loans. Choose Predict.
   
2. Under Choose the prediction type, select Batch prediction, then select Manual.
   
3. Then select loan_data_forecast_dataset from the dataset list, and click Generate predictions.
   
4. You’ll see the following after the batch prediction is complete. Click on the breadcrumb menu next to the Ready status and click on Preview to view the results.
   
5. You can now view the predictions and download them as CSV.
   
6. You can also generate single predictions for one row of data at a time. Under Choose the prediction type, select Single Prediction and then change the values for any of the input fields that you’d like, and choose Update.
   

Analyze the predictions

We will now show you how to use Quicksight to visualize the predictions data from SageMaker canvas to further gain insights from your data. SageMaker Canvas has direct integration with QuickSight, which is a cloud-powered business analytics service that helps employees within an organization to build visualizations, perform ad-hoc analysis, and quickly get business insights from their data, anytime, on any device.

1. With the preview page up, choose Send to Amazon QuickSight.
2. Enter a QuickSight user name you want to share the results to.
   
3. Choose Send and you should see confirmation saying the results were sent successfully.
   
4. Now, you can create a QuickSight dashboard for predictions.
   1. Go to the QuickSight console by entering QuickSight in your console services search bar and choose QuickSight.
      
   2. Under Datasets, select the SageMaker Canvas dataset that was just created.
   3. Choose Edit Dataset.
   4. Under the State field, change the data type to State.
      
   5. Choose Create with Interactive sheet selected.
      
   6. Under visual types, choose the Filled map
   7. Select the State and Probability
   8. Under Field wells, choose Probability and change the Aggregate to Average and Show as to Percent.
      
   9. Choose Filter and add a filter for loan_status to include fully paid loans only. Choose Apply.
      
   10. At the top right in the blue banner, choose Share and Publish Dashboard.
   11. We use the name Average probability for fully paid loan by state, but feel free to use your own.
   12. Choose Publish dashboard and you’re done. You would now be able to share this dashboard with your predictions to other analysts and consumers of this data.

Clean up

Use the following steps to avoid any extra cost to your account:

1. Sign out of SageMaker Canvas
2. In the AWS console, delete the CloudFormation stack you launched earlier in the post.

Conclusion

We believe integrating your cloud data warehouse (Amazon Redshift) with SageMaker Canvas opens the door to producing many more robust ML solutions for your business at faster and without needing to move data and with no ML experience.

You now have business analysts producing valuable business insights, while letting data scientists and ML engineers help refine, tune, and extend models as needed. SageMaker Canvas integration with Amazon Redshift provides a unified environment for building and deploying machine learning models, allowing you to focus on creating value with your data rather than focusing on the technical details of building data pipelines or ML algorithms.

Additional reading:

1. SageMaker Canvas Workshop
2. re:Invent 2022 – SageMaker Canvas
3. Hands-On Course for Business Analysts – Practical Decision Making using No-Code ML on AWS

About the Authors

Suresh Patnam is Principal Sales Specialist  AI/ML and Generative AI at AWS. He is passionate about helping businesses of all sizes transform into fast-moving digital organizations focusing on data, AI/ML, and generative AI.

Sohaib Katariwala is a Sr. Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service. His interests are in all things data and analytics. More specifically he loves to help customers use AI in their data strategy to solve modern day challenges.

Michael Hamilton is an Analytics & AI Specialist Solutions Architect at AWS. He enjoys all things data related and helping customers solution for their complex use cases.

Nabil Ezzarhouni is an AI/ML and Generative AI Solutions Architect at AWS. He is based in Austin, TX and  passionate about Cloud, AI/ML technologies, and Product Management. When he is not working, he spends time with his family, looking for the best taco in Texas. Because…… why not?

Read More

In this post, we set up an agent using Amazon Bedrock Agents to act as a software application builder assistant.

Agentic workflows are a fresh new perspective in building dynamic and complex business use- case based workflows with the help of large language models (LLM) as their reasoning engine or brain. These agentic workflows decompose the natural language query-based tasks into multiple actionable steps with iterative feedback loops and self-reflection to produce the final result using tools and APIs.

Amazon Bedrock Agents helps you accelerate generative AI application development by orchestrating multistep tasks. Amazon Bedrock Agents uses the reasoning capability of foundation models (FMs) to break down user-requested tasks into multiple steps. They use the developer-provided instruction to create an orchestration plan and then carry out the plan by invoking company APIs and accessing knowledge bases using Retrieval Augmented Generation (RAG) to provide a final response to the end user. This offers tremendous use case flexibility, enables dynamic workflows, and reduces development cost. Amazon Bedrock Agents is instrumental in customization and tailoring apps to help meet specific project requirements while protecting private data and securing their applications. These agents work with AWS managed infrastructure capabilities and Amazon Bedrock, reducing infrastructure management overhead. Additionally, agents streamline workflows and automate repetitive tasks. With the power of AI automation, you can boost productivity and reduce cost.

Amazon Bedrock is a fully managed service that offers a choice of high-performing FMs from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Solution overview

Typically, a three-tier software application has a UI interface tier, a middle tier (the backend) for business APIs, and a database tier. The generative AI–based application builder assistant from this post will help you accomplish tasks through all three tiers. It can generate and explain code snippets for UI and backend tiers in the language of your choice to improve developer productivity and facilitate rapid development of use cases. The agent can recommend software and architecture design best practices using the AWS Well-Architected Framework for the overall system design.

The agent can generate SQL queries using natural language questions using a database schema DDL (data definition language for SQL) and execute them against a database instance for the database tier.

We use Amazon Bedrock Agents with two knowledge bases for this assistant. Amazon Bedrock Knowledge Bases inherently uses the Retrieval Augmented Generation (RAG) technique. A typical RAG implementation consists of two parts:

• A data pipeline that ingests data from documents typically stored in Amazon Simple Storage Service (Amazon S3) into a knowledge base, namely a vector database such as Amazon OpenSearch Serverless, so that it’s available for lookup when a question is received
• An application that receives a question from the user, looks up the knowledge base for relevant pieces of information (context), creates a prompt that includes the question and the context, and provides it to an LLM for generating a response

The following diagram illustrates how our application builder assistant acts as a coding assistant, recommends AWS design best practices, and aids in SQL code generation.

Based on the three workflows in the preceding figure, let’s explore the type of task you need for different use cases:

• Use case 1 – If you want to write and validate a SQL query against a database, use the existing DDL schemas set up as knowledge base 1 to come up with the SQL query. The following are sample user queries:
  • What are the total sales amounts by year?
  • What are the top five most expensive products?
  • What is the total revenue for each employee?
• Use case 2 – If you want recommendations on design best practices, look up the AWS Well-Architected Framework knowledge base (knowledge base 2). The following are sample user queries:
  • How can I design secure VPCs?
  • What are some S3 best practices?
• Use case 3 – You might want to author some code, such as helper functions like validate email, or use existing code. In this case, use prompt engineering techniques to call the default agent LLM and generate the email validation code. The following are sample user queries:
  • Write a Python function to validate email address syntax.
  • Explain the following code in lucid, natural language to me. $code_to_explain (this variable is populated using code contents from any code file of your choice. More details can be found in the notebook).

Prerequisites

To run this solution in your AWS account, complete the following prerequisites:

1. Clone the GitHub repository and follow the steps explained in the README.
2. Set up an Amazon SageMaker notebook on an ml.t3.medium Amazon Elastic Compute Cloud (Amazon EC2) instance. For this post, we have provided an AWS CloudFormation template, available in the GitHub repository. The CloudFormation template also provides the required AWS Identity and Access Management (IAM) access to set up the vector database, SageMaker resources, and AWS Lambda
3. Acquire access to models hosted on Amazon Bedrock. Choose Manage model access in the navigation pane on the Amazon Bedrock console and choose from the list of available options. We use Anthropic’s Claude v3 (Sonnet) on Amazon Bedrock and Amazon Titan Embeddings Text v2 on Amazon Bedrock for this post.

Implement the solution

In the GitHub repository notebook, we cover the following learning objectives:

1. Choose the underlying FM for your agent.
2. Write a clear and concise agent instruction to use one of the two knowledge bases and base agent LLM. (Examples given later in the post.)
3. Create and associate an action group with an API schema and a Lambda function.
4. Create, associate, and ingest data into the two knowledge bases.
5. Create, invoke, test, and deploy the agent.
6. Generate UI and backend code with LLMs.
7. Recommend AWS best practices for system design with the AWS Well-Architected Framework guidelines.
8. Generate, run, and validate the SQL from natural language understanding using LLMs, few-shot examples, and a database schema as a knowledge base.
9. Clean up agent resources and their dependencies using a script.

Agent instructions and user prompts

The application builder assistant agent instruction looks like the following.

Hello, I am AI Application Builder Assistant. I am capable of answering the following three categories of questions:

- Best practices for design of software applications using the content inside the AWS best practices 
and AWS well-architected framework Knowledge Base. I help customers understand AWS best practices for 
building applications with AWS services.

- Generate a valid SQLite query for the customer using the database schema inside the Northwind DB knowledge base 
and then execute the query that answers the question based on the [Northwind] dataset. If the Northwind DB Knowledge Base search 
function result did not contain enough information to construct a full query try to construct a query to the best of your ability 
based on the Northwind database schema.

- Generate and Explain code for the customer following standard programming language syntax</p><p>Feel free to ask any questions 
along those lines!

Each user question to the agent by default includes the following system prompt.

Note: The following system prompt remains the same for each agent invocation, only the {user_question_to_agent} gets replaced with user query.

Question: {user_question_to_agent} 

Given an input question, you will use the existing Knowledge Bases on AWS 
Well-Architected Framework and Northwind DB Knowledge Base.

- For building and designing software applications, you will use the existing Knowledge Base on AWS well-architected framework 
to generate a response of the most relevant design principles and links to any documents. This Knowledge Base response can then be passed 
to the functions available to answer the user question. The final response to the direct answer to the user question. 
It has to be in markdown format highlighting any text of interest. Remove any backticks in the final response.

- To generate code for a given user question,  you can use the default Large Language model to come up with the response. 
This response can be in code markdown format. You can optionally provide an explanation for the code.

- To explain code for a given user question, you can use the default Large Language model to come up with the response.

- For SQL query generation you will ONLY use the existing database schemas in the Northwind DB Knowledge Base to create a syntactically 
correct SQLite query and then you will EXECUTE the SQL Query using the functions and API provided to answer the question.

Make sure to use ONLY existing columns and tables based on the Northwind DB database schema. Make sure to wrap table names with 
square brackets. Do not use underscore for table names unless that is part of the database schema. Make sure to add a semicolon after 
the end of the SQL statement generated.</p><p>Remove any backticks and any html tags like <table><th><tr> in the 
final response.

Here are a few examples of questions I can help answer by generating and then executing a SQLite query:

- What are the total sales amounts by year?</p>
- What are the top 5 most expensive products?</p>
- What is the total revenue for each employee?</p>

Cost considerations

The following are important cost considerations:

• This current implementation has no separate charges for building resources using Amazon Bedrock Knowledge Bases or Amazon Bedrock Agents.
• You will incur charges for embedding model and text model invocation on Amazon Bedrock. For more details, refer to Amazon Bedrock pricing.
• You will incur charges for Amazon S3 and vector DB usage. For more details, see Amazon S3 pricing and Amazon OpenSearch Service Pricing, respectively.

Clean up

To avoid incurring unnecessary costs, the implementation automatically cleans up resources after an entire run of the notebook. You can check the notebook instructions in the Clean-up Resources section on how to avoid the automatic cleanup and experiment with different prompts.

The order of resource cleanup is as follows:

1. Disable the action group.
2. Delete the action group.
3. Delete the alias.
4. Delete the agent.
5. Delete the Lambda function.
6. Empty the S3 bucket.
7. Delete the S3 bucket.
8. Delete IAM roles and policies.
9. Delete the vector DB collection policies.
10. Delete the knowledge bases.

Conclusion

This post demonstrated how to query and integrate workflows with Amazon Bedrock Agents using multiple knowledge bases to create a generative AI–based software application builder assistant that can author and explain code, generate SQL using DDL schemas, and recommend design suggestions using the AWS Well-Architected Framework.

Beyond code generation and explanation of code as demonstrated in this post, to run and troubleshoot application code in a secure test environment, you can refer to Code Interpreter setup with Amazon Bedrock Agents

For more information on creating agents to orchestrate workflows, see Amazon Bedrock Agents.

Acknowledgements

The author thanks all the reviewers for their valuable feedback.

About the Author

Shayan Ray is an Applied Scientist at Amazon Web Services. His area of research is all things natural language (like NLP, NLU, NLG). His work has been focused on conversational AI, task-oriented dialogue systems and LLM-based agents. His research publications are on natural language processing, personalization, and reinforcement learning.

Read More