The tools we use to build AI are evolving fast, with PyTorch at the heart of many advances. But unless we evolve the way we approach building AI systems, we risk amplifying harm as fast as we’re scaling up performance. Building AI responsibly means designing systems that not only perform well but do so fairly, safely, and transparently—like making sure an AI hiring tool doesn’t favor one demographic over another.
One useful approach to developing responsible AI systems is Yellow Teaming—a proactive exercise that surfaces potential unintended consequences before deployment. Yellow Teaming helps companies stand out in a crowded market by making more thoughtful, impact-aware design choices that lead to an overall better product.
In this blog, we show how you can quickly create a PyTorch-based LLM Yellow Teaming assistant running on AWS Graviton4 with a reusable system prompt. We also give you an example to show you how to use your new assistant to explore unintended business-critical consequences of feature design and ultimately build better products.
Let’s get started.
What is Yellow Teaming:
You may already be aware of the more popular term Red Teaming in cybersecurity, which involves simulating how adversaries might attack your product and fixing vulnerabilities before launch. Other color-coded approaches exist (like Blue Teams that defend against attacks), but Yellow Teaming is distinct in focusing on thoughtful design and implementation from the start of the product’s lifecycle. Red Teaming practices have already been adapted to the AI domain. Yellow Teaming principles are now becoming an important part of AI development as well.
The practice of Yellow Teaming asks a set of probing questions to help reveal the broader, unintended impacts of your product on your business, your users, and society at large. This application of Yellow Teaming, and the rationale behind it, are explained eloquently in the Development in Progress essay by The Consilience Project. A closely related practice is also offered in the module, Minimizing Harmful Consequences, in the Center for Humane Technology free course.
Why Does Yellow Teaming Matter?
The central idea is that by analyzing the consequences of your product decisions with a wide view, you can design better products that create positive feedback loops for your company’s bottom line and your users’ well-being. For example, it helps you avoid building a chatbot that unintentionally reinforces bias.
Traditional product development practices often solve for narrowly defined success metrics. Creating specific product measurables is good for focus and accountability, but can lead to over-optimization on metrics while ignoring other signals that matter to your company. For instance, building an app with AI-driven recommendations that boosts engagement in the short term but makes people feel worse and fails to retain users over time.
Narrow product optimization tends to cause unmeasured negative effects. These include users getting burnt out or frustrated when using your product, reputational harm or less overall engagement with your company, and society fracturing from lack of trust and meaningful communication.
In many cases, what looks like product success on paper is actually harming your users, your company, and your long-term goals.
How to Implement Yellow Teaming Practices
Yellow Teaming is straightforward and powerful. Pick a product you are building, and systematically evaluate the various consequences for your users, your business, and society when adopted at scale. Start with direct consequences, then move to second- and third-order consequences by asking ‘what happens as a result of the previous effects?’ You should think through these consequences across multiple axis:
- Good and bad
- Short-term and long-term
- Intended and unintended
- Your company and your users
- A single user and groups of users
These types of questions help foster productive brainstorming:
- What kinds of behaviors will this feature incentivize in users?
- What affordances does this technology provide (what can users now do that they couldn’t before, even if unintended)?
- Will this improve or degrade trust in our platform?
- What social groups might benefit—or be left behind?
Yellow Teaming is based on complex systems thinking and externality analysis—fields that have traditionally felt far removed from engineering workflows. But by incorporating a lightweight Yellow Teaming assistant to help your ideation processes, it can become an intuitive, high ROI part of product development.
Building Your PyTorch YellowTeamGPT
The good news is that you don’t need a PhD in philosophy or a panel of advisors to Yellow Team your AI project. You just need to be willing to act and, in this implementation of Yellow Teaming, use a good LLM with the right prompt. There are several advantages to running your LLM locally. The biggest is that you can safely feed in confidential product plans without worrying about your data being leaked. Another benefit is that the smaller model is not perfect and makes mistakes, forcing us as users to apply critical thinking to every output, and putting us in the right headspace to analyze non-obvious product consequences.
Here is how you can set up a PyTorch-based 8-billion parameter Llama3 model on your Graviton instance. First, create a r8g.4xlarge instance running Ubuntu 24.04 with at least 50 GB of storage, then follow these three steps:
1. Set up your machine with the torchchat repo and other requirements:
sudo apt-get update && sudo apt install gcc g++ build-essential python3-pip python3-venv google-perftools -y
git clone https://github.com/pytorch/torchchat.git && cd torchchat
python3 -m venv .venv && source .venv/bin/activate
./install/install_requirements.sh
2. Download the model from Hugging Face (HF) by entering your HF access token (note the max sequence length parameter, which you can increase to enable longer conversations with a linear increase in memory usage):
pip install -U "huggingface_hub[cli]"
huggingface-cli login
python torchchat.py export llama3.1 --output-dso-path exportedModels/llama3.1.so --device cpu --max-seq-length 8192
3. Run the model with Arm CPU optimizations and 700 max token length per response:
LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libtcmalloc.so.4 TORCHINDUCTOR_CPP_WRAPPER=1 TORCHINDUCTOR_FREEZING=1 OMP_NUM_THREADS=16 python torchchat.py generate llama3.1 --dso-path exportedModels/llama3.1.so --device cpu --max-new-tokens 700 --chat
For more details on these commands and additional code snippets to add a UI to this chatbot, review this Arm Learning Path.
You can then enter a custom system prompt. Below is a simple prompt that turns your local LLM into a Yellow Teaming assistant. Feel free to review and tweak it to get the most out of it for your specific needs. Here’s what it does:
- Gathers key product details: What you’re building, how it makes money, who your users are.
- Analyzes direct and indirect consequences: YellowTeamGPT presents one at a time, considering non-obvious impacts to your business, users, and beyond (you’ll likely start to think of more impacts on your own).
- Iterates with you: You are in control, telling YellowTeamGPT to continue listing general direct consequences, identifying specific company risks, moving to 2nd-order effects, and even brainstorming features to make your product better.
Here is the YellowTeamGPT system prompt for you to copy. If directly copying, make sure to copy as one line into your terminal or the new lines may cause issues.
You are an expert in complex systems thinking and AI product design, called YellowTeamGPT. You help technologists build better products that users love, and lower company risk. You do this by helping the user evaluate their product design decisions via the Yellow Teaming methodology, which identifies the consequences of design decisions on their business, their users, and society.
You will request from the user information about their product under development. Once you have enough information, you will analyze the product’s consequences that arise if deployed at scale. Structure your thinking to first review direct consequences, then 2nd order consequences that follow from the identified direct effects (by asking ‘what might happen next as a result?’). Consider consequences that impact the company, users, and society; are short and long term; are across categories like truth and understanding, human well-being, capability growth, economics, and more.
You are here to constructively challenge users, not reinforce their existing ideas. Play devil’s advocate to help users think in ways they are not currently.
You will output in this format: For each identified consequence, tie the impact to product quality, and prompt the user with a question that helps them design the product better to mitigate that consequence (or turn a negative impact into a positive one). List one consequence at a time and ask the user to continue listing them or explore that consequence further.
Example Yellow Teaming
Give your LLM the provided system prompt and hit enter. Next, your YellowTeamGPT assistant will ask for some product details. Here is a hypothetical example product I used:
I’m building an app that turns a group chat conversation into a catchy pop song. Targeting any user, like WhatsApp users. Key functionality is importing a group chat conversation and outputting a tune with lyrics and beat to match. It is an app on any smartphone. Ideally, millions of users. Would make money by targeted advertising of the users.
You’ll notice, as YellowTeamGPT thinks and generates its reply, that it is notably slower than ChatGPT or other popular GPTs. Like the model’s inaccuracy, its slow speed can be perceived as a benefit. The point of this exercise is to slow down, think through non-obvious product impacts, and brainstorm enhancements that create positive value across the systems your product touches. While your YellowTeamingGPT is ‘thinking,’ you should be too.
And below are snippets of my conversation. First, it starts with one direct consequence:
I then instruct it to continue to another consequence:
I ask to explore the second-order effects of having misinformation spread at scale from this app:
Finally, I ask for help brainstorming product features to mitigate this harm. It generates a few interesting concepts that are not product-ready, but easily spark further ideation:
Using YellowTeamGPT for this use case, we were able to rapidly explore product impacts we may not have considered. We could then brainstorm features solving previously unconsidered problems, leading to an improved product experience that also mitigates the risk of reputational harm to our hypothetical company.
Integrating Yellow Teaming Into Your Practices
Anywhere you’re making decisions that shape your product’s features and the user experience, Yellow Teaming fits. Here are a few examples of where you can leverage your new YellowTeamGPT:
- New product ideation sessions to expand your thinking.
- Feature planning docs to stress-test your specs.
- Code review workflows for flagging potential misuse.
- Sprint retrospectives to reflect on design choices at scale.
- Product pitch decks to show responsible AI due diligence.
It can be as formal or informal as you want. The more you and your team think about unintended, Nth-order product consequences across multiple axis, the better your product will be. By incorporating Yellow Teaming into your work, you don’t just do the right thing, you build products that:
- Users engage with and trust more
- Mitigate harmful impacts
- Minimize company risk
- Create lasting business value
Let’s stop thinking of responsible AI practices as something to check off a list and start thinking of it as what it really is –a competitive edge that creates positive outcomes for your company, for your users, and for our shared society.
Read More
