A new era of autonomous vehicle technology, known as AV 2.0, has emerged, marked by large, unified AI models that can control multiple parts of the vehicle stack, from perception and planning to control.
Wayve, a London-based autonomous driving technology company, is leading the surf.
In the latest episode of NVIDIA’s AI Podcast, host Katie Burke Washabaugh spoke with the company’s cofounder and CEO, Alex Kendall, about what AV 2.0 means for the future of self-driving cars.
Unlike AV 1.0’s focus on perfecting a vehicle’s perception capabilities using multiple deep neural networks, AV 2.0 calls for comprehensive in-vehicle intelligence to drive decision-making in real-world, dynamic environments.
Embodied AI — the concept of giving AI a physical interface to interact with the world — is the basis of this new AV wave.
Kendall pointed out that it’s a “hardware/software problem — you need to consider these things separately,” even as they work together. For example, a vehicle can have the highest-quality sensors, but without the right software, the system can’t use them to execute the right decisions.
Generative AI plays a key role, enabling synthetic data generation so AV makers can use a model’s previous experiences to create and simulate novel driving scenarios.
It can “take crowds of pedestrians and snow and bring them together” to “create a snowy, crowded pedestrian scene” that the vehicle has never experienced before.
According to Kendall, that will “play a huge role in both learning and validating the level of performance that we need to deploy these vehicles safely” — all while saving time and costs.
In June, Wayve unveiled GAIA-1, a generative world model for developing autonomous vehicles.
The company also recently announced LINGO-1, an AI model that allows passengers to use natural language to enhance the learning and explainability of AI driving models.
Looking ahead, the company hopes to scale and further develop its solutions, improving the safety of AVs to deliver value, build public trust and meet customer expectations. Kendall views embodied AI as playing a definitive role in the future of the AI landscape, pushing pioneers to “build better” and “build further” to achieve the “next big breakthroughs.”
Teaching the AI brains of autonomous vehicles to understand the world as humans do requires billions of miles of driving experience—the road to achieving this astronomical level of driving leads to the virtual world. Learn how Waabi uses powerful high-fidelity simulations to train and develop production-level autonomous vehicles.
Driving enjoyment and autonomous driving capabilities can complement one another in intelligent, sustainable vehicles. Learn about the automaker’s plans to unveil its third vehicle, the Polestar 3, the tech inside it, and what the company’s racing heritage brings to the intersection of smarts and sustainability.
Humans playing games against machines is nothing new, but now computers can develop games for people to play. Programming enthusiast and social media influencer Harrison Kinsley created GANTheftAuto, an AI-based neural network that generates a playable chunk of the classic video game Grand Theft Auto V.
SUBHEAD: Subscribe to the AI Podcast, Now Available on Amazon Music
Editor’s note: This post is part of our weekly In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows.
‘Tis the season for friends, family and beautifully rendered Santa animations from this week’s In the NVIDIA Studio artist, 3D expert Božo Balov.
This week also marks an incredible milestone, with over 500 NVIDIA RTX-powered games and creative apps now available with support for ray tracing and AI-powered technologies like NVIDIA DLSS. Over 120 of the most popular apps — including the Adobe Creative Cloud suite, Autodesk Maya, Blender, Blackmagic Design’s Davinci Resolve, OBS, Unity and more — use RTX to accelerate workflows by orders of magnitude, power new AI tools and enhancements and enable real-time, ray-traced previews.
To celebrate, NVIDIA GeForce is hosting a giveaway for gift cards, rare, sought-after #RTXON keyboard keycaps and more. Follow GeForce on Facebook, Instagram, TikTok or X (formerly known as Twitter) for instructions on how to enter.
THANK YOU FOR 500 RTX GAMES & APPS
To celebrate use #RTX500 ALL December across our social channels for a chance to win $500 gift cards courtesy of Green Man Gaming + other great prize giveaways.
Say it ain’t snow: the NVIDIA Studio #WinterArtChallenge is back. Through the end of the year, share winter-themed art on Facebook, Instagram or X for a chance to be featured on NVIDIA Studio social media channels. Be sure to tag #WinterArtChallenge to join.
Share your winter-themed art (like this incredible one created on an RTX GPU by @rafianimates) using the hashtag for a chance to be featured on our social channels!
Finally, 80 Level — the creative community for digital artists, animators and computer-generated imagery specialists — is hosting its Community Metasites Challenge. Artists can showcase their creativity by applying unique aesthetics to a simple block level via characters, game mechanics, visual effects and more — with a chance to win a new NVIDIA Studio laptop. Register today.
Special Thanks to @NVIDIAStudio who is offering this incredible prize for the first place winner of the 80 Level Community Metasites Challenge. An ASUS – ProArt Studiobook 16 OLED H7600 16″ Laptop!
Balov’s Christmas Rush 3D animation reimagines Santa as a resident of the coastal city of Split, Croatia — but with a harsher, less jolly edge.
Balov jumped straight into modeling edgy Saint Nick in the virtual-reality modeling software Quill. He deployed vertex-painting techniques and used a photogrammetry scan of a Vespa as a base, adding brushstrokes to blend it with the rest of the scene.
To achieve a flickering effect on Santa’s clothing, Balov created a custom texture with different brush strokes in Adobe Photoshop. The texture doubles as an alpha map, which intentionally clips the geometry.
“When it comes to rendering 3D graphics, nothing really comes close to NVIDIA GPUs.” — Božo Balov
He then used Adobe Photoshop to paint monochromatic background layers. Balov’s GeForce RTX 3080 Ti GPU unlocked over 30 GPU-accelerated features, including blur gallery, liquify, smart sharpen and perspective warp.
Balov then converted the files to the FBX adaptable file format for 3D software before importing them into Blender, where he animated the layers to move in the opposite direction of the character to create a sense of speed. He kept the lighting fairly simple, with one light source as the base and a few supplemental ones to emphasize specific parts of the scene.
Balov prefers working in Blender’s real-time engine EEVEE to animate his scene, cutting wait times. RTX-accelerated OptiX ray tracing in the viewport enabled greater interactivity with smoother movement, speeding his ideation and creative workflow.
Extraordinary detail.
“Rendering is a joy on NVIDIA RTX cards,” said Balov. “Since OptiX made its debut, rendering times have been cut in half or more — Blender Cycles feels like a real-time engine.”
When asked for advice to give aspiring artists, Balov emphasized the importance of individual passion.
“Pursue what matters to you,” he said. “Don’t spend time fulfilling other people’s ideas of what art should be.”
To enhance the gaming experience, studios and developers spend tremendous effort creating photorealistic, immersive in-game environments.
But non-playable characters (NPCs) often get left behind. Many behave in ways that lack depth and realism, making their interactions repetitive and forgettable.
Inworld AI is changing the game by using generative AI to drive NPC behaviors that are dynamic and responsive to player actions. The Mountain View, Calif.-based startup’s Character Engine, which can be used with any character design, is helping studios and developers enhance gameplay and improve player engagement.
Elevate Gaming Experiences: Achievement Unlocked
The Inworld team aims to develop AI-powered NPCs that can learn, adapt and build relationships with players while delivering high-quality performance and maintaining in-game immersion.
To make it easier for developers to integrate AI-based NPCs into their games, Inworld built Character Engine, which uses generative AI running on NVIDIA technology to create immersive, interactive characters. It’s built to be production-ready, scalable and optimized for real-time experiences.
The Character Engine comprises three layers: Character Brain, Contextual Mesh and Real-Time AI.
The layer also enables AI-based NPCs to learn and adapt, navigate relationships and perform motivated actions. For example, users can create triggers using the “Goals and Action” feature to program NPCs to behave in a certain way in response to a given player input.
Contextual Mesh allows developers to set parameters for content and safety mechanisms, custom knowledge and narrative controls. Game developers can use the “Relationships” feature to create emergent narratives, such that an ally can turn into an enemy or vice versa based on how players treat an NPC.
One big challenge developers face when using generative AI is keeping NPCs in-world and on-message. Inworld’s Contextual Mesh layer helps overcome this hurdle by rendering characters within the logic and fantasy of their worlds, effectively avoiding the hallucinations that commonly appear when using large language models (LLMs).
The Real-Time AI layer ensures optimal performance and scalability for real-time experiences.
Inworld used the open-source NVIDIA Triton Inference Server software to standardize other non-generative machine learning model deployments required to power Character Brain features, such as emotions. The startup also plans to use the open-source NVIDIA TensorRT-LLM library to optimize inference performance. Both NVIDIA Triton Inference Server and TensorRT-LLM are available with the NVIDIA AI Enterprise software platform, which provides security, stability and support for production AI.
Inworld also used NVIDIA A100 GPUs within Slurm-managed bare-metal machines for its production training pipelines. Similar machines wrapped in Kubernetes help manage character interactions during gameplay. This setup delivers real-time generative AI at the lowest possible cost.
“We chose to use NVIDIA A100 GPUs because they provided the best, most cost-efficient option for our machine learning workloads compared to other solutions,” said Igor Poletaev, vice president of AI at Inworld.
“Our customers and partners are looking to find novel and innovative ways to drive player engagement metrics by integrating AI NPC functionalities into their gameplay,” said Poletaev. “There’s no way to achieve real-time performance without hardware accelerators, which is why we required GPUs to be integrated into our backend architecture from the very beginning.”
Inworld’s generative AI-powered NPCs have enabled dynamic, evergreen gaming experiences that keep players coming back. Developers and gamers alike have reported enhanced player engagement, satisfaction and retention.
Inworld has powered AI-based NPC experiences from Niantic, LG UPlus, Alpine Electronics and more. One open-world virtual reality game using the Inworld Character Engine saw a 5% increase in playtime, while a detective-themed indie game garnered over $300,000 in free publicity after some of the most popular Twitch streamers discovered it.
GPUs have been called the rare Earth metals — even the gold — of artificial intelligence, because they’re foundational for today’s generative AI era.
Three technical reasons, and many stories, explain why that’s so. Each reason has multiple facets well worth exploring, but at a high level:
GPUs employ parallel processing.
GPU systems scale up to supercomputing heights.
The GPU software stack for AI is broad and deep.
The net result is GPUs perform technical calculations faster and with greater energy efficiency than CPUs. That means they deliver leading performance for AI training and inference as well as gains across a wide array of applications that use accelerated computing.
In its recent report on AI, Stanford’s Human-Centered AI group provided some context. GPU performance “has increased roughly 7,000 times” since 2003 and price per performance is “5,600 times greater,” it reported.
A 2023 report captured the steep rise in GPU performance and price/performance.
The report also cited analysis from Epoch, an independent research group that measures and forecasts AI advances.
“GPUs are the dominant computing platform for accelerating machine learning workloads, and most (if not all) of the biggest models over the last five years have been trained on GPUs … [they have] thereby centrally contributed to the recent progress in AI,” Epoch said on its site.
A 2020 study assessing AI technology for the U.S. government drew similar conclusions.
“We expect [leading-edge] AI chips are one to three orders of magnitude more cost-effective than leading-node CPUs when counting production and operating costs,” it said.
NVIDIA GPUs have increased performance on AI inference 1,000x in the last ten years, said Bill Dally, the company’s chief scientist in a keynote at Hot Chips, an annual gathering of semiconductor and systems engineers.
ChatGPT Spread the News
ChatGPT provided a powerful example of how GPUs are great for AI. The large language model (LLM), trained and run on thousands of NVIDIA GPUs, runs generative AI services used by more than 100 million people.
Since its 2018 launch, MLPerf, the industry-standard benchmark for AI, has provided numbers that detail the leading performance of NVIDIA GPUs on both AI training and inference.
For example, NVIDIA Grace Hopper Superchips swept the latest round of inference tests. NVIDIA TensorRT-LLM, inference software released since that test, delivers up to an 8x boost in performance and more than a 5x reduction in energy use and total cost of ownership. Indeed, NVIDIA GPUs have won every round of MLPerf training and inference tests since the benchmark was released in 2019.
In February, NVIDIA GPUs delivered leading results for inference, serving up thousands of inferences per second on the most demanding models in the STAC-ML Markets benchmark, a key technology performance gauge for the financial services industry.
A RedHat software engineering team put it succinctly in a blog: “GPUs have become the foundation of artificial intelligence.”
AI Under the Hood
A brief look under the hood shows why GPUs and AI make a powerful pairing.
An AI model, also called a neural network, is essentially a mathematical lasagna, made from layer upon layer of linear algebra equations. Each equation represents the likelihood that one piece of data is related to another.
For their part, GPUs pack thousands of cores, tiny calculators working in parallel to slice through the math that makes up an AI model. This, at a high level, is how AI computing works.
Highly Tuned Tensor Cores
Over time, NVIDIA’s engineers have tuned GPU cores to the evolving needs of AI models. The latest GPUs include Tensor Cores that are 60x more powerful than the first-generation designs for processing the matrix math neural networks use.
Along the way, each GPU generation has packed more memory and optimized techniques to store an entire AI model in a single GPU or set of GPUs.
Models Grow, Systems Expand
The complexity of AI models is expanding a whopping 10x a year.
The current state-of-the-art LLM, GPT4, packs more than a trillion parameters, a metric of its mathematical density. That’s up from less than 100 million parameters for a popular LLM in 2018.
In a recent talk at Hot Chips, NVIDIA Chief Scientist Bill Dally described how single-GPU performance on AI inference expanded 1,000x in the last decade.
GPU systems have kept pace by ganging up on the challenge. They scale up to supercomputers, thanks to their fast NVLink interconnects and NVIDIA Quantum InfiniBand networks.
For example, the DGX GH200, a large-memory AI supercomputer, combines up to 256 NVIDIA GH200 Grace Hopper Superchips into a single data-center-sized GPU with 144 terabytes of shared memory.
Each GH200 superchip is a single server with 72 Arm Neoverse CPU cores and four petaflops of AI performance. A new four-way Grace Hopper systems configuration puts in a single compute node a whopping 288 Arm cores and 16 petaflops of AI performance with up to 2.3 terabytes of high-speed memory.
And NVIDIA H200 Tensor Core GPUs announced in November pack up to 288 gigabytes of the latest HBM3e memory technology.
Software Covers the Waterfront
An expanding ocean of GPU software has evolved since 2007 to enable every facet of AI, from deep-tech features to high-level applications.
The NVIDIA AI platform includes hundreds of software libraries and apps. The CUDA programming language and the cuDNN-X library for deep learning provide a base on top of which developers have created software like NVIDIA NeMo, a framework to let users build, customize and run inference on their own generative AI models.
Many of these elements are available as open-source software, the grab-and-go staple of software developers. More than a hundred of them are packaged into the NVIDIA AI Enterprise platform for companies that require full security and support. Increasingly, they’re also available from major cloud service providers as APIs and services on NVIDIA DGX Cloud.
SteerLM, one of the latest AI software updates for NVIDIA GPUs, lets users fine tune models during inference.
A 70x Speedup in 2008
Success stories date back to a 2008 paper from AI pioneer Andrew Ng, then a Stanford researcher. Using two NVIDIA GeForce GTX 280 GPUs, his three-person team achieved a 70x speedup over CPUs processing an AI model with 100 million parameters, finishing work that used to require several weeks in a single day.
“Modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods,” they reported.
Andrew Ng described his experiences using GPUs for AI in a GTC 2015 talk.
In a 2015 talk at NVIDIA GTC, Ng described how he continued using more GPUs to scale up his work, running larger models at Google Brain and Baidu. Later, he helped found Coursera, an online education platform where he taught hundreds of thousands of AI students.
Ng counts Geoff Hinton, one of the godfathers of modern AI, among the people he influenced. “I remember going to Geoff Hinton saying check out CUDA, I think it can help build bigger neural networks,” he said in the GTC talk.
The University of Toronto professor spread the word. “In 2009, I remember giving a talk at NIPS [now NeurIPS], where I told about 1,000 researchers they should all buy GPUs because GPUs are going to be the future of machine learning,” Hinton said in a press report.
Fast Forward With GPUs
AI’s gains are expected to ripple across the global economy.
A McKinsey report in June estimated that generative AI could add the equivalent of $2.6 trillion to $4.4 trillion annually across the 63 use cases it analyzed in industries like banking, healthcare and retail. So, it’s no surprise Stanford’s 2023 AI report said that a majority of business leaders expect to increase their investments in AI.
Today, more than 40,000 companies use NVIDIA GPUs for AI and accelerated computing, attracting a global community of 4 million developers. Together they’re advancing science, healthcare, finance and virtually every industry.
Among the latest achievements, NVIDIA described a whopping 700,000x speedup using AI to ease climate change by keeping carbon dioxide out of the atmosphere (see video below). It’s one of many ways NVIDIA is applying the performance of GPUs to AI and beyond.
Let the games begin — this GFN Thursday brings the highly anticipated Call of Duty: Modern Warfare III to the cloud, the first Activision title on GeForce NOW as part of the NVIDIA and Microsoft partnership.
It’s joined by Call of Duty: Modern Warfare II and Call of Duty: Warzone — all three titles can be played from one central location via the Call of Duty logo on GeForce NOW.
And it’s the most wonderful time of the year — over 65 games are joining the GeForce NOW library in December, with 15 available to stream this week.
Plus, stream GeForce NOW on the go and get console-quality controls by simply snapping a mobile device into a Backbone One controller. For a limited time, Backbone is offering a 30% discount for premium GeForce NOW members starting today in the Rewards Portal. Free-level members can claim the discount starting Dec. 7.
The Lobby Awaits
The war has changed.
Call of Duty: Modern Warfare III returns as a direct sequel to the record-breaking Call of Duty: Modern Warfare II and follows the story of Task Force 141 as they face off the ultimate threat.
Dig into the action-packed single-player campaign or head online to defeat the undead in an exciting open-world co-op experience that takes the Zombies mode that fans know and love to the next level. Those that prefer some multiplayer action can dip into a selection of Core Multiplayer maps from the 16 iconic launch maps of 2009’s Call of Duty: Modern Warfare 2 that are being brought over and modernized for Call of Duty: Modern Warfare III.
Plus, stay tuned to GFN Thursday for when other legacy Call of Duty titles as well as additional supported devices (Android, SHIELD TV and TV) will be added to the cloud. Check out the article for more details.
GeForce NOW Ultimate members can get the upper hand with NVIDIA DLSS 3 and Reflex to get the highest frame rates and lowest latencies for the smoothest gameplay by streaming from a GeForce RTX 4080 gaming rig in the cloud. Never worry about upgrading hardware or system specs again with GeForce NOW.
Presents, Galore
Dig, dig, dig!
Break ground in SteamWorld Build from Thunderful Publishing. Dig deep and build wide to excavate long-lost spacefaring technology while ensuring everyone has the resources needed to survive and reach the final frontier. It launches Dec. 1 on Steam and PC Game Pass — check it out with the three free months of PC Game Pass included with the purchase of a six-month Ultimate membership, part of the GeForce NOW holiday bundle.
Members can start their adventures now with 15 newly supported titles in the cloud this week:
World War Z: Aftermath (Xbox, available on the Microsoft Store)
Worms Rumble (Xbox, available on the Microsoft Store)
Worms W.M.D (Xbox, available on the Microsoft Store)
Nicely Done in November
On top of the 54 games announced in October, an additional 23 joined the cloud last month, including this week’s additions: Astrea: Six-Sided Oracles, Galactic Civilizations IV, Halls of Torment, Kona II: Brume, Laika: Aged Through Blood (Epic Games Store), Pillars of Eternity and SteamWorld Build.
Car Mechanic Simulator 2021 (Xbox, available on PC Game Pass)
Veiled Experts will no longer be coming to the service due to the closure of its live services, and Spirttea (PC Game Pass) didn’t make it to GeForce NOW in November due to technical issues. Stay tuned to GFN Thursday for future updates.
What are you planning to play this weekend? Let us know on Twitter or in the comments below.
Amazon Web Services and NVIDIA will bring the latest generative AI technologies to enterprises worldwide.
Combining AI and cloud computing, NVIDIA founder and CEO Jensen Huang joined AWS CEO Adam Selipsky Tuesday on stage at AWS re:Invent 2023 at the Venetian Expo Center in Las Vegas.
Selipsky said he was “thrilled” to announce the expansion of the partnership between AWS and NVIDIA with more offerings that will deliver advanced graphics, machine learning and generative AI infrastructure.
The two announced that AWS will be the first cloud provider to adopt the latest NVIDIA GH200 NVL32 Grace Hopper Superchip with new multi-node NVLink technology, that AWS is bringing NVIDIA DGX Cloud to AWS, and that AWS has integrated some of NVIDIA’s most popular software libraries.
Huang started the conversation by highlighting the integration of key NVIDIA libraries with AWS, encompassing a range from NVIDIA AI Enterprise to cuQuantum to BioNeMo, catering to domains like data processing, quantum computing and digital biology.
The partnership opens AWS to millions of developers and the nearly 40,000 companies who are using these libraries, Huang said, adding that it’s great to see AWS expand its cloud instance offerings to include NVIDIA’s new L4, L40S and, soon, H200 GPUs.
Selipsky then introduced the AWS debut of the NVIDIA GH200 Grace Hopper Superchip, a significant advancement in cloud computing, and prompted Huang for further details.
“Grace Hopper, which is GH200, connects two revolutionary processors together in a really unique way,” Huang said. He explained that the GH200 connects NVIDIA’s Grace Arm CPU with its H200 GPU using a chip-to-chip interconnect called NVLink, at an astonishing one terabyte per second.
Each processor has direct access to the high-performance HBM and efficient LPDDR5X memory. This configuration results in 4 petaflops of processing power and 600GB of memory for each superchip.
AWS and NVIDIA connect 32 Grace Hopper Superchips in each rack using a new NVLink switch. Each 32 GH200 NVLink-connected node can be a single Amazon EC2 instance. When these are integrated with AWS Nitro and EFA networking, customers can connect GH200 NVL32 instances to scale to thousands of GH200 Superchips
“With AWS Nitro, that becomes basically one giant virtual GPU instance,” Huang said.
The combination of AWS expertise in highly scalable cloud computing plus NVIDIA innovation with Grace Hopper will make this an amazing platform that delivers the highest performance for complex generative AI workloads, Huang said.
“It’s great to see the infrastructure, but it extends to the software, the services and all the other workflows that they have,” Selipsky said, introducing NVIDIA DGX Cloud on AWS.
This partnership will bring about the first DGX Cloud AI supercomputer powered by the GH200 Superchips, demonstrating the power of AWS’s cloud infrastructure and NVIDIA’s AI expertise.
Following up, Huang announced that this new DGX Cloud supercomputer design in AWS, codenamed Project Ceiba, will serve as NVIDIA’s newest AI supercomputer as well, for its own AI research and development.
Named after the majestic Amazonian Ceiba tree, the Project Ceiba DGX Cloud cluster incorporates 16,384 GH200 Superchips to achieve 65 exaflops of AI processing power, Huang said.
Ceiba will be the world’s first GH200 NVL32 AI supercomputer built and the newest AI supercomputer in NVIDIA DGX Cloud, Huang said.
Huang described Project Ceiba AI supercomputer as “utterly incredible,” saying it will be able to reduce the training time of the largest language models by half.
NVIDIA’s AI engineering teams will use this new supercomputer in DGX Cloud to advance AI for graphics, LLMs, image/video/3D generation, digital biology, robotics, self-driving cars, Earth-2 climate prediction and more, Huang said.
“DGX is NVIDIA’s cloud AI factory,” Huang said, noting that AI is now key to doing NVIDIA’s own work in everything from computer graphics to creating digital biology models to robotics to climate simulation and modeling.
“DGX Cloud is also our AI factory to work with enterprise customers to build custom AI models,” Huang said. “They bring data and domain expertise; we bring AI technology and infrastructure.”
In addition, Huang also announced that AWS will be bringing four Amazon EC2 instances based on the NVIDIA GH200 NVL, H200, L40S, L4 GPUs, coming to market early next year.
Selipsky wrapped up the conversation by announcing that GH200-based instances and DGX Cloud will be available on AWS in the coming year. You can catch the discussion and Selipsky’s entire keynote on AWS’s YouTube channel.
Researchers and developers at leading pharmaceutical and techbio companies can now easily deploy NVIDIA Clara software and services for accelerated healthcare through Amazon Web Services.
Announced today at AWS re:Invent, the initiative gives healthcare and life sciences developers using AWS cloud resources the flexibility to integrate NVIDIA-accelerated offerings such as NVIDIA BioNeMo — a generative AI platform for drug discovery — coming to NVIDIA DGX Cloud on AWS, and currently available via the AWS ParallelCluster cluster management tool for high performance computing and the Amazon SageMaker machine learning service.
Thousands of healthcare and life sciences companies globally use AWS. They will now be able to access BioNeMo to build or customize digital biology foundation models with proprietary data, scaling up model training and deployment using NVIDIA GPU-accelerated cloud servers on AWS.
Techbio innovators including Alchemab Therapeutics, Basecamp Research, Character Biosciences, Evozyne, Etcembly and LabGenius are among the AWS users already using BioNeMo for generative AI-accelerated drug discovery and development. This collaboration gives them more ways to rapidly scale up cloud computing resources for developing generative AI models trained on biomolecular data.
This announcement extends NVIDIA’s existing healthcare-focused offerings available on AWS — NVIDIA MONAI for medical imaging workflows and NVIDIA Parabricks for accelerated genomics.
New to AWS: NVIDIA BioNeMo Advances Generative AI for Drug Discovery
BioNeMo is a domain-specific framework for digital biology generative AI, including pretrained large language models (LLMs), data loaders and optimized training recipes that can help advance computer-aided drug discovery by speeding target identification, protein structure prediction and drug candidate screening.
Drug discovery teams can use their proprietary data to build or optimize models with BioNeMo and run them on cloud-based high performance computing clusters.
One of these models, ESM-2 — a powerful LLM that supports protein structure prediction — achieves almost linear scaling on 256 NVIDIA H100 Tensor Core GPUs. Researchers can scale to 512 H100 GPUs to complete training in a few days instead of a month, the training time published in the original paper.
Developers can train ESM-2 at scale using checkpoints of 650 million or 3 billion parameters. Additional AI models supported in the BioNeMo training framework include small-molecule generative model MegaMolBART and protein sequence generation model ProtT5.
BioNeMo’s pretrained models and optimized training recipes — which are available using self-managed services like AWS ParallelCluster and Amazon ECS as well as integrated, managed services through NVIDIA DGX Cloud and Amazon SageMaker — can help R&D teams build foundation models that can explore more drug candidates, optimize wet lab experimentation and find promising clinical candidates faster.
Also Available on AWS: NVIDIA Clara for Medical Imaging and Genomics
Project MONAI, cofounded and enterprise-supported by NVIDIA to support medical imaging workflows, has been downloaded more than 1.8 million times and is available for deployment on AWS. Developers can harness their proprietary healthcare datasets already stored on AWS cloud resources to rapidly annotate and build AI models for medical imaging.
These models, trained on NVIDIA GPU-powered Amazon EC2 instances, can be used for interactive annotation and fine-tuning for segmentation, classification, registration and detection tasks in medical imaging. Developers can also harness MRI image synthesis models available in MONAI to augment training datasets.
To accelerate genomics pipelines, Parabricks enables variant calling on a whole human genome in around 15 minutes, compared to a day on a CPU-only system. On AWS, developers can quickly scale up to process large amounts of genomic data across multiple GPU nodes.
More than a dozen Parabricks workflows are available on AWS HealthOmics as Ready2Run workflows, which enable customers to easily run pre-built pipelines.
Get started with NVIDIA Clara on AWS to accelerate AI workflows for drug discovery, genomics and medical imaging.
Developing more intelligent robots in the cloud is about to get a speed multiplier.
NVIDIA Isaac Sim and NVIDIA L40S GPUs are coming to Amazon Web Services, enabling developers to build and deploy accelerated robotics applications in the cloud. Isaac Sim, an extensible simulator for AI-enabled robots, is built on the NVIDIA Omniverse development platform for building and connecting OpenUSD applications.
Combining powerful AI compute with graphics and media acceleration, the L40S GPU is built to power the next generation of data center workloads. Based on the Ada Lovelace architecture, the L40S enables ultrafast real-time rendering delivering up to a 3.8x performance leap for Omniverse compared with the previous generation, boosting engineering and robotics teams.
The generational leap in acceleration results in 2x faster performance than the A40 GPU across a broad set of robotic simulations tasks when using Isaac Sim.
L40S GPUs can also be harnessed for generative AI workloads, from fine-tuning large language models within a matter of hours, to real-time inferencing for text-to-image and chat applications.
New Amazon Machine Images (AMIs) on the NVIDIA L40S in AWS Marketplace will enable roboticists to easily access preconfigured virtual machines to operate Isaac Sim workloads.
Robotics development in simulation is speeding the process of deploying applications, turbocharging industries such as retail, food processing, manufacturing, logistics and more.
Revenue from mobile robots in warehouses worldwide is expected to explode, more than tripling from $11.6 billion in 2023 to $42.2 billion by 2030, according to ABI Research.
Robotics systems have played an important role across fulfillment centers to help meet the demands of online shoppers and provide a better workplace for employees. Amazon Robotics has deployed more than 750,000 robots in its warehouses around the world to improve the experience for employees supporting package fulfillment and its customers.
“Simulation technology plays a critical role in how we develop, test and deploy our robots.” said Brian Basile, head of virtual systems at Amazon Robotics. “At Amazon Robotics we continue to increase the scale and complexity of our simulations. With the new AWS L40S offering we will push the boundaries of simulation, rendering and model training even further.”
Accelerated Robotics Development With Isaac Sim
Robotics systems can demand large datasets for precision operation in deployed applications. Gathering these datasets and testing them in the real world is time-consuming, costly and impractical.
Robotics simulation drives the training and testing of AI-based robotic applications. With synthetic data, simulations are enabling virtual advances like never before. Simulations can help verify, validate and optimize robot designs, systems and their algorithms before operation. It can also be used to optimize facility designs before construction or remodeling starts for maximum efficiencies, reducing costly manufacturing change orders.
Isaac Sim offers access to the latest robotics simulation tools and capabilities as well as cloud access, enabling teams to collaborate more effectively. Access to the Omniverse Replicator synthetic data generation engine in Isaac Sim allows machine learning engineers to build production-ready synthetic datasets for training robust deep learning perception models.
Amazon Robotics has begun using Omniverse to build digital twins for automating, optimizing and planning its autonomous warehouses in virtual environments before deploying them into the real world.
Using Isaac Sim for sensor emulation, Amazon Robotics will accelerate development of its Proteus autonomous mobile robot, improving it to help the online retail giant efficiently manage fulfillment.
Everything about large language models is big — giant models train on massive datasets across thousands of NVIDIA GPUs.
That can pose a lot of big challenges for companies pursuing generative AI. NVIDIA NeMo, a framework for building, customizing and running LLMs, helps overcome these challenges.
A team of experienced scientists and developers at Amazon Web Services creating Amazon Titanfoundation models for Amazon Bedrock, a generative AI service for foundation models, has been using NVIDIA NeMo for over the past several months.
“One key reason for us to work with NeMo is that it is extensible, comes with optimizations that allow us to run with high GPU utilization while also enabling us to scale to larger clusters so we can train and deliver models to our customers faster,” said Leonard Lausen, a senior applied scientist at AWS.
Think Big, Really Big
Parallelism techniques in NeMo enable efficient LLM training at scale. When coupled with the Elastic Fabric Adapter from AWS, it allowed the team to spread its LLM across many GPUs to accelerate training.
EFA provides AWS customers with an UltraCluster Networking infrastructure that can directly connect more than 10,000 GPUs and bypass the operating system and CPU using NVIDIA GPUDirect.
The combination allowed the AWS scientists to deliver excellent model quality — something that’s not possible at scale when relying solely on data parallelism approaches.
Framework Fits All Sizes
“The flexibility of NeMo,” Lausen said, “allowed AWS to tailor the training software for the specifics of the new Titan model, datasets and infrastructure.”
AWS’s innovations include efficient streaming from Amazon Simple Storage Service (Amazon S3) to the GPU cluster. “It was easy to incorporate these improvements because NeMo builds upon popular libraries like PyTorch Lightning that standardize LLM training pipeline components,” Lausen said.
AWS and NVIDIA aim to infuse products like NVIDIA NeMo and services like Amazon Titan with lessons learned from their collaboration for the benefit of customers.
Editor’s note: This post is part of our weekly In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows.
This week’s talented In the NVIDIA Studio artist, Nourhan Ismail, created a literal NVIDIA studio.
Her piece, called Creator by Day, Gamer by Night, was crafted with the isometric art style and impressive graphical fidelity Ismail’s known for, rich with vibrant colors and playful details. It also captures her “work hard, play hard” mentality as a 3D artist, interior designer and game level designer.
The same art style is featured in the NVIDIA Studio Sessions YouTube miniseries led by Ismail, which provides step-by-step tutorials on how to create a low-poly bedroom, from inception to final render.
Facial Animations Made Easier
Reallusion is the maker of Reallusion iClone, real-time 3D animation software built to produce professional animations for films and video games.
To expedite character animation workflows, the company recently launched its AccuFACE plug-in, which accurately captures facial expressions from webcams and conventional video files, without the need for expensive, specialized equipment.
The NVIDIA Maxine software development platform, the foundational technology behind the revolutionary NVIDIA Broadcast app, powers this incredible capability by weighing output and analyzing facial expressions and blendshapes to predict facial mesh animations.
From there, the AccuFACE plug-in converts this data into facial mesh assets for creators to apply seamlessly. It also fine-tunes lip and tongue articulation using proprietary AccuLIPS technology.
Download the plug-in today, available to creators with NVIDIA RTX GPUs.
Turning Pain Into Beauty
Ismail’s creative journey began at age four as a form of escape from the armed conflict occurring in Syria, her homeland. During that time, Ismail’s family faced many difficulties, including the loss of their home.
In the aftermath, she looked to her father, an accomplished artist and fashion designer, as a source of inspiration.
“His encouragement propelled me to showcase the pinnacle of my abilities, reminding me that art has the power to transform pain into beauty,” she said.
That encouragement has guided and fueled Ismail’s creative journey, eventually giving rise to her signature, single-room isometric style, an homage to the power of resilience and finding beauty in adversity.
Warm and homey.
“Starting with a single room, I delve into interior design, crafting spaces that reflect the comfort and joy I yearned for during challenging times,” she said. “To me, overcoming adversity proves that even from the harshest circumstances, beauty can emerge.”
Ismail started as a self-taught 3D artist, driven by a passion to learn the intricacies of creating digital masterpieces.
“Posting my works became a personal gauge of improvement — not for validation, but as a record of my learning curve,” she said.
Beautifully conceived, masterfully executed.
Each of Ismail’s pieces is a testament to her evolving skills, dedication and love for sharing her craft, especially with her father.
In fact, she dedicated her first isometric house to her father. “That was the happiest moment, to create something inspiring and make someone happy,” she said.
Isometric Art
Ismail first collects reference material on Adobe Behance to gain inspiration on ways to mix different art styles.
She then opens Blender and starts sketching in 3D. Blender Cycles’ RTX-accelerated OptiX ray tracing, powered by her GeForce RTX 3080 Ti GPU, ensured smooth viewport movement.
A gorgeous work in progress.
While the models are still fairly rudimentary, Ismail calculates the angles that light should be coming in from.
“Lighting is an emotional element,” she said. “The lighting of each piece evokes different emotions and a certain idiosyncratic introspectiveness, making the experience unique to each person.”
The NVIDIA Studio has NVIDIA Canvas!
Her trick is to regularly switch between rich, colorful scenes and plain color models to measure the emotional weight and visual impact. She either creates the custom textures herself or downloads premade ones online when on a time crunch.
Ismail’s incredible detail on full display.
Then, she plays with camera angles to analyze depth shadows and lighting, setting up animations and sequence shots in Blender. There, Blender Cycles’ RTX-accelerated OptiX ray tracing delivered seamless viewport movement.
Final touch-ups are done in post-production in Adobe After Effects. Over 30 GPU-accelerated effects sped the process, allowed Ismail to complete the project with time to spare.
“Creator by Day, Gamer by Night” in dark mode.
“There will always be hard times, so never give up and keep believing in yourself,” Ismail encourages content creators.
Digital 3D artist Nourhan Ismail.
Check out Ismail’s Instagram for more spectacular isometric art.
THANK YOU FOR 500 RTX GAMES & APPS 
Comment 
NVIDIA GeForce NOW (@NVIDIAGFN) 
