Global leader in convenient foods and beverages PepsiCo is deploying advanced machine vision technology from startup KoiReader Technologies, powered by the NVIDIA AI platform and GPUs, to improve efficiency and accuracy in its distribution process.
PepsiCo has identified KoiReader’s technology as a solution to enable greater efficiency in reading warehouse labels. This AI-powered innovation helps read warehouse labels and barcodes in fast-moving environments where the labels can be in any size, at any angle or even partially occluded or damaged.
This is up and running in a PepsiCo distribution center in the Dallas-Fort Worth area, with plans for broader deployment this year.
“If you find the right lever, you could dramatically improve our throughput,” said Greg Bellon, senior director of digital supply chain at PepsiCo.
KoiReader’s AI-powered innovation helps read warehouse labels and barcodes in fast-moving environments.
KoiReader’s technology is being used to train and run the deep learning algorithms that power PepsiCo’s AI label and barcode scanning system.
Once near-perfect accuracy was achieved, its application is being expanded to validate customer deliveries to ensure 100% accuracy of human-assisted picking operations.
At the Dallas facility where PepsiCo is testing the technology, Koi’s AutonomousOCR technology scans some of the most complex warehouse labels quickly and accurately on fast-moving conveyor belts.
It also is being investigated to assist warehouse workers as they scan pallets of soda and snacks. The same AutonomousOCR technology has also been deployed to automate yard operations as tractors and trailers enter and exit PepsiCo’s distribution center in Texas.
“KoiReader’s capability offers up the potential for many use cases — starting small and demonstrating capability is key to success,” Bellon says.
The system is already generating valuable real-time insights, Bellon reports.
Koi’s technology can accurately track regular or irregularly shaped products, with and without labels, as well as count how long it takes workers to pack boxes, how many items they are packing, and how long it takes them to retrieve items for boxes.
It acts as a real-time industrial engineering study answering many questions about the influence of people, process and technology on throughput.
A broad array of the NVIDIA stack is being used by KoiReader across its diverse solutions portfolio and customer workflows.
And every aspect of Koi’s applications are built cloud-native, using containerization, Kubernetes and microservices.
Additionally, the NVIDIA AI Enterprise software suite promises to help PepsiCo confidently scale up and manage its applications and AI deployments.
“The KoiVision Platform was built to deliver logistics, supply chain, and industrial automation for enterprise customers. Our solution suite is helping PepsiCo improve operational efficiency and accuracy in its distribution process,” said Ashutosh Prasad, founder and CEO of KoiReader.
“We’re testing out object- and activity-detection capabilities and computer vision today to figure out what kind of data we want to collect with that sort of application,” Bellon said.
Bellon said he’s excited for what’s next. “We’re going to be on a journey together,” he said.
It all started with two software engineers and a tomato farmer on a West Coast road trip.
Visiting farms to survey their needs, the three hatched a plan at an apple orchard: build a highly adaptable 3D vision AI system for automating field tasks.
Verdant, based in the San Francisco Bay Area, is developing AI that promises versatile farm assistance in the form of a tractor implement for weeding, fertilizing and spraying.
Founders Lawrence Ibarria, Gabe Sibley and Curtis Garner — two engineers from Cruise Automation and a tomato farming manager — are harnessing the NVIDIA Jetson edge AI platform and NVIDIA Metropolis SDKs such as TAO Toolkit and DeepStream for this ambitious slice of farm automation.
The startup, founded in 2018, is commercially deployed in carrot farms and in trials at apple, garlic, broccoli and lettuce farms in California’s Central Valley and Imperial Valley, as well as in Oregon.
Verdant plans to help with organic farming by lowering production costs for farmers while increasing yields and providing labor support. It employs the tractor operator, who is trained to manage the AI-driven implements. The company’s robot-as-service model, or RaaS, enables farmers to see metrics on yield improvements and reductions in chemical costs, and pay by the acre for results.
“We wanted to do something meaningful to help the environment,” said Ibarria, Verdant’s chief operating officer. “And it’s not only reducing costs for farmers, it’s also increasing their yield.”
The company recently landed more than $46 million in series A funding.
Another recent event at Verdant was hiring as its chief technology officer Frank Dellaert, who is recognized for using graphical models to solve large-scale mapping and 4D reconstruction challenges. A faculty member at Georgia Institute of Technology, Dellaert has led work at Skydio, Facebook Reality Labs and Google AI while on leave from the research university.
“One of the things that was impressed upon me when joining Verdant was how they measure performance in real-time,” remarked Dellaert. “It’s a promise to the grower, but it’s also a promise to the environment. It shows whether we do indeed save from all the chemicals being put into the field.”
Verdant is a member of NVIDIA Inception, a free program that provides startups with technical training, go-to-market support, and AI platform guidance.
Verdant is working with Bolthouse Farms, based in Bakersfield, Calif., to help its carrot-growing business transition to regenerative agriculture practices. The aim is to utilize more sustainable farming practices, including reduction of herbicides.
Verdant is starting with weeding and expanding next into precision fertilizer applications for Bolthouse.
The computation and automation from Verdant have enabled Bolthouse Farms to understand how to achieve its sustainable farming goals, according to the farm’s management team.
Riding With Jetson AGX Orin
Verdant is putting the Jetson AGX Orin system-on-module inside tractor cabs. The company says that Orin’s powerful computing and availability with ruggedized cases from vendors makes it the only choice for farming applications. Verdant is also collaborating with Jetson ecosystem partners, including RidgeRun, Leopard Imaging and others.
The module enables Verdant to create 3D visualizations showing plant treatments for the tractor operator. The company uses two stereo cameras for its field visualizations, for inference and to gather data in the field for training models on NVIDIA DGX systems running NVIDIA A100 Tensor Core GPUs back at its headquarters. DGX performance allows Verdant to use larger training datasets to get better model accuracy in inference.
“We display a model of the tractor and a 3D view of every single carrot and every single weed and the actions we are doing, so it helps customers see what the robot’s seeing and doing,” said Ibarria, noting this can all run on a single AGX Orin module, delivering inference at 29 frames per second in real time.
DeepStream-Powered Apple Vision
Verdant relies on NVIDIA DeepStream as the framework for running its core machine learning to help power its detection and segmentation. It also uses custom CUDA kernels to do a number of tracking and positioning elements of its work.
Verdant’s founder and CEO, Sibley, whose post-doctorate research was in simultaneous localization and mapping has brought this expertise to agriculture. This comes in handy to help present a logical representation of the farm, said Ibarria. “We can see things, and know when and where we’ve seen them,” he said.
This is important for apples, he said. They can be challenging to treat, as apples and branches often overlap, making it difficult to find the best path to spray them. The 3D visualizations made possible by AGX Orin allow a better understanding of the occlusion and the right path for spraying.
“With apples, when you see a blossom, you can’t just spray it when you see it, you need to wait 48 hours,” said Ibarria. “We do that by building a map, relocalizing ourselves saying, ‘That’s the blossom, I saw it two days ago, and so it’s time to spray.’”
NVIDIA TAO for 5x Model Production
Verdant relies on NVIDIA TAO Toolkit for its model building pipeline. The transfer learning capability in TAO Toolkit enables it to take off-the-shelf models and quickly refine them with images taken in the field. For example, this has made it possible to change from detecting carrots to detecting onions, in just a day. Previously, it took roughly five days to build models from scratch that achieved an acceptable accuracy level.
“One of our goals here is to leverage technologies like TAO and transfer learning to very quickly begin to operate in new circumstances,” said Dellaert.
While cutting model building production time by 5x, the company has also been able to hit 95% precision with its vision systems using these methods.
“Transfer learning is a big weapon in our armory,” he said.
The mics were live and tape was rolling in the studio where the Miles Davis Quintet was recording dozens of tunes in 1956 for Prestige Records.
When an engineer asked for the next song’s title, Davis shot back, “I’ll play it, and tell you what it is later.”
Like the prolific jazz trumpeter and composer, researchers have been generating AI models at a feverish pace, exploring new architectures and use cases. Focused on plowing new ground, they sometimes leave to others the job of categorizing their work.
A team of more than a hundred Stanford researchers collaborated to do just that in a 214-page paper released in the summer of 2021.
In a 2021 paper, researchers reported that foundation models are finding a wide array of uses.
They said transformer models, large language models (LLMs) and other neural networks still being built are part of an important new category they dubbed foundation models.
Foundation Models Defined
A foundation model is an AI neural network — trained on mountains of raw data, generally with unsupervised learning — that can be adapted to accomplish a broad range of tasks, the paper said.
“The sheer scale and scope of foundation models from the last few years have stretched our imagination of what’s possible,” they wrote.
Two important concepts help define this umbrella category: Data gathering is easier, and opportunities are as wide as the horizon.
No Labels, Lots of Opportunity
Foundation models generally learn from unlabeled datasets, saving the time and expense of manually describing each item in massive collections.
Earlier neural networks were narrowly tuned for specific tasks. With a little fine-tuning, foundation models can handle jobs from translating text to analyzing medical images.
Foundation models are demonstrating “impressive behavior,” and they’re being deployed at scale, the group said on the website of its research center formed to study them. So far, they’ve posted more than 50 papers on foundation models from in-house researchers alone.
“I think we’ve uncovered a very small fraction of the capabilities of existing foundation models, let alone future ones,” said Percy Liang, the center’s director, in the opening talk of the first workshop on foundation models.
AI’s Emergence and Homogenization
In that talk, Liang coined two terms to describe foundation models:
Emergence refers to AI features still being discovered, such as the many nascent skills in foundation models. He calls the blending of AI algorithms and model architectures homogenization, a trend that helped form foundation models. (See chart below.)
The field continues to move fast.
A year after the group defined foundation models, other tech watchers coined a related term — generative AI. It’s an umbrella term for transformers, large language models, diffusion models and other neural networks capturing people’s imaginations because they can create text, images, music, software and more.
Generative AI has the potential to yield trillions of dollars of economic value, said executives from the venture firm Sequoia Capital who shared their views in a recent AI Podcast.
A Brief History of Foundation Models
“We are in a time where simple methods like neural networks are giving us an explosion of new capabilities,” said Ashish Vaswani, an entrepreneur and former senior staff research scientist at Google Brain who led work on the seminal 2017 paper on transformers.
That work inspired researchers who created BERT and other large language models, making 2018 “a watershed moment” for natural language processing, a report on AI said at the end of that year.
Google released BERT as open-source software, spawning a family of follow-ons and setting off a race to build ever larger, more powerful LLMs. Then it applied the technology to its search engine so users could ask questions in simple sentences.
In 2020, researchers at OpenAI announced another landmark transformer, GPT-3. Within weeks, people were using it to create poems, programs, songs, websites and more.
“Language models have a wide range of beneficial applications for society,” the researchers wrote.
Their work also showed how large and compute-intensive these models can be. GPT-3 was trained on a dataset with nearly a trillion words, and it sports a whopping 175 billion parameters, a key measure of the power and complexity of neural networks.
The growth in compute demands for foundation models. (Source: GPT-3 paper)
“I just remember being kind of blown away by the things that it could do,” said Liang, speaking of GPT-3 in a podcast.
The latest iteration, ChatGPT — trained on 10,000 NVIDIA GPUs — is even more engaging, attracting over 100 million users in just two months. Its release has been called the iPhone moment for AI because it helped so many people see how they could use the technology.
One timeline describes the path from early AI research to ChatGPT. (Source: blog.bytebytego.com)
From Text to Images
About the same time ChatGPT debuted, another class of neural networks, called diffusion models, made a splash. Their ability to turn text descriptions into artistic images attracted casual users to create amazing images that went viral on social media.
The first paper to describe a diffusion model arrived with little fanfare in 2015. But like transformers, the new technique soon caught fire.
Researchers posted more than 200 papers on diffusion models last year, according to a list maintained by James Thornton, an AI researcher at the University of Oxford.
In a tweet, Midjourney CEO David Holz revealed that his diffusion-based, text-to-image service has more than 4.4 million users. Serving them requires more than 10,000 NVIDIA GPUs mainly for AI inference, he said in an interview (subscription required).
Dozens of Models in Use
Hundreds of foundation models are now available. One paper catalogs and classifies more than 50 major transformer models alone (see chart below).
The Stanford group benchmarked 30 foundation models, noting the field is moving so fast they did not review some new and prominent ones.
Startup NLP Cloud, a member of the NVIDIA Inception program that nurtures cutting-edge startups, says it uses about 25 large language models in a commercial offering that serves airlines, pharmacies and other users. Experts expect that a growing share of the models will be made open source on sites like Hugging Face’s model hub.
Experts note a rising trend toward releasing foundation models as open source.
Foundation models keep getting larger and more complex, too.
That’s why — rather than building new models from scratch — many businesses are already customizing pretrained foundation models to turbocharge their journeys into AI.
Foundations in the Cloud
One venture capital firm lists 33 use cases for generative AI, from ad generation to semantic search.
Major cloud services have been using foundation models for some time. For example, Microsoft Azure worked with NVIDIA to implement a transformer for its Translator service. It helped disaster workers understand Haitian Creole while they were responding to a 7.0 earthquake.
In February, Microsoft announced plans to enhance its browser and search engine with ChatGPT and related innovations. “We think of these tools as an AI copilot for the web,” the announcement said.
Google announced Bard, an experimental conversational AI service. It plans to plug many of its products into the power of its foundation models like LaMDA, PaLM, Imagen and MusicLM.
“AI is the most profound technology we are working on today,” the company’s blog wrote.
Startups Get Traction, Too
Startup Jasper expects to log $75 million in annual revenue from products that write copy for companies like VMware. It’s leading a field of more than a dozen companies that generate text, including Writer, an NVIDIA Inception member.
Other Inception members in the field include Tokyo-based rinna that’s created chatbots used by millions in Japan. In Tel Aviv, Tabnine runs a generative AI service that’s automated up to 30% of the code written by a million developers globally.
A Platform for Healthcare
Researchers at startup Evozyne used foundation models in NVIDIA BioNeMo to generate two new proteins. One could treat a rare disease and another could help capture carbon in the atmosphere.
Evozyne and NVIDIA described a hybrid foundation model for creating proteins in a joint paper.
BioNeMo, a software platform and cloud service for generative AI in drug discovery, offers tools to train, run inference and deploy custom biomolecular AI models. It includes MegaMolBART, a generative AI model for chemistry developed by NVIDIA and AstraZeneca.
“Just as AI language models can learn the relationships between words in a sentence, our aim is that neural networks trained on molecular structure data will be able to learn the relationships between atoms in real-world molecules,” said Ola Engkvist, head of molecular AI, discovery sciences and R&D at AstraZeneca, when the work was announced.
Separately, the University of Florida’s academic health center collaborated with NVIDIA researchers to create GatorTron. The large language model aims to extract insights from massive volumes of clinical data to accelerate medical research.
A Stanford center is applying the latest diffusion models to advance medical imaging. NVIDIA also helps healthcare companies and hospitals use AI in medical imaging, speeding diagnosis of deadly diseases.
AI Foundations for Business
Another new framework, NVIDIA NeMo Megatron, aims to let any business create its own billion- or trillion-parameter transformers to power custom chatbots, personal assistants and other AI applications.
It created the 530-parameter Megatron-Turing Natural Language Generation model (MT-NLG) that powers TJ, the Toy Jensen avatar that gave part of the keynote at NVIDIA GTC last year.
Foundation models — connected to 3D platforms like NVIDIA Omniverse — will be key to simplifying development of the metaverse, the 3D evolution of the internet. These models will power applications and assets for entertainment and industrial users.
Factories and warehouses are already applying foundation models inside digital twins, realistic simulations that help find more efficient ways to work.
Foundation models can ease the job of training autonomous vehicles and robots that assist humans on factory floors and logistics centers like the one described below.
New uses for foundation models are emerging daily, as are challenges in applying them.
Several papers on foundation and generative AI models describing risks such as:
amplifying bias implicit in the massive datasets used to train models,
introducing inaccurate or misleading information in images or videos, and
violating intellectual property rights of existing works.
“Given that future AI systems will likely rely heavily on foundation models, it is imperative that we, as a community, come together to develop more rigorous principles for foundation models and guidance for their responsible development and deployment,” said the Stanford paper on foundation models.
Current ideas for safeguards include filtering prompts and their outputs, recalibrating models on the fly and scrubbing massive datasets.
“These are issues we’re working on as a research community,” said Bryan Catanzaro, vice president of applied deep learning research at NVIDIA. “For these models to be truly widely deployed, we have to invest a lot in safety.”
It’s one more field AI researchers and developers are plowing as they create the future.
It’s a thrilling GFN Thursday with GRID Legends racing to the cloud this week. It leads a total of eight new games expanding the GeForce NOW library. New content for Rainbow Six Siege is also now streaming.
Plus, two new cities are now online with GeForce RTX 4080 performance for cloud gaming. Chicago and Montreal have completed upgrades to RTX 4080 SuperPODs, delivering next-generation cloud streaming to GeForce NOW Ultimate members.
Shifting Up
Beyond-fast gaming meets beyond-fast racing.
Jump into the spectacular action of GRID Legends, the racing game from EA with drama at every turn. Battle for glory with a variety of incredible cars on stellar tracks featuring iconic landmarks from the streets of London and Moscow.
Navigate the world of high-stakes racing as a rookie behind the wheel, with a documentary team capturing every sensational moment. Conquer hundreds of events and challenges, and create a dream racing team in the unique, cinematic story mode. Join up online with other legends and race against friends, or challenge them to a race designed in the Race Creator mode.
GeForce NOW members can experience it all with high dynamic range on PC, Mac and SHIELD TV for a smooth, ultra-crisp driving experience, even under intense racing conditions.
Upgrade Roll Call
Order up! Servers are now live in Chicago and Montreal.
The RTX 4080 SuperPODs have been rolling out around the world for a couple months now, so it’s time to check in with a RTX 4080 roll call.
Chicago and Montreal bring the number of cities on the server update map to 10, joining Ashburn, Dallas, Los Angeles and San Jose in the U.S., and Amsterdam, Frankfurt, London and Paris in Europe. Now past it’s halfway point, the rollout is expected to be completed by mid-year.
Here’s a few reasons to upgrade:
NVIDIA DLSS 3 technology is enabled for AI-powered performance boosts on supported games like HITMAN World of Assassination and Marvel’s Midnight Suns. This means 4K streaming from the cloud results in the smoothest game play at up to 120 frames per second, even when settings are cranked to the max.
NVIDIA Reflex delivers ultra-low latency. Paired with DLSS 3, the technology enables Ultimate members to stream games like Rainbow Six Siege and Apex Legends at up to 240 fps on PCs and Macs, with as low as 35 milliseconds of total latency for a streaming experience that feels nearly indistinguishable from being on a local desktop.
Ultrawide resolutions are supported for the first time ever from the cloud, giving Ultimate members the most immersive game play in No Man’s Sky, Cyberpunk 2077 and Assassin’s Creed Valhalla.
Ultimate members in and around the 10 cities on the map are streaming with new performance today, and can take full advantage of these RTX technologies in the cloud. Level up to next-generation cloud streaming today for beyond-fast gaming.
Spring Forward With New Games
“Brava will do what’s needed. Even when it requires sacrifice.”
Operation Commanding Force is the newest season of Year 8 for Tom Clancy’s Rainbow Six Siege, now available for members to stream. The update brings a new attacker named Brava, equipped with the Kludge Drone, a gadget that can disrupt enemy electronics devices and even take over some of them to turn the tides of battle.
That’s on top of the eight games joining the cloud this week:
There’s a whole lot of games streaming from the cloud, and we want to know your top three. Let us know in the comments below or on Twitter and Facebook.
The Academy Award nominations are in — and for the 15th year in a row, NVIDIA technologies worked behind the scenes of every film nominated for Best Visual Effects.
The five VFX contenders for the 95th annual Academy Awards, taking place on Sunday, March 12, include:
All Quiet on the Western Front
Avatar: The Way of Water
The Batman
Black Panther: Wakanda Forever
Top Gun: Maverick
For over a decade, filmmakers and VFX studios around the world have used NVIDIA technologies to power the most advanced, visually rich movies ever made. Today, creators and artists are transforming VFX using advanced capabilities in graphics, like real-time ray tracing, simulation, AI and virtual production — all powered by NVIDIA RTX technologies.
Diving Into Natural Wonders With Cutting-Edge Graphics
Award-winning studio Wētā FX created the stunning visuals for director James Cameron’s much-anticipated sequel, Avatar: The Way of Water. The film is one of Wētā’s largest VFX projects to date. The team created 3,240 shots — which is 98% of the total shots in the film, more than two-thirds of which featured water.
In computer graphics (CG), making water look natural and realistic — from how it moves off a character’s skin to how it drips from clothing — is one of the biggest challenges for visual effects artists. But for this film, Wētā developed and implemented a new water toolset that advanced their capabilities across simulation, rendering and more.
The team started with pre-production and performance capture using a real-time, GPU-based ocean spectrum deformer, which served as a consistent, physically based starting point for water on set. From there, Wētā created a new suite of water solvers — many of them within Loki, the studio’s in-house multiphysics simulation framework. Loki allows coupling of multiple solvers in any configuration. For example, hair, cloth, air and water can all be simulated together.
Other key innovations from Wētā centered on both dry and wet performance capture, new deep learning models to process stereo camera images and generate depth maps for compositing, and neural networks to assist with facial animation and muscle systems.
Creating Captivating Car Chases Through Gritty Gotham
Wētā FX was also behind the cinematic visuals for The Batman. The team, led by VFX supervisor Anders Langlands, worked on the gripping highway chase between Batman and the infamous villain, the Penguin. As they race through the city of Gotham under heavy rainfall, the Penguin sets off a sequence of car crashes and explosions.
To create a feeling of danger and exhilaration, the team put the car chase scene together through heavily enhanced live action and completely CG shots. Rendering the proper lighting; simulating realistic raindrops colliding with multiple surfaces, hydroplaning and wheel spray; and illuminating rain through headlights and streetlights all added to the complexity of these shots. Wētā also worked on background environments for scenes in the Batcave and Gotham’s City Hall.
Taking CGI to the Sky
The practical effects and cinematography behind Top Gun: Maverick was an instant highlight of this heart-pounding Hollywood blockbuster film. But to add more layers of realism to those outstanding aerial shots, VFX Supervisor Ryan Tudhope and the team at Method Studios partnered with the camera department, aerial coordinators and the United States Navy to film extensive air-to-air and ground-to-air footage of real jets. They captured over 800 hours of aerial stunts, mounts and plates to provide their team with a practical foundation for the visual effects work.
The Top Gun: Maverick team implemented various VFX techniques, creating a surprising 2,400 VFX shots for the movie. The visual effects included creating and adding CG planes in scenes, as well as adding missiles, smoke and explosions in various action sequences. The invisible nature of the visual effects in Top Gun: Maverick make it a top contender for the Academy Award for Best Visual Effects.
A New Swimlane for Underwater Worlds
In Black Panther: Wakanda Forever, Wētā FX further demonstrated its leadership in creating photorealistic underwater sequences. Chris White, visual effects supervisor for the film, was tasked with creating the Mesoamerican-inspired Talokan underwater kingdom.
To get a realistic look for the characters in this undersea world, Wētā used a combination of live-action sequences shot in water tanks and dry-for-wet shots that helped capture realistic underwater motion for the characters, clothes and hair.
Wētā also reflected how various skin tones would react to light with the added complexity of a murky underwater environment. The bar for realistic water simulation has once again been raised by Wētā FX in Blank Panther: Wakanda Forever.
All Action on the VFX Front
Movie magic is made when visual effects are so seamless that the audience remains completely immersed in the story, not realizing that what they’re seeing is an effect. This is how VFX supervisor Markus Frank and production company Cine Chromatix earned their Best Visual Effects nomination for All Quiet on the Western Front.
To authentically tell the story of two young soldiers during World War I, Cine Chromatix and the film’s visual effects teams focused on the fine details needed to craft VFX that are hidden in plain sight.
The result is stunning. Even after watching Cine Chromatix’s VFX breakdown reel for the film, viewers may find themselves scrubbing back and forth to decipher fact from fiction.
See How Oscar-Nominated VFX Are Created at GTC
NVIDIA congratulates all of this year’s nominees for the Academy Award for Best Visual Effects.
Learn more about visual effects, AI, virtual production and animation at NVIDIA GTC, a global technology conference taking place online March 20-23. Register for free and hear from industry luminaries creating stunning visuals in film and TV. Check out all the media and entertainment sessions at GTC.
Editor’s note: This post is part of our weekly In the NVIDIA Studio series, which celebrates featured artists, offers creative tips and tricks, and demonstrates how NVIDIA Studio technology improves creative workflows.
An adrenaline-fueled virtual ride in the sky is sure to satisfy all thrill seekers — courtesy of 3D artist Kosei Wano’s sensational animation, Moon Hawk. Wano outlines his creative workflow this week In the NVIDIA Studio.
Plus, join the #GameArtChallenge — running through Sunday, April 30 — by using the hashtag to share video game fan art, character creations and more for a chance to be featured across NVIDIA social media channels.
Original game content can be made with NVIDIA Omniverse — a platform for creating and operating metaverse applications — using the Omniverse Machinima app. This enables users to collaborate in real time when animating characters and environments in virtual worlds.
Who Dares, Wins
Wano often finds inspiration exploring the diversity of flora and fauna. He has a penchant for examining birds — and even knows the difference in wing shapes between hawks and martins, he said. This interest in flying entities extends to his fascination with aircrafts. For Moon Hawk, Wano took on the challenge of visually evolving a traditional, fuel-based fighter jet into an electric one.
With reference material in hand, Wano opened the 3D app Blender to scale the fighter jet to accurate, real-life sizing, then roughly sketched within the 3D design space, his preferred method to formulate models.
“Moon Hawk” in its traditional form.
The artist then deployed several tips and tricks to model more efficiently: adding Blender’s automatic detailing modifier, applying neuro-reflex modeling to change the aircraft’s proportions, then dividing the model’s major 3D shapes into sections to edit individually — a step Wano calls “dividing each difficulty.”
Neuro-reflex modeling enables Wano to change proportions while maintaining model integrity.
Blender Cycles RTX-accelerated OptiX ray tracing, unlocked by the artist’s GeForce RTX 3080 Ti GPU, enabled interactive, photorealistic modeling in the viewport. “Optix’s AI-powered denoiser renders lightly, allowing for comfortable trial and error,” said Wano, who then applied sculpting and other details. Next, Wano used geo nodes to add organic style and customization to his Blender scenes and animate his fighter jet.
Applying geo nodes.
Blender geo nodes make modeling an almost completely procedural process — allowing for non-linear, non-destructive workflows and the instancing of objects — to create incredibly detailed scenes using small amounts of data.
The “Moon Hawk” model is nearly complete.
For Moon Hawk, Wano applied geo nodes to mix materials not found in nature, creating unique textures for the fighter jet. Being able to make real-time base mesh edits without the concern of destructive workflows gave Wano the freedom to alter his model on the fly with an assist from his GPU. “With the GeForce RTX 3080 Ti, there’s no problem, even with a model as complicated as this,” he said.
Animations accelerated at the speed of light with Wano’s GeForce RTX GPU.
Wano kicked off the animation phase by selecting the speed of the fighter jet and roughly designing its flight pattern.
Mapping the flight path in advance.
The artist referenced popular fighter jet scenes in cinema and video games, as well as studied basic rules of physics, such as inertia, to ensure the flight patterns in his animation were realistic. Then, Wano returned to using geo nodes to add 3D lighting effects without the need to simulate or bake. Such lighting modifications helped to make rendering the project simpler in its final stage.
Parameters were edited with ease, in addition to applying particle simulations and manually shaking the camera to add more layers of immersion to the scenes.
Final color edits in Blender.
With the animation complete, Wano added short motion blur. Accelerated motion blur rendering enabled by his RTX GPU and the NanoVBD toolset for easy rendering of volumes let him apply this effect quickly. And RTX-accelerated OptiX ray tracing in Blender Cycles delivered the fastest final frame renders.
Wano imported final files into Blackmagic Design’s DaVinci Resolve application, where GPU-accelerated color grading, video editing and color scopes helped the artist complete the animation in record time.
3D artist Kosei Wano.
Choosing GeForce RTX was a simple choice for Wano, who said, “NVIDIA products have been trusted by many people for a long time.”
View more of Wano’s impressive portfolio on ArtStation.
Who Dares With Photogrammetry, Wins Again
Wano, like most artists, is always growing his craft, refining essential skills and learning new techniques, including photogrammetry — the art and science of extracting 3D information from photographs.
In the NVIDIA Studio artist Anna Natter recently highlighted her passion for photogrammetry, noting that virtually anything can be preserved in 3D and showcasing features that have the potential to save 3D artists countless hours. Wano saw this same potential when experimenting with the technology in Adobe Substance 3D Sampler.
“Photogrammetry can accurately reproduce the complex real world,” said Wano, who would encourage other artists to think big in terms of both individual objects and environments. “You can design an entire realistic space by placing it in a 3D virtual world.”
Try out photogrammetry and post your creations with the #StudioShare hashtag for a chance to be featured across NVIDIA Studio’s social media channels.
Preparing a retailer’s online catalog once required expensive physical photoshoots to capture products from every angle. A Tel Aviv startup is saving brands time and money by transforming these camera clicks into mouse clicks.
Hexa uses GPU-accelerated computing to help companies turn their online inventory into 3D renders that shoppers can view in 360 degrees, animate or even try on virtually to help their buying decisions. The company, which recently announced a $20.5 million funding round, is working with brands in fashion, furniture, consumer electronics and more.
“The world is going 3D,” said Yehiel Atias, CEO of Hexa. “Just a few years ago, the digital infrastructure to do this was still so expensive that it was more affordable to arrange a photographer, models and lighting. But with the advancements of AI and NVIDIA GPUs, it’s now feasible for retailers to use synthetic data to replace physical photoshoots.”
Hexa’s 3D renders are used on major retail websites such as Amazon, Crate & Barrel and Macy’s. The company creates thousands of renders each month, reducing the need for physical photoshoots of every product in a retailer’s catalog. Hexa estimates that it can save customers up to 300 pounds of carbon emissions for each product imaged digitally instead of physically.
From Physical Photoshoots to AI-Accelerated Renders
Hexa can reconstruct a single 2D image, or a set of low-quality 2D images, into a high-fidelity 3D asset. The company uses differing levels of automation for its renders depending on the complexity of the shape, the amount of visual data that needs to be reconstructed, and the similarity of the object to Hexa’s existing dataset.
To automate elements of its workflow, the team uses dozens of AI algorithms that were developed using the PyTorch deep learning framework and run on NVIDIA Tensor Core GPUs in the cloud. If one of Hexa’s artists is reconstructing a 3D toaster, for example, one algorithm can identify similar geometries the team has created in the past to give the creator a head start.
Another neural network can scan a retailer’s website to identify how many of its products Hexa can support with 3D renders. The company’s entire rendering pipeline, too, runs on NVIDIA GPUs available through Amazon Web Services.
“Accessing compute resources through AWS gives us the option to use thousands of NVIDIA GPUs at a moment’s notice,” said Segev Nahari, lead technical artist at Hexa. “If I need 10,000 frames to be ready by a certain time, I can request the hardware I need to meet the deadline.”
Nahari estimates that rendering on NVIDIA GPUs is up to 3x faster than relying on CPUs.
Broadening Beyond Retail, Venturing Into Omniverse
Hexa developers are continually experimenting with new methods for 3D rendering — looking for workflow improvements in preprocessing, object reconstruction and post-processing. The team recently began working with NVIDIA GET3D, a generative AI model by NVIDIA Research that generates high-fidelity, three-dimensional shapes based on a training dataset of 2D images.
By training GET3D on Hexa’s dataset of shoes, the team was able to generate 3D models of novel shoes not part of the training data.
In addition to its work in ecommerce, Hexa’s research and development team is investigating new applications for the company’s AI software.
“It doesn’t stop at retail,” Atias said. “Industries from gaming to fashion and healthcare are finding out that synthetic data and 3D technology is a more efficient way to do things like digitize inventory, create digital twins and train robots.”
The team credits its membership in NVIDIA Inception, a global program that supports cutting-edge startups, as a “huge advantage” in leveling up the technology Hexa uses.
“Being part of Inception opens doors that outsiders don’t have,” Atias said. “For a small company trying to navigate the massive range of NVIDIA hardware and software offerings, it’s a door-opener to all the cool tools we wanted to experiment with and understand the potential they could bring to Hexa.”
Hexa is testing the NVIDIA Omniverse Enterprise platform — an end-to-end platform for building and operating metaverse applications — as a tool to unify its annotating and rendering workflows, which are used by dozens of 3D artists around the globe. Omniverse Enterprise enables geographically dispersed teams of creators to customize their rendering pipelines and collaborate to build 3D assets.
“Each of our 3D artists has a different software workflow that they’re used to — so it can be tough to get a unified output while still being flexible about the tools each artist uses,” said Jonathan Clark, Hexa’s CTO. “Omniverse is an ideal candidate in that respect, with huge potential for Hexa. The platform will allow our artists to use the rendering software they’re comfortable with, while also allowing our team to visualize the final product in one place.”
Accelerated computing — a capability once confined to high-performance computers in government research labs — has gone mainstream.
Banks, car makers, factories, hospitals, retailers and others are adopting AI supercomputers to tackle the growing mountains of data they need to process and understand.
These powerful, efficient systems are superhighways of computing. They carry data and calculations over parallel paths on a lightning journey to actionable results.
GPU and CPU processors are the resources along the way, and their onramps are fast interconnects. The gold standard in interconnects for accelerated computing is NVLink.
So, What Is NVLink?
NVLink is a high-speed connection for GPUs and CPUs formed by a robust software protocol, typically riding on multiple pairs of wires printed on a computer board. It lets processors send and receive data from shared pools of memory at lightning speed.
Now in its fourth generation, NVLink connects host and accelerated processors at rates up to 900 gigabytes per second (GB/s).
That’s more than 7x the bandwidth of PCIe Gen 5, the interconnect used in conventional x86 servers. And NVLink sports 5x the energy efficiency of PCIe Gen 5, thanks to data transfers that consume just 1.3 picojoules per bit.
The History of NVLink
First introduced as a GPU interconnect with the NVIDIA P100 GPU, NVLink has advanced in lockstep with each new NVIDIA GPU architecture.
In 2018, NVLink hit the spotlight in high performance computing when it debuted connecting GPUs and CPUs in two of the world’s most powerful supercomputers, Summit and Sierra.
The systems, installed at Oak Ridge and Lawrence Livermore National Laboratories, are pushing the boundaries of science in fields such as drug discovery, natural disaster prediction and more.
Bandwidth Doubles, Then Grows Again
In 2020, the third-generation NVLink doubled its max bandwidth per GPU to 600GB/s, packing a dozen interconnects in every NVIDIA A100 Tensor Core GPU.
The A100 powers AI supercomputers in enterprise data centers, cloud computing services and HPC labs across the globe.
Today, 18 fourth-generation NVLink interconnects are embedded in a single NVIDIA H100 Tensor Core GPU. And the technology has taken on a new, strategic role that will enable the most advanced CPUs and accelerators on the planet.
A Chip-to-Chip Link
NVIDIA NVLink-C2C is a version of the board-level interconnect to join two processors inside a single package, creating a superchip. For example, it connects two CPU chips to deliver 144 Arm Neoverse V2 cores in the NVIDIA Grace CPU Superchip, a processor built to deliver energy-efficient performance for cloud, enterprise and HPC users.
NVIDIA NVLink-C2C also joins a Grace CPU and a Hopper GPU to create the Grace Hopper Superchip. It packs accelerated computing for the world’s toughest HPC and AI jobs into a single chip.
Alps, an AI supercomputer planned for the Swiss National Computing Center, will be among the first to use Grace Hopper. When it comes online later this year, the high-performance system will work on big science problems in fields from astrophysics to quantum chemistry.
The Grace CPU packs 144 Arm Neoverse V2 cores across two die connected by NVLink-C2C.
Grace and Grace Hopper are also great for bringing energy efficiency to demanding cloud computing workloads.
For example, Grace Hopper is an ideal processor for recommender systems. These economic engines of the internet need fast, efficient access to lots of data to serve trillions of results to billions of users daily.
Recommenders get up to 4x more performance and greater efficiency using Grace Hopper than using Hopper with traditional CPUs.
In addition, NVLink is used in a powerful system-on-chip for automakers that includes NVIDIA Hopper, Grace and Ada Lovelace processors. NVIDIA DRIVE Thor is a car computer that unifies intelligent functions such as digital instrument cluster, infotainment, automated driving, parking and more into a single architecture.
LEGO Links of Computing
NVLink also acts like the socket stamped into a LEGO piece. It’s the basis for building supersystems to tackle the biggest HPC and AI jobs.
For example, NVLinks on all eight GPUs in an NVIDIA DGX system share fast, direct connections via NVSwitch chips. Together, they enable an NVLink network where every GPU in the server is part of a single system.
To get even more performance, DGX systems can themselves be stacked into modular units of 32 servers, creating a powerful, efficient computing cluster.
NVLink is one of the key technologies that let users easily scale modular NVIDIA DGX systems to a SuperPOD with up to an exaflop of AI performance.
Users can connect a modular block of 32 DGX systems into a single AI supercomputer using a combination of an NVLink network inside the DGX and NVIDIA Quantum-2 switched Infiniband fabric between them. For example, an NVIDIA DGX H100 SuperPOD packs 256 H100 GPUs to deliver up to an exaflop of peak AI performance.
To get even more performance, users can tap into the AI supercomputers in the cloud such as the one Microsoft Azure is building with tens of thousands of A100 and H100 GPUs. It’s a service used by groups like OpenAI to train some of the world’s largest generative AI models.
And it’s one more example of the power of accelerated computing.
March is already here and a new month always means new games, with a total of 19 joining the GeForce NOW library.
Set off on a magical journey to restore Disney magic when Disney Dreamlight Valley joins the cloud later this month. Plus, the hunt is on with Capcom’s Monster Hunter Rise now available for all members to stream, as is major new content for Battlefield 2042 and Destiny 2.
Embark on a dream adventure when Disney Dreamlight Valley from Gameloft releases in the cloud on Thursday, March 16. In this life-sim adventure game, Disney and Pixar characters live in harmony until the Forgetting threatens to destroy the wonderful memories created by its inhabitants. Help restore Disney magic to the Valley and go on an enchanting journey — full of quests, exploration and beloved Disney and Pixar friends.
Live the Disney dream life while collecting thousands of decorative items inspired by Disney and Pixar worlds to personalize gamers’ own unique homes in the Valley. The game’s latest free update, “A Festival of Friendship,” brings even more features, items and characters to interact with.
Disney fans of all ages will enjoy seeing their favorite characters, from Disney Encanto’s Mirabel to TheLion King’s Scar, throughout the game when it launches in the cloud later this month. Members can jump onto their PC, Mac and other devices to start the adventure without having to worry about download times, system requirements or storage space.
March Madness
Starting off the month is Capcom’s popular action role-playing game Monster Hunter Rise: Sunbreak, including Free Title Update 4, which brings the return of the Elder Dragon Velkhana, lord of the tundra that freezes all in its path. The game is now available for GeForce NOW members to stream, so new and returning Hunters can seamlessly bring their monster hunting careers to the cloud.
Dominate the battlefield.
New content is also available for members to stream this week for blockbuster titles. Eleventh Hour is the latest season release for Battlefield 2042, including a new map, specialist, weapon and vehicle to help players dominate the battle.
Eyes up, Guardians.
Lightfall, Destiny 2’s latest expansion following last year’s The Witch Queen, brings Guardians one step closer to the conclusion of the “Light and Darkness saga.” Experience a brand new campaign, Exotic gear and weapons, a new six-player raid, and more as players prepare for the beginning of the end.
On top of all that, here are the three new games being added this week:
While February is the shortest month, there was no shortage of games. Four extra games were added to the cloud for GeForce NOW members on top of the 25 games announced:
A few games announced didn’t make it into February due to shifts in their release dates, including Above Snakes and Heads Will Roll: Reforged. Command & Conquer Remastered Collection was removed from GeForce NOW on March 1 due to a technical issue. Additionally, PERISH and the Dark and Darker playtest didn’t make it to the cloud this month. Look for updates in a future GFN Thursday on some of these titles.
Finally, we’ve got a question to start your weekend gaming adventures. Let us know your answer in the comments below or on Twitter and Facebook.
Share your favorite video game companion and why they are the best.
Cloud and edge networks are setting up a new line of defense, called confidential computing, to protect the growing wealth of data users process in those environments.
Confidential Computing Defined
Confidential computing is a way of protecting data in use, for example while in memory or during computation, and preventing anyone from viewing or altering the work.
Using cryptographic keys linked to the processors, confidential computing creates a trusted execution environment or secure enclave. That safe digital space supports a cryptographically signed proof, called attestation, that the hardware and firmware is correctly configured to prevent the viewing or alteration of their data or application code.
In the language of security specialists, confidential computing provides assurances of data and code privacy as well as data and code integrity.
What Makes Confidential Computing Unique?
Confidential computing is a relatively new capability for protecting data in use.
For many years, computers have used encryption to protect data that’s in transit on a network and data at rest, stored in a drive or non-volatile memory chip. But with no practical way to run calculations on encrypted data, users faced a risk of having their data seen, scrambled or stolen while it was in use inside a processor or main memory.
With confidential computing, systems can now cover all three legs of the data-lifecycle stool, so data is never in the clear.
Confidential computing adds a new layer in computer security — protecting data in use while running on a processor.
In the past, computer security mainly focused on protecting data on systems users owned, like their enterprise servers. In this scenario, it’s okay that system software sees the user’s data and code.
With the advent of cloud and edge computing, users now routinely run their workloads on computers they don’t own. So confidential computing flips the focus to protecting the users’ data from whoever owns the machine.
With confidential computing, software running on the cloud or edge computer, like an operating system or hypervisor, still manages work. For example, it allocates memory to the user program, but it can never read or alter the data in memory allocated by the user.
How Confidential Computing Got Its Name
A 2015 research paper was one of several using new Security Guard Extensions (Intel SGX) in x86 CPUs to show what’s possible. It called its approach VC3, for Verifiable Confidential Cloud Computing, and the name — or at least part of it — stuck.
“We started calling it confidential cloud computing,” said Felix Schuster, lead author on the 2015 paper.
Four years later, Schuster co-founded Edgeless Systems, a company in Bochum, Germany, that develops tools so users can create their own confidential-computing apps to improve data protection.
Confidential computing is “like attaching a contract to your data that only allows certain things to be done with it,” he said.
How Does Confidential Computing Work?
Taking a deeper look, confidential computing sits on a foundation called a root of trust, which is based on a secured key unique to each processor.
The processor checks it has the right firmware to start operating with what’s called a secure, measured boot. That process spawns reference data, verifying the chip is in a known safe state to start work.
Next, the processor establishes a secure enclave or trusted execution environment (TEE) sealed off from the rest of the system where the user’s application runs. The app brings encrypted data into the TEE, decrypts it, runs the user’s program, encrypts the result and sends it off.
At no time could the machine owner view the user’s code or data.
One other piece is crucial: It proves to the user no one could tamper with the data or software.
Attestation uses a private key to create security certificates stored in public logs. Users can access them with the web’s transport layer security (TLS) to verify confidentiality defenses are intact, protecting their workloads.
The proof is delivered through a multi-step process called attestation (see diagram above).
The good news is researchers and commercially available services have demonstrated confidential computing works, often providing data security without significantly impacting performance.
A high-level look at how confidential computing works.
Shrinking the Security Perimeters
As a result, users no longer need to trust all the software and systems administrators in separate cloud and edge companies at remote locations.
Confidential computing closes many doors hackers like to use. It isolates programs and their data from attacks that could come from firmware, operating systems, hypervisors, virtual machines — even physical interfaces like a USB port or PCI Express connector on the computer.
The new level of security promises to reduce data breaches that rose from 662 in 2010 to more than 1,000 by 2021 in the U.S. alone, according to a report from the Identity Theft Resource Center.
That said, no security measure is a panacea, but confidential computing is a great security tool, placing control directly in the hands of “data owners”.
Use Cases for Confidential Computing
Users with sensitive datasets and regulated industries like banks, healthcare providers and governments are among the first to use confidential computing. But that’s just the start.
Because it protects sensitive data and intellectual property, confidential computing will let groups feel they can collaborate safely. They share an attested proof their content and code was secured.
Example applications for confidential computing include:
Companies executing smart contracts with blockchains
Research hospitals collaborating to train AI models that analyze trends in patient data
Retailers, telecom providers and others at the network’s edge, protecting personal information in locations where physical access to the computer is possible
Software vendors can distribute products which include AI models and proprietary algorithms while preserving their intellectual property
While confidential computing is getting its start in public cloud services, it will spread rapidly.
Users need confidential computing to protect edge servers in unattended or hard-to-reach locations. Enterprise data centers can use it to guard against insider attacks and protect one confidential workload from another.
Market researchers at Everest Group estimate the available market for confidential computing could grow 26x in five years.
So far, most users are in a proof-of-concept stage with hopes of putting workloads into production soon, said Schuster.
Looking forward, confidential computing will not be limited to special-purpose or sensitive workloads. It will be used broadly, like the cloud services hosting this new level of security.
Indeed, experts predict confidential computing will become as widely used as encryption.
The technology’s potential motivated vendors in 2019 to launch the Confidential Computing Consortium, part of the Linux Foundation. CCC’s members include processor and cloud leaders as well as dozens of software companies.
The group’s projects include the Open Enclave SDK, a framework for building trusted execution environments.
“Our biggest mandate is supporting all the open-source projects that are foundational parts of the ecosystem,” said Jethro Beekman, a member of the CCC’s technical advisory council and vice president of technology at Fortanix, one of the first startups founded to develop confidential computing software.
“It’s a compelling paradigm to put security at the data level, rather than worry about the details of the infrastructure — that should result in not needing to read about data breaches in the paper every day,” said Beekman, who wrote his 2016 Ph.D. dissertation on confidential computing.
A growing sector of security companies is working in confidential computing and adjacent areas. (Source: GradientFlow)
How Confidential Computing Is Evolving
Implementations of confidential computing are evolving rapidly.
At the CPU level, AMD has released Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP). It extends the process-level protection in Intel SGX to full virtual machines, so users can implement confidential computing without needing to rewrite their applications.
Top processor makers have aligned on supporting this approach. Intel’s support comes via new Trusted Domain Extensions. Arm has described its implementation, called Realms.
Proponents of the RISC-V processor architecture are implementing confidential computing in an open-source project called Keystone.
Accelerating Confidential Computing
NVIDIA is bringing GPU acceleration to VM-style confidential computing to market with its Hopper architecture GPUs.
The H100 Tensor Core GPUs enable confidential computing for a broad swath of AI and high performance computing use cases. This gives users of these security services access to accelerated computing.
An example of how GPUs and CPUs work together to deliver an accelerated confidential computing service.
Meanwhile, cloud service providers are offering services today based on one or more of the underlying technologies or their own unique hybrids.
What’s Next for Confidential Computing
Over time, industry guidelines and standards will emerge and evolve for aspects of confidential computing such as attestation and efficient, secure I/O, said Beekman of CCC.
While it’s a relatively new privacy tool, confidential computing’s ability to protect code and data and provide guarantees of confidentiality makes it a powerful one.
Looking ahead, experts expect confidential computing will be blended with other privacy methods like fully homomorphic encryption (FHE), federated learning, differential privacy, and other forms of multiparty computing.
Using all the elements of the modern privacy toolbox will be key to success as demand for AI and privacy grows.
So, there are many moves ahead in the great chess game of security to overcome the challenges and realize the benefits of confidential computing.
Take a Deeper Dive
To learn more, watch “Hopper Confidential Computing: How it Works Under the Hood,” session S51709 at GTC on March 22 or later (free with registration).
Check out “Confidential Computing: The Developer’s View to Secure an Application and Data on NVIDIA H100,” session S51684 on March 23 or later.
You also can attend a March 15 panel discussion at the Open Confidential Computing Conference moderated by Schuster and featuring Ian Buck, NVIDIA’s vice president of hyperscale and HPC. And watch the video below.
NVIDIA GeForce NOW (@NVIDIAGFN) 
