Need for Speed: Researchers Switch on World’s Fastest AI Supercomputer

It will help piece together a 3D map of the universe, probe subatomic interactions for green energy sources and much more.

Perlmutter, officially dedicated today at the National Energy Research Scientific Computing Center (NERSC), is a supercomputer that will deliver nearly four exaflops of AI performance for more than 7,000 researchers.

That makes Perlmutter the fastest system on the planet on the 16- and 32-bit mixed-precision math AI uses. And that performance doesn’t even include a second phase coming later this year to the system based at Lawrence Berkeley National Lab.

More than two dozen applications are getting ready to be among the first to ride the 6,159 NVIDIA A100 Tensor Core GPUs in Perlmutter, the largest A100-powered system in the world. They aim to advance science in astrophysics, climate science and more.

A 3D Map of the Universe

In one project, the supercomputer will help assemble the largest 3D map of the visible universe to date. It will process data from the Dark Energy Spectroscopic Instrument (DESI), a kind of cosmic camera that can capture as many as 5,000 galaxies in a single exposure.

Researchers need the speed of Perlmutter’s GPUs to capture dozens of exposures from one night to know where to point DESI the next night. Preparing a year’s worth of the data for publication would take weeks or months on prior systems, but Perlmutter should help them accomplish the task in as little as a few days.

“I’m really happy with the 20x speedups we’ve gotten on GPUs in our preparatory work,” said Rollin Thomas, a data architect at NERSC who’s helping researchers get their code ready for Perlmutter.

Perlmutter’s Persistence Pays Off

DESI’s map aims to shed light on dark energy, the mysterious physics behind the accelerating expansion of the universe. Dark energy was largely discovered through the 2011 Nobel Prize-winning work of Saul Perlmutter, a still-active astrophysicist at Berkeley Lab who will help dedicate the new supercomputer named for him.

“To me, Saul is an example of what people can do with the right combination of insatiable curiosity and a commitment to optimism,” said Thomas, who worked with Perlmutter on projects following up the Nobel-winning discovery.

Supercomputer Blends AI, HPC

A similar spirit fuels many projects that will run on NERSC’s new supercomputer. For example, work in materials science aims to discover atomic interactions that could point the way to better batteries and biofuels.

Traditional supercomputers can barely handle the math required to generate simulations of a few atoms over a few nanoseconds with programs such as Quantum Espresso. But by combining their highly accurate simulations with machine learning, scientists can study more atoms over longer stretches of time.

“In the past it was impossible to do fully atomistic simulations of big systems like battery interfaces, but now scientists plan to use Perlmutter to do just that,” said Brandon Cook, an applications performance specialist at NERSC who’s helping researchers launch such projects.

That’s where Tensor Cores in the A100 play a unique role. They accelerate both the double-precision floating point math for simulations and the mixed-precision calculations required for deep learning.

Similar work won NERSC recognition in November as a Gordon Bell finalist for its BerkeleyGW program using NVIDIA V100 GPUs. The extra muscle of the A100 promises to take such efforts to a new level, said Jack Deslippe, who led the project and oversees application performance at NERSC.

Software Helps Perlmutter Sing

Software is a strategic component of Perlmutter, too, said Deslippe, noting support for OpenMP and other popular programming models in the NVIDIA HPC SDK the system uses.

Separately, RAPIDS, open-source code for data science on GPUs, will speed the work of NERSC’s growing team of Python programmers. It proved its value in a project that analyzed all the network traffic on NERSC’s Cori supercomputer nearly 600x faster than prior efforts on CPUs.

“That convinced us RAPIDS will play a major part in accelerating scientific discovery through data,” said Thomas.

Coping with COVID’s Challenges

Despite the pandemic, Perlmutter is on schedule. But the team had to rethink critical steps like how it ran hackathons for researchers working from home on code for the system’s exascale-class applications.

Meanwhile, engineers from Hewlett Packard Enterprise helped assemble phase 1 of the system, collaborating with NERSC staff who upgraded their facility to accommodate the new system. “We greatly appreciate the work of those people onsite bringing the system up, especially under all the special COVID protocols,” said Thomas.

At the virtual launch event, NVIDIA CEO Jensen Huang congratulated the Berkeley Lab crew on its plans to advance science with the supercomputer.

“Perlmutter’s ability to fuse AI and high performance computing will lead to breakthroughs in a broad range of fields from materials science and quantum physics to climate projections, biological research and more,” Huang said.

On Time for AI Supercomputing

The virtual ribbon cutting today represents a very real milestone.

“AI for science is a growth area at the U.S. Department of Energy, where proof of concepts are moving into production use cases in areas like particle physics, materials science and bioenergy,” said Wahid Bhimji, acting lead for NERSC’s data and analytics services group.

“People are exploring larger and larger neural-network models and there’s a demand for access to more powerful resources, so Perlmutter with its A100 GPUs, all-flash file system and streaming data capabilities is well timed to meet this need for AI,” he added.

Researchers who want to run their work on Perlmutter can submit a request for access to the system.