ExaStar Takes Astrophysical Research to the Exascale

Researching explosive astrophysical phenomena such as stellar explosions and neutron star mergers gives scientists insight into the cosmic origin of the elements and the behavior of matter. But this type of research is computationally intensive and requires the power of the largest supercomputers available. Within the next few years, exascale computing can take this research to the next level.

However, to harness the power of exascale, researchers will need new code developed specifically for exascale computing. To address this need, the ExaStar project, an application development project within the Department of Energy’s (DOE) Exascale Computing Project (ECP), brings together research teams from Lawrence Berkeley, Oak Ridge, and Argonne national laboratories and Stony Brook University to collaborate on building code for astrophysics simulations that will run on exascale computers.

Berkeley Lab’s team of ExaStar researchers is focused on simulating explosive astrophysical events to address fundamental questions in nuclear astrophysics. They’ve already tackled one of the most difficult issues in this research: simulating how radiation flows through matter.

“Determining how radiation flows through matter is key to understanding the mechanism of supernovae and predicting the detectible signatures,” said Daniel Kasen, a scientist in Berkeley Lab’s Nuclear Science Division and associate professor in the Physics and Astronomy departments at UC Berkeley. “We think that it is neutrino radiation that carries energy from the hot core of the star to the outer layers and blows everything apart; being able to simulate that process has been the biggest hurdle in solving how supernovae work.”

All simulations use approximations; one of the ExaStar team’s goals has been to develop a radiation transport solver using new mathematical algorithms that reduce the degree of approximation. “We’re at the point now where we can start running supernovae calculations that have a higher fidelity treatment of neutrinos than we’ve ever done before,” Kasen said. Toward this end the team is leveraging supercomputing resources at Berkeley Lab’s National Energy Research Scientific Computing Center (NERSC) and the Oak Ridge Leadership Computing Facility.

The ExaStar Software Ecosystem

Codes that the ExaStar team uses -- namely, SedonaEx, Castro, and FLASH5 -- all rely on AMReX, the adaptive mesh refinement (AMR) software framework developed as part of the ECP AMR Co-Design Center, which is led out of Berkeley Lab’s Center for Computational Sciences and Engineering (CCSE). These codes form ExaStar’s software ecosystem for astrophysical modeling at the exascale.  While Castro and FLASH will be used to study the stellar explosions themselves, SedonaEx will enable researchers to connect the simulation results with what observers actually see from Earth. As Don Willcox, a CCSE postdoctoral computational scientist, points out, “You can't directly observe what happens inside a supernova explosion in real time. So to figure out how these supernovae work, you have to put together calculations from different physics algorithms over a very wide range of spatial and time scales. And at the end you compare your model predictions to observations to see how well they match.”

The ECP is funding multiple groups to work on building codes for exascale computers, which will be available within the next few years and promise computing power that is 50 times faster than the most powerful supercomputers available at the start of the project.  

“If our simulations can effectively use that big bump in computer power, we’ll be able to go from running one or two exploratory models of supernovae to running parameter surveys of many models and actually doing predictive science,” said Kasen. “It will be a game changer.”

The explosive astrophysical events that the Berkeley Lab ExaStar team are looking at now provide a way to probe basic events in physics. “These are cosmic laboratories that reach conditions that we could never reach on Earth, so they give us insight into how nature works in these environments of extreme density and gravity,” he added.  

These extreme environments can help scientists address questions of origin—how the matter that makes up the Earth and humans first came into being in the universe. “We know that most of the heavier elements in the universe came from these exploding/colliding stars,” said Kasen. “We still don’t fully understand how those nuclear processes work, so we are trying to simulate the creation of the elements inside supernovae and related phenomena and finally answer that question of where the elements came from.”

In addition, while scientists have known for decades that supernovae exist, they do not yet understand how they work in detail. “Getting exascale machines will give us the raw power, but to make use of it we have to do a lot of work to get our codes effectively running on these new architectures,” said Kasen. 

NERSC is a DOE Office of Science User Facility.