Berkeley Lab-led ECP Co-Design Center Achieves Orders of Magnitude Speed-Up in Latest Software Release

Just one year after the U.S. Department of Energy's (DOE) Exascale Computing Program began funding projects to prepare scientific applications for exascale supercomputers, the Block-Structured Adaptive Mesh Refinement Co-Design Center has released a new version of its software that solves a benchmark problem hundreds of times faster than the original baseline.

The Block-Structured Adaptive Mesh Refinement Co-Design Center is one of five ECP co-design centers, so named because they are structured to create a close interchange of ideas between hardware technologies, software technologies and applications. The goal of the co-design center’s AMReX software framework is to support the development of block-structured adaptive mesh refinement (AMR) algorithms for solving systems of partial differential equations on next-generation architectures.

AMR allows scientists to focus computing power on the most critical parts of the problem in the most computationally efficient way possible. Applications worldwide rely on AMR for a wide variety of applications. Within the ECP specifically, projects in the areas of accelerator modeling, astrophysics, combustion, cosmology, and multiphase flows already rely on AMReX to support their algorithms.

Led by John Bell at Lawrence Berkeley National Laboratory, the AMR Co-Design Center builds on previous block-structured AMR work done by Berkeley Lab’s Center for Computational Sciences and Engineering and the lab’s Applied Numerical Algorithms Group. The groups respectively developed BoxLib and Chombo, two of the leading AMR frameworks used by the U.S. research community.

“The goal of the center is to redesign and re-implement our core block-structured AMR methodology to enable codes using AMReX to run efficiently on exascale computers,” said Bell. “Our latest release of AMReX demonstrates that DOE’s investment in the exascale program is already paying scientific dividends. Researchers using AMReX can now run today’s problems more efficiently.”

Members of the BoxLib and Chombo development teams, as well as developers of FLASH, a DOE-supported application code used in astrophysics and high-energy density physics, have come together with researchers from DOE’s National Renewable Energy Laboratory to build the new framework.

The co-design center released its first version of the AMReX framework in June and this latest release adds the capability to represent solid geometries using a technique known as the embedded boundary technique. This approach can be used to calculate flow past an object inside the domain, such as air over an airplane wing, or can represent the boundaries of a domain, such as the walls of a combustion chamber. To measure the effectiveness of ECP’s investment in AMReX, the team used an existing code to define a baseline, then solved the same problem with AMReX.  After a comprehensive re-design of the core data structures and a new implementation of the embedded boundary methodology, AMReX-based solvers reached the solution almost 200 times faster on the same number of processors.

In addition, while the baseline code had limited support for running in parallel, the AMReX code uses a hybrid parallel approach, an effective way take full advantage of systems such as Cori with manycore processors. 

Members of the co-design center are also working with other groups developing exascale applications to make sure that researchers will be able to effectively use AMR algorithms on exascale systems when they are deployed.

As one example, AMReX is a cornerstone of the Pele combustion codes being developed by the “Transforming Combustion Science and Technology with Exascale Simulations” ECP application project led by Jacqueline Chen of Sandia National Laboratories. This project will enable combustion simulations using direct number simulations (DNS) and hybrid DNS/large eddy simulations in realistic geometries. Ultimately, the aim of the project is to contribute to the design of high efficiency, low-emission combustion engines and gas turbines to reduce emissions and improve fuel efficiency.

“The Pele suite of combustion codes is built on top of the block-structured AMR infrastructure that AMReX is providing,” Chen said. “Notably, AMReX is providing performance portability, embedded boundary methods for complex geometry, and a container for treating particle-based spray, soot and radiation physics modules.”

The importance of AMR to DOE’s research missions was recently recognized as part of the 40^th anniversary of the founding of the Department of Energy. To mark the anniversary, DOE’s Office of Science selected 40 of the most significant research milestones. Among them was the 1989 research paper "Local adaptive mesh refinement for shock hydrodynamics," published in the Journal of Computational Physics and written by Marsha J. Berger of New York University and Phillip Colella, who led the development of Chombo.

Over the years AMR has been used to solve increasingly complex problems and has been implemented in increasingly sophisticated software frameworks.

“AMR has played an important role in DOE since the ‘90s,” said David Brown, director of Berkeley Lab’s Computational Research Division. “With the development of AMReX, numerous scientific applications will have access to AMR at the exascale, which will have an enormous effect on future scientific productivity.”

The work was funded through the Department of Energy’s Exascale Computing Project through the Office of Advanced Scientific Computing Research in the DOE Office of Science.