Exascale Application Project Targets Carbon Capture and Storage

Cold-flow demonstration of a circulating fluidized bed with gas and particle dynamics predicted by MFIX-Exa, rendering generated by Ascent in situ visualization. Six million 1mm diameter particles initially rest in a hopper (cylinder at left), flow down to fill the L-valve and are transported by a jet and sparger to the fluidized bed (cylinder at right), where they are violently fluidized before being transported up the riser, meeting a blind-T connection to the cyclone where the particles swirl around and are recycled to the hopper and the transport gas exits through the vortex finder at the top of the domain. This prototypical CFB is used by the MFIX-Exa team as a testbed for algorithm development aimed at performance improvement. (Credit: Jordan Musser, National Energy Technology Laboratory)

Among the many prospects for exascale computing applications, carbon capture and storage are of particular interest given the potential impact on combating climate change. Carbon capture and storage (CCS) technologies are promising approaches for reducing CO2 emissions, but one of the biggest challenges in deploying them is the scale-up from laboratory design to industrial scale.

The MFIX-Exa software subproject of the U.S. Department of Energy’s (DOE’s) Exascale Computing Project (ECP) is helping achieve that scaling. Utilizing the AMReX software framework developed at Lawrence Berkeley National Laboratory (Berkeley Lab), MFIX-Exa can track billions of individual particles simultaneously to simulate the gas-solid flows typical of a reactor - something standard computing cannot achieve.

The direct scale-up of multiphase flow reactors is known to be unreliable, and the current approach requires building and testing physical systems, which is cost prohibitive. MFIX-Exa will leverage exascale computing power to model emerging CCS technologies to aid in the design and optimization of these systems, helping control costs and reducing the risk of failure. MFIX-Exa can recognize when things go wrong far more rapidly than in a lab or experimental setting.

MFIX-Exa is a joint project between DOE’s National Energy Technology Laboratory (NETL) and Berkeley Lab, powered by a multi-lab team that integrates expertise in HPC directly with expertise in multiphase flow modeling. Jordan Musser, a scientist at NETL, is the MFIX-Exa principal investigator, while Berkeley Lab scientists Weiqun Zhang and Andrew Myers are the lead AMReX developers. Hengjie Wang, a postdoc in Berkeley Lab’s Center for Computational Sciences and Engineering (CCSE), contributes to MFIX-Exa as well, working on balancing the computation load from modeling different regions of the chemical reactors, which has led to significant computing speed improvements

“We had a good amount of experience using AMReX with particle-in-cell codes,” said Myers, a computer systems engineer in CCSE and a member of the AMReX core development team. “But MFIX-Exa was the first AMReX-based code to have direct pairwise particle interactions, so there was a set of tools we needed to build out to support those operations; because these particles are directly interacting with one another, it gets much more computationally expensive.”

Using AMReX allowed the MFIX-Exa project to easily migrate from running on only CPUs to running on hybrid, GPU-accelerated architectures. “When the MFIX-Exa project started, we started with MFIX legacy code, but it could only run on CPUs,” said CCSE senior engineer Weiqun Zhang. “But in HPC the hardware is changing and the next-generation supercomputing uses GPUs. So AMReX played an important role because the code can be used on CPUs or various types of GPUs.”

The code is geared toward simulating commercial chemical looping reactors that can be used to reduce CO2 emissions through capture and storage. The MFIX-Exa code calculates the fluid and particle dynamics in these types of reactors and can simulate the behavior of large-scale commercial reactors during their design phase. This allows engineers to prototype reactors with a high-fidelity model and diagnose issues before a plant is built. MFIX-Exa will allow a ramp-up of problem size, speed, and accuracy on exascale computers over the next decade.

MFIX-Exa is one of six ECP codes built on AMReX; other AMReX-based applications in ECP include accelerator modeling, cosmology and astrophysics, combustion and wind energy modeling. The MFIX-Exa code itself has been used to launch several offshoot codes, which have actually spawned some new research directions, according to Berkeley Lab Applied Mathematics Department Head Ann Almgren.

“AMReX has always had a very symbiotic relationship with the application codes built using AMReX; the application codes benefit immensely from being able to take advantage of the common infrastructure,” she said. “And when one application needs a new feature that AMReX can supply, that feature is immediately available to all the other applications. This synergy has enabled us to go in new directions we never foresaw.”

This article incorporates information from an ECP technology highlight about the MFIX-Exa project.