ECP Pagoda Project Rolls Out First Software Libraries

Just one year after the U.S. Department of Energy's (DOE) Exascale Computing Program (ECP) began funding projects to prepare scientific applications for exascale supercomputers, the Pagoda Project—a three-year ECP software development program based at Lawrence Berkeley National Laboratory—has successfully reached a major milestone: making its open source software libraries publicly available as of September 30, 2017.

The Pagoda software stack.

Led by Scott B. Baden, Group Lead of the Computer Languages and Systems Software (CLaSS) Group within Berkeley Lab’s Computational Research Division, the Pagoda Project’s libraries are designed to support lightweight global address space communication for exascale applications. The libraries take advantage of the Partitioned Global Address Space (PGAS) model to emulate large, distributed shared memories. By employing this model, which allows researchers to treat the physically separate memories of each supercomputer node as one address space, the Pagoda libraries will be able to leverage available global address hardware support to significantly reduce the communication costs of moving data—often a performance bottleneck in large-scale scientific applications, Baden explained.

“Our job is to ensure that the exascale applications reach key performance parameters defined by the DOE,” he added.

Thus this first release of the software is as functionally complete as possible, Baden emphasized, covering a good deal of the specification released last June. “We need to quickly determine if our users, in particular our ECP application developer partners, are satisfied,” he said. “If they can give us early feedback, we can avoid surprises later on.”

GASNet-EX and UPC++

The Pagoda software stack comprises a communication substrate layer, GASNet-Ex, and a productivity layer, UPC++. GASNet-Ex is a communication interface that provides language-independent, low-level networking for PGAS languages such as UPC and Coarray Fortran, the UPC++ library and for the Legion Programming Language. UPC++ is a C++ interface for application programmers that creates “friendlier” PGAS abstractions above GASNet-Ex’s communication services.

“GASNet-Ex, which has been around for over 15 years and is being enhanced to make it more versatile and performant in the exascale environment, is a library intended for developers of tools that are in turn used to develop applications,” Baden explained. “It operates at the network hardware level, which is more challenging to program than at the productivity layer.” The GASNet-Ex effort is led by Pagoda co-PI Paul Hargrove and was originally designed by Dan Bonachea, who jointly develops the software. Both are members of CLaSS.

As the productivity layer, UPC++ sits at a slightly higher level over GASNet-Ex, in a form appropriate for applications programmers. The goal of this layer is to impose minimal overheads in exchange for hiding considerable idiosyncratic detail, so users are satisfied with the benefits obtained by increased productivity.

Over the past year, the Pagoda team worked closely with several Berkeley Lab partners to develop applications and application frameworks, including the Adaptive Mesh Refinement Co-Design Center (AMReX), Sparse Solvers (ECP AD project) and ExaBiome (ECP AD Project). They also worked with several industry partners, including IBM, NVIDIA, HPE and Cray, and over the next few months will be meeting with all of the major vendors who are vying to build the first exascale computer or the components that will go into those computers.

“We are part of a large community of ECP developers,” Baden said. “And the ECP wants to deploy a software stack, a full set of tools, as an integrated package that will enable them to ensure that the pieces are compatible, that they will all work together. I am fortunate to be working with such a talented team that is highly motivated to deliver a vital component of the ECP software stack.” This team includes other members of CLaSS—Steve Hofmeyr and Amir Kamil (at the University of Michigan)—as well John Bachan, Brian van Straalen and Mathias Jacquelin. Bryce Lelbach, now with NVIDIA, also made early contributions.

Now that they are publicly available, the Pagoda libraries are expected to be used by other ECP efforts and supercomputer users in general to meet the challenges posed not only by the first-generation exascale computers but by today’s petascale systems as well.

“Much of the ECP software and programming technology can be leveraged across multiple applications, both within ECP and beyond,” said Kathy Yelick, Associate Lab Director for Computing Sciences at Berkeley Lab, in a recent interview with HPCwire. For example, AMReX, which was launched last November and recently announced its own first milestone, released its new framework to support the development of block-structured AMR algorithms, and at least five of the ECP application projects are using AMR to efficiently simulate fine-resolution features, Yelick noted.

For the remaining two years of the Pagoda project, the team will be focused on application integration and performance enhancements that adeptly leverage low-level hardware support, Baden noted.