CRD’s Oliker New Lead for DOE’s SUPER Project to Improve HPC Performance, Efficiency

CRD's Lenny Oliker is the new PI for the SciDAC SUPER project.

As one of the world’s leading supporters of high performance computing, the U.S. Department of Energy provides thousands of researchers around the world with access to some of the most powerful supercomputers. As part of its research portfolio, DOE also invests in projects aimed at improving the performance of scientific applications on HPC systems, helping scientists make the most effective and efficient use of these systems. 

For almost 15 years, DOE has supported research into HPC performance and related areas through SciDAC, the Scientific Discovery through Advanced Computing program. The latest iteration of SciDAC’s HPC performance project is the Institute for Sustained Performance, Energy, and Resilience, or SUPER. Until recently, SUPER was led by Bob Lucas of the Information Sciences Institute at the University of Southern California. But with Lucas’ decision to step down, the leadership role has been assumed by Leonid “Lenny” OIiker of the Computational Research Division at Lawrence Berkeley National Laboratory (Berkeley Lab).

Oliker’s leadership is a continuation of Berkeley Lab’s key role in the project. When the initial round of SciDAC projects was announced in August 2001, David Bailey of Berkeley Lab was named as the principal investigator for the Performance Evaluation Research Center (PERC) project. One of the hallmarks of PERC and other SciDAC projects was a multi-institution approach. PERC integrated several previous efforts and also built ties with several other SciDAC scientific projects to ensure that the techniques and tools developed by PERC were truly useful to the broader DoE Office of Science community.

Five years later, the second round of SciDAC projects included PERI, the Performance Engineering Research Institute, which continued the work of PERC. Lucas, who had previously served as head of the High Performance Computing Research Department at Berkeley Lab, was announced as the project leader. In addition to helping researches make the most efficient and effective use of petascale systems, the project developed methods to optimize both the systems and the applications for highest performance.

SUPER builds on these past successes and now includes research into performance auto-tuning, energy efficiency, resilience, multi-objective optimization, and end-to-end tool integration. Leading the project dovetails neatly with Oliker’s research interests, which include optimization of scientific methods on emerging multi-core systems, ultra-efficient designs of domain-optimized computational platforms and performance evaluation of extreme-scale applications on leading supercomputers. Oliker recently took time to answer a handful of questions about his new role, the collaborative nature of the SUPER project and some of its recent accomplishments.

Question: First, what's your reaction to being named lead for SUPER?
Answer: We were disappointed to hear that Bob Lucas would be retiring as director of the project, after so many years of dedicated service and intellectual leadership to the community, he will surely be missed. Sadly, David Bailey also retired rather recently as well. He was the PI of the original SciDAC-1 institute and a leader in the subsequent institutes.  Stepping up as director of SUPER is an honor and a wonderful opportunity to work with leading computer science experts on a project that is strategically positioned to broadly impact the SciDAC program. Although the idea of coordinating an effort spread across 14 institutions sounds daunting, we are fortunate that many of these investigators have been working together for over a dozen years as part of SciDAC. These established relationships enable distributed technical leadership that immensely facilitates effective collaborations, both within SUPER’s technical research areas as well as interfacing with the broader SciDAC and DOE communities. It’s been a pleasure working with this team as a researcher over the years, and together with the new project deputy Jeff Hollingsworth (University of Maryland), we look forward to advancing SUPER’s ambitious high-end computing goals.

Q: What's your research role in SUPER?
A: Since my graduate school days, I have had a keen interest in understanding and advancing performance behavior on massively parallel systems.  Needless to say this never gets dull given the ever-changing architectural landscape and application characteristics.  SUPER provides unique opportunities to collaborate with some of the best computational scientists in the world and to make an actual performance impacts on their codes. This enables unprecedented higher-fidelity simulations with the potential for new science discoveries. Lately as part of my SUPER engagement, I have been fortunate enough to work with the NWChem quantum chemistry code, whose BES (DOE’s Office of Basic Energy Sciences) SciDAC project is led by Chris Cramer (University of Minnersota).

Additionally, I have been participating in the optimization of MPAS-Ocean as part of a BER (DOE’s Office of Biological and Environmental Research) SciDAC project called MULTISCSALE led by Bill Collins (LBNL). This has been a particularly fun project, since numerous SUPER folks have had the opportunity to particulate and help develop new algorithms and optimizations, as well as leverage the code knowledge to drive new performance tools and energy-minimization schemes. I have also been closely involved with a performance modeling and characterization project called Roofline, originally developed by Sam Williams here at LBNL. We are working to develop tools for the community to help drive optimization by allowing computational scientists extract the performance expectations for a given code and architectural platform.  

Q: How would you describe the project in a nutshell?
A: We are following our successful model of leveraging the research investments DOE and others have made and integrating the results to create new capabilities beyond the reach of any one group. To accomplish this, SUPER is organized into multi-institution teams conducting a variety of important research activities, including performance auto-tuning, energy efficiency, resilience, multi-objective optimization, and end-to-end tool integration. These are led by Mary Hall (University of Utah), Laura Carrington (SDSC), Paul Hovland (ANL) and Bronis de Supinski (LLNL) respectively. Additionally, a critical part of SUPER is our application engagement led by Pat Worley (ORNL), where we are actively collaborating with 13 of the 19 SciDAC scientific application partnerships, plus two additional supplemental codes.

Our role here is to provide expertise in measurement, analysis and performance to help application developers understand and optimize performance of their codes. Overall, our collaborations focus on the real challenges of scientific computing for existing and emerging petascale systems, with a goal of ensuring the broad and immediate impact of our research findings. Once again, our long history of working together allows for effective distributed leadership across the key research areas.  

Q: Can you describe some of the project’s accomplishments
A: There have been numerous success stories over the years, so let me just give a brief example of recent accomplishments as I presented in June at the 2015 SciDAC PI meeting. For the FES XCG1 fusion code led by C.S Chang (PPPL), designed for simulating edge plasma in tokamaks, we devised a new load balancing optimization strategy that improves performance up to 40 percent for production runs on OLCF’s Titan system.

We improved the Fock matrix construction and the associated TEXAS integral package of the quantum chemistry NWChem code by 1.6x by integrating an optimized threading strategy on the Xeon Phi platform. Looking at the electronic excitations application of Martin Head-Gordon (UC Berkeley and LBNL), we leverage and optimized a distributed memory tensor contraction methodology that sped up performance by up to 150x on ALCF’s Mira compared to the original shared-memory implementation. Our collaboration with Wick Haxton (UC Berkeley) on BIGSTICK, the nuclear physics configuration interaction code, resulted up to an 8x speed up on NERSC’s Edison, due to significant intra- and inter-processor optimizations. For the MPAS-Ocean multiscale code, we attained an improvement of up to 4x at high concurrencies by using SUPER tools to isolate the performance bottlenecks together with a combination of established and novel optimizations.  

Finally, for the GEANT4 library, which is used to simulate the interaction between physics particles and materials, optimization efforts resulted in a 5 percent  improvement for certain important configurations. While this might not sound like much, it has been estimated that running GEANT4 consumes $100M of computation resources per year, so a 5 percent improvement translates to $5M of savings per year.

This is only a small snapshot of our application accomplishments, we also made important contributions in tool integration for bottleneck detection, dynamic and static energy minimization strategies that preserve application performance, auto-tuning techniques for high performance on SciDAC-driven computational methods, and resilience solutions to detect and mitigate both soft and hard-errors on HPC systems.

Q: Have you seen any impacts in the community from the project?
A: Over the past 14 years, SUPER as well as PERC and PERI, have provided a forum for the benchmarking and performance engineering communities to effectively collaborate. It has enabled new collaborations, especially in emerging performance areas such auto-tuning. The integration of state-of-the-art performance tools certainly wouldn’t have happened without SciDAC. SUPER and its predecessors were explicitly funded to be the center of mass of the performance community, and those collaborative innovations have extended beyond the scope of SUPER to successfully impact the broader HPC community.