CRD’s Helen He Develops Method for Easier Use of Community Climate System Model

As part of a comprehensive national project to provide U.S. climate researchers with state-of-the-art modeling capabilities, Yun “Helen” He of CRD’s Scientific Computing Group has developed a “single executable” mode for the Community Climate System Model (CCSM) on NERSC’s IBM SP. CCSM consists of separate programs for modeling atmosphere, ocean, sea ice and land surface and a flux coupler linking the four components. These components are integrated together in original multiple executable codes as a single computational system while keeping each component as a standalone executable.

However, CCSM in this configuration could only run on computer architectures which support the MPMD (multiple program, multiple data) programming model. This meant that newer architectures, such as IBM’s BlueGene/L supercomputer, could not run the CCSM. With the single executable model, the CCSM will be able to run on a wider variety of systems, including small Linux clusters.

He developed the single executable method as part of the project for Collaborative Design and Development of the Community Climate System Model for Terascale Computers under DOE’s Scientific Discovery through Advanced Computing (SciDAC) program.

The new single executable method was developed by redesigning the top level code structures. It also maintains the coexistence of the original multiple executable and new single executable versions, and ensures minimal impact on users with modified build scripts. The new CCSM single executable also improves the way of setting number of processors and threads for each model component from static to dynamic. Another technical difficulty is the name conflicts originating from the same name modules and subroutines, etc., used in different components within the single executable code. He proposed a module-based approach, using a wrapper module to localize global symbols, to resolve this issue.

 The new method is based on the MPH Library developed by He and Chris Ding, also of the Scientific Computing Group. MPH stands for Multi-Program Handshaking and the library was developed to address the growing trend toward integrating separate programs (i.e., get them to shake hands) to take advantage of terascale distributed memory supercomputers. MPH provides the key infrastructure for component name registration, resource allocation and initializes communication channels between independent components. MPH provides a flexible, versatile mechanism for these tasks and thus could be a basis for larger software tools/frameworks. Often, such components are developed independently of one another by researchers at different institutions without consideration for integrating them to run as one application.

In the case of CCSM, while model development is of community effort, the atmosphere component, the Community Atmosphere Model (CAM) and the land component, the Community Land Model (CLM),  were mostly developed at the National Center for Atmospheric Research (NCAR). The ocean component, the Parallel Ocean Program (POP) was mostly developed at Los Alamos National Lab (LANL), while the Community Sea Ice Model (CSIM) was also mostly developed at NCAR and LANL.

“Software lasts much longer than specific computer systems, so finding a method to adapt existing applications to run on new architectures protects that intellectual investment in software,” He said. “In addition to making the CCSM code easier to use for a broader range of users, MPH can also be applied to applications in fields such as materials sciences to integrate model components or to perform ensemble simulations. Another example of applicability would be where you run one application to generate data and run another one to analyze the results.”

The single executable version of CCSM has been ported to the IBM systems at NCAR and is currently being tested to run on the Cray X1 at Oak Ridge national lab and the BlueGene/L at NCAR. However, before releasing the application for testing, He needed to do a bit-by-bit comparison to ensure the accuracy of the new version

In addition to the CCSM, MPH has many users such as NCAR/Weather Research and Forecasting Model (WRF), European Center for Medium-Range Weather Forecasts (ECMWF), Argonne National Lab (ANL), Berkeley Lab, University of Michigan, and Edinburgh Parallel Computing Centre (EPCC). Other researchers from NASA, NOAA, Germany and China have expressed interest in using MPH. He also gave a presentation on MPH and its applications at the 8th International Workshop on Next Generation Climate Models for Advanced High Performance Computing Facilities in Albuquerque at the end of February.

“Thank you for all of your efforts on this project!,” wrote Nancy Norton of the CCSM Software Engineering Group at NCAR. “I especially appreciated your responsiveness to our requests, your flexibility as the project requirements evolved, your good humor as we iterated on the pop2 source code and setup, and your dedication to completing this project in a timely fashion.”

Mariana Vertenstein, manager of the software engineering group, added, “The work you have done should be extremely valuable as we go forwards to create single executable concurrent versions of all CCSM development configurations.”