# High Performance Geometric Multigrid

High Performance Geometric Multigrid (HPGMG-FV) is a benchmark designed to proxy the finite volume based geometric multigrid linear solvers found in adaptive mesh refinement (AMR) based applications like the Low Mach Combustion Code (LMC). HPGMG-FV is being used to conduct computer science (e.g. Top500 benchmarking, programming models, compilers, performance optimization, and auto-tuners), computer architecture, and applied math research.

HPGMG-FV solves variable-coefficient elliptic problems (-b div beta grad u = f) on isotropic Cartesian grids using the finite volume method (FV) and Full Multigrid (FMG). The method is fourth-order accurate in the max norm, as demonstrated by the FMG convergence. FMG interpolation (prolongation) is quartic, V-cycle interpolation is quadratic, and restriction is piecewise constant. Recursive decomposition is used to construct a space filling curve akin to Z-Mort in order to distribute work among processes. Out-of-place Gauss-Seidel, Red-Black is used for smoothing, preconditioned by the diagonal. FMG convergence is observed with a using a V(3,3) cycle. Thus convergence is reached in a total of 13 fine-grid operator applications (3 pre-smooth GSRBs, residual, 3 post-smooth GSRBs). This makes HPGMG-FV extremely fast, accurate, and energy efficient.

The Publications page includes an introductory presentation to HPGMG-FV that many users may find useful when attempting to optimize or port the code. Moreover, the reference HPGMG-FV implementation includes optional second order discretizations, constant-coefficient variants, and many alternate smoothers including Jacobi and Chebyshev. Finally, there is an HPGMG mailing list with discussions of various releases and performance.

- Software There are two public versions of HPGMG. Both implement the 4th order (v0.3) solver. The former is MPI+OpenMP3 while the latter adds CUDA support for the key kernels in order to leverage the computing power of NVIDIA's GPUs: hpgmg on bitbucket (HPGMG-FV is in the ./finite-volume/source sub directory) hpgmg-cuda on bitbucket (v0.3 compliant MPI2+OpenMP3+CUDA7 implementation). Please see NVIDIA's Parallel ForAll blog for more details. Instructions A reference MPI+OpenMP implementation of…
- HPGMG Submission Guidelines Those wishing to submit HPGMG-FV results should email them to Samuel Williams. The current submission guidelines are as follows: Submissions must run the variable-coefficient fourth-order, "Laplacian" using the out-of-place GSRB smoother (3 pre-smooths and 3 post-smooths) with fourth-order homogeneous Dirichlet boundary conditions using the (F-Cycle) FMG solver (i.e. the defaults). Moreover, second-order, 1:2 volume-averaged prolongation (27pt) is required within the v-cycles and…
- Results HPGMG-FV as an alternate Top500 Benchmark Both the CPU implementation of HPGMG-FV as well as the GPU-accelerated version are configured to run as an alternate to the Top500's HPL benchmark (see hpgmg.org). Results may be submitted for ranking (peak DOF/s) and comparisons. The current submission guidelines may be found here. Below are the results as presented at the Supercomputing BoF's. Each machine was allowed to use any amount of memory per node. Unfortunately, due to scheduling and…
- People A number of researchers have and continue to contribute to the HPGMG effort. Samuel Williams Mark Adams Brian Van Straalen Erich Strohmaier Jed Brown …
- Reference Implementation A detailed description of the reference MPI+OpenMP implementation is forthcoming. introduction to finite volume representations and geometric multigrid boxes levels blocks ghost zone (halo) exchanges stencils and intra-level operators restriction, prolongation (interpolation), and inter-level operators U-Cycles, V-Cycles, and F-Cycles (FMG) timers and understanding the output alternate smoothers alternate operators and discretizations alternate boundary conditions alternate coarse-grid…
- Research Configuring HPGMG-FV as a Co-Design Proxy AMR combustion applications like LMC from the Combustion Co-Design Center create several AMR levels of refinement, and within each level, maintain many variables for every point in space. As the code evolves, we expect to track up to 100 chemical species (e.g. CO, NO, CO2, etc...). Each of these mass fractions require a MG diffusion solve. One may use HPGMG-FV to proxy these diffusion solves. However, as there are up to 10 AMR levels each with 100…
- Publications A number of efforts have leverage HPGMG in order to benchmark system and programming model performance and productivity as well as conduct research into alternate architectures, programming models, languages, and…