High Performance Geometric Multigrid

High Performance Geometric Multigrid (HPGMG-FV) is a benchmark designed to proxy the finite volume based geometric multigrid linear solvers found in adaptive mesh refinement (AMR) based applications like the Low Mach Combustion Code (LMC). HPGMG-FV is being used to conduct computer science (e.g. Top500 benchmarking, programming models, compilers, performance optimization, and auto-tuners), computer architecture, and applied math research.

HPGMG-FV solves variable-coefficient elliptic problems (-b div beta grad u = f) on isotropic Cartesian grids using the finite volume method (FV) and Full Multigrid (FMG). The method is fourth-order accurate in the max norm, as demonstrated by the FMG convergence. FMG interpolation (prolongation) is quartic, V-cycle interpolation is quadratic, and restriction is piecewise constant. Recursive decomposition is used to construct a space-filling curve akin to Z-Mort in order to distribute work among processes. Out-of-place Gauss-Seidel, Red-Black is used for smoothing, preconditioned by the diagonal. FMG convergence is observed with a using a V(3,3) cycle. Thus convergence is reached in a total of 13 fine-grid operator applications (3 pre-smooth GSRBs, residual, 3 post-smooth GSRBs). This makes HPGMG-FV extremely fast, accurate, and energy-efficient.

The project publications page includes an introductory presentation to HPGMG-FV that many users may find useful when attempting to optimize or port the code. Moreover, the reference HPGMG-FV implementation includes optional second order discretizations, constant-coefficient variants, and many alternate smoothers including Jacobi and Chebyshev.  Finally, there is an HPGMG mailing list with discussions of various releases and performance. 

• Software There are two public versions of HPGMG. Both implement the 4th order (v0.3) solver. The former is MPI+OpenMP3 while the latter adds CUDA support for the key kernels in order to leverage the computing power of NVIDIA's GPUs:
  
  hpgmg on bitbucket (HPGMG-FV is in the ./finite-volume/source sub directory)
  hpgmg-cuda on bitbucket (v0.3 compliant MPI2+OpenMP3+CUDA7 implementation). Please see NVIDIA's Parallel ForAll blog for more details.
• HPGMG Submission Guidelines Those wishing to submit HPGMG-FV results should email them to Samuel Williams. This page details current submission guidelines.
• Results HPGMG-FV as an Alternate Top500 BenchmarkBoth the CPU implementation of HPGMG-FV as well as the GPU-accelerated version are configured to run as an alternate to the Top500's HPL benchmark. Results may be submitted for ranking (peak DOF/s) and comparisons. Visit the HPGMG website for current submission guidelines.Below are the results as presented at the Supercomputing BoF's. Each machine was allowed to use any amount of memory per node. Unfortunately, due to scheduling and allocation…
  • November 2018 Results
  • June 2018 Results
  • November 2017 Results
  • June 2017 Results
  • November 2016 Results
  • June 2016 Results
  • November 2015 Results
  • November 2015 Results (2nd order)
  • November 2014 Results (2nd Order)
• People A number of researchers contribute to the HPGMG effort including the following: Samuel Williams Mark Adams Brian Van Straalen Erich Strohmaier Jed…
• Reference Implementation A detailed description of the reference MPI+OpenMP implementation is forthcoming. introduction to finite volume representations and geometric multigrid boxes levels blocks ghost zone (halo) exchanges stencils and intra-level operators restriction, prolongation (interpolation), and inter-level operators U-Cycles, V-Cycles, and F-Cycles (FMG) timers and understanding the output alternate smoothers alternate operators and discretizations alternate boundary conditions alternate coarse-grid…
• Research Configuring HPGMG-FV as a Co-Design ProxyAMR combustion applications like LMC from the Combustion Co-Design Center create several AMR levels of refinement, and within each level, maintain many variables for every point in space. As the code evolves, we expect to track up to 100 chemical species (e.g. CO, NO, CO2, etc...). Each of these mass fractions require a MG diffusion solve. One may use HPGMG-FV to proxy these diffusion solves. However, as there are up to 10 AMR levels each with 100…
• Publications A number of efforts have leveraged HPGMG in order to benchmark system and programming model performance and productivity as well as to conduct research into alternate architectures, programming models, languages, and…