CMB Simulations At Scale
Using a KNL-optimized version of the TOAST CMB simulation and analysis framework, and running on 600,000 cores of NERSC's Cori-2 supercomputer, we were able to simulate observations of the sky signal, instrument noise, and atmospheric fluctuations made by 50,000 detectors spread over 7 frequencies, scanning 20% of the sky for 1 year from the Chajnantor Plateau in the Atacama Desert in Chile. This is at the scale of the planned Simons Observatory, and an important milestone towards simulating CMB-S4.
Collectively, the detectors made over 30 trillion observations - 35 times more than the full Planck mission. This entire data volume was held in memory so that it could be passed through multiple reduction pipelines, avoiding the prohibitive cost of either writing the simulation to disk or regenerating it anew for each reduction.
Animation 1: The telescope's view through one realization of turbulent, wind-blown, atmospheric water vapor. The volume of atmosphere being simulated depended on (a) the scan width and duration and (b) the wind speed and direction, both of which changed every 20 minutes. The entire observation used about 5000 such realizations.
Animation 2: Cumulative daily maps of the sky temperature and polarization at each frequency showing how the atmosphere and noise integrate down over time. The year-long campaign spanned 129 observation-days during which the ACTpol SS patch was available for a 13-hour constant elevation scan. To make these maps, the signal, noise, and atmosphere observations were combined (including percent level detector calibration error), filtered with a 3rd order polynomial, and binned into pixels.
|Frequency||Temperature (T)||Polarization (P)|
This work was supported by the LBNL LDRD project: "Developing a Scalable Simulation and Analysis Framework for CMB-S4 on Xeon Phi Based Supercomputers", and benefited from extraordinary help from the NERSC, Cray and Intel teams.