The LBNL CMB Software Module

Commonly used general-purpose software is made available to NERSC users through the use of modules. In the same way, we are now making the most popular CMB-specific software available to NERSC CMB users through a dedicated module.

Unlike general purpose software, the cmb module is not supported by NERSC, but by CMB data analysts in the Computational Research Division at Berkeley Lab; if you have any problems with this module, or would like to see new software added to it, you should therefore contact us, and not NERSC support.

The latest release of the CMB module is installed at NERSC on franklin, jacquard, bassi, and davinci. See notes at the bottom of this page describing how to use this software on each machine. Also, this software has been installed on several machines outside of NERSC. These machines also have useage notes below.


Individual Software Packages

The CMB module is made up of a number of packages, each of which is also a module in its own right. These fall into 2 categories:

  1. Packages for which we support the installation, but not the software itself. We are "package maintainers" for this software, similar to package maintainers for various GNU/Linux distributions. We take the original source tarball, apply a patch which allows the software to build in our environment, and then build and install the software.


  2. Packages for which we support both the NERSC installation and the software itself.

All of the software has been compiled in 64-bit mode when possible, and with extensive optimization flags. Codes using OpenMP have been compiled with this feature enabled.


Using the CMB module

The precise location of the installed software modules varies on each system. In order to gain access to the modules, a few lines will need to be added to your shell resource file(s). See the machine-specific notes below to see how to initialize the modules environment on each system.

After you have set up your environment, the modules should show up at the top of the list when you execute the module avail command: 

%> module avail
You should see a listing for both the "cmb" and "cmbpath" modules. You can disregard the cmbpath module unless you are trying to do specialized things with the modules. You can see which modules are currently loaded by typing: 
%> module list
Now load the cmb module and verify that it has in turn loaded the modules for all of the individual software packages. 
%> module load cmb
%> module list
More information can be found in the "help" page for each module listed above. For example, after using "module avail" to see what modules are available, you could then get more information about specific modules: 
%> module help healpix

All of the executables contained within these modules are now in your default PATH. You can run them just by typing the name of the program. If you want to verify exactly which executable is going to launch when you type the name of a program, use the "which" command: 

%> which anafast
Many of the tools in the CMB module provide libraries that you can use for building your own software. The necessary include and linking flags for these libraries are provided by the "cmbconfig" script. After loading the cmb or cmbpath modules, you will have access to this script. Running this script without any arguments displays useage information: 
%> cmbconfig


<************************************************************>
  Usage:  cmbconfig
          --comp (print compiler)
          --cflags (print compile options)
          --incl (print include directories)
          --lflags (print linking directories)
          --libs (print libraries)
          --lang < C | CXX | F77 | F90 | MPIC | MPICXX | MPIF77 | MPIF90 >
          -I (DEPRECATED:  same as --incl)
          -L (DEPRECATED:  same as --lflags --libs)
          [package1] [package2] [package3]...

          Where the following package names are accepted:
            cfitsio (print the flags for CFITSIO)
            xml2 (print the flags for XML2)
            healpix (print the flags for F90 HEALPix)
            chealpix (print the flags for C HEALPix)
            cxxhealpix (print the flags for C++ HEALPix)
            hpic (print the flags for HPIC)
            gcp (print the flags for GCP)
            fftw2 (print the flags for FFTW2)
            fftw2_mpi (print the flags for MPI FFTW2)
            fftw3 (print the flags for FFTW3)
            camb (print the flags for CAMB)
            lenspix (print the flags for LensPix)
            wmap (print the flags for WMAP)
            m3 (print the flags for M3)
            m3_mpi (print the flags for MPI M3)
            scalapack (print the flags for scalapack)
            madspec (print the flags for MADspec)
            lapack (print the flags for LaPACK)
<************************************************************>
For example, to compile a C++ program called "mycode" that requires the FFTW3, healpix, and GCP libraries, here is how you would use cmbconfig on bassi. Note that we make use of the "backtick" shell operator to insert the output of the cmbconfig script into our compile command: 
%> xlC_r -O3 `cmbconfig --incl fftw3 gcp healpix` -o mycode mycode.cpp \
`cmbconfig --lflags --libs fftw3 gcp healpix`
You can also use cmbconfig inside the Makefiles of your own software projects. In order to create a portable Makefile, you could use cmbconfig to obtain the compiler name as well. For example, a simple Makefile for the the "mycode" project might look like this: 
CXX = $(shell cmbconfig --lang CXX --comp)
CXXFLAGS = $(shell cmbconfig --lang CXX --cflags)
INCL = $(shell cmbconfig --incl fftw3 gcp healpix)
LINK = $(shell cmbconfig --lflags --libs fftw3 gcp healpix)


mycode : mycode.o
    $(CXX) -o mycode mycode.o $(LINK)


mycode.o : mycode.c
    $(CXX) $(CXXFLAGS) $(INCL) -o mycode.o -c mycode.c
If you use cmbconfig in your Makefile, then be sure to load the cmb module before running make.



NERSC - Machine Specific Notes

On all supported machines, you must initialize the cmb modules in your shell resource file(s). This consists of adding a couple lines to the resource file which "sources" (inserts) a piece of shell code into your shell initialization process. There are 2 files that can be sourced depending on whether you are using a bourne-type shell or a C-type shell. If you don't know what your default shell is, just log into the machine and type: 

%> echo $SHELL
After determining your shell, follow the directions below for the machine of interest. NOTE: if you set up only one type of shell (e.g. bash) to access the cmb modules, remember that any batch scripts which use a different shell (like csh) may not have access to the cmb modules.



franklin.nersc.gov

On franklin, the CMB project directory on NGF is not yet available on the compute nodes. Because of this, the CMB module software has been installed to Ted's home directory. This is a temporary fix that will disappear once franklin's compute nodes can access NGF. For now, you should add the following lines to your shell resource files:
bash
(~/.bashrc.ext) 
. /u0/k/kisner/software/cmb/cmbinit.sh
(t)csh
(~/.cshrc.ext) 
source /u0/k/kisner/software/cmb/cmbinit.csh
The next time you log in, the cmb module directory will be added to the list of available modules. In addition to the standard cmb module, a recent version of subversion (1.4.5) has been installed as a module. It is not loaded automatically, but after setting up the cmb module as above, you can load it with: 
%> module load svn

bassi.nersc.gov

If you have previously run the "setupcmb.pl" script, then you can either do nothing (future updates to the modules should show up fine), or you can switch your startup configuration to the new mechanism. If you are switching to the new system, edit your ~/.modules file and remove the lines that say "module use <...>". After clearing out these old lines (or if you never ran the old setupcmb script), you should then add the following lines to your shell resource files:
bash
(~/.bashrc.ext) 
. /project/projectdirs/cmb/modules/bassi/cmbinit.sh
(t)csh
(~/.(t)cshrc.ext) 
source /project/projectdirs/cmb/modules/bassi/cmbinit.csh
The next time you log in, the cmb module directory will be added to the list of available modules. NOTE: The loadleveler batch system on bassi has a limit on the size of the shell environment at the time of job submission. This means that you may have to unload the cmb module before submitting a job. However, it is possible to just use a "module load" command within the batch script itself.

jacquard.nersc.gov

If you have previously run the "setupcmb.pl" script, then you can either do nothing (future updates to the modules should show up fine), or you can switch your startup configuration to the new mechanism. If you are switching to the new system, edit your ~/.modules file and remove the lines that say "module use <...>". After clearing out these old lines (or if you never ran the old setupcmb script), you should then add the following lines to your shell resource files:
bash
(~/.bashrc.ext) 
. /project/projectdirs/cmb/modules/jacquard/cmbinit.sh
(t)csh
(~/.(t)cshrc.ext) 
source /project/projectdirs/cmb/modules/jacquard/cmbinit.csh
The next time you log in, the cmb module directory will be added to the list of available modules.

davinci.nersc.gov

If you have previously run the "setupcmb.pl" script, then you can either do nothing (future updates to the modules should show up fine), or you can switch your startup configuration to the new mechanism. If you are switching to the new system, edit your ~/.modules file and remove the lines that say "module use <...>". After clearing out these old lines (or if you never ran the old setupcmb script), you should then add the following lines to your shell resource files:
bash
(~/.bashrc.ext) 
. /project/projectdirs/cmb/modules/davinci/cmbinit.sh
(t)csh
(~/.(t)cshrc.ext) 
source /project/projectdirs/cmb/modules/davinci/cmbinit.csh
The next time you log in, the cmb module directory will be added to the list of available modules. In addition to the standard cmb module, the KST data visualization tool is also available and included when you load the cmb module. Davinci is the preferred machine for visualization tools like IDL, kst, etc, since it has 32 Itanium2 processors with 190GB of shared RAM. The internal KDE libraries sometimes generate a lot of output to the console, which can be annoying. To avoid this, you can use the alias "kstclean", which simple pipes this output to /dev/null: 
%> kstclean <arguments>

Other Systems - Machine Specific Notes

planck01.ipac.caltech.edu

The current version of the installed intel fortran compiler (9.1.031) has problems when building the OpenMPI F90 wrapper compiler. Although the cmb module is installed on this system, any MPI-enabled F90 codes will crash with errors about an "invalid communicator". This problem will be addressed shortly, and this website will be updated to reflect the current state of the situation. Add the following lines to your shell resource files to use the cmb module:
bash
(~/.bashrc.ext) 
. /planck/tools/NERSC/cmb/cmbinit.sh
(t)csh
(~/.(t)cshrc.ext) 
source /planck/tools/NERSC/cmb/cmbinit.csh
The next time you log in, the cmb module directory will be added to the list of available modules. This machine is an 8-core shared memory machine. There is no batch system, so you can launch MPI executable directly with: 
%> mpirun -np 4 <executable> <arguments>
where the number of processors specified should be no more than 8.

ent.oats.inaf.it

On this machine, newer versions of supporting tools have been installed. These supporting tools (like gcc-4.2.2) have been installed as modules separately from the cmb modules. Add the following lines to your shell resource files to use the cmb module:
bash
(~/.bashrc) 
export PATH=$PATH:/lfi_dpc_test/kisner/software/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lfi_dpc_test/kisner/software/lib
. /lfi_dpc_test/kisner/software/Modules/default/init/bash
. /lfi_dpc_test/kisner/software/cmb_gcc/cmbinit.sh
(t)csh
(~/.(t)cshrc) 
setenv PATH ${PATH}:/lfi_dpc_test/kisner/software/bin
setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/lfi_dpc_test/kisner/software/lib
source /lfi_dpc_test/kisner/software/Modules/default/init/(t)csh
source /lfi_dpc_test/kisner/software/cmb_gcc/cmbinit.csh
The next time you log in, the cmb module directory will be added to the list of available modules. The cmb module will automatically load the gcc-4.2.2 environment module, the Intel MKL module, and the OpenMPI 1.2.4 module. A new version of Subversion (1.4.5) has also been installed, and may be loaded separately. When running jobs after loading the cmb module, the "mpirun" command that is used will be the one built as part of OpenMPI.

In order for programs in the cmb module to run properly inside of batch jobs, their modules must be loaded in your shell resource file. This is not an ideal solution, and I am investigating more elegant solutions, such as a wrapper around mpirun. In the mean time, if you want to run batch jobs with executables in the cmb module, you must add the following to your shell resource file:
bash
(~/.bashrc) 
module load cmb                   
(t)csh
(~/.(t)cshrc) 
module load cmb                   
To run a job, use the "new" version of mpirun from OpenMPI. You no longer need to specify the "-machinefile" option to mpirun; the mpirun command automatically knows which processors are available:
PBS Script 
mpirun [-np <# of processors>] <executable> <arguments>


Experiment-Specific Modules

The module system is also a convenient way to provide access to experiment-specific or proprietary software to a defined set of users. The first such module, specifically for Planck users, is already available. You can find documentation about this module here. Other users/experiments interested in this approach should contact us for more information.

Return to the main Module Page