The NERSC CMB Data Analysis Repositories

Repository Resources

Repositories are annual allocations of high performance computing resources, comprising

CPU cycles measured in massively parallel processing (MPP) hours, and
HPSS data archiving measured in storage resource units (SRU) which includes both data transfer and storage.

There are two repositories for CMB data analysis

mp107 is for the analysis of data from sub-orbital experiments. It is currently shared by about 30 users from 9 experiments, and its initial 2007 allocation is of 290,000 MPP-hours and 50,000 SRUs.

planck is for the analysis of data from the Planck satellite mission. It is currently shared by about 50 users, and its initial 2007 allocation is of 500,000 MPP-hours and 500,000 SRUs.

Each user in a repository is given access to a fraction (typically 5%) of the total allocation. If you exhaust your allowance please contact me about increasing your limit.

If you are a member of both repositories please be careful to charge each of your jobs to the appropriate one.

To check the current status of the allocation you can use the getnim command on any of the NERSC machines (with "repo" set to either mp107 or planck as appropriate):

for the total remaining allocation in hours: getnim -R repo
for the remaining allocation available to a particular user: getnim -R repo -U username
for the amount of time used so far by each user: getnim -R repo -L

Jobs using a large number of CPUs can quickly burn a large number of MPP hours.
Please take a moment to check the number of MPP hours that you are requesting
(number of processors x number of hours x machine-specific charge factor)
whenever you submit a batch job.

Individual & Project Disk Spaces

Individual users have disk quotas (of both space and inodes) on each of the NERSC machines Seaborg, Jacquard, Bassi & Da Vinci. If you exhaust your quota - and you have archived everything you can - contact me about requesting an increased quota.

In addition each repository has a project directory on the NERSC global filesystem (NGF) which is mounted on all of the NERSC machines. Unlike the individual scratch directories this space is not automatically purged - please take care to archive and/or delete files that are not in active use. Subdirectories have been set up for each experiment using the NERSC facilities that are accessible only to members of the corresponding filegroup.

The project directory is a limited resource shared by all users.
It should be used primarily for data files that are required on multiple machines.
Please estimate your disk space requirements before generating or copying files,
and consult me if it they are significant (>100GB).

Filegroups

Every experiment sharing the mp107 repository has a UNIX filegroup defined for it to allow proprietory and/or sensitive files to be shared between members of that filegroup without them becoming accessible to everyone in the mp107 uber-group.

to check the group(s) that you belong use: groups

If you are not in a filegroup that you should be in please contact me.

By default you will access NERSC facilities in the mp107 group and your file/directory group permissions will be set accordingly. However you can

change the group of any file/directory using: chgrp filegroup file/directory-name
set the file/directory group permissions to make it readable using: chmod g+rX file/directory-name

You can also set a different filegroup for particular session, which will ensure that all the files/directories created during that session will have the required group properties.

to change your group use: newgrp filegroup

Filegroup directories have also been set up

as subdirectories of our NGF project space at /project/projectdirs/mp107/filegroup
as subdirectories of our project archive space on the default hpss system archive at /nersc/projects/cmb/filegroup

with each directory having access restricted to the members of that group.

Modules

Popular analysis software is avaiable through a CMB module on all the NERSC machines. Certain experiments also have their own modules for dedicated software; to find out more contact me.