BLCR (Berkeley Lab Checkpoint/Restart) allows programs running on Linux to be "checkpointed" (written entirely to a file), and then later "restarted". BLCR can be found at http://ftg.lbl.gov/checkpoint.
Putting a process to sleep (via the SIGSTOP signal) implies stopping its execution. Taking a checkpoint writes a snapshot of a process to disk: the process may either be allowed to continue running after the checkpoint is complete, or you can kill the process to release all of its resources .
With sleep, a process's resources are not all fully released (such as virtual memory, network connections, process id, etc.). Checkpointing then killing a process fully releases all system resources.
Restarts from checkpoint files can be used across machine reboots, and/or even on different machines than the one that the checkpoint was taken on. This is not true for SIGCONT.
BLCR performs checkpointing and restarting inside the linux kernel. While this makes it less portable than solutions that use user-level libraries, it also means that it has full access to all kernel resources, and can thus restore resources (like process IDs) that user-level libraries cannot. This also allows BLCR to checkpoint/restart groups of processes (such as shell scripts and their subprocesses), together with the pipes that connect them.
BLCR runs on x86 systems running Linux 2.4 or 2.6 kernels and x86_64 (Opteron/EM64T) systems running Linux 2.6 kernels. With the 0.5.0 release, we believe the following to work:
- 2.4.0 through 2.4.34 on x86 only (and no plans to support 2.4 on x86_64).
- 2.6.0 through 2.6.19 on x86 and x86_64.
We know that 0.5.0 will not compile on the following, and may have issues on kernels that are similar/related:
We hope to resolve these issues in future patch releases.
- 2.6.17-5mdv (Mandriva 2007.0) for x86 (x86_86 appears OK)
- 2.6.20 on both x86 and x86_64
BLCR uses a set of configure-based tests to determine which kernel features are available, and so in principle, BLCR should work with any distribution that uses a supported cpu/kernel combination (see above).
Historically, BLCR has been tested with kernels from versions 9.x and 10.0 of SuSE Linux; Red Hat 7.2 through 9; Red Hat Enterprise Linux version 3; CentOS 3.1; Fedora Core 2, 3 and 4; and many vanilla Linux kernels (from kernel.org) from 2.4.0 on up. We have not tested every single version of the kernel from every vendor, nor is each BLCR release retested against all distributions tested in the past. However, we believe that BLCR should work on most distributions using kernels in the ranges given above (except where vendors may have applied patches that bring in problematic changes from kernels outside that range).
If after reading this question and the one above you believe your platform should be supported but cannot get BLCR to work, then please consult our bug database for possible solutions and then report the problem if you don't find it already reported. We can't try every possible platform ourselves and count on user's bug reports to let us know when our testing has missed something.
Yes, BLCR can checkpoint both single- and multithreaded (pthreads) programs. This support works with both the LinuxThreads and NPTL implementations of pthreads. However, BLCR has not been tested with other threading packages, such as those used by some Java runtimes. We are interested in hearing of both success and failure with other threading packages.
Yes, starting with version 0.5.0 BLCR is able to save and restore groups of related processes together with the pipes that connect them. To do this, BLCR must be given a single request that covers all the processes involved. Currently there are three ways to specify a group request to BLCR:
- A process "tree", consisting of a process and all its non-orphaned descendants. (An orphaned process is one who's parent has exited, leaving it as a child of the init process).
- A POSIX process group, consisting of all processes with a given PGID (including orphaned processes, if any).
- A POSIX session, consisting of all processes with a given SID (including orphaned processes, if any).
While BLCR can save and restore the pipes used for IPC among processes in these groups, it is unable at this time to deal with most other IPC mechanisms (see next FAQ).
Yes. BLCR does not support checkpointing certain process resources. Most notably, BLCR will not checkpoint and/or restore open sockets (TCP/IP, Unix domain, etc.), or SysV IPC objects (man 5 ipc). Such resources are silently ignored at checkpoint time and are not restored.Additionally, BLCR does not correctly handle memory shared among processes that was obtained via mmap(MAP_SHARED) when there is no backing file (MAP_ANONYMOUS or when the backing file has been deleted). The restart may appear to succeed because the memory is restored to each process, but the sharing is lost. This is a high priority issue we will address in a future release.
Applications can arrange to save any necessary information and reacquire such resources at restart time (see next FAQ).
BLCR supports adding 'callbacks' to user-level code, which are called when a checkpoint is about to be performed, and when it is restarted (or continues on after the checkpoint). This is how MPI communication can be handled (see next FAQ).Full documentation of the callback interface is not yet documented because some of the interfaces are still subject to change. However, the comments in the file libcr.h should provide enough to get started.
Not by itself. But by using checkpoint callbacks (see previous FAQ). some MPI implementations, including LAM/MPI 7.x and MVAPICH 0.9.8, have made themselves checkpointable by BLCR. You can checkpoint/restart an MPI program running across an entire cluster of machines with BLCR, without any code modifications, if you use one of these MPI implementations.Support in OpenMPI (the successor to LAM/MPI) is not currently available (as of early February 2007). Support is planned for a later OpenMPI release.
Root access is needed to install the BLCR kernel modules. However, once these are installed, any user can checkpoint and restart their own programs without needing root permission.
Currently BLCR is integrated into the job system provided by the SciDAC Scalable Systems Software Suite (SSS).
Information on integration with SGE can be found here.
Work is ongoing by third parties to integrate BLCR into other batch systems. If you are interested in adding BLCR support to a job launcher/scheduler, please contact us!
Most of the architecture-specific code in BLCR is confined to small set of logic to save and restore the CPU-specific register set (vmadump) and some gcc inline assembly for atomic operations and special system calls. The majority of BLCR's code base is entirely processor-independent.
The BLCR team is doing a PPC port and is aware of a group doing a port to ARM. If you are interested in seeing BLCR run on other chips, and are able to devote programmer resources, please contact us! The Alpha platform is likely to be the easiest since vmadump already supports this architectures for Linux 2.6 kernels.
No. All of the kernel logic used by BLCR is implemented within kernel modules. You can thus compile BLCR and load it into a running kernel (with modprobe or insmod) without needing to recompile your kernel or reboot.
A machine that is running a supported architecture (only x86 and x86_64 for now) and Linux kernel (2.4 or 2.6 series).A set of configured kernel headers that matches the kernel you wish to build against. By configured, we mean that include/linux/version.h and the files in include/linux/modules/ match the target kernel. For many distributions (Red Hat in particular) a kernel-source package is often enough if using the vendor's kernel. For a custom kernel, the actual kernel build directory is usually required.
The kernel's symbol table. Normally the file /boot/System.map, or equivalent will serve this purpose.
BLCR needs to be able to examine a linux kernel source tree that has been configured, and this configuration must match the kernel that you will run BLCR against.If you do not have a configured linux kernel source tree, you may be able to create one fairly easily. Many distributions provide a 'config' file that is all you need to easily produce a configured kernel source tree. Good places to look for a config file include /boot/config-2.4.26-1-386 or /config-2.4.26-1-386. In some distributions, the kernel is actually setup to include its configuration in /proc/config.gz (or /proc/config.bz2). If you can find any one of these files then we can proceed with the following steps:
- Make a copy of the unconfigured source for the linux kernel you are using, and copy in the file you located:
$ cp -a /usr/src/kernel-source-2.4.26 /tmp/linux-2.4.26-1-386 $ cd /tmp/linux-2.4.26-1-386 $ cp [CONFIG_FILE] .config- Configure it using one of the following:
- For kernels 2.6.6 and newer:
$ make modules_prepare- For 2.6.x kernels prior to 2.6.6:
$ make prepare-all scripts- For 2.4.x kernels:
$ make oldconfig $ make dep $ make include/linux/version.h EXTRAVERSION=-1-386The 'make dep' step may take a significant amount of time. Note here that "-1-386" is whatever appears after the "2.X.Y" in the output of "uname -r".- Once that is done, you should be able to configure BLCR using the newly configured kernel source. You should continue to use the System.map file from your running kernel. What you want is probably something like
$ ./configure --with-system-map=/boot/System.map-2.4.26-1-386 --with-linux=/tmp/linux-2.4.26-1-386.
Please try rebuilding blcr after commenting out the following six lines near the top of the files vmadump/vmadump.c and blcr_imports/imports.c:
#if defined(CONFIG_MODVERSIONS) && ! defined(MODVERSIONS) #define MODVERSIONS #endif #if defined(MODVERSIONS) #include <linux/modversions.h> #endifLet us know if your compilation still doesn't work.
To build, BLCR needs to read the System.map file that correponds to the kernel you will use BLCR with. Generally, BLCR will find this file "automagically" during ./configure, but some distributions do not provide it, and/or you may not keep yours in a standard place.If you know where the correct System.map file is, use
$ ./configure --with-system-map=PATH_TO_YOUR_SYSTEM.MAPIf your System.map is absent, it may still be available as an optional RPM. Under Red Hat, for instance you may be able to get it by installing (depending on the release) either the kernel-source or kernel-devel RPMs for the kernel you will use BLCR with. For example, to build for a Red Hat 2.4.20-20.9 kernel, you need the kernel-source-2.4.20-20.9 RPM, and the kernel-2.4.20-20.9 RPM. Once these RPMs are installed, the BLCR configure script will generally find the files it needs automatically, so you won't need to pass any additional arguments to configure.
However, Fedora Core 2 and some of its derivatives are shipping a "stripped-down" System.map file. If this is the case, BLCR will abort during configuration with an error stating that the System.map cannot be used. You must install an additional RPM which contains a full System.map in order to build BLCR. In Fedora Core 2 the 'kernel-debuginfo' RPM contains a full System.map file, which it will install into the /usr/lib/debug/boot directory. BLCR's configure script will search this directory, but just to be certain you may still wish to pass '--with-system-map' to point configure at the correct System.map file.
Important Note: If you need to install the kernel-debuginfo RPM, make sure the correct version is installed. Specifically, the 'arch' type must be the same. If your kernel was built for the 'i386' (or 'i586', or 'i686'), the kernel-debuginfo RPM must have the same value. Thus, for an i586 kernel, install 'kernel-debuginfo-2.6.5-1.358.i586.rpm'. To determine which kernel version you have, use$ rpm -q kernel --qf '%{version}-%{release}.%{arch}\n'To make sure that you have installed compatible kernel and kernel-debuginfo RPMs, use$ rpm -q kernel kernel-debuginfo --qf '%{version}-%{release}.%{arch}\n'(replace 'kernel' with 'kernel-smp' if you are using an SMP kernel). You should see the same string, repeated twice.If you try to use BLCR with the wrong System.map, BLCR will build without complaints, but will probably detect the problem when the blcr.o kernel module is loaded (it does this by comparing some well-known exported kernel symbols' addresses to those provided by the System.map file), and the module load will be aborted.
Recent kernels check at module load time that the same compiler version (major and minor numbers) were used to build both the module and the kernel. This is the error you will see if they don't match. When this happens, you will need to reconfigure and build BLCR with the correct compiler. When a module fails to load due to a version mismatch, you should be able to find a message in the system logs indicating the required compiler version. Alternatively, the following should find the "signature" in existing kernel modules:$ find /lib/modules/`uname -r` -name '*.ko' -print | head -1 | xargs strings | grep gcc vermagic=2.6.17 SMP mod_unload PENTIUM4 gcc-3.2In this case a gcc-3.2.X version is required.Regardless of which method is used to find the correct version, you will need to reconfigure BLCR to use the correct compiler. To do so, rerun configure with the addition of "CC=/path/to/the/correct/gcc" to the command line.
This is because a resource needed into order to restart the process is already in use. The most common problem is that another process already exists with the same pid (process ID)--the operating system will not allow you to create two programs with the same pid. Very frequently this is because a user is trying to 'restart' a process from a checkpoint, when the original process they took the checkpoint of is still running!If you are unlucky enough that some other, unrelated process has grabbed the PID of your application, you must figure out some way to get rid of that process. If you own the process, you can of course simply kill it (or checkpoint it!). Otherwise, consider becoming root, or consulting your system administrator. BLCR will not kill another process for you (this 'feature' would raise certain security issues).
You should be able to restart a BLCR-checkpointed job on a different node (or set of nodes, for a parallel job), provided that all the nodes involved provide the exact same libraries and other files that your executable needs. BLCR does not save the contents of shared libraries that your program uses, nor does it save the contents of any files your program has open()'ed. But so long as all of these libraries and/or files exist on another node, your program should restart fine.Note that libraries must be exactly the same for a restart to work; if they are not the same size, for instance, restart will not work. If you've installed the same version of the operating system to all of your nodes (and you've updated them all the same way), you should be fine.
Finally, some versions of Linux are now using "prelinking", which is a method for assigning fixed addresses for shared libraries to load into executables. Prelinking is a feature--it enables applications that use many shared libraries to load faster. The fixed address used by the same library on different nodes is often deliberately randomized (in order to defeat buffer overflow attacks that could otherwise rely on standard libraries being loaded at the same address on every machine with the same OS version). Alas, if the prelinked addresses are different, you will not be able to restart BLCR checkpoints on another node. The solution for this is to turn off prelinking. This is a systemwide setting, so you will need to be root. On Fedora Core 2, at least, the fix is to edit /etc/sysconfig/prelink and set 'PRELINKING=no'. The comments claim that this will cause prelinking to be undone automatically the next night. We've never been patient, and instead "undo" prelinking immediately by running (as root)
# /usr/sbin/prelink --undo --all".
This error was possible in older releases of BLCR when an applciation was not checkpointable. This should not happen in release 0.5.0 or newer and should be reported as a bug if seen.
By default, LAM/MPI will use $HOME as the location for storing the checkpoint files for all the processes involved in an MPI job, unless it was configured at build time with 'configure --with-cr-file-dir=/somewhere/else', in which case '/somewhere/else' will be the default location. So rebuilding LAM is one (rather slow and painful) way to change where checkpoints are stored.A much easier solution is to set the LAM 'cr_base_dir' SSI parameter for each individual job that you wish to have use a different directory for storing checkpoints. This can either be done by setting the '$LAM_MPI_SSI_cr_base_dir' environment variable to the full path of the directory you want to use, or by setting the 'cr_base_dir' parameter on the command line:
$ mpirun -np 2 -ssi cr_base_dir /somewhere/else a.outSee the LAM Documentation for more details, especially the "Available MPI Modules | Checkpoint/Restart of MPI Jobs" section in the User's Guide.
To download the BLCR software, or for links to all the available information about BLCR, please visit http://ftg.lbl.gov/checkpoint.
There is a mailing list of BLCR developers and some of the users at checkpoint@lbl.gov and which is archived here.This list is managed my majordomo. So, to subscribe just e-mail "subscribe checkpoint" (in the message body and without the quotes) to majordomo@lbl.gov.
If you think you've found a bug in BLCR, please do let us know about it. There are many kernel-dependencies in BLCR and we could easly have missed testing on a system like yours. We count on user's bug reports to help ensure wide testing coverage.The BLCR bug database is managed by a Bugzilla, located at http://mantis.lbl.gov/bugzilla.
Before reporting a bug, you are encouraged to search the database to see if a bug report exists for your problem. For some issues a solution can be found in just a day. So, a patch to fix your problem may already be attached to an existing bug report. BLCR is just one of a group of projects managed on this server, so be sure to select product "BLCR" in your queries.