Berkeley eXtensible Processing Engine (BXPE)
FPGA-Accelerated Supercomputing on the Edge for NCEM and ALS High Performance Sensors
With experimental facilities data production rates exponentially increasing, there is a greater need to move computation closer to the experimental source. To this end, we have developed the Berkeley eXtensible Processing Engine (BXPE) Framework, which allows experimentalists to utilize FPGAs stitched into the network pipeline to process data in real-time as it flows over the network. We provide a Python-tooling front-end to allow for development of algorithms using this framework. In this particular design, we ported the NCEM Center-of-Mass and the ALS convolution applications to this edge computing framework.
Experimental facilities like the Advanced Light Source (ALS) and National Center for Electron Microscopy (NCEM) are experiencing double-exponential increases in data production rates from emerging detectors. The volume and rate of these increasingly large experimental dataset threatens to overwhelm both wide area networks and HPC center ingest rates, as shown in Fig. 2. For example, the 4D Camera at NCEM utilizes 48 10Gb/s interfaces for transmitting data acquired from the A/D over the link. Much like how experiments today benefit from custom sensors and associated accelerators/algorithms, future experiments will benefit from specialized compute to analyze and reduce the incoming data. Edge computing techniques offer an opportunity to reduce the load by moving some of the preprocessing closer to the detectors.
We asked our collaborators for examples of preprocessing currently in use. NCEM provided us a phase contrast electron microscopy imaging technique based on the Center-of-Mass of many detector frames, as shown in Fig. 3, as well as a sample of data obtained from their detector. ALS similarly provided us a Convolution application, as well as sample data. Both of these were implemented as Jupyter Notebooks, which provides a simple Python interface for their application. Since most of their development is in Python, maintaining that interface is critical for ease of use.
However, developing directly on FPGA hardware typically requires a hardware-design background, and it is outside the scope for domain scientists. There is great need to efficiently migrate algorithms developed in Python to the FPGA. Thus any developed framework must provide an easy interface for domain scientists to develop and to test their algorithms on this framework, as shown in Fig. 4.
We developed the Berkeley eXtensible Processing Engine (BXPE) Framework, an FPGA-based edge network processing pipeline for real-time data processing. Based on the Xilinx Alveo U250/U280 FPGA-based accelerator cards, this edge computing framework provides two 100Gb/s Ethernet connections as well as up to ~10,000 DSP primitives. To provide access to these resources, we developed a Python-based tooling to allow users to port their algorithms to this framework with a software-based simulator and compiler, as well as configuring the FPGA-based hardware array. Using a co-design methodology, we developed a DSP processing engine with the BXPE framework based on the NCEM Center-of-Mass and ALS convolution applications. A block diagram of the hardware architecture is shown in Fig. 1.
For more details about the project, please read the LDRD Final Report.
- Chin Guok (PI - ESnet)
- Anastasiia Butko
- Peter Ercius (NCEM)
- Farzad Fatollahi-Fard
- Yatish Kumar (ESnet)
- Chris Neely (Xilinx/AMD)
- Nirmalendu Bikash Patra
- John Shalf
- David Shapiro (ALS)
Download File »
FPGA-Accelerated Supercomputing on the Edge for NCEM and ALS
LDRD Final Report for the FPGA-Accelerated Supercomputing on the Edge for NCEM and ALS Project
About Berkeley Lab
Founded in 1931 on the belief that the biggest scientific challenges are best addressed by teams, Lawrence Berkeley National Laboratory and its scientists have been recognized with 14 Nobel Prizes. Today, Berkeley Lab researchers develop sustainable energy and environmental solutions, create useful new materials, advance the frontiers of computing, and probe the mysteries of life, matter, and the universe. Scientists from around the world rely on the Lab’s facilities for their own discovery science. Berkeley Lab is a multiprogram national laboratory, managed by the University of California for the U.S. Department of Energy’s Office of Science.
DOE’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit energy.gov/science.