Berkeley eXtensible Processing Engine (BXPE)

With experimental facilities data production rates exponentially increasing, there is a greater need to move computation closer to the experimental source. To this end, we have developed the Berkeley eXtensible Processing Engine (BXPE) Framework, which allows experimentalists to utilize FPGAs stitched into the network pipeline to process data in real-time as it flows over the network. We provide a Python-tooling front-end to allow for development of algorithms using this framework. In this particular design, we ported the NCEM Center-of-Mass and the ALS convolution applications to this edge computing framework.

Fig. 2: Current HPC Centers cannot keep up with current data rates.

Experimental facilities like the Advanced Light Source (ALS) and National Center for Electron Microscopy (NCEM) are experiencing double-exponential increases in data production rates from emerging detectors. The volume and rate of these increasingly large experimental dataset threatens to overwhelm both wide area networks and HPC center ingest rates, as shown in Fig. 2. For example, the 4D Camera at NCEM utilizes 48 10Gb/s interfaces for transmitting data acquired from the A/D over the link. Much like how experiments today benefit from custom sensors and associated accelerators/algorithms, future experiments will benefit from specialized compute to analyze and reduce the incoming data. Edge computing techniques offer an opportunity to reduce the load by moving some of the preprocessing closer to the detectors.

Fig. 3: Example Center-of-Mass Application Jupyter Notebook from NCEM.

We asked our collaborators for examples of preprocessing currently in use. NCEM provided us a phase contrast electron microscopy imaging technique based on the Center-of-Mass of many detector frames, as shown in Fig. 3, as well as a sample of data obtained from their detector. ALS similarly provided us a Convolution application, as well as sample data. Both of these were implemented as Jupyter Notebooks, which provides a simple Python interface for their application. Since most of their development is in Python, maintaining that interface is critical for ease of use.

Fig. 4: The edge is part of a continuum of computing workflows.

However, developing directly on FPGA hardware typically requires a hardware-design background, and it is outside the scope for domain scientists. There is great need to efficiently migrate algorithms developed in Python to the FPGA. Thus any developed framework must provide an easy interface for domain scientists to develop and to test their algorithms on this framework, as shown in Fig. 4.

We developed the Berkeley eXtensible Processing Engine (BXPE) Framework, an FPGA-based edge network processing pipeline for real-time data processing. Based on the Xilinx Alveo U250/U280 FPGA-based accelerator cards, this edge computing framework provides two 100Gb/s Ethernet connections as well as up to ~10,000 DSP primitives. To provide access to these resources, we developed a Python-based tooling to allow users to port their algorithms to this framework with a software-based simulator and compiler, as well as configuring the FPGA-based hardware array. Using a co-design methodology, we developed a DSP processing engine with the BXPE framework based on the NCEM Center-of-Mass and ALS convolution applications. A block diagram of the hardware architecture is shown in Fig. 1. 

For more details about the project, please read the LDRD Final Report.

Project Participants

• Chin Guok (PI - ESnet)
• Anastasiia Butko
• Peter Ercius (NCEM)
• Farzad Fatollahi-Fard
• Yatish Kumar (ESnet)
• Chris Neely (Xilinx/AMD)
• Nirmalendu Bikash Patra
• John Shalf
• David Shapiro (ALS)