A Data Science Pioneer, Deb Agarwal Retires From Berkeley Lab

Deborah Agarwal has always taken a pragmatic approach to her career. Throughout her life, she had three guiding principles for determining whether she’d accept or stay at a job: Is the work fun? Is the research topic interesting? Will I be working with people I like?

The answer to these questions was a resounding “yes” in 1992 when Agarwal accepted an internship to work with Van Jacobson in Lawrence Berkeley National Laboratory’s (Berkeley Lab’s) Networking Research Group; and again in 1994 when she turned down academic job offers to work with William (Bill) Johnston in the Lab’s Information and Computing Sciences Division.

Now more than three decades since she first arrived as an intern, Agarwal will retire as Director of Berkeley Lab’s Scientific Data Division on July 1. A celebration of her Lab career will be held on July 7 in Berkeley.

“Looking back, I’ve really loved the opportunity to live the ethos of Ernest Lawrence’s original plan: big teams working on big problems. I’ve enjoyed immersing myself in team science, and I’m proud of having been able to help revitalize some of that ethos throughout my career at Berkeley Lab,” said Agarwal.

Winding Road Through Engineering

Deb Agarwal, named women at the Lab in 2013

When Agarwal was growing up, many of the engineering fields she would eventually work in (robotics, networking, data science) didn’t exist or were in their infancy, so she never strove for careers in them. Instead, she found her place in these domains by sticking to her three guiding questions.

Although Agarwal came from an engineering family—her father was an electrical engineer at General Motors (GM) Research Laboratories and patented numerous inventions; all her sisters became successful engineers—she didn’t always aspire to be an engineer. But that changed when she was invited to a week-long summer program at the Rensselaer Polytechnic Institute designed to introduce high school students to different engineering disciplines.

That summer, Agarwal was one of 30 students participating in the program. “The University had a big mainframe computer, and we spent the mornings learning to program and playing games on the system. The afternoons were dedicated to exposing us to a particular engineering field: industrial engineering one day, civil engineering the next, electrical, and so on; we’d meet the professors, learn about their research, then visit their Labs. It was very cool,” she said.

After completing the program, she methodically went through the disciplines presented that week and decided to pursue her undergraduate degree in mechanical engineering at Purdue University. Before applying for graduate school, Agarwal joined GM’s advanced engineering staff and put her skills toward helping the company figure out how to quickly and effectively integrate robotics into their automotive assembly line.

“I picked up a lot of computer science skills during my time at GM because I was working on offline programming these robots before they were integrated into the assembly line,” she said.

Agarwal planned to continue with mechanical engineering and automated assembly research when she arrived at UC Santa Barbara for graduate school, but her professor didn’t have any expertise in robotics or manufacturing. His background was in controls theory, so after reading about the challenging problems of fly-by-wire for jets, she decided it’d be fun to work on that. To better understand digital systems, she took an undergraduate course in digital electronics, where she learned things like assembly language programming, digital logic, how to create field programmable gate arrays, etc.

Deb Agarwal welcomes Berkeley Lab Computing Science's 2023 Summer Students.

“At that point, I was not enjoying my graduate-level mechanical engineering courses, but this digital electronics course was so much fun. I thought about it for a while and decided to switch my major to electrical and computer engineering,” said Agarwal. “Fortunately, my digital electronics professor was also assistant dean of the UCSB College of Engineering and helped me to switch my major.”

Once she changed her major, Agarwal knew it would be a while before she obtained her Ph.D. So she took an analytical approach to selecting her graduate student research project. “I wanted to like the people I worked with, and I wanted an interesting research topic. After these two criteria, I looked at who had funding, then began doing informational interviews,” she said. “The people who ended up being my professors were in the multicast protocol space, which was a really interesting topic in the early ‘90s, so I decided to do that.”

The pivot into multicast networking eventually led Agarwal to accept an internship in Berkeley Lab’s Network Research Group in 1992, where she spent a month working on—the Multicast Backbone (M-Bone)—with Van Jacobson. In addition to enabling researchers worldwide to collaborate in real-time, throughout the 1990s M-Bone’s early videoconferencing tool was also used to live-stream a Rolling Stones Voodoo Lounge Concert and connect US elementary school students to South Pole researchers for the first time.

Berkeley Lab’s Remote Access Grid Entity (RAGE), a roaming telepresence robot to enhance two-way interactive audio and video communication between groups connected by the Internet, was unveiled in 2001 and awarded an R&D100 Award. Here’s the development team: Front row, left to right: Deb Agarwal, Eve Edelson, Marcia Perry. Back row, left to right: Dan Gunter, Clayton Bagwell, Keith R. Jackson, Martin Stoufer, Zach Radding, Joshua Boverhof, and John Shalf.

Building a Career at Berkeley Lab, “I’ll Leave When It Stops Being Fun”

After earning her Ph.D., Agarwal thought she’d pursue an academic career. She applied for professorships at several universities and even received some offers, but then Berkeley Lab’s Bill Johnston reached out to her with an intriguing project.

“Bill had just gotten a project funded to enable remote experimentation at the Advanced Light Source’s (ALS) beamline 7, and he wanted me to run the project,” said Agarwal. “I was pretty tired after getting my Ph.D., and I knew it would be a lot of work to build an academic career: teaching classes, building up a team of graduate students, and applying for funding. But with this Berkeley Lab job, I’d get to apply my thesis work to a problem; I wouldn’t have to write a proposal; I’d just jump in and do the fun part.”

In 2005, Agarwal and Craig Leres built the Keck Remote Observing Facility, a room in Building 50B filled with computers that allowed researchers to control telescopes located 3,800 miles away in Mauna Kea, Hawaii. Agarwal is pictured here with Tony Spadafora, the scientific coordinator of the Keck Remote Observing Facility project at Berkeley Lab.

Building on her graduate work, where she developed a reliable multicast protocol, Agarwal spent the next few years applying her expertise to developing collaborative software tools that allowed researchers at different locations to collaborate on experiments as if they were in the same laboratory. And the ALS’s Spectro-Microscopy Facility Beamline 7.0 at Berkeley Lab was the pilot application for these tools.

Deb Agarwal was named one of the Top 25 Women of the Web in the May 2000 cover story by Upside magazine for her work with the Comprehensive Test Ban Treaty.

In the late 1990s, the Comprehensive Test Ban Treaty Organization asked Agarwal to help them determine if “reliable multicast” technology was a cost-effective way of handling the large amount of data involved in verifying that no nation detonates a nuclear device. For this work, Agarwal was named one of the Top 25 Women of the Web in the May 2000 cover story by Upside magazine.

Once Agawal’s ALS project finished, another exciting opportunity popped up with DOE 2000. At the turn of the millennium, the Department of Energy sought to fund research on technologies essential to supporting collaborations with experiments.

“I felt really well situated for this initiative because I had the experience of working on similar projects at the ALS. So we proposed follow-up work to some tools we developed for the ALS, and we did very well,” said Agarwal. “We got some really fun projects, so instead of racing back to academia, my perspective shifted to ‘I will leave when I stop having fun.’”

A Reluctant Leader Pioneers a Path Forward

For Agarwal, the fun almost stopped in 2006. That’s when she learned that her DOE program manager was retiring, and their replacement would be steering funding away from the collaborative research that she enjoyed doing and focusing more on network research related to optical fibers, a topic that didn’t interest her as much. This pivot also meant that many of her colleagues would lose their funding. Around the same time, Johnston became the Energy Sciences Network’s Director; and Associate Laboratory Director for Computing Sciences Horst Simon asked Agarwal to head the Distributed Systems Department.

“My first reaction was no; I don’t want to be a department head. But after giving it some thought, I realized that our department would lose all our funding in 18 months; it would take at least six months to find and hire someone for the job, which means whoever gets hired would come into the job with one year of funding left and not know anybody, this wasn’t a good position to put our people in,” said Agarwal. “So I accepted the department head job to make sure that someone was taking care of our people. I figured I’d help close things down and leave within a year, hoping that academia would still be interested.”

After accepting the role of department head, Agarwal enrolled in the Executive Training Program at UC Berkeley’s Haas Business School, where they taught the importance of strategic planning. She took the concept back to her department colleagues and asked them: “If we could reinvent ourselves, what would you be?”

“One said: I want to matter to science…If you look back at the tools we’ve built to date, I will bet that science only uses about 10 to 20 percent of them. My first reaction was: no, that can’t be true! I just worked on this huge ALS project, but when I started asking around, I found that this observation was accurate,” said Agarwal. “It made a lot of sense; ASCR (DOE’s Office of Advanced Scientific Computing Research) only really funded us to build the prototype. So we might create something fairly usable, but we weren’t funded to make it production or maintain it. So we couldn’t be useful to science because they couldn’t depend on our software; we moved on to our next project once we were done with the prototype.”

There was also the last mile problem. “Once we’ve built the prototype, we’ve done the hard part but may not have solved the whole problem. They may still need a data format translator or something else to use our tool. We assumed they could hire a student to build that,” said Agarwal. “But they never hired the student to build the missing component or to maintain the software, and our tools weren’t used.”

The Microsoft e-Science team 2007: Front row, Microsoft’s Catharine van Ingen (left), and Stuart Ozer. Second row (l-r) Matt Rodriguez, Deb Agarwal, and Monte Good with the Computational Research Division.

That’s when Agarwal and the department started engaging directly with scientific research groups, asking what they needed and then asking them to fund her team to build and maintain the technology. Their first project was with researchers in Berkeley Lab’s Particle Data Group to revamp the web interface and infrastructure of the Particle Data Reference.

“At the time, a father and son team had built a prototype in a framework that no one in the rest of the world ever used, so this system was unmaintainable. Without these two people, the system couldn’t be updated or changed. So the physicists offered to pay our staff to redo the interface,” said Agarwal. “With this project and a few others, we enabled about half of the department to stay; others had enough warning that most landed great jobs at universities, other groups at Berkeley Lab, and in industry. Over time, some even came back.”

This pioneering pivot to user-centric design and development —where computing researchers work closely with domain scientists to understand how they think and work and then develop software solutions tailored to their needs—combined with the exponential growth in data produced by experimental and observational science instruments and advancements in data science transformed Agarwal’s department. Over the next decade, its staff built partnerships in every scientific area at the Lab; and it grew so much in personnel and stature that it eventually helped pave the way for a new Data Science Division at Berkeley Lab in 2021 with Agarwal as the interim lead.

“Realizing that we weren’t really helping science fundamentally shifted our perspective. We recognized that we couldn’t just use our hammer or screwdriver to solve a science problem; we needed to understand the problem and context to then decide if there is an existing tool or if we have to build a new solution,” said Agarwal.

She adds that “being a part of this effort to rebuild the department in a challenging time and helping it become something that, I think, is so much better: an organization that tries to live Lawrence’s team science philosophy has been one of the biggest highlights of my career. I can only claim to have advocated for this philosophy; the team came up with the concept and made it happen.”

Deb Agarwal at the Open and Transparent Water Data Act (AB 1755) workshop on WED, NOV 29, 2017 in Berkeley, Calif.

Around the time of the department’s revitalization, Agarwal was approached to participate in several Microsoft e-Science project collaborations that were fun, interesting, and involved working with people she liked. The projects—Berkeley Water Center, Ameriflux, and Fluxnet—involved developing systems for amassing and organizing environmental data from diverse sources to accelerate climate change research. These projects’ successes eventually led to DOE-funded efforts like the Environmental System – Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE). Agarwal’s work also led to an appointment on the inaugural steering committee of the California Water Data Consortium (CWDC), where she would help define a statewide integrated water data platform, develop protocols for data sharing, documentation, quality control, public access, and promotion of open-source platforms and decision support tools related to California’s water data.

In 2019, Agarwal was part of an LDRD-funded team developing a computational tool that will allow groundwater managers to plan and manage their precious water resources more sustainably, leading to improved resistance to droughts. The team is pictured here: Juliane Mueller, Bhavna Arora, Charuleka Varadharajan, Deborah Agarwal, and Reetik Sahu.

In the last few years, Agarwal has also held leadership roles in the Institute for the Design of Advanced Energy Systems (IDAES) project focused on improving the efficiency of power plants and DOE’s Carbon Capture Simulation Initiative (CCSI) that accelerates the development and deployment cycle for new carbon capture technologies. She was also briefly the Deputy Task Area Lead for Modeling and Simulation for the new National Alliance for Water Innovation (NAWI), a $100 million hub to advance technologies to treat nontraditional water sources.

Diversifying Science and Engineering

Throughout her Berkeley Lab career, Agarwal has also advocated for diversifying the scientific and engineering workforce. She has actively organized and participated in mentoring programs to help underrepresented groups enter and succeed in these domains.

“Up to the point of getting my Ph.D. I always believed that to succeed, you just have to be better; that’s life. Then, I went to the first Grace Hopper Conference in 1994, and my perspective completely changed,” said Agarwal. “I met a lot of brilliant people; one was a woman in a wheelchair, she had to overcome many things to succeed, and it was clear that working harder wasn’t the right answer for her—the playing field wasn’t fair. That’s when I realized that I had to care about inclusion, and I needed to do something to change perceptions in the field.”

Agarwal (front row right) with the Computing Sciences Staff at the 2013 Grace Hopper Conference.

Agarwal adds that the conference was one of the first times she realized that there were role models in the field of computer science—like Frances Allen and Barbara Simons—that she could admire, and they even came to her poster presentation.

This experience inspired Agarwal to participate in and advocate for a Berkeley Lab presence at the Grace Hopper Conference and the Tapia Celebration of Diversity in Computing for many years. She also played a leading role in developing the Computing Sciences Mentorship Program and has been a champion of many other Lab IDEA (inclusion, diversity, equity, and accountability) initiatives.

“We’ve reached a point where we’re getting more women to apply for our job openings, and we’ve gotten more women in our senior ranks—it’s better. But, we’ve got a long way to go toward making progress in increasing representation among other historically marginalized groups,” said Agarwal. She’s so invested in this effort that her first act in retirement is participating in a workshop on how the Lab and the field can progress in this area.

In the first few months of her retirement, Agarwal and her wife will take a road trip across the country to camp at national parks and with family in Michigan.