Publications

Hector G. Martin, Tijana Radivojevic, Jeremy Zucker, Kristofer Bouchard, Jess Sustarich, Sean Peisert, Dan Arnold, Nathan Hillson, Gyorgy Babnigg, Jose M. Marti, Christopher J. Mungall, Gregg T. Beckham, Lucas Waldburger, James Carothers, ShivShankar Sundaram, Deb Agarwal, Blake A. Simmons, Tyler Backman, Deepanwita Banerjee, Deepti Tanjore, Lavanya Ramakrishnan, Anup Singh, "Perspectives for Self-Driving Labs in Synthetic Biology", Current Opinion in Biotechnology, February 2023, doi: 10.1016/j.copbio.2022.102881

MB Simmonds, WJ Riley, DA Agarwal, X Chen, S Cholia, R Crystal-Ornelas, ET Coon, D Dwivedi, VC Hendrix, M Huang, A Jan, Z Kakalia, J Kumar, CD Koven, L Li, M Melara, L Ramakrishnan, DM Ricciuto, AP Walker, W Zhi, Q Zhu, C Varadharajan, Guidelines for Publicly Archiving Terrestrial Model Data to Enhance Usability, Intercomparison, and Synthesis, Data Science Journal, 2022, doi: 10.5334/dsj-2022-003

C Varadharajan, VC Hendrix, DS Christianson, M Burrus, C Wong, SS Hubbard, DA Agarwal, BASIN-3D: A brokering framework to integrate diverse environmental data, Computers and Geosciences, 2022, doi: 10.1016/j.cageo.2021.105024

B Faybishenko, R Versteeg, G Pastorello, D Dwivedi, C Varadharajan, D Agarwal, Challenging problems of quality assurance and quality control (QA/QC) of meteorological time series data, Stochastic Environmental Research and Risk Assessment, Pages: 1049--1062 2022, doi: 10.1007/s00477-021-02106-w

F Molz, B Faybishenko, D Agarwal, A broad exploration of nonlinear dynamics in microbial systems motivated by chemostat experiments producing deterministic chaos., 2022,

C Varadharajan, Z Kakalia, E Alper, EL Brodie, M Burrus, RWH Carroll, D Christianson, W Dong, V Hendrix, M Henderson, S Hubbard, D Johnson, R Versteeg, KH Williams, DA Agarwal, The Colorado East River Community Observatory Data Collection, Hydrological Processes 35(6), 2021, doi: 10.22541/au.161962485.54378235/v1

D. A. Agarwal, J. Damerow, C. Varadharajan, D. S. Christianson, G. Z. Pastorello, Y.-W. Cheah, L. Ramakrishnan, "Balancing the needs of consumers and producers for scientific data collections", Ecological Informatics, 2021, 62:101251, doi: 10.1016/j.ecoinf.2021.101251

J Müller, B Faybishenko, D Agarwal, S Bailey, C Jiang, Y Ryu, C Tull, L Ramakrishnan, Assessing data change in scientific datasets, Concurrency and Computation: Practice and Experience, 2021, doi: 10.1002/cpe.6245

SL Brantley, T Wen, DA Agarwal, JG Catalano, PA Schroeder, K Lehnert, C Varadharajan, J Pett-Ridge, M Engle, AM Castronova, RP Hooper, X Ma, L Jin, K McHenry, E Aronson, AR Shaughnessy, LA Derry, J Richardson, J Bales, EM Pierce, The future low-temperature geochemical data-scape as envisioned by the U.S. geochemical community, Computers and Geosciences, 2021, doi: 10.1016/j.cageo.2021.104933

JE Damerow, C Varadharajan, K Boye, EL Brodie, M Burrus, KD Chadwick, R Crystal-Ornelas, H Elbashandy, RJ Eloy Alves, KS Ely, AE Goldman, T Haberman, V Hendrix, Z Kakalia, KM Kemner, AB Kersting, N Merino, F O Brien, Z Perzan, E Robles, P Sorensen, JC Stegen, RL Walls, P Weisenhorn, M Zavarin, D Agarwal, Sample identifiers and metadata to support data management and reuse in multidisciplinary ecosystem sciences, Data Science Journal, 2021, doi: 10.5334/dsj-2021-011

G. Z. Pastorello, C. Trotta, E. Canfora, H. Chu, D. Christianson, Y.-W. Cheah, C. Poindexter, J. Chen, A. Elbashandy, M. Humphrey, P. Isaac, D. Polidori, M. Reichstein, A. Ribeca, C. van Ingen, N. Vuichard, L. Zhang, B. Amiro, C. Ammann, M. A. Arain, J. Ardö, T. Arkebauer, S. K. Arndt, N. Arriga, M. Aubinet, M. Aurela, D. Baldocchi, A. Barr, E. Beamesderfer, L. B. Marchesini, O. Bergeron, J. Beringer, C. Bernhofer, D. Berveiller, D. Billesbach, T. A. Black, P. D. Blanken, G. Bohrer, J. Boike, P. V. Bol stad, D. Bonal, J.-M. Bonnefond, D. R. Bowling, R. Bracho, J. Brodeur, C. Brümmer, N. Buchmann, B. Burban, S. P. Burns, P. Buysse, P. Cale, M. Cavagna, P. Cellier, S. Chen, I. Chini, T. R. Chris tensen, J. Cleverly, A. Collalti, C. Consalvo, B. D. Cook, D. Cook, C. Coursolle, E. Cremonese, P. S. Curtis, E. D’Andrea, H. da Rocha, X. Dai, K. J. Davis, B. D. Cinti, A. de Grandcourt, A. D. Ligne, R. C. D. Oliveira, N. Delpierre, A. R. Desai, C. M. D. Bella, P. di Tommasi, H. Dolman, F. Domingo, G. Dong, S. Dore, P. Duce, E. Dufrêne, A. Dunn, J. Dušek, D. Eamus, U. Eichelmann, H. A. M. ElKhidir, W. Eugster, C. M. Ewenz, B. Ewers, D. Famulari, S. Fares, I. Feigenwinter, A. Feitz, R. Fensholt, G. Fil ippa, M. Fischer, J. Frank, M. Galvagno, M. Gharun, D. Gianelle, B. Gielen, B. Gioli, A. Gitelson, I. Goded, M. Goeckede, A. H. Goldstein, C. M. Gough, M. L. Goulden, A. Graf, A. Griebel, C. Gruening, T. Grünwald, A. Hammerle, S. Han, X. Han, B. U. Hansen, C. Hanson, J. Hatakka, Y. He, M. Hehn, B. Heinesch, N. Hinko-Najera, L. Hörtnagl, L. Hutley, A. Ibrom, H. Ikawa, M. Jackowicz-Korczynski, D. Janouš, W. Jans, R. Jassal, S. Jiang, T. Kato, M. Khomik, J. Klatt, A. Knohl, S. Knox, H. Kobayashi, G. Koerber, O. Kolle, Y. Kosugi, A. Kotani, A. Kowalski, B. Kruijt, J. Kurbatova, W. L. Kutsch, H. Kwon, S. Launiainen, T. Laurila, B. Law, R. Leuning, Y. Li, M. Liddell, J.-M. Limousin, M. Lion, A. J. Liska, A. Lohila, A. López-Ballesteros, E. López-Blanco, B. Loubet, D. Loustau, A. Lucas-Moffat, J. Lüers, S. Ma, C. Macfarlane, V. Magliulo, R. Maier, I. Mammarella, G. Manca, B. Marcolla, H. A. Margolis, S. Mar ras, W. Massman, M. Mastepanov, R. Matamala, J. H. Matthes, F. Mazzenga, H. McCaughey, I. McHugh, A. M. S. McMillan, L. Merbold, W. Meyer, T. Meyers, S. D. Miller, S. Minerbi, U. Moderow, R. K. Monson, L. Montagnani, C. E. Moore, E. Moors, V. Moreaux, C. Moureaux, J. W. Munger, T. Nakai, J. Neirynck, Z. Nesic, G. Nicolini, A. Noormets, M. Northwood, M. Nosetto, Y. Nouvellon, K. Novick, W. Oechel, J. E. Olesen, J.-M. Ourcival, S. A. Papuga, F.-J. Parmentier, E. Paul-Limoges, M. Pavelka, M. Peichl, E. Pendall, R. P. Phillips, K. Pilegaard, N. Pirk, G. Posse, T. Powell, H. Prasse, S. M. Prober, S. Ram bal, U. Rannik, N. Raz-Yaseef, D. Reed, V. R. de Dios, N. Restrepo-Coupe, B. R. Reverter, M. Roland, S. Sabbatini, T. Sachs, S. R. Saleska, E. P. S.-C. nete, Z. M. Sanchez-Mejia, H. P. Schmid, M. Schmidt, K. Schneider, F. Schrader, I. Schroder, R. L. Scott, P. Sedlák, P. Serrano-Ortíz, C. Shao, P. Shi, I. Shironya, L. Siebicke, L. Šigut, R. Silberstein, C. Sirca, D. Spano, R. Steinbrecher, R. M. Stevens, C. Sturtevant, A. Suyker, T. Tagesson, S. Takanashi, Y. Tang, N. Tapper, J. Thom, F. Tiedemann, M. Tomassucci, J.-P. Tuovinen, S. Urbanski, R. Valentini, M. van der Molen, E. van Gorsel, K. van Huissteden, A. Varlagin, J. Verfaillie, T. Vesala, C. Vincke, D. Vitale, N. Vygodskaya, J. P. Walker, E. Walter-Shea, H. Wang, R. Weber, S. Westermann, C. Wille, S. Wofsy, G. Wohlfahrt, S. Wolf, W. Woodgate, Y. Li, R. Zampedri, J. Zhang, G. Zhou, D. Zona, D. Agarwal, S. Biraud, M. Torn, D. Papale, "The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data", Scientific Data, 2020, 7:225, doi: 10.1038/s41597-020-0534-3

P. Linton, W. Melodia, A. Lazar, D. Agarwal, L. Bianchi, D. Ghoshal, K. Wu, G. Pastorello, L. Ramakrishnan, "Identifying Time Series Similarity in Large-Scale Earth System Datasets", The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19), 2019,

C Varadharajan, S Cholia, C Snavely, V Hendrix, C Procopiou, D Swantek, W Riley, D Agarwal, "Launching an Accessible Archive of Environmental Data", Eos, January 1, 2019, 100, doi: 10.1029/2019eo111263

Payton A Linton, William M Melodia, Alina Lazar, Deborah Agarwal, Ludovico Bianchi, Devarshi Ghoshal, Kesheng Wu, Gilberto Pastorello, Lavanya Ramakrishnan, "Identifying Time Series Similarity in Large-Scale Earth System Datasets", 2019,

Payton Linton, William Melodia, Alina Lazar, Deborah Agarwal, Ludovico Bianchi, Devarshi Ghoshal, Gilberto Pastorello, Lavanya Ramakrishnan, Kesheng Wu, Understanding Data Similarity in Large-Scale Scientific Datasets, 2019 IEEE International Conference on Big Data (Big Data), Pages: 4525--4531 2019,

C Varadharajan, B Faybishenko, A Henderson, M Henderson, VC Hendrix, SS Hubbard, Z Kakalia, A Newman, B Potter, H Steltzer, R Versteeg, DA Agarwal, KH Williams, C Wilmer, Y Wu, W Brown, M Burrus, RWH Carroll, DS Christianson, B Dafflon, D Dwivedi, BJ Enquist, Challenges in Building an End-to-End System for Acquisition, Management, and Integration of Diverse Data from Sensor Networks in Watersheds: Lessons from a Mountainous Community Observatory in East River, Colorado, IEEE Access, Pages: 182796--18 2019, doi: 10.1109/ACCESS.2019.2957793

B Faybishenko, F Molz, D Agarwal, "Nonlinear dynamics simulations of microbial ecological processes: Model, diagnostic parameters of deterministic chaos, and sensitivity analysis", Springer Proceedings in Mathematics and Statistics, ( 2018) Pages: 437--465 doi: 10.1007/978-3-030-02825-1_19

DC Miller, JD Siirola, D Agarwal, AP Burgard, A Lee, JC Eslick, B Nicholson, C Laird, LT Biegler, D Bhattacharyya, NV Sahinidis, IE Grossmann, CE Gounaris, D Gunter, "Next Generation Multi-Scale Process Systems Engineering Framework", Computer Aided Chemical Engineering, 2018, 44:2209--2214, doi: 10.1016/B978-0-444-64241-7.50363-3

D Ghoshal, L Ramakrishnan, D Agarwal, "Dac-Man: Data Change Management for Scientific Datasets on HPC Systems", SC ’18, Piscataway, NJ, USA, IEEE Press, 2018, 72:1--72:1,

Gilberto Z. Pastorello, Dario Papale, Housen Chu, Carlo Trotta, Deb A. Agarwal, Eleonora Canfora, Dennis D. Baldocchi, M. S. Torn, "A new data set to keep a sharper eye on land-air exchanges", Eos, 2017, 98:28-32, doi: 10.1029/2017EO071597

Danielle S. Christianson, Charuleka Varadharajan, Bradley Christoffersen, Matteo Detto, Faybishenko, Bruno O. Gimenez, Val C. Hendrix, Kolby J. Jardine, Robinson Negron-Juarez, Z. Pastorello, Thomas L. Powell, Megha Sandesh, Jeffrey M. Warren, Brett T. Wolfe, Jeffrey Q. Chambers, Lara M. Kueppers, Nathan G. McDowell, Deborah A. Agarwal, "A metadata reporting framework (FRAMES) for synthesis of ecohydrological observations", Ecological Informatics, 2017, 42:148-158, doi: 10.1016/j.ecoinf.2017.06.002

DA Agarwal, C Varadharajan, S Cholia, C Snavely, VC Hendrix, D Gunter, WJ Riley, M jones, AE budden, D Vieglas, Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE)-A New US DOE Data Archive, 2017,

D Guyon, AC Orgerie, C Morin, D Agarwal, "How Much Energy Can Green HPC Cloud Users Save?", Proceedings - 2017 25th Euromicro International Conference on Parallel, Distributed and Network-Based Processing, PDP 2017, 2017, 416--420, doi: 10.1109/PDP.2017.62

YW Cheah, J Boverhof, A Elbashandy, D Agarwal, J Leek, T Epperly, J Eslick, D Miller, "Data management and simulation support accelerating carbon capture through computing", Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016, 2017, 389--398, doi: 10.1109/eScience.2016.7870924

Gilberto Z. Pastorello, Dan K. Gunter, Housen Chu, Danielle S. Christianson, Carlo Trotta, Eleonora Canfora, Boris Faybishenko, You-Wei Cheah, Norm Beekwilder, Stephen W. Chan, Sigrid Dengel, Trevor Keenan, Fianna O Brien, Abderahman Elbashandy, Cristina M. Poindexter, Marty Humphrey, Dario Papale, Deb A. Agarwal, "Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures", Proceedings of the 13th IEEE International Conference on e-Science (e-Science 2017), Auckland, New Zealand, 2017, doi: 10.1109/eScience.2017.64

Deborah A Agarwal, Boris Faybishenko, Vicky L Freedman, Harinarayan Krishnan, Gary Kushner, Carina Lansing, Ellen Porter, Alexandru Romosan, Arie Shoshani, Haruko Wainwright, others, "A science data gateway for environmental management", Concurrency and Computation: Practice and Experience, 2016, 28:1994--2004,

DC Miller, D Agarwal, D Bhattacharyya, J Boverhof, YW Cheah, Y Chen, J Eslick, J Leek, J Ma, P Mahapatra, B Ng, NV Sahinidis, C Tong, SE Zitney, "Innovative computational tools and models for the design, optimization and control of carbon capture processes", Computer Aided Chemical Engineering, 2016, 38:2391--2396, doi: 10.1016/B978-0-444-63428-3.50403-3

V Hendrix, L Ramakrishnan, Y Ryu, C Van Ingen, KR Jackson, D Agarwal, "CAMP: Community access MODIS pipeline", Future Generation Computer Systems, 2014, 36:418--429, doi: 10.1016/j.future.2013.09.023

Gilberto Z. Pastorello, Deb A. Agarwal, Taghrid Samak, Dario Papale, Trotta, Alessio Ribeca, Cristina M. Poindexter, Boris Faybishenko, Dan K. Gunter, Rachel Hollowgrass, Eleonora Canfora, "Observational data patterns for time series data quality assessment", Proceedings of the 10th IEEE International Conference on e-Science (e-Science 2014), Guaruja, Brazil, 2014, doi: 10.1109/eScience.2014.45

DC Miller, M Syamlal, DS Mebane, C Storlie, D Bhattacharyya, NV Sahinidis, D Agarwal, C Tong, SE Zitney, A Sarkar, X Sun, S Sundaresan, E Ryan, D Engel, C Dale, "Carbon capture simulation initiative: A case study in multiscale modeling and new challenges", Annual Review of Chemical and Biomolecular Engineering, 2014, 5:301--323, doi: 10.1146/annurev-chembioeng-060713-040321

Lavanya Ramakrishnan, Sarah S. Poon, Val C. Hendrix, Dan K. Gunter, Gilberto Z. Pastorello, Deb A. Agarwal, "Experiences with User-Centered Design for the Tigres Workflow API", Proceedings of the 10th IEEE International Conference on e-Science (e-Science 2014), Guaruja, Brazil, 2014, doi: 10.1109/eScience.2014.56

JR Balderrama, M Simonin, L Ramakrishnan, V Hendrix, C Morin, D Agarwal, C Tedeschi, "Combining workflow templates with a shared space-based execution model", Proceedings of WORKS 2014: The 9th Workshop on Workflows in Support of Large-Scale Science - held in conjunction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2014, 50--58, doi: 10.1109/WORKS.2014.14

Karen L. Schuchardt, Deborah A. Agarwal, Stefan A. Finsterle, Carl W. Gable, Ian Gorton, Luke J. Gosink, Elizabeth H. Keating, Carina S. Lansing, Joerg Meyer, William A.M. Moeglein, George S.H. Pau, Ellen A. Porter, Sumit Purohit, Mark L. Rockhold, Arie Shoshani, and Chandrika Sivaramakrishnan, Akuna, "Integrated Toolsets Supporting Advanced Subsurface Flow and Transport Simulations for Environmental Management", XIX International Conference on Computational Methods in Water Resources (CMWR 2012), University of Illinois at Urbana-Champaign, June 2012,

Download File: CMWR-2012.pdf (pdf: 1.9 MB)

Karen L. Schuchardt, Deborah A. Agarwal, Stefan A. Finsterle, Carl W. Gable, Ian Gorton, Luke J. Gosink, Elizabeth H. Keating, Carina S. Lansing, Joerg Meyer, William A.M. Moeglein, George S.H. Pau, Ellen A. Porter, Sumit Purohit, Mark L. Rockhold, Arie Shoshani, Chandrika Sivaramakrishnan, "Akuna-Integrated Toolsets Supporting Advanced Subsurface Flow and Transport Simulations for Environmental Management", XIX International Conference on Computational Methods in Water Resources (CMWR 2012), University of Illinois at Urbana-Champaign, June 17-22, 2012, 2012,

D Baldocchi, M Reichstein, D Papale, L Koteen, R Vargas, D Agarwal, R Cook, The role of trace gas flux networks in the biogeosciences, Eos, Pages: 217--218 2012, doi: 10.1029/2012EO230001

V Hendrix, J Li, K Jackson, L Ramakrishnan, Y Ryu, K Beattie, C Morin, D Skinner, C van Ingen, D Agarwal, "Community Access to MODIS Satellite Reprojection and Reduction Pipeline and Data Sets", AGU Fall Meeting, 2012,

D Agarwal, YW Cheah, D Fay, J Fay, D Guo, T Hey, M Humphrey, K Jackson, Jie Li, C Poulain, Y Ryu, C Van Ingen, "Data-intensive science: The Terapixel and MODISAzure projects", International Journal of High Performance Computing Applications, 2011, 25:304--316, doi: 10.1177/1094342011414746

D. Agarwal, M. Humphrey,N., Beekwilder, K. Jackson, M. Goode, and C. van Ingen, "A Data Centered Collaboration Portal to Support Global Carbon-Flux Analysis", Concurrency and Computation: Practice and Experience - Successes in Furthering Scientific Discovery, June 2010, LBNL 1827E,

Y Simmhan, E Soroush, C Van Ingen, D Agarwal, L Ramakrishnan, "BReW: Blackbox resource selection for e-Science workflows", 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010, 2010, doi: 10.1109/WORKS.2010.5671857

L Ramakrishnan, C Guok, K Jackson, E Kissel, DM Swany, D Agarwal, "On-demand overlay networks for large scientific data transfers", CCGrid 2010 - 10th IEEE/ACM International Conference on Cluster, Cloud, and Grid Computing, 2010, 359--367, doi: 10.1109/CCGRID.2010.82

M Bishop, J Cummins, S Peisert, A Singh, B Bhumiratana, D Agarwal, D Frincke, M Hogarth, "Relationships and data sanitization: A study in scarlet", Proceedings New Security Paradigms Workshop, January 2010, 151--163, doi: 10.1145/1900546.1900567

DA Agarwal, M Humphrey, NF Beekwilder, KR Jackson, MM Goode, C Van Ingen, "A data-centered collaboration portal to support global carbon-flux analysis", Concurrency Computation Practice and Experience, 2010, 22:2323--2334, doi: 10.1002/cpe.1600

J Li, M Humphrey, D Agarwal, K Jackson, C Van Ingen, Y Ryu, EScience in the cloud: A MODIS satellite data reprojection and reduction pipeline in the Windows Azure platform, Proceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, IPDPS 2010, 2010, doi: 10.1109/IPDPS.2010.5470418

J Li, M Humphrey, YW Cheah, Y Ryu, D Agarwal, K Jackson, C Van Ingen, Fault tolerance and scaling in e-Science cloud applications: Observations from the continuing development of MODISAzure, Proceedings - 2010 6th IEEE International Conference on e-Science, eScience 2010, Pages: 246--253 2010, doi: 10.1109/eScience.2010.47

DA Agarwal, DA Tran, MD Yarvis, Guest editorial, International Journal of Pervasive Computing and Communications, Pages: 18 2009, doi: 10.1108/ijpcc.2009.36105daa.001

M Humphrey, D Agarwal, C Van Ingen, Fluxdata.org: Publication and curation of shared scientific climate and earth sciences data, e-Science 2009 - 5th IEEE International Conference on e-Science, Pages: 118--125 2009, doi: 10.1109/e-Science.2009.25

William T.C. Kramer, John M. Shalf, E. Wes Bethel, D. Agarwal, Michael Banda, John Hules, Juan C. Meza, Leonid Oliker, Horst Simon, David Skinner, Francesca Verdier, Howard Walter, Michael Wehner, and Katherine Yelick, "HPC in 2016: A View Point from NERSC", Proceedings of the Cray User Group Conference, Helsinki, Finland, 2008,

M Humphrey, N Beekwilder, D Agarwal, D Baldocchi, M Goode, C Ingen, D Papale, M Reichstein, M Rodriguez, R Vargas, N Li, J Gupchup, Y Ryu, Creating and Accessing the Global Fluxnet Data Set, 2008,

PV Sundareshwar, R Murtugudde, G Srinivasan, S Singh, KJ Ramesh, R Ramesh, SB Verma, D Agarwal, D Baldocchi, CK Baru, KK Baruah, GR Chowdhury, VK Dadhwal, CBS Dutt, J Fuentes, PK Gupta, WW Hargrove, M Howard, CS Jha, S Lal, WK Michener, AP Mitra, JT Morris, RR Myneni, M Naja, R Nemani, R Purvaja, S Raha, SK Santhana Vanan, M Sharma, A Subramaniam, R Sukumar, RR Twilley, PR Zimmerman, Environment: Environmental monitoring network for India, Science, Pages: 204--205 2007, doi: 10.1126/science.1137417

WTC Kramer, A Shoshani, DA Agarwal, BR Draney, G Jin, GF Butler, JA Hules, Deep scientific computing requires deep data, IBM Journal of Research and Development, Pages: 209--232 2004, doi: 10.1147/rd.482.0209

S Peggs, T Satogata, D Agarwal, D Rice, Remote operations in A global accelerator network, Proceedings of the IEEE Particle Accelerator Conference, Pages: 278--282 2003,

DA Agarwal, LE Moser, PM Melliar-Smith, RK Budhia, The Totem Multiple-Ring Ordering and Topology Maintenance Protocol, ACM Transactions on Computer Systems, Pages: 93--132 1998, doi: 10.1145/279227.279228

Y Amir, LE Moser, PM Melliar-Smith, DA Agarwal, P Ciarfella, "The Totem Single-Ring Ordering and Membership Protocol", ACM Transactions on Computer Systems (TOCS), 1995, 13:311--342, doi: 10.1145/210223.210224

Jan Balewski, Mercy G Amankwah, Roel Van Beeumen, E Wes Bethel, Talita Perciano, Daan Camps, "Quantum-parallel vectorized data encodings and computations on trapped-ion and transmon QPUs", Journal, February 10, 2024, 14, doi: https://doi.org/10.1038/s41598-024-53720-x

GM Wallace, Z Bai, N Bertelli, EW Bethel, T Perciano, S Shiraiwa, JC Wright, "Towards Fast, Accurate Predictions of RF Simulations via Data-driven Modeling: Forward and Lateral Models", Conference, AIP Publishing, August 1, 2023, 2984, doi: https://doi.org/10.1063/5.0162422

Gregory Wallace, Zhe Bai, Robbie Sadre, Talita Perciano, Nicola Bertelli, Syun'ichi Shiraiwa, Wes Bethel, John Wright, "Towards fast and accurate predictions of radio frequency power deposition and current profile via data-driven modelling: applications to lower hybrid current drive", Journal of Plasma Physics, August 18, 2022, 88:895880401, doi: 10.1017/S0022377822000708

M. G. Amankwah, D. Camps, E. W. Bethel, R. Van Beeumen, T. Perciano, "Quantum pixel representations and compression for N-dimensional images", Nature Scientific Reports, May 11, 2022, 12:7712, doi: 10.1038/s41598-022-11024-y

S. Zhang, R. Sadre, B. A. Legg, H. Pyles, T. Perciano, E. W. Bethel, D. Baker, O. Rübel, J. J. D. Yoreo, "Rotational dynamics and transition mechanisms of surface-adsorbed proteins", Proceedings of the National Academy of Sciences, April 11, 2022, 119:e202024211, doi: 10.1073/pnas.2020242119

E. Wes Bethel, Burlen Loring, Utkarsh Ayachit, P. N. Duque, Nicola Ferrier, Joseph Insley, Junmin Gu, Kress, Patrick O’Leary, Dave Pugmire, Silvio Rizzi, Thompson, Will Usher, Gunther H. Weber, Brad Whitlock, Wolf, Kesheng Wu, "Proximity Portability and In Transit, M-to-N Data Partitioning and Movement in SENSEI", In Situ Visualization for Computational Science, ( 2022) doi: 10.1007/978-3-030-81627-8_20

E. Wes Bethel, Burlen Loring, Utkarsh Ayatchit, David Camp, P. N. Duque, Nicola Ferrier, Joseph Insley, Junmin Gu, Kress, Patrick O’Leary, David Pugmire, Silvio Rizzi, Thompson, Gunther H. Weber, Brad Whitlock, Matthew Wolf, Kesheng Wu, "The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale", In Situ Visualization for Computational Science, ( 2022) doi: 10.1007/978-3-030-81627-8_13

E. W. Bethel, C. Heinemann, and T. Perciano, "Performance Tradeoffs in Shared-memory Platform Portable Implementations of a Stencil Kernel", Eurographics Symposium on Parallel Graphics and Visualization, June 14, 2021,

Stefano Marchesini, Anuradha Trivedi, Pablo Enfedaque, Talita Perciano, Dilworth Parkinson, "Sparse Matrix-Based HPC Tomography", Computational Science -- ICCS 2020, Cham, Springer International Publishing, 2020, 248--261, doi: 10.1007/978-3-030-50371-0_18

E. Wes Bethel, David Camp, Talita Perciano, Colleen Heinemann, "Performance Analysis of Traditional and Data-Parallel Primitive Implementations of Visualization and Analysis Kernels", Berkeley, CA, USA, 94720, 2020,

H. Childs, S. Ahern, J. Ahrens, A. C. Bauer, J. Bennett, E. W. Bethel, P.-T. Bremer, E. Brugger, J. Cottam, M. Dorier, S. Dutta, J. Favre, T. Fogal, S. Frey, C. Garth, B. Geveci, W. F. Godoy, C. D. Hansen, C. Harrison, B. Hentschel, J. Insley, C. Johnson, S. Klasky, A. Knoll, J. Kress, M. Larsen, J. Lofstead, K.-L. Ma, P. Malakar, J. Meredith, K. Moreland, P. Navratil, P. O Leary, M. Parashar, V. Pascucci, J. Patchett, T. Peterka, S. Petruzza, N. Podhorszki, D. Pugmire, M. Rasquin, S. Rizzi, D. H. Rogers, S. Sane, F. Sauer, R. Sisneros, H.-W. Shen, W. Usher, R. Vickery, V. Vishwanath, I. Wald, R. Wang, G. H. Weber, B. Whitlock, M. Wolf, H. Yu, S. B. Ziegler, "A Terminology for In Situ Visualization and Analysis Systems", International Journal of High Performance Computing Applications, 2020, 34:676--691, doi: 10.1177/1094342020935991

Junmin Gu, Burlen Loring, Kesheng Wu, E Wes Bethel, "HDF5 as a vehicle for in transit data movement", Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, 2019, 39--43,

B Loring, A Myers, D Camp, EW Bethel, "Python-based in situ analysis and visualization", Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV 18, ACM Press, 2018, doi: 10.1145/3281464.3281465

B Lessley, T Perciano, C Heinemann, D Camp, H Childs, EW Bethel, "DPP-PMRF: Rethinking Optimization for a Probabilistic Graphical Model Using Data-Parallel Primitives", The 8th IEEE Symposium on Large Data Analysis and Visualization - LDAV 2018, 2018,

C Heinemann, T Perciano, D Ushizima, EW Bethel, "Distributed memory parallel Markov random fields using graph partitioning", Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017, 2018, 2018-Jan:3332--3341, doi: 10.1109/BigData.2017.8258318

E. Wes Bethel, In Situ Processing Overview and Relevance to the HPC Community, SIAM Conference on Computational Science and Engineering, MS74: In Situ Methods and Infrastructures: Faster Insight Through Smarter Computing, 2017,

U Ayachit, B Whitlock, M Wolf, B Loring, B Geveci, D Lonie, EW Bethel, "The SENSEI generic in situ interface", Proceedings of ISAV 2016: 2nd Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2017, 40--44, doi: 10.1109/ISAV.2016.13

B Lessley, T Perciano, M Mathai, H Childs, EW Bethel, "Maximal clique enumeration with data-parallel primitives", 2017 IEEE 7th Symposium on Large Data Analysis and Visualization, LDAV 2017, 2017, 2017-Dec:16--25, doi: 10.1109/LDAV.2017.8231847

T Perciano, D Ushizima, H Krishnan, D Parkinson, N Larson, DM Pelt, W Bethel, F Zok, J Sethian, "Insight into 3D micro-CT data: Exploring segmentation algorithms through performance metrics", Journal of Synchrotron Radiation, 2017, 24:1065--1077, doi: 10.1107/S1600577517010955

E Bethel, zorder-lib: Library API for Z-Order Memory Layout:, 2017,

EW Bethel, Towards a data-centric research and development roadmap for large-scale science user facilities, Proceedings - 13th IEEE International Conference on eScience, eScience 2017, Pages: 462--464 2017, doi: 10.1109/eScience.2017.72

Andrew C. Bauer, Kenneth E. Jansen, E. Wes Bethel, Utkarsh Ayachit, Michel Rasquin, Benjamin Matthews, Steve Jordan, "In Situ Analysis and Visualization at Scale with PHASTA and ParaView Catalyst on Mira and Theta", SC16 Scientific Visualization Showcase, 2016,

E. Wes Bethel, Martin Greenwald, Kerstin Kleese Dam, Manish Parashar, Stefan M. Wild, H. Steven Wiley, "2016 IEEE 12th International Conference on e-Science", 2016 IEEE 12th International Conference on e-Science, Baltimore, MD, USA, 2016, 213--222,

Hoa Nguyen, D\ aith\ i Stone, E. Wes Bethel, "Statistical Projections for Multi-dimensional Visual Data Exploration", 6th IEEE Symposium on Large Data Analysis and Visualization, 2016,

moderator E. Wes Bethel (organizer, Hank Childs, Ken Moreland, Dave Pugmire, Matt Larsen, Matthieu Dorier, In Situ Efforts and Challenges in Large Data Analysis and Visualization, IEEE Symposium on Large Data Analysis and Visualization (LDAV), 2016,

Utkarsh Ayachit, Andrew Bauer, Earl PN Duque, Greg Eisenhauer, Nicola Ferrier, Junmin Gu, Kenneth E Jansen, Burlen Loring, Zarija Lukic, Suresh Menon, others, "Performance analysis, design considerations, and applications of extreme-scale in situ infrastructures", SC 16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2016, 921--932, LBNL 1007264,

T Perciano, DM Ushizima, EW Bethel, YD Mizrahi, D Parkinson, JA Sethian, "Reduced-complexity image segmentation under parallel Markov Random Field formulation using graph partitioning", Proceedings - International Conference on Image Processing, ICIP, 2016, 2016-Aug:1259--1263, doi: 10.1109/ICIP.2016.7532560

O Rübel, B Loring, JL Vay, DP Grote, R Lehe, S Bulanov, H Vincenti, EW Bethel, "WarpIV: In Situ Visualization and Analysis of Ion Accelerator Simulations", IEEE Computer Graphics and Applications, 2016, 36:22--35, doi: 10.1109/MCG.2016.62

DM Ushizima, HA Bale, EW Bethel, P Ercius, BA Helms, H Krishnan, LT Grinberg, M Haranczyk, AA Macdowell, K Odziomek, DY Parkinson, T Perciano, RO Ritchie, C Yang, "IDEAL: Images Across Domains, Experiments, Algorithms and Learning", JOM, 2016, 68:2963--2972, doi: 10.1007/s11837-016-2098-4

David Camp, Hari Krishnan, David Pugmire, Christoph Garth, Ian Johnson, E. Wes Bethel, Kenneth I. Joy, and Hank Childs., "GPU Acceleration of Particle Advection Workloads in a Parallel, Distributed Memory Setting", Proceedings of Eurographics Symposium on Parallel Graphics and Visualization (EGPGV), May 5, 2013,

Hank Childs, Berk Geveci, William J. Schroeder, Jeremy S. Meredith, Kenneth Moreland, Christopher Sewell, Torsten Kuhlen, E.Wes Bethel, "Research Challenges for Visualization Software", IEEE Computer, May 1, 2013, 46:34-43, LBNL 6239E,

E Wes Bethel, Prabhat Prabhat, Suren Byna, Oliver R\ ubel, K John Wu, Michael Wehner, "Why high performance visual data analytics is both relevant and difficult", Visualization and Data Analysis 2013, January 2013, 8654:86540B, LBNL LBNL-6063E,

Download File: LBNL-6063E.pdf (pdf: 3.6 MB)

Kesheng Wu, E Bethel, Ming Gu, David Leinweber, Oliver R\ ubel, "A big data approach to analyzing market volatility", Algorithmic Finance, 2013, 2:241--267, LBNL LBNL-6382E,

Understanding the microstructure of the financial market requires the processing of a vast amount of data related to individual trades, and sometimes even multiple levels of quotes. Analyzing such a large volume of data requires tremendous computing power that is not easily available to financial academics and regulators. Fortunately, public funded High Performance Computing (HPC) power is widely available at the National Laboratories in the US. In this paper we demonstrate that the HPC resource and the techniques for data-intensive sciences can be used to greatly accelerate the computation of an early warning indicator called Volume-synchronized Probability of Informed trading (VPIN). The test data used in this study contains five and a half year's worth of trading data for about 100 most liquid futures contracts, includes about 3 billion trades, and takes 140GB as text files. By using (1) a more efficient file format for storing the trading records, (2) more effective data structures and algorithms, and (3) parallelizing the computations, we are able to explore 16,000 different ways of computing VPIN in less than 20 hours on a 32-core IBM DataPlex machine. Our test demonstrates that a modest computer is sufficient to monitor a vast number of trading activities in real-time -- an ability that could be valuable to regulators.

Our test results also confirm that VPIN is a strong predictor of liquidity-induced volatility. With appropriate parameter choices, the false positive rates are about 7% averaged over all the futures contracts in the test data set. More specifically, when VPIN values rise above a threshold (CDF > 0.99), the volatility in the subsequent time windows is higher than the average in 93% of the cases.

Kesheng Wu, Wes Bethel, Ming Gu, David, Oliver R\ ubel, Testing VPIN on Big Data, Available at SSRN 2318259, 2013,

O Rübel, A Greiner, S Cholia, K Louie, EW Bethel, TR Northen, BP Bowen, "OpenMSI: A high-performance web-based platform for mass spectrometry imaging", Analytical Chemistry, 2013, 85:10354--103, doi: 10.1021/ac402540a

E Wes Bethel, Prabhat Prabhat, Suren Byna, R\ ubel, K John Wu, Michael Wehner, Why high performance visual data analytics is both and difficult, IS\&T/SPIE Electronic Imaging, Pages: 86540B-865 2013, doi: 10.1117/12.2010980

DN Williams, T Bremer, C Doutriaux, J Patchett, S Williams, G Shipman, R Miller, DR Pugmire, B Smith, C Steed, EW Bethel, H Childs, H Krishnan, P Prabhat, M Wehner, CT Silva, E Santos, D Koop, T Ellqvist, J Poco, B Geveci, A Chaudhary, A Bauer, A Pletzer, D Kindig, GL Potter, TP Maxwell, Ultrascale visualization of climate data, Computer, Pages: 68--76 2013, doi: 10.1109/MC.2013.119

E. Wes Bethel and Mark Howison, "Multi-core and Many-core Shared-memory Parallel Raycasting Volume Rendering Optimization and Tuning", International Journal of High Performance Computing Applications, 2012, LBNL 5362E,

Hank Childs, Eric Brugger, Brad Whitlock, Jeremy Meredith, Sean Ahern, David Pugmire, Kathleen Biagas, Mark Miller, Cyrus Harrison, Gunther H. Weber, Hari Krishnan, Thomas Fogal, Allen Sanderson, Christoph Garth, E. Wes Bethel, David Camp, Oliver Rubel, Marc Durant, Jean M. Favre, Paul Navratil, "VisIt: An End-User Tool For Visualizing and Analyzing Very Large Data", High Performance Visualization---Enabling Extreme-Scale Scientific Insight, ( October 2012) Pages: 357--372

E. Wes Bethel, David Camp, Hank Childs, Christoph Garth, Mark Howison, Kenneth I. Joy, David Pugmire, "Hybrid Parallelism", High Performance Visualization---Enabling Extreme-Scale Scientific Insight, ( October 2012) Pages: 261--290

Hank Childs, David Pugmire, Sean Ahern, Brad Whitlock, Mark Howison, Prabhat, Gunther Weber, E. Wes Bethel, "Visualization at Extreme Scale Concurrency", High Performance Visualization---Enabling Extreme-Scale Scientific Insight, ( October 2012) Pages: 291--306

E. Wes Bethel, Hank Childs, Charles Hansen (editors), High Performance Visualization---Enabling Extreme-Scale Scientific Insight, Chapman & Hall, CRC Computational Science, (CRC Press/Francis--Taylor Group: October 2012)

Oliver Rübel, Cameron, G. R. Geddes, Min Chen, Estelle Cormier-Michel, and E. Wes Bethel, "Query-driven Analysis of Plasma-based Particel Acceleration Data", Poster Abstracts of IEEE VisWeek, October 2012,

Mehmet Balman, Eric Pouyoul, Yushu Yao, E. Wes Bethel, Burlen Loring, Prabhat, John Shalf, Alex Sim, and Brian L. Tierney, "Experiences with 100G Network Applications", In Proceedings of the Fifth international Workshop on Data-intensive Distributed Computing, in conjunction with ACM High Performance Distributing Computing (HPDC) Conference, 2012, Delft, Netherlands, June 2012, LBNL 5603E, doi: 10.1145/2286996.2287004

Download File: didc-report.pdf (pdf: 11 MB)

100Gbps networking has finally arrived, and many research and educational in- stitutions have begun to deploy 100Gbps routers and services. ESnet and Internet2 worked together to make 100Gbps networks available to researchers at the Super- computing 2011 conference in Seattle Washington. In this paper, we describe two of the first applications to take advantage of this network. We demonstrate a visu- alization application that enables remotely located scientists to gain insights from large datasets. We also demonstrate climate data movement and analysis over the 100Gbps network. We describe a number of application design issues and host tuning strategies necessary for enabling applications to scale to 100Gbps rates.

E. Wes Bethel, David Camp, Hank Childs, Mark Howison, Hari Krishnan, Burlen Loring, Joerg Meyer, Prabhat, Oliver Ruebel, Daniela Ushizima, Gunther Weber, "Towards Exascale: High Performance Visualization and Analytics – Project Status Report. Technical Report", DOE Exascale Research Conference, April 2012,

Ushizima, D.M., Weber, G., Morozov, D., Bethel, W., Sethian, J.A., "Algorithms for Microstructure Description applied to Microtomography", Carbon Cycle 2.0 Symposium, February 10, 2012,

Surendra Byna, Jerry Chou, Oliver Rubel, Homa Karimabadi, William S Daughter, Vadim Roytershteyn, E Wes Bethel, Mark Howison, Ke-Jou Hsu, Kuan-Wu Lin, others, "Parallel I/O, analysis, and visualization of a trillion particle simulation", SC 12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, January 2012, 1--12,

Oliver R\ ubel, Surendra Byna, Kesheng Wu, Fuyu Li, Michael Wehner, Wes Bethel, others, "Teca: A parallel toolkit for extreme climate analysis", Procedia Computer Science, Elsevier, January 2012, 9:866--876, LBNL 5352E,

We present TECA, a parallel toolkit for detecting extreme events in large climate datasets. Modern climate datasets expose parallelism across a number of dimensions: spatial locations, timesteps and ensemble members. We design TECA to exploit these modes of parallelism and demonstrate a prototype implementation for detecting and tracking three classes of extreme events: tropical cyclones, extra-tropical cyclones and atmospheric rivers. We process a modern TB-sized CAM5 simulation dataset with TECA, and demonstrate good runtime performance for the three case studies.

E. W. Bethel, Surendra Byna, Jerry Chou, Cormier-Michel, Cameron G. R. Geddes, Howison, Fuyu Li, Prabhat, Ji Qiang, R\ ubel, Rob D. Ryne, Michael Wehner, Wu, "Big Data Analysis and Visualization: What Do LINACS Tropical Storms Have In Common?", 11th International Computational Accelerator Physics ICAP 2012, Germany, 2012,

EW Bethel, S. Byna, J. Chou, E., CGR Geddes, M. Howison, F. Li J. Q. Prabhat, O. R\ ubel, RD Ryne and, Big Data Analysis and Visualization: What Do LINACS Tropical Storms Have In Common?, 11th International Computational Accelerator Physics ICAP 2012, 2012,

O. R\ ubel, S. Byna, K. Wu, F. Li, M., W. Bethel, others, "TECA: A Parallel Toolkit for Extreme Climate", Procedia Computer Science, Elsevier, 2012, 9:866--876, doi: 10.1016/j.procs.2012.04.093

E. W. Bethel and D. Leinweber and O. Rubel and K. Wu, "Federal Market Information Technology in the Post Flash Crash Era: Roles of Supercomputing", The Journal of Trading, 2012, 7:9-24, LBNL 5263E, doi: 10.3905/jot.2012.7.2.009

E. Wes Bethel, David Leinweber, Oliver Rübel Kesheng Wu, Federal Market Information Technology in the Crash Era: Roles for Supercomputing, The Journal of Trading, Pages: 9--25 2012, doi: 10.3905/jot.2012.7.2.009

E. Wes Bethel, "Exploration of Optimization Options for Increasing Performance of a GPU Implementation of a Three-dimensional Bilateral Filter", 2012, LBNL 5406E,

Download File: LBNL-5406E.pdf (pdf: 438 KB)

O. Rübel, S.V.E. Keränen, M.D. Biggin, D.W. Knowles, G.H. Weber, H. Hagen, B. Hamann, and E.W. Bethel, "Linking Advanced Visualization and MATLAB for the Analysis of 3D Gene Expression Data", Mathematics and Visualization, Visualization in Medicine and Life Sciences II, Progress and New Challenges, edited by L. Linsen and B. Hamann and H. Hagen and H.-C. Hege, (Springer Verlag: 2012) Pages: 267-285, LBNL 4891E,

Download File: LBNL-4891E.pdf (pdf: 15 MB)

Mark Howison, E. Wes Bethel, Hank Childs, "Hybrid Parallelism for Volume Rendering on Large, Multi- and Many-core Systems", IEEE Transactions on Visualization and Computer Graphics, January 2012, 18:17-29, LBNL 4370E,

Surendra Byna, Jerry Chou, Oliver R\ ubel, , Homa Karimabadi, William S. Daughton, Roytershteyn, E. Wes Bethel, Mark Howison Ke-Jou Hsu, Kuan-Wu Lin, Arie Shoshani, Uselton, Kesheng Wu, Parallel I/O, Analysis, and Visualization of a Particle Simulation, Proceedings of SuperComputing 2012, 2012,

J Meyer, EW Bethel, JL Horsman, SS Hubbard, H Krishnan, A Romosan, EH Keating, L Monroe, R Strelitz, P Moore, G Taylor, B Torkian, TC Johnson, I Gorton, Visual data analysis as an integral part of environmental management, IEEE Transactions on Visualization and Computer Graphics, Pages: 2088--2094 2012, doi: 10.1109/TVCG.2012.278

H Krishnan, J Meyer, A Romosan, H Childs, EW Bethel, Enabling advanced environmental management via remote and distributed visual data exploration and analysis, Computing and Visualization in Science, Pages: 123--133 2012, doi: 10.1007/s00791-013-0204-5

M Balman, E Pouyoul, Y Yao, EW Bethel, B Loring, Prabhat, J Shalf, A Sim, BL Tierney, "Experiences with 100Gbps network applications", DIDC 12 - 5th International Workshop on Data-Intensive Distributed Computing, 2012, 33--42, doi: 10.1145/2286996.2287004

D Ushizima, D Morozov, GH Weber, AGC Bianchi, JA Sethian, EW Bethel, "Augmented topological descriptors of pore networks for material science", IEEE Transactions on Visualization and Computer Graphics, 2012, 18:2041--2050, LBNL 5964E, doi: 10.1109/TVCG.2012.200

Download File: LBNL-5964E.pdf (pdf: 5.4 MB)

E. Wes Bethel, David Leinweber, Oliver Rübel, Kesheng Wu, "Federal Market Information Technology in the Post Flash Crash Era: Roles of Supercomputing", Workshop on High Performance Computational Finance at SC11, Seattle, WA, USA, November 2011, LBNL 5263E,

Ushizima, D.M., Parkinson, D., Nico, P., Ajo-Franklin, J., Macdowell, A., Kocar, B., Bethel E.W, Sethian J.A., "Statistical segmentation and porosity quantification of 3D x-ray microtomography", XXXIV Applications of Digital Image Processing: Proceeding of SPIE 2011, San Diego, CA, USA, August 2011,

Ushizima, D.M., Weber, G.H., Ajo-Franklin, J., Kim, Y., Macdowell, A., Morozov, D., Nico, P., Parkinson, D., Trebotich, D., Wan, J., and Bethel E.W., "Analysis and visualization for multiscale control of geologic CO2", Journal of Physics: Conference Series, Proceedings of SciDAC 2011, July 2011, LBNL Denver, CO, USA,

Jerry Chou, Mark Howison, Brian Austin, Kesheng Wu, Ji Qiang, E Wes Bethel, Arie Shoshani, Oliver R\ ubel, Rob D Ryne, "Parallel index and query for large scale data analysis", Proceedings of 2011 international conference for high performance computing, networking, storage and analysis, 2011, 1--11, LBNL 5317E,

J. Chou, K. Wu, O. R\ ubel, M. Howison, Qiang, Prabhat, B. Austin, E. W. Bethel, D. Ryne, A. Shoshani, "Parallel Index and Query for Large Scale Data", SC11, 2011, doi: 10.1145/2063384.2063424

M Prabhat, S Byna, C Paciorek, G Weber, K Wu, T Yopes, MF Wehner, G Ostrouchov, D Pugmire, R Strelitz, others, "Pattern Detection and Extreme Value Analysis on Large Climate Data", AGUFM, Pages: IN41C--03 January 2011,

Mark Howison, Mike McGreevy, Bruce Palmer, Oliver Ruebel, Kesheng Wu, "ExaHDF5: An I/O Platform for Exascale Data Models, Analysis and Performance", 2011,

Download File: Prabhat.pdf (pdf: 784 KB)

Luke J. Gosink, Christoph Garth, John C. Anderson, E. Wes Bethel, and Kenneth I. Joy, "An Application of Multivariate Statistical Analysis for Query-Driven Visualization", IEEE Transactions on Visualization and Computer Graphics, 2011, 11:264-275, LBNL 3536E, doi: 10.1109/TVCG.2010.80

Download File: LBNL-3536E.pdf (pdf: 13 MB)

E. Wes Bethel, John van Rosendale, Dale Southard, Kelly Gaither, Hank Childs, Eric Brugger, Sean Ahern, "Visualization at Supercomputing Centers: The Tale of Little Big Iron and the Three Skinny Guys", IEEE Computer Graphics and Applications, January 2011, 31:90-95, LBNL 4180E,

Download File: skinnyGuys.pdf (pdf: 428 KB)

Maciej Haranczyk, Richard L. Martin, Prabhat, James A. Sethian & E. Wes Bethel, "Computational Approaches for the High-Throughput Analysis of Porous materials for Energy related applications", Scientific Discovery through Advanced Computing 2011, 2011,

Prabhat, S. Byna, C. Paciorek, G. Weber, Wu, T. Yopes, M. Wehner, W. Collins, G., R. Strelitz, E. W. Bethel, Pattern Detection and Extreme Value Analysis on Large Data, DOE/BER Climate and Earth System Modeling PI Meeting, 2011,

E. W. Bethel, D. Leinweber, O. Rübel, K., Federal Market Information Technology in the Post Crash Era: Roles for Supercomputing, WHPCF, Pages: 23--30 2011, doi: 10.1145/2088256.2088267

Prabhat, Quincey Koziol, Karen Schuchardt, E. Wes Bethel, Jerry Chuo, Mark Howison, Mike, Bruce Palmer, Oliver Ruebel, Kesheng, ExaHDF5: An I/O Platform for Exascale Data Analysis and Performance, SciDAC 2011, 2011,

Mark Howison, Andreas Adelmann, E. Wes Bethel, Achim Gsell, Benedikt Oswald, Prabhat, "H5hut: A High-Performance I/O Library for Particle-Based Simulations", Proceedings of 2010 Workshop on Interfaces and Abstractions for Scientific Data Storage (IASDS10), Heraklion, Crete, Greece, September 2010, LBNL 4021E,

Download File: LBNL-4021E.pdf (pdf: 716 KB)

M. Howison, E. W. Bethel, and H. Childs, "Hybrid Parallelism for Volume Rendering on Large, Multi-core Systems", Journal of Physics: Conference Series, Proceedings of SciDAC 2010, Chattanooga, TN, USA, July 2010, LBNL 4024E,

Download File: LBNL-4024E.pdf (pdf: 4.2 MB)

M. Howison, E. W. Bethel, and H. Childs, "Hybrid Parallelism for Volume Rendering on Large, Multi-core Systems", Proceedings of Astronum 2010, San Diego, CA, USA, June 2010, LBNL 4024E,

Mark Howison, E. Wes Bethel, Hank Childs, "MPI-hybrid Parallelism for Volume Rendering on Large, Multi-core Systems", In Proceedings of the Eurographics Symposium on Parallel Graphics and Visualization (EGPGV), Norköping, Sweden, May 2010, LBNL 3297E,

Download File: LBNL-3297E.pdf (pdf: 5 MB)

O. Rübel, G. H. Weber, M-Y Huang, E. W. Bethel, M. D. Biggin, C. C. Fowlkes, C. Luengo Hendriks, S. V. E. Keränen, M. Eisen, D. Knowles, J. Malik, H. Hagen and B. Hamann, "Integrating Data Clustering and Visualization for the Analysis of 3D Gene Expression Data", IEEE Transactions on Computational Biology and Bioinformatics, March 2010, 7:64-79, LBNL 382E,

Download File: LBNL-382E-TCBB.pdf (pdf: 11 MB)

Daniela Ushizima, Cameron Geddes, Estelle Cormier-Michel, E. Wes Bethel, Janet Jacobsen, Prabhat, Oliver Rubel, Gunther Weber, Bernard Hamann, Peter Messmer, Hans Hagen, "Automated detection and analysis of particle beams in laser-plasma accelerator simulations", Machine Learning, edited by Yagang Zhang, (In-Teh: February 2010) Pages: 367-389, LBNL 3845E,

Oliver R\ ubel, Sean Ahern, E Wes Bethel, Mark D Biggin, Hank Childs, Estelle Cormier-Michel, Angela DePace, Michael B Eisen, Charless C Fowlkes, Cameron GR Geddes, others, "Coupling visualization and data analysis for knowledge discovery from multi-dimensional scientific data", Procedia computer science, Elsevier, January 2010, 1:1757--1764, LBNL 3669E,

Download File: ICCS2010Ruebel.pdf (pdf: 3.7 MB)

Gunther Weber, "Recent advances in visit: Amr streamlines and query-driven visualization", 2010,

Download File: LBNL-3185E.pdf (pdf: 2.1 MB)

Hank Childs, David Pugmire, Sean Ahern, Brad Whitlock, Mark Howison, Prabhat, Gunther Weber, E. Wes Bethel, "Extreme Scaling of Production Visualization Software on Diverse Architectures", IEEE Computer Graphics and Applications, January 2010, 30:22--31, LBNL 3403E, doi: 10.1109/MCG.2010.51

E. Wes Bethel, "Using wesBench to Study the Rendering Performance of Graphics Processing Units", 2010, LBNL 3025E,

Download File: LBNL-3025E.pdf (pdf: 1.2 MB)

Oliver Rübel, Sean Ahern, E. Wes Bethel, D. Biggin, Hank Childs, Estelle, Angela DePace, Michael B. Eisen Charless C. Fowlkes, Cameron G. R. Geddes, Hagen, Bernd Hamann, Min-Yu Huang, Soile E. Keränen, David W. Knowles, Cris L. Hendriks, Jitendra Malik, Jeremy Meredith Peter Messmer, Prabhat, Daniela Ushizima, H. Weber, Kesheng Wu, "Coupling visualization and data analysis for knowledge from multi-dimensional scientific data", Procedia Computer Science, 2010, 1:1751--1758, doi: 10.1016/j.procs.2010.04.197

E. Wes Bethel, Hank Childs, Ajith Mascarenhas, Valerio Pascucci, and Prabhat, "Scientific Data Management Challenges in High Performance Visual Data Analysis", Scientific Data Management: Challenges, Existing Technology, and Deployment, edited by Arie Shoshani and Doron Rotem, (Chapman & Hall CRC Press: December 2009) LBNL 1449E,

E. W. Bethel, C. Johnson, S. Ahern, J. Bell, P.-T. Bremer, H. Childs, E. Cormier-Michel, M. Day, E. Deines, T. Fogal, C. Garth, C. G. R. Geddes, H. Hagen, B. Hamann, C. Hansen, J. Jacobsen, K. Joy, J. Kruger, J. Meredith, P. Messmer, G. Ostrouchov, V. Pascucci, K. Potter, Prabhat, D. Pugmire, O. Rubel, A. Sanderson, C. Silva, D. Ushizima, G. Weber, B. Whitlock, K. Wu, "Occam's Razor and Petascale Visual Data Analysis", SciDAC 2009, J. of Physics: Conference Series, San Diego, California, July 2009, LBNL 2210E,

Download File: BethelETALSciDAC2009.pdf (pdf: 23 MB)

Luke Gosink, Kesheng Wu, E. Wes Bethel, John D. Owens, Kenneth I. Joy, "Data Parallel Bin-based Indexing for Answering Queries on Multi-core Architecture", Proceedings of the 21st International Conference on Scientific and Statistical Database Management (SSDBM), June 2009, 5566:110-129, LBNL 2211E,

Download File: LBNL-2211E.pdf (pdf: 219 KB)

Shoaib Kamil, Cy Chan, Samuel Williams, Leonid Oliker, John Shalf, Mark Howison, E. Wes Bethel, Prabhat, "A Generalized Framework for Auto-tuning Stencil Computations", BEST PAPER AWARD - Cray User Group Conference (CUG), Atlanta, GA, May 4, 2009, LBNL 2078E,

Download File: cug09-autotune.pdf (pdf: 354 KB)

Best Paper Award

K Wu, S Ahern, EW Bethel, J Chen, H Childs, C Geddes, J Gu, H Hagen, B Hamann, J Lauret, others, "FastBit: Interactively Searching Massive Data", Proc. of SciDAC 2009, 2009, LBNL 2164E,

Download File: LBNL-2164E.pdf (pdf: 3.2 MB)

Oliver R\ ubel, Cameron GR Geddes, Estelle Cormier-Michel, Kesheng Wu, Gunther H Weber, Daniela M Ushizima, Peter Messmer, Hans Hagen, Bernd Hamann, Wes Bethel, others, "Automatic beam path analysis of laser wakefield particle acceleration data", Computational Science \& Discovery, January 2009, 2:015005, LBNL 2734E,

Download File: LBNL-2734E.pdf (pdf: 15 MB)

C. G. R. Geddes, E Cormier-Michel, E. H. Esarey, C. B. Schroeder, J.-L. Vay, W. P. Leemans, D. L.. Bruhwiler, J. R. Cary, B. Cowan, M. Durant, P. Hamill, P. Messmer, P. Mullowney, C. Nieter, K. Paul, S. Shasharina, S. Veitzer, G. Weber, O. Rübel, D. Ushizima, Prabhat, E. W.Bethel, K. Wu, Large Fields for Smaller Facility Sources, SciDAC Review, Pages: 13-21, 2009,

Download File: geddes.pdf (pdf: 344 KB)

E Bethel, "Modern Scientific Visualization is More than Just Pretty Pictures", January 2009, LBNL 1450E,

Download File: LBNL-1450E.pdf (pdf: 9.8 MB)

Luke J Gosink, Kesheng Wu, E Wes Bethel, John D Owens, Kenneth I Joy, "Data parallel bin-based indexing for answering queries on multi-core architectures", International Conference on Scientific and Statistical Database Management, 2009, 110--129,

E Wes Bethel, Chris Johnson, Sean Ahern, John Bell, Peer-Timo Bremer, Hank Childs, Estelle Cormier-Michel, Marc Day, Eduard Deines, Tom Fogal, others, "Occam s razor and petascale visual data analysis", Journal of Physics: Conference Series, 2009, 180:012084,

C. Garth, E. Deines, K. Joy, E. W. Bethel, H. Childs, G. Weber, S. Ahern, D. Pugmire, A. Sanderson, C. Johnson, Twists and Turns: Vector Field Visual Data Analysis for Petascale Computational Science, SciDAC Review, Pages: 10-21, 2009,

Download File: vacet.pdf (pdf: 1.5 MB)

E. Wes Bethel, "High Performance, Three-Dimensional Bilateral Filtering", 2009, LBNL 1601E,

Luke J. Gosink, Kesheng Wu, E. Wes Bethel, D. Owens, Kenneth I. Joy, Data Parallel Bin-Based Indexing for Answering Queries Multi-core Architectures, Lecture Notes in Computer Science, Pages: 110--129 2009, doi: 10.1007/978-3-642-02279-1_9

Oliver R\ ubel, Cameron G R Geddes, Estelle, Kesheng Wu, Prabhat, Gunther H, Daniela M Ushizima, Peter Messmer, Hans, Bernd Hamann, Wes Bethel, "Automatic beam path analysis of laser wakefield acceleration data", Computational Science \& Discovery, 2009, 2:015005,

O. Rübel, Prabhat, K. Wu, H. Childs, J. Meredith, C.G.R. Geddes, E. Cormier-Michel, S. Ahern, G.H. Weber, P. Messmer, H. Hagen, B. Hamann and E.W. Bethel, "High Performance Multivariate Visual Data Exploration for Extemely Large Data", Supercomputing (SC), Austin, Texas, USA, November 2008, LBNL 716E,

Download File: LBNL-716E.pdf (pdf: 8.7 MB)

O. Rübel, Prabhat, K. Wu, H. Childs, J. Meredith, C.G.R. Geddes, E. Cormier-Michel, S. Ahern, G.H. Weber, P. Messmer, H. Hagen, B. Hamann and E.W. Bethel, "Application of High-performance Visual Analysis Methods to Laser Wakefield Particle Acceleration Data", IEEE Visualization 2008, October 2008,

Download File: LBNL-952E.pdf (pdf: 1.3 MB)

B. Paul, S. Ahern, E. W. Bethel, E. Brugger, R. Cook, J. Daniel, K. Lewis, J. Owen, and D. Southard, "Chromium Renderserver: Scalable and Open Remote Rendering Infrastructure", IEEE Transactions on Visualization and Computer Graphics, May 2008, 14:627-639, LBNL 63693,

Download File: LBNL-63693-CRRS.pdf (pdf: 2.7 MB)

G.H. Weber, V. Beckner, H. Childs, T. Ligocki, M. Miller, B. van Straalen, E.W. Bethel, "Visualization of Scalar Adaptive Mesh Refinement Data", Numerical Modeling of Space Plasma Flows: Astronum-2007 (Astronomical Society of the Pacific Conference Series), April 2008, 385:309-320, LBNL 220E,

Download File: LBNL-220E.pdf (pdf: 1.5 MB)

J. Chen, I. Yoon and E. W. Bethel, "Interactive, Internet Delivery of Visualization via Structured, Prerendered Multiresolution Imagery.", IEEE Transactions on Visualization and Computer Graphics, March 2008, 14:302-312, LBNL 62252,

Download File: LBNL-62252-chen-tvcg.pdf (pdf: 2.1 MB)

William T.C. Kramer, John M. Shalf, E. Wes Bethel, D. Agarwal, Michael Banda, John Hules, Juan C. Meza, Leonid Oliker, Horst Simon, David Skinner, Francesca Verdier, Howard Walter, Michael Wehner, and Katherine Yelick, "HPC in 2016: A View Point from NERSC", Proceedings of the Cray User Group Conference, Helsinki, Finland, 2008,

Luke J Gosink, "Bin-hash indexing: A parallel method for fast query processing", 2008, LBNL 729E,

Download File: LBNL-729E.pdf (pdf: 511 KB)

E. Wes Bethel, Oliver Rübel, Prabhat, Wu, Gunther H. Weber, Valerio Pascucci Hank Childs, Ajith Mascarenhas, Jeremy, Sean Ahern, "Modern Scientific Visualization is More than Just Pictures", Numerical Modeling of Space Plasma Flows: (Astronomical Society of the Pacific Series), St. Thomas, USVI, 2008, 301--317,

Luke J. Gosink, Kesheng Wu, E. Wes Bethel, D. Owens, Kenneth I. Joy, Bin-Hash Indexing: A Parallel Method For Fast Processing, 2008,

Oliver R\ ubel, Prabhat, Kesheng Wu, Hank, Jeremy Meredith, Cameron G. R. Geddes, Cormier-Michel, Sean Ahern, Gunther H., Peter Messmer, Hans Hagen, Bernd Hamann E. Wes Bethel, Application of High-performance Visual Analysis to Laser Wakefield Particle Acceleration Data, IEEE Visualization 2008, 2008,

Oliver R\ ubel, Prabhat, Kesheng Wu, Hank, Jeremy Meredith, Cameron G. R. Geddes, Cormier-Michel, Sean Ahern, Gunther H., Peter Messmer, Hans Hagen, Bernd Hamann E. Wes Bethel, High Performance Multivariate Visual Data Exploration Extemely Large Data, SuperComputing 2008 (SC08), Pages: 51 2008,

Daniela Ushizima, Oliver Rübel, Prabhat, Gunther Weber, E. Wes Bethel, Cecilia Aragon, Cameron Geddes, Estelle Cormier-Michel, Bernd Hamann, Peter Messmer, Hans Hagen, "Automated Analysis for Detecting Beams in Laser Wakefield Simulations", 2008 Seventh International Conference on Machine Learning and Applications, Proceedings of IEEE ICMLA'08, 2008, 382-387, LBNL 960E,

Download File: LBNL-960E.pdf (pdf: 528 KB)

Luke J. Gosink, John C. Anderson, E. Wes Bethel, Kenneth I. Joy, "Query-Driven Visualization of Time-Varying Adaptive Mesh Refinement Data", IEEE Transactions on Visualization and Computer Graphics, 2008, 14:1715-1722, LBNL 803E,

Download File: LBNL-803E.pdf (pdf: 4.3 MB)

E. Wes Bethel, Chris Johnson, Charles Hansen, Claudio Silva, Steven Parker, Allen Sanderson, Lee Myers, Martin Cole, Xavier Tricoche, Sean Ahern, George Ostrouchov, Dave Pugmire, Jamison Daniel, Jeremy Meredith, Valerio Pascucci, Hank Childs, Peer-Timo Bremer, Ajith Mascarenhas, Ken Joy, Bernd Hamann, Christoph Garth, Cecilia Aragon, Gunther Weber, and Prabhat, Seeing the Unseeable, SciDAC Review, Pages: 24-33, 2008,

Download File: vacet2.pdf (pdf: 1.1 MB)

O. Rübel, G. H. Weber, M-Y Huang, E. W. Bethel, S. V. E. Keränen, C. C. Fowlkes, C. L. Luengo Hendriks, A. H. DePace, L. Simirenko, M. B. Eisen, M. D. Biggin, H. Hagen, J. Malik, D. W. Knowles and B. Hamann, "PointCloudXplore 2: Visual Exploration of 3D Gene Expression", Visualization of Large and Unstructured Data Sets, edited by C. Garth, H. Hagen, M. Hering-Bertram, (Gesellschaft fuer Informatik (GI): 2008) LBNL 249E,

Download File: LBNL-249E.pdf (pdf: 2.9 MB)

K. Wu, K. Stockinger, A. Shoshani, Wes, "FastBit--Helps Finding the Proverbial Needle in a", 2006, LBNL LBNL-PUB/963,

Luke Gosink, John Shalf, Kurt Stockinger, Wu, Wes Bethel, "HDF5-FastQuery: Accelerating Complex Queries on Datasets using Fast Bitmap Indices", SSDBM 2006, Vienna, Austria, July 2006, IEEE Computer Society Press., 2006, 149--158,

Luke Gosink, John Shalf, Kurt Stockinger, Kesheng Wu, Wes Bethel, HDF5-FastQuery: Accelerating complex queries on HDF datasets using fast bitmap indices, 18th International Conference on Scientific and Statistical Database Management (SSDBM 06), Pages: 149--158 2006,

E. Wes Bethel, Scott Campbell, Eli Dart, Kurt Stockinger, Kesheng Wu, Accelerating Network Traffic Analysis Using Visualization, Symposium on Visual Analytics Science and Technology Baltimore, Maryland, USA, October 31 - November 2006, Pages: 115--122 2006,

E. Wes Bethel, Scott Campbell, Eli Dart, John Shalf, Kurt Stockinger, Kesheng Wu, High Performance Visualization using Query-Driven and Analytics, 2006,

Kurt Stockinger, E. Wes Bethel, Scott Campbell, Eli Dart, Kesheng Wu, Detecting distributed scans using high-performance visualization, SC 06, Pages: 82 2006,

E. Wes Bethel, Scott Campbell, Eli Dart, Lee, Steven A. Smith, Kurt Stockinger, Tierney, Kesheng Wu, "Interactive Analysis of Large Network Data Collections Query-Driven Visualization", 2005,

Kurt Stockinger, John Shalf, Kesheng Wu, E Wes Bethel, "Query-driven visualization of large data sets", VIS 05. IEEE Visualization, 2005., 2005, 167--174,

Kurt Stockinger, John Shalf, Wes Bethel, Kesheng Wu, DEX: Increasing the Capability of Scientific Data Analysis by Using Efficient Bitmap Indices to Accelerate Scientific Visualization, SSDBM, Pages: 35-44 2005,

Kurt Stockinger, Kesheng Wu, Scott Campbell, Lau, Mike Fisk, Eugene Gavrilov, Alex, Christopher E. Davis, Rick Olinger, Rob, Jim Prewett, Paul Weber, Thomas P., E. Wes Bethel, Steve Smith, Network Traffic Analysis With Query Driven, SC 2005, 2005,

Horst D. Simon, William T.C. Kramer, David H. Bailey, Michael J. Banda, E. Wes Bethel, Jonathan T. Carter, James M. Craw, William J. Fortney, John A. Hules, Nancy L. Meyer, Juan C. Meza, Esmond G. Ng, Lynn E. Rippe, William C. Saphir, Francesca Verdier, Howard A. Walter, Katherine A. Yelick, "Science-Driven Computing: NERSC's Plan for 2006-2010", 2005,

Gunther H. Weber, Martin Öhler, Oliver Kreylos, John Shalf, Wes Bethel, Bernd Hamann, Gerik Scheuermann, "Parallel Cell Projection Rendering of Adaptive Mesh Refinement Data", IEEE Symposium on Parallel and Large-Data Visualization and Graphics, 2003, 51-60,

E. Wes Bethel, Greg Abram, John Shalf, Randall Frank, Jim Ahrens, Steve Parker, N. Samatova, Mark Miller, Interoperability of Visualization Software and Data Models is NOT an Achievable Goal, IEEE Visualization, Pages: 607-610 2003,

T. J. Jankun-Kelly, Kreylos, Ma, Hamann, I. Joy, John Shalf, E. Wes Bethel, Deploying Web-Based Visual Exploration Tools on the Grid, IEEE Computer Graphics and Applications, Pages: 40-50 2003,

E. Wes Bethel, John Shalf, Grid-Distributed Visualizations Using Connectionless Protocols, IEEE Computer Graphics and Applications, Pages: 51-59 2003,

John Shalf, E. Wes Bethel, The Grid and Future Visualization System Architectures, IEEE Computer Graphics and Applications, Pages: 6-9 2003,

Reinhard Gentz, Sean Peisert, Joshua Boverhof, Daniel Gunter, "SPARCS: Stream-Processing Architecture applied in Real-time Cyber-physical Security", Proceedings of the 15th IEEE International Conference on e-Science (eScience), San Diego, CA, IEEE, September 2019, doi: 10.1109/eScience.2019.00028

YW Cheah, J Boverhof, A Elbashandy, D Agarwal, J Leek, T Epperly, J Eslick, D Miller, "Data management and simulation support accelerating carbon capture through computing", Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016, 2017, 389--398, doi: 10.1109/eScience.2016.7870924

S Peisert, R Gentz, J Boverhof, C McParland, S Engle, A Elbashandy, D Gunter, "LBNL Open Power Data", January 2017, doi: 10.21990/C21599

DC Miller, D Agarwal, D Bhattacharyya, J Boverhof, YW Cheah, Y Chen, J Eslick, J Leek, J Ma, P Mahapatra, B Ng, NV Sahinidis, C Tong, SE Zitney, "Innovative computational tools and models for the design, optimization and control of carbon capture processes", Computer Aided Chemical Engineering, 2016, 38:2391--2396, doi: 10.1016/B978-0-444-63428-3.50403-3

S Cholia, D Skinner, J Boverhof, "NEWT: A RESTful service for building High: Performance Computing web applications", 2010 Gateway Computing Environments Workshop, GCE 2010, 2010, doi: 10.1109/GCE.2010.5676125

K Chard, W Tan, J Boverhof, R Madduri, I Foster, Wrap Scientific Applications as WSRE Grid Services using gRAVI, 2009 IEEE INTERNATIONAL CONFERENCE ON WEB SERVICES, VOLS 1 AND 2, Pages: 83--+ 2009, doi: 10.1109/ICWS.2009.110

A Baranovski, K Beattie, S Bharathi, J Boverhof, J Bresnahan, A Chervenak, I Foster, T Freeman, D Gunter, K Keahey, C Kesselman, R Kettimuthu, N Leroy, M Link, M Livny, R Madduri, G Oleynik, L Pearlman, R Schuler, B Tierney, Enabling petascale science: Data management, troubleshooting, and scalable science services, Journal of Physics: Conference Series, 2008, doi: 10.1088/1742-6596/125/1/012068

M Humphrey, G Wasson, J Gawor, J Bester, S Lang, I Foster, S Pickles, M McKeown, K Jackson, J Boverhof, M Rodriguez, L Berkeley, S Meder, IEEE, State and events for web services: A comparison of five WS-resource framework and WS-notification implementations, 14th IEEE International Symposium on High Performance Distributed Computing, Proceedings, Pages: 3--13 2005, doi: 10.1109/HPDC.2005.1520928

L Smale, J Boverhof, The suprachiasmatic nucleus and intergeniculate leaflet of Arvicanthis niloticus, a diurnal murid rodent from east Africa, JOURNAL OF COMPARATIVE NEUROLOGY, Pages: 190--208 1999, doi: 10.1002/(SICI)1096-9861(19990111)403:2<190::AID-CNE4>3.0.CO;2-K

D.K. Sung, Y. Son, A. Sim, K. Wu, S. Byna, H. Tang, H. Eom, C. Kim, S. Kim, "A2FL: Autonomous and Adaptive File Layout in HPC through Real-time Access Pattern Analysis", 38th IEEE International Parallel & Distributed Processing Symposium (IPDPS2024), 2024,

Neeraj Rajesh, Keith Bateman, Jean Luca Bez, Suren Byna, Anthony Kougkas, Xian-He Sun, "TunIO: An AI-powered Framework for Optimizing HPC I/O", 38th IEEE International Parallel & Distributed Processing Symposium, San Fransicso, CA, US, May 27, 2024,

Hammad Ather, Jean Luca Bez, Yankun Xia, Suren Byna, "Drilling Down I/O Bottlenecks with Cross-layer I/O Profile Exploration", 38th IEEE International Parallel & Distributed Processing Symposium, San Francisco, CA, USA, May 27, 2024,

Wei Zhang, Houjun Tang, Suren Byna, "IDIOMS: Index-powered Distributed Object-centric Metadata Search for Scientific Data Management", The 24th IEEE/ACM international Symposium on Cluster, Cloud and Internet Computing. Philadelphia, 2024 (CCGrid 2024), May 9, 2024,

Download File: 956600a598.pdf (pdf: 782 KB)

Jean Luca Bez, Houjun Tang, Scot Breitenfeld, Huihuo Zheng, Wei-Keng Liao, Kaiyuan Hou, Zanhua Huang, Suren Byna, "h5bench: Exploring HDF5 Access Patterns Performance in Pre-Exascale Platforms", Concurrency and Computation: Practice and Experience (CCPE), January 31, 2024,

Jean Luca Bez, Suren Byna, Shadi Ibrahim, "I/O Access Patterns in HPC Applications: A 360-Degree Survey", ACM Computing Surveys, September 15, 2023, 56, doi: 10.1145/3611007

Bin Dong, Jean Luca Bez, Suren Byna, "AIIO: Using Artificial Intelligence for Job-Level and Automatic I/O Performance Bottleneck Diagnosis.", In Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’23), June 16, 2023,

Download File: IODiagnose-final.pdf (pdf: 1.9 MB)

Hammad Ather, Jean Luca Bez, Boyana Norris, Suren Byna, "Illuminating the I/O Optimization Path of Scientific Applications", High Performance Computing: 38th International Conference, ISC High Performance 2023, Hamburg, Germany, May 21–25, 2023, Proceedings, Hamburg, Germany, Springer-Verlag, May 21, 2023, 22–41, doi: https://doi.org/10.1007/978-3-031-32041-5_2

The existing parallel I/O stack is complex and difficult to tune due to the interdependencies among multiple factors that impact the performance of data movement between storage and compute systems. When performance is slower than expected, end-users, developers, and system administrators rely on I/O profiling and tracing information to pinpoint the root causes of inefficiencies. Despite having numerous tools that collect I/O metrics on production systems, it is not obvious where the I/O bottlenecks are (unless one is an I/O expert), their root causes, and what to do to solve them. Hence, there is a gap between the currently available metrics, the issues they represent, and the application of optimizations that would mitigate performance slowdowns. An I/O specialist often checks for common problems before diving into the specifics of each application and workload. Streamlining such analysis, investigation, and recommendations could close this gap without requiring a specialist to intervene in every case. In this paper, we propose a novel interactive, user-oriented visualization, and analysis framework, called Drishti. This framework helps users to pinpoint various root causes of I/O performance problems and to provide a set of actionable recommendations for improving performance based on the observed characteristics of an application. We evaluate the applicability and correctness of Drishti using four use cases from distinct science domains and demonstrate its value to end-users, developers, and system administrators when seeking to improve an application’s I/O performance.

Hammad Ather, Jean Luca Bez, Boyana Norris, Suren Byna, "Illuminating the I/O Optimization Path of Scientific Applications", International Conference on High Performance Computing (ISC'23), Springer Nature Switzerland, May 10, 2023, 22-41, doi: https://doi.org/10.1007/978-3-031-32041-5_2

S. Kim, A. Sim, K. Wu, S. Byna, Y. Son, H. Eom, "Design and Implementation of I/O Performance Prediction Scheme on HPC Systems through Large-scale Log Analysis", Journal of Big Data, 2023, 10(65), doi: 10.1186/s40537-023-00741-4

Jean Luca Bez, Hammad Ather, Suren Byna, "Drishti: Guiding End-Users in the I/O Optimization Journey", PDSW 2022, held in conjunction with SC22, 2022,

Jean Luca Bez, Suren Byna, April 2019 Darshan counters from the Cori supercomputer [Data set], Zenodo, 2022, doi: 10.5281/zenodo.6476501

Sunggon Kim, Alex Sim, Kesheng Wu, Suren Byna, Yongseok Son, "Design and implementation of dynamic I/O control scheme for large scale distributed file systems", Cluster Computing, 2022, 25(6):1--16, doi: 10.1007/s10586-022-03640-0

Download File: wu2022.bib (bib: 22 KB)

Jean Luca Bez, Ahmad Maroof Karimi, Arnab K. Paul, Bing Xie, Suren Byna, Philip Carns, Sarp Oral, Feiyi Wang, Jesse Hanley, "Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production Load", 31st International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC '22), Association for Computing Machinery, June 27, 2022, 43–55, doi: 10.1145/3502181.3531461

D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, W. Arndt, J. Blaschke, S. Byna, R. Cheema, S. Cholia, M. Day, B. Enders, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, T. Lehman, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, L. Stephey, R. Thomas, G. Torok, "LBNL Superfacility Project Report", Lawrence Berkeley National Laboratory, 2022, doi: 10.48550/arXiv.2206.11992

Jean Luca Bez, Suren Byna, Understanding I/O Behavior with Interactive Darshan Log Analysis, Exascale Computing Project (ECP) Community Days BoF, 2022,

Houjun Tang, Quincey Koziol, John Ravi, and Suren Byna,, "Transparent Asynchronous Parallel I/O using Background Threads", IEEE Transactions on Parallel and Distributed Systems, April 4, 2022, 33, doi: 10.1109/TPDS.2021.3090322

Qiao Kang, Scot Breitenfeld, Kaiyuan Hou, Wei-keng Liao, Robert Ross, and Suren Byna,, "Optimizing Performance of Parallel I/O Accesses to Non-contiguous Blocks in Multiple Array Variables", IEEE BigData 2021 conference, December 19, 2021,

J. Bang, C. Kim, K. Wu, A. Sim, S. Byna, H. Sung, H. Eom, "An In-Depth I/O Pattern Analysis in HPC Systems", IEEE International Conference on High Performance Computing, Data & Analytics (HiPC2021), 2021, doi: 10.1109/HiPC53243.2021.00056

Houjun Tang, Bing Xie, Suren Byna, Phillip Carns, Quincey Koziol, Sudarsun Kannan, Jay Lofstead, and Sarp Oral,, "SCTuner: An Auto-tuner Addressing Dynamic I/O Needs on Supercomputer I/O Sub-systems", 6th International Parallel Data Systems Workshop (PDSW) 2021, held in conjunction with SC21, November 21, 2021,

Cong Xu, Suparna Bhattacharya, Martin Foltin, Suren Byna, and Paolo Faraboschi, "Data-Aware Storage Tiering for Deep Learning", 6th International Parallel Data Systems Workshop (PDSW) 2021, held in conjunction with SC21, November 21, 2021,

Wei Zhang, Suren Byna, Hyogi Sim, Sangkeun Lee, Sudharshan Vazhkudai, and Yong Chen,, "Exploiting User Activeness for Data Retention in HPC Systems", International Conference for High Performance Computing, Networking, Storage and Analysis (SC '21), November 21, 2021, doi: https://doi.org/10.1145/3458817.3476201

Download File: 3458817.3476201-2.pdf (pdf: 1.5 MB)

Bo Fang, Daoce Wang, Sian Jin, Quincey Koziol, Zhao Zhang, Qiang Guan, Suren Byna, Sriram Krishnamoorthy, and Dingwen Tao,, "Characterizing Impacts of Storage Faults on HPC Applications: A Methodology and Insights", IEEE Cluster 2021, September 1, 2021,

Suren Byna, Houjun Tang, and Quincey Koziol,, Automatic and Transparent Scientific Data Management with Object Abstractions, PASC 2021, in a Minisymposium on "Data Movement Orchestration on HPC Systems", July 31, 2021,

Bing Xie, Houjun Tang, Suren Byna, Jesse Hanley, Quincey Koziol, Tonglin Li, Sarp Oral,, "Battle of the Defaults: Extracting Performance Characteristics of HDF5 under Production Load", CCGrid 2021, May 31, 2021,

Tonglin Li, Suren Byna, Quincey Koziol, Houjun Tang, Jean Luca Bez, Qiao Kang, "h5bench: HDF5 I/O Kernel Suite for Exercising HPC I/O Patterns", Cray User Group (CUG) 2021, January 1, 2021,

Jean Luca Bez, Houjun Tang, Bing Xie, David Williams-Young, Rob Latham, Rob Ross, Sarp Oral, Suren Byna, "I/O Bottleneck Detection and Tuning: Connecting the Dots using Interactive Log Analysis", 2021 IEEE/ACM Sixth International Parallel Data Systems Workshop (PDSW), January 1, 2021, 15-22, doi: 10.1109/PDSW54622.2021.00008

D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, S. Byna, R. Cheema, S. Cholia, M. Day, B. Enders, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, R. Thomas, G. Torok, The Superfacility project: automated pipelines for experiments and HPC, International Conference for High Performance Computing, Networking, Storage, and Analysis (SC20), State of the Practice (SOP), 2020,

B. Enders, D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, S. Byna, R. Cheema, S. Cholia, M. Day, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, R. Thomas, G. Torok, "Cross-facility science with the Superfacility Project at LBNL", 2nd Workshop on Large-scale Experiment-in-the-Loop Computing (XLOOP 2020), in conjunction with the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 20), 2020, doi: 10.1109/XLOOP51963.2020.00006

Bin Dong, Ver\ onica Rodr\ \iguez Tribaldos, Xin Xing, Suren Byna, Jonathan Ajo-Franklin, Kesheng Wu, "DASSA: Parallel DAS Data Storage and Analysis for Subsurface Event Detection", 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS), July 14, 2020, 254--263,

Download File: paper.pdf (pdf: 3.9 MB)

Sunggon Kim, Alex Sim, Kesheng Wu, Suren Byna, Yongseok Son, Hyeonsang Eom, "Towards hpc i/o performance prediction through large-scale log analysis", Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2020), 2020, 77--88, doi: 10.1145/3369583.3392678

Jiwoo Bang, Chungyong Kim, Kesheng Wu, Alex Sim, Suren Byna, Sunggon Kim, Hyeonsang Eom, "HPC Workload Characterization Using Feature Selection and Clustering", ACM International Workshop on System and Network Telemetry and Analysis (SNTA 2020), in conjunction with The 29th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC 2020), 2020, 33--40, doi: 10.1145/3391812.3396270

Houjun Tang, Suren Byna, Bin Dong, Quincey Koziol, "Parallel Query Service for Object-centric Data Management Systems", 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), IEEE, May 18, 2020, 406-415,

Suren Byna, M. Scot Breitenfeld, Bin Dong, Quincey Koziol, Elena Pourmal, Dana Robinson, Jerome Soumagne, Houjun Tang, Venkatram Vishwanath, and Richard Warren, "ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems", Journal of Computer Science and Technology 2020, 35(1): 145-160, February 2, 2020, doi: 10.1007/s11390-020-9822-9

Richard Warren, Jerome Soumagne, Jingqing Mu, Houjun Tang, Suren Byna, Bin Dong, Quincey Koziol, "Analysis in the Data Path of an Object-centric Data Management System", 26th IEEE International Conference on High Performance Computing, Data, and Analytics (HiPC 2019), December 18, 2019,

Houjun Tang, Suren Byna, Stephen Bailey, Zarija Lukic, Jialin Liu, Quincey Koziol, Bin Dong, "Tuning Object-centric Data Management Systems for Large Scale Scientific Applications", 26th IEEE International Conference on High Performance Computing, Data, and Analytics (HiPC 2019), December 18, 2019,

Wei Zhang, Suren Byna, Chenxu Niu, Yong Chen, "Exploring Metadata Search Essentials for Scientific Data Management", 26th IEEE International Conference on High Performance Computing, Data, and Analytics (HiPC 2019), December 17, 2019,

Tirthak Patel, Suren Byna, Glenn K. Lockwood, Devesh Tiwari, "Revisiting I/O Behavior in Large-Scale Storage Systems: The Expected and the Unexpected", Supercomputing 2019 (SC19), November 24, 2019, doi: 10.1145/3295500.3356183

Donghe Kang, Oliver Rübel, Suren Byna, Spyros Blanas, "Comparison of Array Management Library Performance - A Neuroscience Use Case", SC19 Poster, November 20, 2019,

Wei Zhang, Suren Byna, Houjun Tang, Brody Williams, Yong Chen, "MIQS: Metadata Indexing and erying Service for Self-Describing File Formats", The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19), November 19, 2019,

Download File: 3295500.3356146.pdf (pdf: 1 MB)

Houjun Tang, Quincey Koziol, Suren Byna, John Mainzer, Tonglin Li, "Enabling Transparent Asynchronous I/O using Background Threads", 2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW 2019), November 19, 2019, doi: DOI 10.1109/PDSW49588.2019.00006

Megha Agarwal, Divyansh Singhvi, Preeti Malakar, Suren Byna, "Active Learning-based Automatic Tuning and Prediction of Parallel I/O Performance", 2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW), November 19, 2019, doi: DOI 10.1109/PDSW49588.2019.00007

Glenn K. Lockwood, Shane Snyder, Suren Byna, Philip Carns, Nicholas J. Wright, "Understanding Data Motion in the Modern HPC Data Center", 2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW), November 19, 2019, doi: DOI 10.1109/PDSW49588.2019.00012

S. Kim, A. Sim, K. Wu, S. Byna, T. Wang, Y. Son, H. Eom, "DCA-IO: A Dynamic I/O Control Scheme for Parallel and Distributed File System", 19th Annual IEEE/ACM International Symposium in Cluster, Cloud, and Grid Computing (CCGrid 2019), 2019, doi: 10.1109/CCGRID.2019.00049

Teng Wang, Suren Byna, Glenn Lockwood, Philip Carns, Shane Snyder, Sunggon Kim, Nicholas Wright, "A Zoom-in Analysis of I/O Logs to Detect Root Causes of I/O Performance Bottlenecks", IEEE/ACM CCGrid 2019, May 14, 2019,

Tonglin Li, Quincey Koziol, Houjun Tang, Jialin Liu, Suren Byna, "I/O Performance Analysis of Science Applications Using HDF5 File-level Provenance", Cray User Group (CUG) 2019, May 10, 2019,

Jingqing Mu, Jerome Soumagne, Suren Byna, Quincey Koziol, Houjun Tang, Richard Warren, "Interfacing HDF5 with A Scalable Object-centric Storage System on Hierarchical Storage", Cray User Group (CUG) 2019, May 7, 2019,

Babak Behzad, Suren Byna, Prabhat, and Marc Snir, "Optimizing I/O Performance of HPC Applications with Autotuning", ACM Transactions on Parallel Computing (TOPC), February 28, 2019,

Beytullah Yildiz, Kesheng Wu, Suren Byna, Arie Shoshani, "Parallel membership queries on very large scientific data sets using bitmap indexes", Concurrency and Computation: Practice and Experience, January 1, 2019, 31:e5157,

Many scientific applications produce very large amounts of data as advances in hardware fuel computing and experimental facilities. Managing and analyzing massive quantities of scientific data is challenging as data are often stored in specific formatted files, such as HDF5 and NetCDF, which do not offer appropriate search capabilities. In this research, we investigated a special class of search capability, called membership query, to identify whether queried elements of a set are members of an attribute. Attributes that naturally have classification values appear frequently in scientific domains such as category and object type as well as in daily life such as zip code and occupation. Because classification attribute values are discrete and require random data access, performing a membership query on a large scientific data set creates challenges. We applied bitmap indexing and parallelization to membership queries to overcome these challenges. Bitmap indexing provides high performance not only for low cardinality attributes but also for high cardinality attributes, such as floating‐point variables, electric charge, or momentum in a particle physics data set, due to compression algorithms such as Word‐Aligned Hybrid. We conducted experiments, in a highly parallelized environment, on data obtained from a particle accelerator model and a synthetic data set.

Bin Dong, Kesheng Wu, Suren Byna, Houjun Tang, "SLOPE: Structural Locality-Aware Programming Model for Composing Array Data Analysis", International Conference on High Performance Computing, January 1, 2019, 61--80,

Download File: slope-cr.pdf (pdf: 623 KB)

Bin Dong, Patrick Kilian, Xiaocan Li, Fan Guo, Suren Byna, Kesheng Wu, "Terabyte-scale Particle Data Analysis: An ArrayUDF Case Study", Proceedings of the 31st International Conference on Scientific and Statistical Database Management, January 1, 2019, 202--205,

Download File: VPIC-SSDBM-Camera-ready.pdf (pdf: 271 KB)

Suren Byna, Quincey Koziol, Venkatram Vishwanath, Jerome Soumagne, Houjun Tang, Kimmy Mu, Richard Warren, François Tessier, Bin Dong, Teng Wang, and Jialin Liu, Proactive Data Containers (PDC): An object-centric data store for large-scale computing systems, AGU Fall Meeting, December 13, 2018,

Glenn Lockwood, Shane Snyder, Teng Wang, Suren Byna, Phil Carns, and Nicholas Wright, "A Year in the Life of a Parallel File System", International Conference for High Performance Computing, Networking, and Storage (SC'18), IEEE / ACM, November 15, 2018,

Fahim Chowdhury, Jialin Liu, Quincey Koziol, Thorsten Kurth, Steven Farrell, Suren Byna, Prabhat, Weikuan Yu,, Initial Characterization of I/O in Large-Scale Deep Learning Applications, 3rd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems (PDSW-DISCS), November 13, 2018,

Jialin Liu, Quincey Koziol, Gregory Butler, Neil Fortner, Mohamad Chaarawi, Houjun Tang, Suren Byna, Glenn Lockwood, Ravi Cheema, Kristy Kallback-Rose, Damian Hazen, Prabhat, "Evaluation of HPC Application I/O on Object Storage Systems", 3rd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems (PDSW-DISCS), November 12, 2018,

Wei Zhang, Houjun Tang, Suren Byna, Yong Chen, "DART: Distributed Adaptive Radix Tree for Efficient Affix-based Keyword Search on HPC Systems", Proceedings of the 27th International Conference on Parallel Architectures and Compilation Techniques, November 1, 2018, 24,

Download File: 3243176.3243207.pdf (pdf: 1.1 MB)

Kimmy Mu, Jerome Soumagne, Houjun Tang, Suren Byna, Quincey Koziol, Richard Warren, "A Server-managed Transparent Object Storage Abstraction for HPC", 2018 IEEE International Conference on Cluster Computing (CLUSTER), September 10, 2018,

Teng Wang, Suren Byna, Glenn Lockwood, Nicholas Wright, Phil Carns, and Shane Snyder,, "IOMiner: Large-scale Analytics Framework for Gaining Knowledge from I/O Logs", IEEE Cluster 2018, September 10, 2018,

Teng Wang, Suren Byna, Bin Dong, and Houjun Tang, "UniviStor: Integrated Hierarchical and Distributed Storage for HPC", IEEE Cluster 2018., September 1, 2018,

Houjun Tang, Suren Byna, Francois Tessier, Teng Wang, Bin Dong, Jingqing Mu, Quincey Koziol, Jerome Soumagne, Venkatram Vishwanath, Jialin Liu, and Richard Warren, "Toward Scalable and Asynchronous Object-centric Data Management for HPC", 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) 2018, May 1, 2018,

Haoyuan Xing, Sofoklis Floratos, Spyros Blanas, Suren Byna, Prabhat, Kesheng Wu, and Paul Brown,, "ArrayBridge: Interweaving declarative array processing with imperative high-performance computing", 34th IEEE International Conference on Data Engineering (ICDE) 2018, April 17, 2018,

Bharti Wadhwa, Suren Byna, Ali R. Butt, "Toward Transparent Data Management in Multi-layer Storage Hierarchy for HPC Systems", IEEE International Conference on Cloud Engineering 2018 (IC2E 2018), April 17, 2018,

Haoyuan Xing, Sofoklis Floratos, Spyros Blanas, Suren Byna, M Prabhat, Kesheng Wu, Paul Brown, "ArrayBridge: Interweaving declarative array processing in SciDB with imperative HDF5-based programs", 2018 IEEE 34th International Conference on Data Engineering (ICDE), 2018, 977--988,

Bin Dong, Teng Wang, Houjun Tang, Quincey Koziol, Kesheng Wu, Suren Byna, "ARCHIE: Data analysis acceleration with array caching in hierarchical storage", 2018 IEEE International Conference on Big Data (Big Data), January 1, 2018, 211--220,

Download File: DataElevator-ARCHIE.pdf (pdf: 613 KB)

Kesheng Wu, Surendra Byna, Bin Dong, others, VPIC IO utilities, 2018,

Kesheng Wu, Bin Dong, Surendra Byna, "Scientific Data Services Framework for Plasma Physics", APS, 2018, 2018:BM10--006,

Glenn Lockwood, Shane Snyder, Wucherl Yoo, Kevin Harms, Zachary Nault, Suren Byna, Philip Carns, Nicholas Wright, "UMAMI: A Recipe for Generating Meaningful Metrics through Holistic I/O Performance Analysis", 2nd Joint International Workshop on Parallel Data Storage & Data Intensive Scalable Computing Systems (PDSW-DISCS), 2017 (Held in conjunction with SC17), November 14, 2017,

Houjun Tang, Suren Byna, Bin Dong, Jialin Liu, and Quincey Koziol, "SoMeta: Scalable Object-centric Metadata Management for High Performance Computing", IEEE Cluster 2017, September 5, 2017,

Suren Byna, Mohamad Chaarawi, Quincey Koziol, John Mainzer, and Frank Willmore,, "Tuning HDF5 subfiling performance on parallel file systems", Cray User Group (CUG) meeting 2017, May 10, 2017,

Cong Xu, Shane Snyder, Omkar Kulkarni, Vishwanath Venkatesan, Philip Carns, Suren Byna, Robert Sisneros, and Kalyana Chadalavada,, "DXT: Darshan eXtended Tracing", Cray User Group (CUG) meeting 2017, May 10, 2017,

Jialin Liu, Quincey Koziol, Houjun Tang, François Tessier, Wahid Bhimji, Brandon Cook, Brian Austin, Suren Byna, Bhupender Thakur, Glenn Lockwood, Jack Deslippe, Prabhat, "Understanding the I/O Performance Gap Between Cori KNL and Haswell", Cray User Group Conference 2017 (CUG 2017), May 1, 2017,

Bin Dong, Kesheng Wu, Surendra Byna, Jialin Liu, Weijie Zhao, Florin Rusu, "ArrayUDF: User-defined scientific data analysis on arrays", Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing, January 1, 2017, 53--64,

Download File: hpdc02.pdf (pdf: 921 KB)

M. Bryson, S. Byna (Advisor), A. Sim (Advisor), K. Wu (Advisor), "The Search for Missing Parallel IO Performance on the Cori Supercomputer", International Conference for High Performance Computing, Networking, Storage and Analysis (SC’16), ACM Student Research Competition (SRC), 2016,

Bin Dong, Surendra Byna, Kesheng Wu, "SDS-Sort: Scalable Dynamic Skew-aware Parallel", HPDC 16, New York, NY, USA, ACM, 2016, 57--68, doi: 10.1145/2907294.2907300

Md. Mostofa Ali Patwary, Nadathur Rajagopalan Satish, Narayanan Sundaram, Jialin Liu, Peter Sadowski, Evan Racah, Suren Byna, Craig Tull, Wahid Bhimji, Prabhat, and Pradeep Dubey,, "PANDA: Extreme Scale Parallel K-Nearest Neighbor on Distributed Architectures", 30th IEEE International Parallel & Distributed Processing Symposium (IPDPS) 2016, Chicago, May 23, 2016,

Wahid Bhimji, Debbie Bard, Melissa Romanus, David Paul, Andrey Ovsyannikov, Brian Friesen, Matt Bryson, Joaquin Correa, Glenn K. Lockwood, Vakho Tsulaia, Suren Byna, Steve Farrell, Doga Gursoy, Chris Daley, Vince Beckner, Brian Van Straalen, Nicholas Wright, Katie Antypas, Prabhat,, "Accelerating Science with the NERSC Burst Buffer Early User Program", Cray User Group (CUG) 2016, May 10, 2016,

Cong Xu, Suren Byna, Vishwanath Venkatesan, Robert Sisneros, Omkar Kulkarni, Mohamad Chaarawi, and Kalyana Chadalavada, "LIOProf: Exposing Lustre File System Behavior for I/O Middleware", Cray User Group (CUG) 2016, May 10, 2016,

Dharshi Devendran, Suren Byna, Bin Dong, Brian van Straalen, Hans Johansen, Noel Keen, and Nagiza Samatova,, "Collective I/O Optimizations for Adaptive Mesh Refinement Data Writes on Lustre File System", Cray User Group (CUG) 2016, May 10, 2016,

Harinarayan Krishnan, Burlen Loring, Suren Byna, Michael F. Wehner, Travis A. O'Brien, Prabhat, Chris Paciorek, and Daithi Stone, "Enabling End-to-End Climate Science Workflows in High Performance Computing Environments", The AMS (American Meteorological Society) 96th Annual Meeting, January 6, 2016,

Burlen Loring, Suren Byna, Prabhat, Junmin Gu, Hari Krishnan, Michael Wehner, and Oliver Ruebel, "TECA an Extreme Event Detection and Climate Analysis Package for High Performance Computing", The AMS (American Meteorological Society) 96th Annual Meeting, January 6, 2016,

Houjun Tang, Suren Byna, Steve Harenberg, Xiaocheng Zou, Wenzhao Zhang, Kesheng Wu, Bin Dong, Oliver Rubel, Kristofer Bouchard, Scott Klasky, others, "Usage pattern-driven dynamic data layout reorganization", 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), January 1, 2016, 356--365,

Wenzhao Zhang, Houjun Tang, Steve Harenberg, Surendra Byna, Xiaocheng Zou, Dharshi Devendran, Daniel F Martin, Kesheng Wu, Bin Dong, Scott Klasky, others, "Amrzone: A runtime amr data sharing framework for scientific applications", 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), January 1, 2016, 116--125,

Bin Dong, Surendra Byna, Kesheng Wu, "Sds-sort: Scalable dynamic skew-aware parallel sorting", Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing, January 1, 2016, 57--68,

Download File: SDS-Sort.pdf (pdf: 450 KB)

Xiaocheng Zou, David A Boyuka II, Dhara Desai, Daniel F Martin, Suren Byna, Kesheng Wu, "AMR-aware in situ indexing and scalable querying", Proceedings of the 24th High Performance Computing Symposium, January 1, 2016, 26,

Bin Dong, Suren Byna, Kesheng Wu, Hans Johansen, Jeffrey N Johnson, Noel Keen, others, "Data elevator: Low-contention data movement in hierarchical storage system", 2016 IEEE 23rd international conference on high performance computing (HiPC), January 1, 2016, 152--161,

Download File: 201612-DataElevator-HiPC2016-Bin-Byna.pdf (pdf: 765 KB)

Houjun Tang, Suren Byna, Steve Harenberg, Wenzhao Zhang, Xiaocheng Zou, Daniel F Martin, Bin Dong, Dharshi Devendran, Kesheng Wu, David Trebotich, others, "In situ storage layout optimization for amr spatio-temporal read accesses", 2016 45th International Conference on Parallel Processing (ICPP), January 1, 2016, 406--415,

Wenzhao Zhang, Houjun Tang, Stephen Ranshous, Surendra Byna, Daniel F Mart\ \in, Kesheng Wu, Bin Dong, Scott Klasky, Nagiza F Samatova, "Exploring memory hierarchy and network topology for runtime AMR data sharing across scientific applications", 2016 IEEE International Conference on Big Data (Big Data), January 1, 2016, 1359--1366,

Hari Krishnan, Suren Byna, Michael Wehner, Junmin Gu, Travis O'Brien, Burlen Loring, Daithi Stone, William Collins, Prabhat, Yunjie Liu, Jeffrey Johnson, and Christopher Paciorek, "Enabling Efficient Climate Science Workflows in High Performance Computing Environments", AGU Fall Meeting, 2015, December 13, 2015,

Soyoung Jeon, Prabhat, Suren Byna, Junmin Gu, William Collins, and Michael Wehner,, "Characterization of extreme precipitation within atmospheric river events over California", Advances in Statistical Climatology, Meteorology and Oceanography (ASCMO), November 21, 2015, 1:45-57, doi: 10.5194/ascmo-1-45-2015

Md. Mostofa Ali Patwary, Suren Byna, Nadathur Rajagopalan Satish, Narayanan Sundaram, Zarija Lukic, Vadim Roytershteyn, Michael J. Anderson, Yushu Yao, Mr Prabhat, and Pradeep Dubey, "BD-CATS: Big Data Clustering at Trillion Particle Scale", Supercomputing 2015 (SC15), Supercomputing 2015 (SC15), November 17, 2015,

Babak Behzad, Suren Byna, Prabhat and Marc Snir, "Pattern-driven Parallel I/O Tuning", 10th Parallel Data Storage Workshop (PDSW) 2015, held in conjunction with SC15, 10th Parallel Data Storage Workshop (PDSW) 2015, to be held in conjunction with SC15, November 16, 2015,

Shane Snyder, Philip Carns, Robert Latham, Misbah Mubarak, Chris Carothers, Babak Behzad, Huong Vu Thanh Luu, Suren Byna, and Prabhat, "Techniques for Modeling Large-scale HPC I/O Workloads", the 6th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS15), in conjunction with SC15, the 6th International Workshop on Performance Modeling, Benchmarking and Simulation of High Performa, November 15, 2015,

Jinoh Kim, Bin Dong, Suren Byna, and Kesheng Wu, "Security for the Scientific Data Service Framework", 2nd International Workshop on Privacy and Security of Big Data (PSBD 2015), in conjunction with IEEE BigData 2015, 2015,

Prabhat, Suren Byna, Venkat Vishwanath, Eli Dart, Michael Wehner, and William Collins,, "TECA: Petscale Pattern Recognition for Climate Science", 16th International Conference on Computer Analysis of Images and Patterns (CAIP) 2015, 2015,

Babak Behzad, Suren Byna, Stefan Wild, Prabhat and Marc Snir, "Dynamic Model-driven Parallel I/O Performance Tuning", IEEE Cluster 2015, 2015,

Xiaocheng (Chris) Zou, Suren Byna, Hans Johansen, Daniel Martin, Nagiza F. Samatova, Arie Shoshani, John Wu, "Six-fold Speedup of Ice Calving Detection Achieved by AMR-aware Parallel Connected Component Labeling", SciDAC PI Meeting, July 2015, 2015,

H. Luu, M. Winslett, W. Gropp, R. Ross, P. Carns, K. Harms, Prabhat, S. Byna, Y. Yao,, "A Multi-platform Study of I/O Behavior on Petascale Supercomputers", The 24th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) 2015, 2015,

Xiaocheng Zou, Kesheng Wu, David A. Boyuka, Daniel F. Martin, Suren Byna, Houjun, Kushal Bansal, Terry J. Ligocki, Hans Johansen, and Nagiza F. Samatova, "Parallel In Situ Detection of Connected Components Adaptive Mesh Refinement Data", Proceedings of the Cluster, Cloud and Grid Computing (CCGrid) 2015, 2015,

Suren Byna, Robert Sisneros, Kalyana Chadalavada, Quincey Koziol, "Tuning Parallel I/O on Blue Waters for Writing 10 Trillion Particles", Cray User Group (CUG) meeting 2015, 2015,

Suren Byna, Brian Austin, "Evaluation of Parallel I/O Performance and Energy Consumption with Frequency Scaling on Cray XC30", Cray User Group (CUG) meeting 2015, 2015,

Bin Dong, Surendra Byna, Kesheng Wu, "Heavy-tailed distribution of parallel I/O system response time", Proceedings of the 10th Parallel Data Storage Workshop, 2015, 37--42,

Bin Dong, Surendra Byna, Kesheng Wu, "Spatially clustered join on heterogeneous scientific data sets", 2015 IEEE International Conference on Big Data (Big Data), 2015, 371--380,

Soyoung Jeon, Christopher Paciorek, Prabhat, Surendra Byna, William Collins, Michael Wehner, "Uncertainty Quantification for Characterizing Spatial Tail Dependence under Statistical Framework", AGU, Fall Meeting 2014, 2014,

Babak Behzad, Surendra Byna, Stefan M. Wild, Mr. Prabhat, Marc Snir, "Improving Parallel I/O Autotuning with Performance Modeling", ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC 2014), New York, NY, USA, ACM, 2014, 253--256, doi: 10.1145/2600212.2600708

Spyros Blanas, Kesheng Wu, Surendra Byna, Bin Dong, Arie Shoshani, "Parallel Data Analysis Directly on Scientific File Formats", SIGMOD 14, 2014, 385--396, doi: 10.1145/2588555.2612185

Spyros Blanas, Kesheng Wu, Surendra Byna, Bin Dong, Arie Shoshani, "Parallel data analysis directly on scientific file formats", Proceedings of the 2014 ACM SIGMOD international conference on Management of data, January 1, 2014, 385--396,

M Scot Breitenfeld, Kalyana Chadalavada, Robert Sisneros, Surendra Byna, Quincey Koziol, Neil Fortner, Prabhat, Venkat Vishwanath, "Recent Progress in Tuning Performance of Large-scale I/O with Parallel HDF5", The 9th Parallel Data Storage Workshop (PDSW) held in conjunction with SC14, 2014,

Hsuan-Te Chiu, Jerry Chou, Venkat Vishwanath, Surendra Byna, Kesheng Wu, "Simplifying index file structure to improve I/O performance of parallel indexing", 2014 20th IEEE International Conference on Parallel and Distributed Systems (ICPADS), 2014, 576--583,

Ted Habermann, Andrew Collette, Steve Vincena, Jay Jay Billings, Matt Gerring, Konrad Hinsen, Werner Benger, Filipe RNC Maia, Suren Byna, Pierre de Buyl, "The Hierarchical Data Format (HDF): A Foundation for Sustainable Data and Software", 2nd Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE2), in conjunction with Supercomputing 2014 (SC14), 2014,

Jialin Liu, Surendra Byna, Bin Dong, Kesheng Wu, Yong Chen, "Model-driven data layout selection for improving read performance", 2014 IEEE International Parallel \& Distributed Processing Symposium Workshops, 2014, 1708--1716,

Bin Dong, Surendra Byna, Kesheng Wu, "Parallel query evaluation as a Scientific Data Service", 2014 IEEE International Conference on Cluster Computing (CLUSTER), January 1, 2014, 194--202,

Jialin Liu, S. Byna, Bin Dong, Kesheng Wu, Chen, "Model-Driven Data Layout Selection for Improving Read", Parallel Distributed Processing Symposium Workshops 2014 IEEE International, 2014, 1708--1716, doi: 10.1109/IPDPSW.2014.190

Bin Dong, S. Byna, Kesheng Wu, Parallel query evaluation as a Scientific Data, Cluster Computing (CLUSTER), 2014 IEEE International on, Pages: 194--202 2014, doi: 10.1109/CLUSTER.2014.6968765

Hsuan-Te Chiu, Jerry Chou, Venkat Vishwanath, Byna, Kesheng Wu, Simplifying Index File Structure to Improve I/O of Parallel Indexing, The 20th IEEE International Conference on Parallel and Systems (ICPADS 2014), 2014,

Babak Behzad, Huong Vu Thanh Luu, Joseph Huchette, Surendra Byna, Prabhat, Ruth Aydt, Quincey Koziol, and Marc Snir, "Taming parallel I/O complexity with auto-tuning", In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC '13), 2013,

Babak Behzad, Joseph Huchette, Huong Vu Thanh Luu, Ruth Aydt, Surendra Byna, Yushu Yao, Quincey Koziol, and Prabhat, "A framework for auto-tuning HDF5 applications", Proceedings of the 22nd international symposium on High-performance parallel and distributed computing (HPDC), 2013,

E Wes Bethel, Prabhat Prabhat, Suren Byna, Oliver R\ ubel, K John Wu, Michael Wehner, "Why high performance visual data analytics is both relevant and difficult", Visualization and Data Analysis 2013, January 2013, 8654:86540B, LBNL LBNL-6063E,

Download File: LBNL-6063E.pdf (pdf: 3.6 MB)

Bin Dong, Surendra Byna, Kesheng Wu, "SDS: a framework for scientific data services", Proceedings of the 8th Parallel Data Storage Workshop, January 1, 2013, 27--32,

Bin Dong, Surendra Byna, Kesheng Wu, "Expediting scientific data analysis with reorganization of data", 2013 IEEE International Conference on Cluster Computing (CLUSTER), January 1, 2013, 1--8,

Kuan-Wu Lin, Surendra Byna, Jerry Chou, Wu, "Optimizing FastQuery performance on Lustre file", Proceedings of the 25th International Conference on and Statistical Database Management, 2013, 29,

Bin Dong, S. Byna, Kesheng Wu, Expediting scientific data analysis with of data, Cluster Computing (CLUSTER), 2013 IEEE International on, Pages: 1--8 2013, doi: 10.1109/CLUSTER.2013.6702675

Babak Behzad, Joey Huchette, Huong Luu, Ruth Aydt, Quincey Koziol, Prabhat, Suren Byna, Mohamad Chaarawi, Yushu Yao, "Auto-Tuning of Parallel IO Parameters for HDF5 Applications", Proceedings of the 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, 2012,

Y. Yin, S. Byna, H. Song, X.-H. Sun, and R. Thakur, "Boosting Application-Specific Parallel I/O Optimization Using IOSIG", IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), Ottowa, Canada, May 13, 2012,

Surendra Byna, Jerry Chou, Oliver Rubel, Homa Karimabadi, William S Daughter, Vadim Roytershteyn, E Wes Bethel, Mark Howison, Ke-Jou Hsu, Kuan-Wu Lin, others, "Parallel I/O, analysis, and visualization of a trillion particle simulation", SC 12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, January 2012, 1--12,

Oliver R\ ubel, Surendra Byna, Kesheng Wu, Fuyu Li, Michael Wehner, Wes Bethel, others, "Teca: A parallel toolkit for extreme climate analysis", Procedia Computer Science, Elsevier, January 2012, 9:866--876, LBNL 5352E,

We present TECA, a parallel toolkit for detecting extreme events in large climate datasets. Modern climate datasets expose parallelism across a number of dimensions: spatial locations, timesteps and ensemble members. We design TECA to exploit these modes of parallelism and demonstrate a prototype implementation for detecting and tracking three classes of extreme events: tropical cyclones, extra-tropical cyclones and atmospheric rivers. We process a modern TB-sized CAM5 simulation dataset with TECA, and demonstrate good runtime performance for the three case studies.

E. W. Bethel, Surendra Byna, Jerry Chou, Cormier-Michel, Cameron G. R. Geddes, Howison, Fuyu Li, Prabhat, Ji Qiang, R\ ubel, Rob D. Ryne, Michael Wehner, Wu, "Big Data Analysis and Visualization: What Do LINACS Tropical Storms Have In Common?", 11th International Computational Accelerator Physics ICAP 2012, Germany, 2012,

EW Bethel, S. Byna, J. Chou, E., CGR Geddes, M. Howison, F. Li J. Q. Prabhat, O. R\ ubel, RD Ryne and, Big Data Analysis and Visualization: What Do LINACS Tropical Storms Have In Common?, 11th International Computational Accelerator Physics ICAP 2012, 2012,

O. R\ ubel, S. Byna, K. Wu, F. Li, M., W. Bethel, others, "TECA: A Parallel Toolkit for Extreme Climate", Procedia Computer Science, Elsevier, 2012, 9:866--876, doi: 10.1016/j.procs.2012.04.093

Mehmet Balman, Suredra Byna, "Open Problems in network-aware data management in exa-scale computing and terabit networking era", In Proceedings of the First international Workshop on Network-Aware Data Management, in conjunction with ACM/IEEE international Conference For High Performance Computing, Networking, Storage and Analysis, 2011, Seattle, WA, November 11, 2011, LBNL 6176E, doi: http://dx.doi.org/10.1145/2110217.2110229

Download File: ndm12ppbalman.pdf (pdf: 141 KB)

Accessing and managing large amounts of data is a great challenge in collaborative computing environments where resources and users are geographically distributed. Recent advances in network technology led to next-generation high- performance networks, allowing high-bandwidth connectiv- ity. Efficient use of the network infrastructure is necessary in order to address the increasing data and compute require- ments of large-scale applications. We discuss several open problems, evaluate emerging trends, and articulate our per- spectives in network-aware data management.

Kesheng Wu, Surendra Byna, Doron Rotem, Arie, "Scientific Data Services -- A High-Performance I/O with Array Semantics", HPCDB, IEEE, 2011, doi: 10.11v45/2125636.2125640

Surendra Byna, Michael F Wehner, Kesheng John Wu, "Detecting atmospheric rivers in large climate datasets", Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities, 2011, 7--14,

Extreme precipitation events on the western coast of North America are often traced to an unusual weather phenomenon known as atmospheric rivers. Although these storms may provide a significant fraction of the total water to the highly managed western US hydrological system, the resulting intense weather poses severe risks to the human and natural infrastructure through severe flooding and wind damage. To aid the understanding of this phenomenon, we have developed an efficient detection algorithm suitable for analyzing large amounts of data. In addition to detecting actual events in the recent observed historical record, this detection algorithm can be applied to global climate model output providing a new model validation methodology. Comparing the statistical behavior of simulated atmospheric river events in models to observations will enhance confidence in projections of future extreme storms. Our detection algorithm is based on a thresholding condition on the total column integrated water vapor established by Ralph et al. (2004) followed by a connected component labeling procedure to group the mesh points into connected regions in space. We develop an efficient parallel implementation of the algorithm and demonstrate good weak and strong scaling. We process a 30-year simulation output on 10,000 cores in under 3 seconds.

M Prabhat, S Byna, C Paciorek, G Weber, K Wu, T Yopes, MF Wehner, G Ostrouchov, D Pugmire, R Strelitz, others, "Pattern Detection and Extreme Value Analysis on Large Climate Data", AGUFM, Pages: IN41C--03 January 2011,

Md Kamal Hossain Chowdhury, Houjun Tang, Jean Luca Bez, Purushotham V. Bangalore, Suren Byna, "Efficient Asynchronous I/O with Request Merging", 2023 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), St. Petersburg, FL, USA, IEEE, December 31, 1969, 628-636, doi: 10.1109/IPDPSW59300.2023.00107

E. Wes Bethel, Burlen Loring, Utkarsh Ayatchit, David Camp, P. N. Duque, Nicola Ferrier, Joseph Insley, Junmin Gu, Kress, Patrick O’Leary, David Pugmire, Silvio Rizzi, Thompson, Gunther H. Weber, Brad Whitlock, Matthew Wolf, Kesheng Wu, "The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale", In Situ Visualization for Computational Science, ( 2022) doi: 10.1007/978-3-030-81627-8_13

B Loring, A Myers, D Camp, EW Bethel, "Python-based in situ analysis and visualization", Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV 18, ACM Press, 2018, doi: 10.1145/3281464.3281465

David Camp, Hari Krishnan, David Pugmire, Christoph Garth, Ian Johnson, E. Wes Bethel, Kenneth I. Joy, and Hank Childs., "GPU Acceleration of Particle Advection Workloads in a Parallel, Distributed Memory Setting", Proceedings of Eurographics Symposium on Parallel Graphics and Visualization (EGPGV), May 5, 2013,

Hank Childs, Eric Brugger, Brad Whitlock, Jeremy Meredith, Sean Ahern, David Pugmire, Kathleen Biagas, Mark Miller, Cyrus Harrison, Gunther H. Weber, Hari Krishnan, Thomas Fogal, Allen Sanderson, Christoph Garth, E. Wes Bethel, David Camp, Oliver Rubel, Marc Durant, Jean M. Favre, Paul Navratil, "VisIt: An End-User Tool For Visualizing and Analyzing Very Large Data", High Performance Visualization---Enabling Extreme-Scale Scientific Insight, ( October 2012) Pages: 357--372

E. Wes Bethel, David Camp, Hank Childs, Christoph Garth, Mark Howison, Kenneth I. Joy, David Pugmire, "Hybrid Parallelism", High Performance Visualization---Enabling Extreme-Scale Scientific Insight, ( October 2012) Pages: 261--290

David Camp, Hank Childs, Christoph Garth, David Pugmire, Kenneth I. Joy, "Parallel Stream Surface Computation for Large Data Sets", Proceedings of IEEE Symposium on Large Data Analysis and Visualization (LDAV), October 2012, 39--47, LBNL 5776E,

E. Wes Bethel, David Camp, Hank Childs, Mark Howison, Hari Krishnan, Burlen Loring, Joerg Meyer, Prabhat, Oliver Ruebel, Daniela Ushizima, Gunther Weber, "Towards Exascale: High Performance Visualization and Analytics – Project Status Report. Technical Report", DOE Exascale Research Conference, April 2012,

David Camp, Christoph Garth, Hank Childs, Dave Pugmire, Kenneth I. Joy, "Streamline Integration Using MPI-Hybrid Parallelism on a Large Multicore Architecture", IEEE Transactions on Visualization and Computer Graphics, November 2011, 17:1702-1713, LBNL 4563E, doi: http://doi.ieeecomputersociety.org/10.1109/TVCG.2010.259

Download File: LBNL-4563E.pdf (pdf: 3.5 MB)

David Camp, Hank Childs, Amit Chourasia, Christoph Garth, Kenneth Joy, "Evaluating the Benefits of An Extended Memory Hierarchy for Parallel Streamline Algorithms", Proceedings IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV) 2011, Providence, RI, USA, IEEE Press, October 2011, 57-64, LBNL 5503E,

Download File: LBNL-camp.pdf (pdf: 1.1 MB)

D. A. Agarwal, J. Damerow, C. Varadharajan, D. S. Christianson, G. Z. Pastorello, Y.-W. Cheah, L. Ramakrishnan, "Balancing the needs of consumers and producers for scientific data collections", Ecological Informatics, 2021, 62:101251, doi: 10.1016/j.ecoinf.2021.101251

G. Z. Pastorello, C. Trotta, E. Canfora, H. Chu, D. Christianson, Y.-W. Cheah, C. Poindexter, J. Chen, A. Elbashandy, M. Humphrey, P. Isaac, D. Polidori, M. Reichstein, A. Ribeca, C. van Ingen, N. Vuichard, L. Zhang, B. Amiro, C. Ammann, M. A. Arain, J. Ardö, T. Arkebauer, S. K. Arndt, N. Arriga, M. Aubinet, M. Aurela, D. Baldocchi, A. Barr, E. Beamesderfer, L. B. Marchesini, O. Bergeron, J. Beringer, C. Bernhofer, D. Berveiller, D. Billesbach, T. A. Black, P. D. Blanken, G. Bohrer, J. Boike, P. V. Bol stad, D. Bonal, J.-M. Bonnefond, D. R. Bowling, R. Bracho, J. Brodeur, C. Brümmer, N. Buchmann, B. Burban, S. P. Burns, P. Buysse, P. Cale, M. Cavagna, P. Cellier, S. Chen, I. Chini, T. R. Chris tensen, J. Cleverly, A. Collalti, C. Consalvo, B. D. Cook, D. Cook, C. Coursolle, E. Cremonese, P. S. Curtis, E. D’Andrea, H. da Rocha, X. Dai, K. J. Davis, B. D. Cinti, A. de Grandcourt, A. D. Ligne, R. C. D. Oliveira, N. Delpierre, A. R. Desai, C. M. D. Bella, P. di Tommasi, H. Dolman, F. Domingo, G. Dong, S. Dore, P. Duce, E. Dufrêne, A. Dunn, J. Dušek, D. Eamus, U. Eichelmann, H. A. M. ElKhidir, W. Eugster, C. M. Ewenz, B. Ewers, D. Famulari, S. Fares, I. Feigenwinter, A. Feitz, R. Fensholt, G. Fil ippa, M. Fischer, J. Frank, M. Galvagno, M. Gharun, D. Gianelle, B. Gielen, B. Gioli, A. Gitelson, I. Goded, M. Goeckede, A. H. Goldstein, C. M. Gough, M. L. Goulden, A. Graf, A. Griebel, C. Gruening, T. Grünwald, A. Hammerle, S. Han, X. Han, B. U. Hansen, C. Hanson, J. Hatakka, Y. He, M. Hehn, B. Heinesch, N. Hinko-Najera, L. Hörtnagl, L. Hutley, A. Ibrom, H. Ikawa, M. Jackowicz-Korczynski, D. Janouš, W. Jans, R. Jassal, S. Jiang, T. Kato, M. Khomik, J. Klatt, A. Knohl, S. Knox, H. Kobayashi, G. Koerber, O. Kolle, Y. Kosugi, A. Kotani, A. Kowalski, B. Kruijt, J. Kurbatova, W. L. Kutsch, H. Kwon, S. Launiainen, T. Laurila, B. Law, R. Leuning, Y. Li, M. Liddell, J.-M. Limousin, M. Lion, A. J. Liska, A. Lohila, A. López-Ballesteros, E. López-Blanco, B. Loubet, D. Loustau, A. Lucas-Moffat, J. Lüers, S. Ma, C. Macfarlane, V. Magliulo, R. Maier, I. Mammarella, G. Manca, B. Marcolla, H. A. Margolis, S. Mar ras, W. Massman, M. Mastepanov, R. Matamala, J. H. Matthes, F. Mazzenga, H. McCaughey, I. McHugh, A. M. S. McMillan, L. Merbold, W. Meyer, T. Meyers, S. D. Miller, S. Minerbi, U. Moderow, R. K. Monson, L. Montagnani, C. E. Moore, E. Moors, V. Moreaux, C. Moureaux, J. W. Munger, T. Nakai, J. Neirynck, Z. Nesic, G. Nicolini, A. Noormets, M. Northwood, M. Nosetto, Y. Nouvellon, K. Novick, W. Oechel, J. E. Olesen, J.-M. Ourcival, S. A. Papuga, F.-J. Parmentier, E. Paul-Limoges, M. Pavelka, M. Peichl, E. Pendall, R. P. Phillips, K. Pilegaard, N. Pirk, G. Posse, T. Powell, H. Prasse, S. M. Prober, S. Ram bal, U. Rannik, N. Raz-Yaseef, D. Reed, V. R. de Dios, N. Restrepo-Coupe, B. R. Reverter, M. Roland, S. Sabbatini, T. Sachs, S. R. Saleska, E. P. S.-C. nete, Z. M. Sanchez-Mejia, H. P. Schmid, M. Schmidt, K. Schneider, F. Schrader, I. Schroder, R. L. Scott, P. Sedlák, P. Serrano-Ortíz, C. Shao, P. Shi, I. Shironya, L. Siebicke, L. Šigut, R. Silberstein, C. Sirca, D. Spano, R. Steinbrecher, R. M. Stevens, C. Sturtevant, A. Suyker, T. Tagesson, S. Takanashi, Y. Tang, N. Tapper, J. Thom, F. Tiedemann, M. Tomassucci, J.-P. Tuovinen, S. Urbanski, R. Valentini, M. van der Molen, E. van Gorsel, K. van Huissteden, A. Varlagin, J. Verfaillie, T. Vesala, C. Vincke, D. Vitale, N. Vygodskaya, J. P. Walker, E. Walter-Shea, H. Wang, R. Weber, S. Westermann, C. Wille, S. Wofsy, G. Wohlfahrt, S. Wolf, W. Woodgate, Y. Li, R. Zampedri, J. Zhang, G. Zhou, D. Zona, D. Agarwal, S. Biraud, M. Torn, D. Papale, "The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data", Scientific Data, 2020, 7:225, doi: 10.1038/s41597-020-0534-3

You-Wei Cheah, Danielle Svehla Christianson, Housen Chu, Gilberto Pastorello, Fianna O’Brien, Yeongshnn Ong, Catharine van Ingen, Margaret Torn, Deb Agarwal, AmeriFlux BADM: Implementing lessons from 12 years of long-tail data management into next generation earth science systems (IN34A-03), 2018 AGU Fall Meeting, Washington, D.C., December 12, 2018,

Cheah You-Wei, Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, Bringing Data Science to Qualitative Analysis, 2018 IEEE 14th International Conference on e-Science, Pages: 325-326 2018, doi: 10.1109/eScience.2018.00076

YW Cheah, J Boverhof, A Elbashandy, D Agarwal, J Leek, T Epperly, J Eslick, D Miller, "Data management and simulation support accelerating carbon capture through computing", Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016, 2017, 389--398, doi: 10.1109/eScience.2016.7870924

Gilberto Z. Pastorello, Dan K. Gunter, Housen Chu, Danielle S. Christianson, Carlo Trotta, Eleonora Canfora, Boris Faybishenko, You-Wei Cheah, Norm Beekwilder, Stephen W. Chan, Sigrid Dengel, Trevor Keenan, Fianna O Brien, Abderahman Elbashandy, Cristina M. Poindexter, Marty Humphrey, Dario Papale, Deb A. Agarwal, "Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures", Proceedings of the 13th IEEE International Conference on e-Science (e-Science 2017), Auckland, New Zealand, 2017, doi: 10.1109/eScience.2017.64

DC Miller, D Agarwal, D Bhattacharyya, J Boverhof, YW Cheah, Y Chen, J Eslick, J Leek, J Ma, P Mahapatra, B Ng, NV Sahinidis, C Tong, SE Zitney, "Innovative computational tools and models for the design, optimization and control of carbon capture processes", Computer Aided Chemical Engineering, 2016, 38:2391--2396, doi: 10.1016/B978-0-444-63428-3.50403-3

You-Wei Cheah, Richard Canon, Beth Plale, Lavanya Ramakrishnan, "Milieu: Provenance Collection and Query Framework for High Performance Computing Systems", IEEE Big Data Congress, 2013,

D Agarwal, YW Cheah, D Fay, J Fay, D Guo, T Hey, M Humphrey, K Jackson, Jie Li, C Poulain, Y Ryu, C Van Ingen, "Data-intensive science: The Terapixel and MODISAzure projects", International Journal of High Performance Computing Applications, 2011, 25:304--316, doi: 10.1177/1094342011414746

J Li, M Humphrey, YW Cheah, Y Ryu, D Agarwal, K Jackson, C Van Ingen, Fault tolerance and scaling in e-Science cloud applications: Observations from the continuing development of MODISAzure, Proceedings - 2010 6th IEEE International Conference on e-Science, eScience 2010, Pages: 246--253 2010, doi: 10.1109/eScience.2010.47

D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, W. Arndt, J. Blaschke, S. Byna, R. Cheema, S. Cholia, M. Day, B. Enders, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, T. Lehman, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, L. Stephey, R. Thomas, G. Torok, "LBNL Superfacility Project Report", Lawrence Berkeley National Laboratory, 2022, doi: 10.48550/arXiv.2206.11992

MB Simmonds, WJ Riley, DA Agarwal, X Chen, S Cholia, R Crystal-Ornelas, ET Coon, D Dwivedi, VC Hendrix, M Huang, A Jan, Z Kakalia, J Kumar, CD Koven, L Li, M Melara, L Ramakrishnan, DM Ricciuto, AP Walker, W Zhi, Q Zhu, C Varadharajan, Guidelines for Publicly Archiving Terrestrial Model Data to Enhance Usability, Intercomparison, and Synthesis, Data Science Journal, 2022, doi: 10.5334/dsj-2022-003

D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, S. Byna, R. Cheema, S. Cholia, M. Day, B. Enders, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, R. Thomas, G. Torok, The Superfacility project: automated pipelines for experiments and HPC, International Conference for High Performance Computing, Networking, Storage, and Analysis (SC20), State of the Practice (SOP), 2020,

B. Enders, D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, S. Byna, R. Cheema, S. Cholia, M. Day, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, R. Thomas, G. Torok, "Cross-facility science with the Superfacility Project at LBNL", 2nd Workshop on Large-scale Experiment-in-the-Loop Computing (XLOOP 2020), in conjunction with the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 20), 2020, doi: 10.1109/XLOOP51963.2020.00006

C Varadharajan, S Cholia, C Snavely, V Hendrix, C Procopiou, D Swantek, W Riley, D Agarwal, "Launching an Accessible Archive of Environmental Data", Eos, January 1, 2019, 100, doi: 10.1029/2019eo111263

Wahid Bhimji, Steven Farrell, Oliver Evans, Matthew Henderson, Shreyas Cholia, Aaron Vose, Mr Prabhat, Rollin Thomas, Richard Shane Canon, "Interactive HPC Deep Learning with Jupyter Notebooks", Supercomputing 2018, Dallas, TX, November 2018,

Shreyas Cholia, Matthew Henderson, Oliver Evans, Fernando Perez, "Kale: A System for Enabling Human-in-the-loop Interactivity in HPC Workflows", Science Gateways 2018, figshare, September 26, 2018, doi: 10.6084/m9.figshare.7067075.v3

S. Farrell, A. Vose, O. Evans, M. Henderson, S. Cholia, W. Bhimji, R. Thomas, S.
Canon, and Prabhat,, "Interactive Distributed Deep Learning with Jupyter Notebooks", ISC Workshop on Interactive High-Performance Computing, June 28, 2018,

Dáithí A Stone, Mark D Risser, Oliver M Angélil, Michael F Wehner, Shreyas Cholia, Noel Keen, Harinarayan Krishnan, Travis A O Brien, William D Collins, "A basis set for exploration of sensitivity to prescribed ocean conditions for estimating human contributions to extreme weather in CAM5. 1-1degree", Weather and climate extremes, 2018, 19:10--19,

Anubhav Jain, Joseph Montoya, Shyam Dwaraknath, Nils ER Zimmermann, John Dagdelen, Matthew Horton, Patrick Huck, Donny Winston, Shreyas Cholia, Shyue Ping Ong, others, "The Materials Project: Accelerating Materials Design Through Theory-Driven Data and Tools", Handbook of Materials Modeling: Methods: Theory and Modeling, (Springer: 2018) Pages: 1--34

Shreyas Cholia, Matthew Henderson, Oliver Evans, Demo: Extending Jupyter to Support Interactive High Performance Computing, Science Gateways 2017, October 2017, doi: 10.6084/m9.figshare.5501137.v1

DA Agarwal, C Varadharajan, S Cholia, C Snavely, VC Hendrix, D Gunter, WJ Riley, M jones, AE budden, D Vieglas, Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE)-A New US DOE Data Archive, 2017,

Rollin Thomas, Shane Canon, Shreyas Cholia, Lisa Gerhardt, Evan Racah, "Toward Interactive Supercomputing at NERSC with Jupyter", Cray User Group (CUG) Conference Proceedings, 2017,

Patrick Huck, Dan Gunter, Shreyas Cholia, Donald Winston, AT N Diaye, Kristin Persson, "User applications driven by the community contribution framework MPContribs in the Materials Project", Concurrency and Computation: Practice and Experience, 2016, 28:1982--1993,

A Greiner, E Racah, R Canon, J Liu, Y Liu, D Bard, L Gerhardt, R Thomas, S Cholia, J Porter, W Bhimji, Q Koziol, Prabhat, "Data-Intensive Supercomputing for Science", 2016,

Stone, D., H. Shiogama, P. Wolski, O. Angélil, S. Cholias, N. Christidis, A. Dittus, C. Folland, A. King, J. Kinter, H. Krishnan, S.-K. Min, M. Wehner, "The C20C+ Detection and Attribution Project", Fall Meeting of the American Geophysical Union, 2015,

Download File: StoneDAGU2015.pdf (pdf: 4.7 MB)

P Huck, D Gunter, S Cholia, D Winston, A N Diaye, KA Persson, "User Applications Driven by the Community Contribution Framework MPContribs in the Materials Project.", CoRR, 2015, abs/1510,

SP Ong, S Cholia, A Jain, M Brafman, D Gunter, G Ceder, KA Persson, "The Materials Application Programming Interface (API): A simple, flexible and efficient API for materials data based on REpresentational State Transfer (REST) principles", Computational Materials Science, 2015, 97:209--215, doi: 10.1016/j.commatsci.2014.10.037

S Cholia, T Sun, "The NEWT platform: An extensible plugin framework for creating ReSTful HPC APIs", Proceedings of GCE 2014: 9th Gateway Computing Environments Workshop, held in conjunction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2015, 17--20, doi: 10.1109/GCE.2014.14

R Madduri, A Rodriguez, T Uram, K Heitmann, T Malik, S Sehrish, R Chard, S Cholia, M Paterno, J Kowalkowski, S Habib, "PDACS: A Portal for Data Analysis Services for Cosmological Simulations", Computing in Science and Engineering, 2015, 17:18--26, doi: 10.1109/MCSE.2015.87

NJ Wright, SS Dosanjh, AK Andrews, KB Antypas, B Draney, RS Canon, S Cholia, CS Daley, KM Fagnan, RA Gerber, L Gerhardt, L Pezzaglia, A Prabhat, KH Schafer, J Srinivasan, "Cori: A pre-exascale supercomputer for big data and HPC applications", Advances in Parallel Computing, 2015, 26:82--100, doi: 10.3233/978-1-61499-583-8-82

R Chard, S Sehrish, A Rodriguez, R Madduri, TD Uram, M Paterno, K Heitmann, S Cholia, J Kowalkowski, S Habib, "PDACS-A portal for data analysis services for cosmological simulations", Proceedings of GCE 2014: 9th Gateway Computing Environments Workshop, held in conjunction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2015, 30--33, doi: 10.1109/GCE.2014.7

Dáithí Stone, Michael Wehner, Shreyas Cholia, Harinarayan Krishnan, Piotr Wolski, Mark Tadross, Chris Folland, Nikos Christidis, Hideo Shiogama, "The C20C+ Detection and Attribution Project", Integrated Climate Modeling Principal Investigator Meeting 2014, 2014,

R Cowles, C Jackson, V Welch, S Cholia, "A model for identity management in future scientific collaboratories", Proceedings of Science, 2014, 23-28-Ma,

A Jain, SP Ong, G Hautier, W Chen, WD Richards, S Dacek, S Cholia, D Gunter, D Skinner, G Ceder, KA Persson, "Commentary: The materials project: A materials genome approach to accelerating materials innovation", APL Materials, 2013, 1, doi: 10.1063/1.4812323

SP Ong, WD Richards, A Jain, G Hautier, M Kocher, S Cholia, D Gunter, VL Chevrier, KA Persson, G Ceder, "Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis", Computational Materials Science, 2013, 68:314--319, doi: 10.1016/j.commatsci.2012.10.028

O Rübel, A Greiner, S Cholia, K Louie, EW Bethel, TR Northen, BP Bowen, "OpenMSI: A high-performance web-based platform for mass spectrometry imaging", Analytical Chemistry, 2013, 85:10354--103, doi: 10.1021/ac402540a

M Di Pierro, J Hetrick, S Cholia, J Simone, C Deta, "The new Gauge Connection at NERSC", Proceedings of Science, 2013, 29-July-,

D Gunter, S Cholia, A Jain, M Kocher, K Persson, L Ramakrishnan, SP Ong, G Ceder, "Community accessible datastore of high-throughput calculations: Experiences from the materials project", Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, 2012, 1244--1251, doi: 10.1109/SC.Companion.2012.150

A Jain, G Hautier, SP Ong, C Moore, B Kang, H Chen, X Ma, JC Kim, M Kocher, D Gunter, S Cholia, A Greiner, DH Bailey, D Skinner, K Persson, G Ceder, "Materials Project: A public materials database and its application to lithium ion battery cathode design", ABSTRACTS OF PAPERS OF THE AMERICAN CHEMICAL SOCIETY, 2012, 243,

MD Pierro, J Hetrick, S Cholia, D Skinner, "Making QCD lattice data accessible and organized through advanced web interfaces", Proceedings of Science, 2011, 139,

Lavanya Ramakrishnan, Keith Jackson, Shane Canon, Shreyas Cholia, John Shalf, "Defining Future Platform Requirements for e-Science Cloud (Position paper)", ACM Symposium on Cloud Computing 2010 (ACM SOCC 2010), Indianapolis, Indiana, 2010,

S Cholia, D Skinner, J Boverhof, "NEWT: A RESTful service for building High: Performance Computing web applications", 2010 Gateway Computing Environments Workshop, GCE 2010, 2010, doi: 10.1109/GCE.2010.5676125

KR Jackson, L Ramakrishnan, K Muriki, S Canon, S Cholia, J Shalf, HJ Wasserman, NJ Wright, "Performance analysis of high performance computing applications on the Amazon Web Services cloud", Proceedings - 2nd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2010, 2010, 159--168, doi: 10.1109/CloudCom.2010.69

M Pala, S Cholia, SA Rea, SW Smith, "Interoperable PKI data distribution in computational grids", International Journal of Grid and High Performance Computing, 2009, 1:56--73, doi: 10.4018/jghpc.2009040105

S Cholia, RJ Porter, "Publication and protection of sensitive site information in a grid infrastructure", Proceedings CCGRID 2008 - 8th IEEE International Symposium on Cluster Computing and the Grid, 2008, 639--644, doi: 10.1109/CCGRID.2008.86

M Pala, S Cholia, SA Rea, SW Smith, "Extending PKI interoperability in computational grids", Proceedings CCGRID 2008 - 8th IEEE International Symposium on Cluster Computing and the Grid, 2008, 645--650, doi: 10.1109/CCGRID.2008.70

C Varadharajan, VC Hendrix, DS Christianson, M Burrus, C Wong, SS Hubbard, DA Agarwal, BASIN-3D: A brokering framework to integrate diverse environmental data, Computers and Geosciences, 2022, doi: 10.1016/j.cageo.2021.105024

C Varadharajan, AP Appling, B Arora, DS Christianson, VC Hendrix, V Kumar, AR Lima, J Müller, S Oliver, M Ombadi, T Perciano, JM Sadler, H Weierbach, JD Willard, Z Xu, J Zwart, "Can machine learning accelerate process understanding and decision-relevant predictions of river water quality?", Hydrological Processes, January 1, 2022, 36, doi: 10.1002/hyp.14565

H Weierbach, AR Lima, JD Willard, VC Hendrix, DS Christianson, M Lubich, C Varadharajan, Stream Temperature Predictions for River Basin Management in the Pacific Northwest and Mid-Atlantic Regions Using Machine Learning, Water (Switzerland), 2022, doi: 10.3390/w14071032

C Varadharajan, Z Kakalia, E Alper, EL Brodie, M Burrus, RWH Carroll, D Christianson, W Dong, V Hendrix, M Henderson, S Hubbard, D Johnson, R Versteeg, KH Williams, DA Agarwal, The Colorado East River Community Observatory Data Collection, Hydrological Processes 35(6), 2021, doi: 10.22541/au.161962485.54378235/v1

D. A. Agarwal, J. Damerow, C. Varadharajan, D. S. Christianson, G. Z. Pastorello, Y.-W. Cheah, L. Ramakrishnan, "Balancing the needs of consumers and producers for scientific data collections", Ecological Informatics, 2021, 62:101251, doi: 10.1016/j.ecoinf.2021.101251

G. Z. Pastorello, C. Trotta, E. Canfora, H. Chu, D. Christianson, Y.-W. Cheah, C. Poindexter, J. Chen, A. Elbashandy, M. Humphrey, P. Isaac, D. Polidori, M. Reichstein, A. Ribeca, C. van Ingen, N. Vuichard, L. Zhang, B. Amiro, C. Ammann, M. A. Arain, J. Ardö, T. Arkebauer, S. K. Arndt, N. Arriga, M. Aubinet, M. Aurela, D. Baldocchi, A. Barr, E. Beamesderfer, L. B. Marchesini, O. Bergeron, J. Beringer, C. Bernhofer, D. Berveiller, D. Billesbach, T. A. Black, P. D. Blanken, G. Bohrer, J. Boike, P. V. Bol stad, D. Bonal, J.-M. Bonnefond, D. R. Bowling, R. Bracho, J. Brodeur, C. Brümmer, N. Buchmann, B. Burban, S. P. Burns, P. Buysse, P. Cale, M. Cavagna, P. Cellier, S. Chen, I. Chini, T. R. Chris tensen, J. Cleverly, A. Collalti, C. Consalvo, B. D. Cook, D. Cook, C. Coursolle, E. Cremonese, P. S. Curtis, E. D’Andrea, H. da Rocha, X. Dai, K. J. Davis, B. D. Cinti, A. de Grandcourt, A. D. Ligne, R. C. D. Oliveira, N. Delpierre, A. R. Desai, C. M. D. Bella, P. di Tommasi, H. Dolman, F. Domingo, G. Dong, S. Dore, P. Duce, E. Dufrêne, A. Dunn, J. Dušek, D. Eamus, U. Eichelmann, H. A. M. ElKhidir, W. Eugster, C. M. Ewenz, B. Ewers, D. Famulari, S. Fares, I. Feigenwinter, A. Feitz, R. Fensholt, G. Fil ippa, M. Fischer, J. Frank, M. Galvagno, M. Gharun, D. Gianelle, B. Gielen, B. Gioli, A. Gitelson, I. Goded, M. Goeckede, A. H. Goldstein, C. M. Gough, M. L. Goulden, A. Graf, A. Griebel, C. Gruening, T. Grünwald, A. Hammerle, S. Han, X. Han, B. U. Hansen, C. Hanson, J. Hatakka, Y. He, M. Hehn, B. Heinesch, N. Hinko-Najera, L. Hörtnagl, L. Hutley, A. Ibrom, H. Ikawa, M. Jackowicz-Korczynski, D. Janouš, W. Jans, R. Jassal, S. Jiang, T. Kato, M. Khomik, J. Klatt, A. Knohl, S. Knox, H. Kobayashi, G. Koerber, O. Kolle, Y. Kosugi, A. Kotani, A. Kowalski, B. Kruijt, J. Kurbatova, W. L. Kutsch, H. Kwon, S. Launiainen, T. Laurila, B. Law, R. Leuning, Y. Li, M. Liddell, J.-M. Limousin, M. Lion, A. J. Liska, A. Lohila, A. López-Ballesteros, E. López-Blanco, B. Loubet, D. Loustau, A. Lucas-Moffat, J. Lüers, S. Ma, C. Macfarlane, V. Magliulo, R. Maier, I. Mammarella, G. Manca, B. Marcolla, H. A. Margolis, S. Mar ras, W. Massman, M. Mastepanov, R. Matamala, J. H. Matthes, F. Mazzenga, H. McCaughey, I. McHugh, A. M. S. McMillan, L. Merbold, W. Meyer, T. Meyers, S. D. Miller, S. Minerbi, U. Moderow, R. K. Monson, L. Montagnani, C. E. Moore, E. Moors, V. Moreaux, C. Moureaux, J. W. Munger, T. Nakai, J. Neirynck, Z. Nesic, G. Nicolini, A. Noormets, M. Northwood, M. Nosetto, Y. Nouvellon, K. Novick, W. Oechel, J. E. Olesen, J.-M. Ourcival, S. A. Papuga, F.-J. Parmentier, E. Paul-Limoges, M. Pavelka, M. Peichl, E. Pendall, R. P. Phillips, K. Pilegaard, N. Pirk, G. Posse, T. Powell, H. Prasse, S. M. Prober, S. Ram bal, U. Rannik, N. Raz-Yaseef, D. Reed, V. R. de Dios, N. Restrepo-Coupe, B. R. Reverter, M. Roland, S. Sabbatini, T. Sachs, S. R. Saleska, E. P. S.-C. nete, Z. M. Sanchez-Mejia, H. P. Schmid, M. Schmidt, K. Schneider, F. Schrader, I. Schroder, R. L. Scott, P. Sedlák, P. Serrano-Ortíz, C. Shao, P. Shi, I. Shironya, L. Siebicke, L. Šigut, R. Silberstein, C. Sirca, D. Spano, R. Steinbrecher, R. M. Stevens, C. Sturtevant, A. Suyker, T. Tagesson, S. Takanashi, Y. Tang, N. Tapper, J. Thom, F. Tiedemann, M. Tomassucci, J.-P. Tuovinen, S. Urbanski, R. Valentini, M. van der Molen, E. van Gorsel, K. van Huissteden, A. Varlagin, J. Verfaillie, T. Vesala, C. Vincke, D. Vitale, N. Vygodskaya, J. P. Walker, E. Walter-Shea, H. Wang, R. Weber, S. Westermann, C. Wille, S. Wofsy, G. Wohlfahrt, S. Wolf, W. Woodgate, Y. Li, R. Zampedri, J. Zhang, G. Zhou, D. Zona, D. Agarwal, S. Biraud, M. Torn, D. Papale, "The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data", Scientific Data, 2020, 7:225, doi: 10.1038/s41597-020-0534-3

C Varadharajan, B Faybishenko, A Henderson, M Henderson, VC Hendrix, SS Hubbard, Z Kakalia, A Newman, B Potter, H Steltzer, R Versteeg, DA Agarwal, KH Williams, C Wilmer, Y Wu, W Brown, M Burrus, RWH Carroll, DS Christianson, B Dafflon, D Dwivedi, BJ Enquist, Challenges in Building an End-to-End System for Acquisition, Management, and Integration of Diverse Data from Sensor Networks in Watersheds: Lessons from a Mountainous Community Observatory in East River, Colorado, IEEE Access, Pages: 182796--18 2019, doi: 10.1109/ACCESS.2019.2957793

Danielle S. Christianson, Charuleka Varadharajan, Bradley Christoffersen, Matteo Detto, Faybishenko, Bruno O. Gimenez, Val C. Hendrix, Kolby J. Jardine, Robinson Negron-Juarez, Z. Pastorello, Thomas L. Powell, Megha Sandesh, Jeffrey M. Warren, Brett T. Wolfe, Jeffrey Q. Chambers, Lara M. Kueppers, Nathan G. McDowell, Deborah A. Agarwal, "A metadata reporting framework (FRAMES) for synthesis of ecohydrological observations", Ecological Informatics, 2017, 42:148-158, doi: 10.1016/j.ecoinf.2017.06.002

Gilberto Z. Pastorello, Dan K. Gunter, Housen Chu, Danielle S. Christianson, Carlo Trotta, Eleonora Canfora, Boris Faybishenko, You-Wei Cheah, Norm Beekwilder, Stephen W. Chan, Sigrid Dengel, Trevor Keenan, Fianna O Brien, Abderahman Elbashandy, Cristina M. Poindexter, Marty Humphrey, Dario Papale, Deb A. Agarwal, "Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures", Proceedings of the 13th IEEE International Conference on e-Science (e-Science 2017), Auckland, New Zealand, 2017, doi: 10.1109/eScience.2017.64

Danielle S Christianson, Cari G Kaufman, "Effects of sample design and landscape features on a measure of environmental heterogeneity", Methods in Ecology and Evolution, 2016, 7:770--782,

ZM Subin, WJ Riley, J Jin, DS Christianson, MS Torn, LM Kueppers, "Ecosystem feedbacks to climate change in California: development, Testing, and analysis using a coupled regional atmosphere and land surface model (WRF3--CLM3. 5)", Earth Interactions, 2011, 15:1--38,

Bin Dong, Jean Luca Bez, Suren Byna, "AIIO: Using Artificial Intelligence for Job-Level and Automatic I/O Performance Bottleneck Diagnosis.", In Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing (HPDC ’23), June 16, 2023,

Download File: IODiagnose-final.pdf (pdf: 1.9 MB)

Jonathan Ajo‐Franklin, Verónica Rodríguez Tribaldos, Avinash Nayak, Feng Cheng, Robert Mellors, Benxin Chi, Todd Wood, Michelle Robertson, Cody Rotermund, Eric Matzel, Dennise C. Templeton, Christina Morency, Kesheng Wu, Bin Dong, Patrick Dobson;, "The Imperial Valley Dark Fiber Project: Toward Seismic Studies Using DAS and Telecom Infrastructure for Geothermal Applications", Seismological Research Letters, June 24, 2022,

Bin Dong, Alex Popescu, Veronica Rodriguez Tribaldos, Suren Byna, Jonathan Ajo-Franklin, Kesheng Wu, "Real-time and post-hoc compression for data from Distributed Acoustic Sensing", Computers \& Geosciences, June 24, 2022, 105181,

Download File: wu2022.bib (bib: 22 KB)

Runzhou Han, Suren Byna, Houjun Tang, Bin Dong, and Mai Zheng,, "PROV-IO: An I/O-Centric Provenance Framework for Scientific Data on HPC Systems", HPDC 2022, June 23, 2022,

John Wu, Bin Dong, Alex Sim, Automating Data Management Through Unified Runtime Systems, DOE ASCR Workshop on the Management and Storage of Scientific Data, 2022, doi: 10.2172/1843500

Bin Dong, Kesheng Wu, Suren Byna, User-Defined Tensor Data Analysis, SpringerBrief, (January 1, 2022)

Screen Shot 2022 06 24 at 1.24.03 PM

Jonathan Blair Ajo-Franklin, Ver\ onica Rodr\ \iguez Tribaldos, Avinash Nayak, Nathaniel J Lindsey, Feng Cheng, Benxin Chi, Bin Dong, Kesheng Wu, Inder Monga, Distributed Acoustic Sensing (DAS) at the Plot to Basin Scale: Connecting Near-Surface Sensing and Seismology with a Common Observational Tool, AGU Fall Meeting 2020, 2020,

Bin Dong, Ver\ onica Rodr\ \iguez Tribaldos, Xin Xing, Suren Byna, Jonathan Ajo-Franklin, Kesheng Wu, "DASSA: Parallel DAS Data Storage and Analysis for Subsurface Event Detection", 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS), July 14, 2020, 254--263,

Download File: paper.pdf (pdf: 3.9 MB)

Houjun Tang, Suren Byna, Bin Dong, Quincey Koziol, "Parallel Query Service for Object-centric Data Management Systems", 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), IEEE, May 18, 2020, 406-415,

Suren Byna, M. Scot Breitenfeld, Bin Dong, Quincey Koziol, Elena Pourmal, Dana Robinson, Jerome Soumagne, Houjun Tang, Venkatram Vishwanath, and Richard Warren, "ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems", Journal of Computer Science and Technology 2020, 35(1): 145-160, February 2, 2020, doi: 10.1007/s11390-020-9822-9

Richard Warren, Jerome Soumagne, Jingqing Mu, Houjun Tang, Suren Byna, Bin Dong, Quincey Koziol, "Analysis in the Data Path of an Object-centric Data Management System", 26th IEEE International Conference on High Performance Computing, Data, and Analytics (HiPC 2019), December 18, 2019,

Houjun Tang, Suren Byna, Stephen Bailey, Zarija Lukic, Jialin Liu, Quincey Koziol, Bin Dong, "Tuning Object-centric Data Management Systems for Large Scale Scientific Applications", 26th IEEE International Conference on High Performance Computing, Data, and Analytics (HiPC 2019), December 18, 2019,

Bin Dong, Kesheng Wu, Suren Byna, Houjun Tang, "SLOPE: Structural Locality-Aware Programming Model for Composing Array Data Analysis", International Conference on High Performance Computing, January 1, 2019, 61--80,

Download File: slope-cr.pdf (pdf: 623 KB)

Bin Dong, Patrick Kilian, Xiaocan Li, Fan Guo, Suren Byna, Kesheng Wu, "Terabyte-scale Particle Data Analysis: An ArrayUDF Case Study", Proceedings of the 31st International Conference on Scientific and Statistical Database Management, January 1, 2019, 202--205,

Download File: VPIC-SSDBM-Camera-ready.pdf (pdf: 271 KB)

Suren Byna, Quincey Koziol, Venkatram Vishwanath, Jerome Soumagne, Houjun Tang, Kimmy Mu, Richard Warren, François Tessier, Bin Dong, Teng Wang, and Jialin Liu, Proactive Data Containers (PDC): An object-centric data store for large-scale computing systems, AGU Fall Meeting, December 13, 2018,

Teng Wang, Suren Byna, Bin Dong, and Houjun Tang, "UniviStor: Integrated Hierarchical and Distributed Storage for HPC", IEEE Cluster 2018., September 1, 2018,

Houjun Tang, Suren Byna, Francois Tessier, Teng Wang, Bin Dong, Jingqing Mu, Quincey Koziol, Jerome Soumagne, Venkatram Vishwanath, Jialin Liu, and Richard Warren, "Toward Scalable and Asynchronous Object-centric Data Management for HPC", 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) 2018, May 1, 2018,

Bin Dong, Teng Wang, Houjun Tang, Quincey Koziol, Kesheng Wu, Suren Byna, "ARCHIE: Data analysis acceleration with array caching in hierarchical storage", 2018 IEEE International Conference on Big Data (Big Data), January 1, 2018, 211--220,

Download File: DataElevator-ARCHIE.pdf (pdf: 613 KB)

Kesheng Wu, Surendra Byna, Bin Dong, others, VPIC IO utilities, 2018,

Kesheng Wu, Bin Dong, Surendra Byna, "Scientific Data Services Framework for Plasma Physics", APS, 2018, 2018:BM10--006,

Weijie Zhao, Florin Rusu, Bin Dong, Kesheng Wu, Anna YQ Ho, Peter Nugent, "Distributed caching for processing raw arrays", Proceedings of the 30th International Conference on Scientific and Statistical Database Management, 2018, 1--12,

Xin Xing, Bin Dong, Jonathan Ajo-Franklin, Kesheng Wu, "Automated Parallel Data Processing Engine with Application to Large-Scale Feature Extraction", 2018 IEEE/ACM Machine Learning in HPC Environments (MLHPC), January 1, 2018, 37--46,

Download File: arrayudf-das.pdf (pdf: 2.7 MB)

Houjun Tang, Suren Byna, Bin Dong, Jialin Liu, and Quincey Koziol, "SoMeta: Scalable Object-centric Metadata Management for High Performance Computing", IEEE Cluster 2017, September 5, 2017,

Bin Dong, Kesheng Wu, Surendra Byna, Jialin Liu, Weijie Zhao, Florin Rusu, "ArrayUDF: User-defined scientific data analysis on arrays", Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing, January 1, 2017, 53--64,

Download File: hpdc02.pdf (pdf: 921 KB)

Weijie Zhao, Florin Rusu, Bin Dong, Kesheng Wu, Peter Nugent, "Incremental view maintenance over array data", Proceedings of the 2017 ACM International Conference on Management of Data, January 1, 2017, 139--154,

Tzuhsien Wu, Jerry Chou, Shyng Hao, Bin Dong, Scott Klasky, Kesheng Wu, "Optimizing the query performance of block index through data analysis and I/O modeling", Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, January 1, 2017, 1--10,

Bin Dong, Surendra Byna, Kesheng Wu, "SDS-Sort: Scalable Dynamic Skew-aware Parallel", HPDC 16, New York, NY, USA, ACM, 2016, 57--68, doi: 10.1145/2907294.2907300

Houjun Tang, Suren Byna, Steve Harenberg, Xiaocheng Zou, Wenzhao Zhang, Kesheng Wu, Bin Dong, Oliver Rubel, Kristofer Bouchard, Scott Klasky, others, "Usage pattern-driven dynamic data layout reorganization", 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), January 1, 2016, 356--365,

Wenzhao Zhang, Houjun Tang, Steve Harenberg, Surendra Byna, Xiaocheng Zou, Dharshi Devendran, Daniel F Martin, Kesheng Wu, Bin Dong, Scott Klasky, others, "Amrzone: A runtime amr data sharing framework for scientific applications", 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), January 1, 2016, 116--125,

Bin Dong, Surendra Byna, Kesheng Wu, "Sds-sort: Scalable dynamic skew-aware parallel sorting", Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing, January 1, 2016, 57--68,

Download File: SDS-Sort.pdf (pdf: 450 KB)

Xiaocheng Zou, David A Boyuka II, Dhara Desai, Daniel F Martin, Suren Byna, Kesheng Wu, "AMR-aware in situ indexing and scalable querying", Proceedings of the 24th High Performance Computing Symposium, January 1, 2016, 26,

Bin Dong, Suren Byna, Kesheng Wu, Hans Johansen, Jeffrey N Johnson, Noel Keen, others, "Data elevator: Low-contention data movement in hierarchical storage system", 2016 IEEE 23rd international conference on high performance computing (HiPC), January 1, 2016, 152--161,

Download File: 201612-DataElevator-HiPC2016-Bin-Byna.pdf (pdf: 765 KB)

Houjun Tang, Suren Byna, Steve Harenberg, Wenzhao Zhang, Xiaocheng Zou, Daniel F Martin, Bin Dong, Dharshi Devendran, Kesheng Wu, David Trebotich, others, "In situ storage layout optimization for amr spatio-temporal read accesses", 2016 45th International Conference on Parallel Processing (ICPP), January 1, 2016, 406--415,

Wenzhao Zhang, Houjun Tang, Stephen Ranshous, Surendra Byna, Daniel F Mart\ \in, Kesheng Wu, Bin Dong, Scott Klasky, Nagiza F Samatova, "Exploring memory hierarchy and network topology for runtime AMR data sharing across scientific applications", 2016 IEEE International Conference on Big Data (Big Data), January 1, 2016, 1359--1366,

Tzuhsien Wu, Hao Shyng, Jerry Chou, Bin Dong, Kesheng Wu, "Indexing blocks to reduce space and time requirements for searching large data files", 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), January 1, 2016, 398--402,

Weijie Zhao, Florin Rusu, Bin Dong, Kesheng Wu, "Similarity Join over Array Data", SIGMOD, January 1, 2016, 2007--2022,

Jinoh Kim, Bin Dong, Suren Byna, and Kesheng Wu, "Security for the Scientific Data Service Framework", 2nd International Workshop on Privacy and Security of Big Data (PSBD 2015), in conjunction with IEEE BigData 2015, 2015,

Bin Dong, Surendra Byna, Kesheng Wu, "Heavy-tailed distribution of parallel I/O system response time", Proceedings of the 10th Parallel Data Storage Workshop, 2015, 37--42,

Bin Dong, Surendra Byna, Kesheng Wu, "Spatially clustered join on heterogeneous scientific data sets", 2015 IEEE International Conference on Big Data (Big Data), 2015, 371--380,

Bin Dong, Xiuqiao Li, Limin Xiao, Li Ruan, "Towards minimizing disk I/O contention: A partitioned file assignment approach", Future Generation Computer Systems, Volume 37, July 2014, Pages 178-190, 2014,

Spyros Blanas, Kesheng Wu, Surendra Byna, Bin Dong, Arie Shoshani, "Parallel Data Analysis Directly on Scientific File", SIGMOD 14, 2014, 385--396, doi: 10.1145/2588555.2612185

Spyros Blanas, Kesheng Wu, Surendra Byna, Bin Dong, Arie Shoshani, "Parallel data analysis directly on scientific file formats", Proceedings of the 2014 ACM SIGMOD international conference on Management of data, January 1, 2014, 385--396,

Bin Dong, Surendra Byna, Kesheng Wu, "Parallel query evaluation as a Scientific Data Service", 2014 IEEE International Conference on Cluster Computing (CLUSTER), January 1, 2014, 194--202,

Jialin Liu, S. Byna, Bin Dong, Kesheng Wu, Chen, "Model-Driven Data Layout Selection for Improving Read", Parallel Distributed Processing Symposium Workshops 2014 IEEE International, 2014, 1708--1716, doi: 10.1109/IPDPSW.2014.190

Bin Dong, S. Byna, Kesheng Wu, Parallel query evaluation as a Scientific Data, Cluster Computing (CLUSTER), 2014 IEEE International on, Pages: 194--202 2014, doi: 10.1109/CLUSTER.2014.6968765

Bin Dong, Surendra Byna, Kesheng Wu, "SDS: a framework for scientific data services", Proceedings of the 8th Parallel Data Storage Workshop, January 1, 2013, 27--32,

Bin Dong, Surendra Byna, Kesheng Wu, "Expediting scientific data analysis with reorganization of data", 2013 IEEE International Conference on Cluster Computing (CLUSTER), January 1, 2013, 1--8,

Bin Dong, S. Byna, Kesheng Wu, Expediting scientific data analysis with of data, Cluster Computing (CLUSTER), 2013 IEEE International on, Pages: 1--8 2013, doi: 10.1109/CLUSTER.2013.6702675

Bin Dong, Xiuqiao Li, Qimeng Wu, Limin Xiao, Li Ruan, "A dynamic and adaptive load balancing strategy for parallel file system with large-scale I/O servers", Journal of Parallel and Distributed Computing (JPDC), Volume 72, Issue 10, October 2012, Pages 1254-1268, 2012,

Bin Dong, Xiuqiao Li, Limin Xiao, Li Ruan, "A New File-Specific Stripe Size Selection Method for Highly Concurrent Data Access", The 13th ACM/IEEE International Conference on Grid Computing (Grid 2012), 2012, 2012,

Devarshi Ghoshal, Drew Paine, Gilberto Pastorello, Abdelrahman Elbashandy, Dan Gunter, Oluwamayowa Amusat, Lavanya Ramakrishnan, "Experiences with Reproducibility: Case Studies from Scientific Workflows", (P-RECS'21) Proceedings of the 4th International Workshop on Practical Reproducible Evaluation of Computer Systems, ACM, June 21, 2021, doi: 10.1145/3456287.3465478

Reproducible research is becoming essential for science to ensure transparency and for building trust. Additionally, reproducibility provides the cornerstone for sharing of methodology that can improve efficiency. Although several tools and studies focus on computational reproducibility, we need a better understanding about the gaps, issues, and challenges for enabling reproducibility of scientific results beyond the computational stages of a scientific pipeline. In this paper, we present five different case studies that highlight the reproducibility needs and challenges under various system and environmental conditions. Through the case studies, we present our experiences in reproducing different types of data and methods that exist in an experimental or analysis pipeline. We examine the human aspects of reproducibility while highlighting the things that worked, that did not work, and that could have worked better for each of the cases. Our experiences capture a wide range of scenarios and are applicable to a much broader audience who aim to integrate reproducibility in their everyday pipelines.

G. Z. Pastorello, C. Trotta, E. Canfora, H. Chu, D. Christianson, Y.-W. Cheah, C. Poindexter, J. Chen, A. Elbashandy, M. Humphrey, P. Isaac, D. Polidori, M. Reichstein, A. Ribeca, C. van Ingen, N. Vuichard, L. Zhang, B. Amiro, C. Ammann, M. A. Arain, J. Ardö, T. Arkebauer, S. K. Arndt, N. Arriga, M. Aubinet, M. Aurela, D. Baldocchi, A. Barr, E. Beamesderfer, L. B. Marchesini, O. Bergeron, J. Beringer, C. Bernhofer, D. Berveiller, D. Billesbach, T. A. Black, P. D. Blanken, G. Bohrer, J. Boike, P. V. Bol stad, D. Bonal, J.-M. Bonnefond, D. R. Bowling, R. Bracho, J. Brodeur, C. Brümmer, N. Buchmann, B. Burban, S. P. Burns, P. Buysse, P. Cale, M. Cavagna, P. Cellier, S. Chen, I. Chini, T. R. Chris tensen, J. Cleverly, A. Collalti, C. Consalvo, B. D. Cook, D. Cook, C. Coursolle, E. Cremonese, P. S. Curtis, E. D’Andrea, H. da Rocha, X. Dai, K. J. Davis, B. D. Cinti, A. de Grandcourt, A. D. Ligne, R. C. D. Oliveira, N. Delpierre, A. R. Desai, C. M. D. Bella, P. di Tommasi, H. Dolman, F. Domingo, G. Dong, S. Dore, P. Duce, E. Dufrêne, A. Dunn, J. Dušek, D. Eamus, U. Eichelmann, H. A. M. ElKhidir, W. Eugster, C. M. Ewenz, B. Ewers, D. Famulari, S. Fares, I. Feigenwinter, A. Feitz, R. Fensholt, G. Fil ippa, M. Fischer, J. Frank, M. Galvagno, M. Gharun, D. Gianelle, B. Gielen, B. Gioli, A. Gitelson, I. Goded, M. Goeckede, A. H. Goldstein, C. M. Gough, M. L. Goulden, A. Graf, A. Griebel, C. Gruening, T. Grünwald, A. Hammerle, S. Han, X. Han, B. U. Hansen, C. Hanson, J. Hatakka, Y. He, M. Hehn, B. Heinesch, N. Hinko-Najera, L. Hörtnagl, L. Hutley, A. Ibrom, H. Ikawa, M. Jackowicz-Korczynski, D. Janouš, W. Jans, R. Jassal, S. Jiang, T. Kato, M. Khomik, J. Klatt, A. Knohl, S. Knox, H. Kobayashi, G. Koerber, O. Kolle, Y. Kosugi, A. Kotani, A. Kowalski, B. Kruijt, J. Kurbatova, W. L. Kutsch, H. Kwon, S. Launiainen, T. Laurila, B. Law, R. Leuning, Y. Li, M. Liddell, J.-M. Limousin, M. Lion, A. J. Liska, A. Lohila, A. López-Ballesteros, E. López-Blanco, B. Loubet, D. Loustau, A. Lucas-Moffat, J. Lüers, S. Ma, C. Macfarlane, V. Magliulo, R. Maier, I. Mammarella, G. Manca, B. Marcolla, H. A. Margolis, S. Mar ras, W. Massman, M. Mastepanov, R. Matamala, J. H. Matthes, F. Mazzenga, H. McCaughey, I. McHugh, A. M. S. McMillan, L. Merbold, W. Meyer, T. Meyers, S. D. Miller, S. Minerbi, U. Moderow, R. K. Monson, L. Montagnani, C. E. Moore, E. Moors, V. Moreaux, C. Moureaux, J. W. Munger, T. Nakai, J. Neirynck, Z. Nesic, G. Nicolini, A. Noormets, M. Northwood, M. Nosetto, Y. Nouvellon, K. Novick, W. Oechel, J. E. Olesen, J.-M. Ourcival, S. A. Papuga, F.-J. Parmentier, E. Paul-Limoges, M. Pavelka, M. Peichl, E. Pendall, R. P. Phillips, K. Pilegaard, N. Pirk, G. Posse, T. Powell, H. Prasse, S. M. Prober, S. Ram bal, U. Rannik, N. Raz-Yaseef, D. Reed, V. R. de Dios, N. Restrepo-Coupe, B. R. Reverter, M. Roland, S. Sabbatini, T. Sachs, S. R. Saleska, E. P. S.-C. nete, Z. M. Sanchez-Mejia, H. P. Schmid, M. Schmidt, K. Schneider, F. Schrader, I. Schroder, R. L. Scott, P. Sedlák, P. Serrano-Ortíz, C. Shao, P. Shi, I. Shironya, L. Siebicke, L. Šigut, R. Silberstein, C. Sirca, D. Spano, R. Steinbrecher, R. M. Stevens, C. Sturtevant, A. Suyker, T. Tagesson, S. Takanashi, Y. Tang, N. Tapper, J. Thom, F. Tiedemann, M. Tomassucci, J.-P. Tuovinen, S. Urbanski, R. Valentini, M. van der Molen, E. van Gorsel, K. van Huissteden, A. Varlagin, J. Verfaillie, T. Vesala, C. Vincke, D. Vitale, N. Vygodskaya, J. P. Walker, E. Walter-Shea, H. Wang, R. Weber, S. Westermann, C. Wille, S. Wofsy, G. Wohlfahrt, S. Wolf, W. Woodgate, Y. Li, R. Zampedri, J. Zhang, G. Zhou, D. Zona, D. Agarwal, S. Biraud, M. Torn, D. Papale, "The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data", Scientific Data, 2020, 7:225, doi: 10.1038/s41597-020-0534-3

YW Cheah, J Boverhof, A Elbashandy, D Agarwal, J Leek, T Epperly, J Eslick, D Miller, "Data management and simulation support accelerating carbon capture through computing", Proceedings of the 2016 IEEE 12th International Conference on e-Science, e-Science 2016, 2017, 389--398, doi: 10.1109/eScience.2016.7870924

Gilberto Z. Pastorello, Dan K. Gunter, Housen Chu, Danielle S. Christianson, Carlo Trotta, Eleonora Canfora, Boris Faybishenko, You-Wei Cheah, Norm Beekwilder, Stephen W. Chan, Sigrid Dengel, Trevor Keenan, Fianna O Brien, Abderahman Elbashandy, Cristina M. Poindexter, Marty Humphrey, Dario Papale, Deb A. Agarwal, "Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures", Proceedings of the 13th IEEE International Conference on e-Science (e-Science 2017), Auckland, New Zealand, 2017, doi: 10.1109/eScience.2017.64

S Peisert, R Gentz, J Boverhof, C McParland, S Engle, A Elbashandy, D Gunter, "LBNL Open Power Data", January 2017, doi: 10.21990/C21599

GH Weber, MS Bandstra, DH Chivers, HH Elgammal, V Hendrix, J Kua, JS Maltz, K Muriki, Y Ong, K Song, MJ Quinlan, L Ramakrishnan, BJ Quiter, "Web-based visual data exploration for improved radiological source detection", Concurrency Computation, 2017, 29, doi: 10.1002/cpe.4203

Chris Lawson, Jose Manuel Martí, Tijana Radivojevic, Sai Vamshi R. Jonnalagadda, Reinhard Gentz, Nathan J. Hillson, Sean Peisert, Joonhoon Kim, Blake A. Simons, Christopher J. Petzold, Steven W. Singer, Aindrila Mukhopadhyay, Deepti Tanjore, Josh Dunn, Héctor García Martín,, "Machine Learning for Metabolic Engineering: A Review", Metabolic Engineering, November 19, 2020,

Reinhard Gentz, Sean Peisert, "An Examination and Survey of Random Bit Flips and Scientific Computing", Trusted CI Report, December 20, 2019,

Mahdi Jamei, Raksha Ramakrishna, Teklemariam Tesfay, Reinhard Gentz, Ciaran Roberts, Anna Scaglione, Sean Peisert, "Phasor Measurement Units Optimal Placement and Performance Limits for Fault Localization", IEEE Journal on Selected Areas in Communications (J-SAC), Special Issue on Communications and Data Analytics in Smart Grid, November 6, 2019, 38(1):180-192, doi: 10.1109/jsac.2019.2951971

Thomas W. Edgar, Aditya Ashok, Garret E. Seppala, K.M. Arthur-Durrett, M. Engels, Reinhard Gentz, Sean Peisert, "An Automated Disruption-Tolerant Key Management Framework for Critical Systems", Journal of Information Warfare, October 8, 2019, 18(4):85-124, doi: https://www.jinfowar.com/journal/volume-18-issue-4/automated-disruption-tolerant-device-authentication-key-management-framework-critical-systems

Reinhard Gentz, Sean Peisert, Joshua Boverhof, Daniel Gunter, "SPARCS: Stream-Processing Architecture applied in Real-time Cyber-physical Security", Proceedings of the 15th IEEE International Conference on e-Science (eScience), San Diego, CA, IEEE, September 2019, doi: 10.1109/eScience.2019.00028

Reinhard Gentz, Héctor García Martin, Edward Baidoo, Sean Peisert, "Workflow Automation in Liquid Chromatography Mass Spectrometry", Proceedings of the 15th IEEE International Conference on e-Science (eScience), San Diego, CA, IEEE, September 2019, doi: 10.1109/eScience.2019.00095

Melissa Stockman, Dipankar Dwivedi, Reinhard Gentz, Sean Peisert, "Detecting Programmable Logic Controller Code Using Machine Learning", International Journal of Critical Infrastructure Protection, September 2019, vol. 26,, doi: 10.1016/j.ijcip.2019.100306

Ciaran Roberts, Anna Scaglione, Mahdi Jamei, Reinhard Gentz, Sean Peisert, Emma M. Stewart, Chuck McParland, Alex McEachern, Daniel Arnold, "Learning Behavior of Distribution System Discrete Control Devices for Cyber-Physical Security", IEEE Transaction on Smart Grid, August 1, 2019, 11(1):749-761, doi: 0.1109/TSG.2019.2936016

Sean Peisert, Ciaran Roberts, Anna Scaglione, Mahdi Jamei, Reinhard Gentz, Charles McParland, Alex McEachren, Galen Rasche, Aaron Snyder, "Supporting Cyber Security of Power Distribution Systems by Detecting Differences Between Real-time Micro-Synchrophasor Measurements and Cyber-Reported SCADA - Final Report", October 15, 2018,

S Peisert, R Gentz, J Boverhof, C McParland, S Engle, A Elbashandy, D Gunter, "LBNL Open Power Data", January 2017, doi: 10.21990/C21599

L. Ferrari, A. Scaglione, R. Gentz, Y. W. P. Hong, "Convergence Results on Pulse Coupled Oscillator Protocols in Locally Connected Networks", IEEE/ACM Transactions on Networking, 2017, 25:1004-1019, doi: 10.1109/TNET.2016.2611379

R. Gentz, A. Scaglione, L. Ferrari, Y. W. P. Hong, "PulseSS: A Pulse-Coupled Synchronization and Scheduling Protocol for Clustered Wireless Sensor Networks", IEEE Internet of Things Journal, 2016, 3:1222-1234, doi: 10.1109/JIOT.2016.2576923

R. Gentz, S. X. Wu, H. T. Wai, A. Scaglione, A. Leshem, "Data Injection Attacks in Randomized Gossiping", IEEE Transactions on Signal and Information Processing over Networks, 2016, 2:523-538, doi: 10.1109/TSIPN.2016.2614898

Georgia Koutsandria, Reinhard Gentz, Mahdi Jamei, Anna Scaglione, Sean Peisert, Chuck McParland, "A real-time testbed environment for cyber-physical security on the power grid", Proceedings of the First ACM Workshop on Cyber-Physical Systems-Security and/or PrivaCy, January 1, 2015, 67--78, doi: 10.1145/2808705.2808707

Reinhard Gentz, Anna Scaglione, Yao-Win Hong, Lorenzo Ferrari, "PulseSS: A Microcontroller Implementation of Pulse-Coupled Scheduling and Synchronization Protocol for Cluster-BasedWireless Sensor Networks", 2015,

R. Gentz, H. T. Wai, A. Scaglione, A. Leshem, "Detection of data injection attacks in decentralized learning", 2015 49th Asilomar Conference on Signals, Systems and Computers, 2015, 350-354, doi: 10.1109/ACSSC.2015.7421145

Lorenzo Ferrari, Reinhard Gentz, Anna Scaglione, Masood Parvania, "The Pulse Coupled Phasor Measurement Units", IEEE Smartgridcomm, 2014,

Devarshi Ghoshal, Ludovico Bianchi, Abdelilah Essiari, Drew Paine, Sarah Poon, Michael Beach, Alpha N'Diaye, Patrick Huck, Lavanya Ramakrishnan, "Science Capsule: Towards Sharing and Reproducibility of Scientiﬁc Workﬂows", 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS), November 15, 2021, doi: 10.1109/WORKS54523.2021.00014

Workflows are increasingly processing large volumes of data from scientific instruments, experiments and sensors. These workflows often consist of complex data processing and analysis steps that might include a diverse ecosystem of tools and also often involve human-in-the-loop steps. Sharing and reproducing these workflows with collaborators and the larger community is critical but hard to do without the entire context of the workflow including user notes and execution environment. In this paper, we describe Science Capsule, which is a framework to capture, share, and reproduce scientific workflows. Science Capsule captures, manages and represents both computational and human elements of a workflow. It automatically captures and processes events associated with the execution and data life cycle of workflows, and lets users add other types and forms of scientific artifacts. Science Capsule also allows users to create `workflow snapshots' that keep track of the different versions of a workflow and their lineage, allowing scientists to incrementally share and extend workflows between users. Our results show that Science Capsule is capable of processing and organizing events in near real-time for high-throughput experimental and data analysis workflows without incurring any significant performance overheads.

Devarshi Ghoshal, Drew Paine, Gilberto Pastorello, Abdelrahman Elbashandy, Dan Gunter, Oluwamayowa Amusat, Lavanya Ramakrishnan, "Experiences with Reproducibility: Case Studies from Scientific Workflows", (P-RECS'21) Proceedings of the 4th International Workshop on Practical Reproducible Evaluation of Computer Systems, ACM, June 21, 2021, doi: 10.1145/3456287.3465478

Reproducible research is becoming essential for science to ensure transparency and for building trust. Additionally, reproducibility provides the cornerstone for sharing of methodology that can improve efficiency. Although several tools and studies focus on computational reproducibility, we need a better understanding about the gaps, issues, and challenges for enabling reproducibility of scientific results beyond the computational stages of a scientific pipeline. In this paper, we present five different case studies that highlight the reproducibility needs and challenges under various system and environmental conditions. Through the case studies, we present our experiences in reproducing different types of data and methods that exist in an experimental or analysis pipeline. We examine the human aspects of reproducibility while highlighting the things that worked, that did not work, and that could have worked better for each of the cases. Our experiences capture a wide range of scenarios and are applicable to a much broader audience who aim to integrate reproducibility in their everyday pipelines.

Devarshi Ghoshal, Ludovico Bianchi, Abdelilah Essiari, Michael Beach, Drew Paine, Lavanya Ramakrishnan, "Science Capsule - Capturing the Data Life Cycle", Journal of Open Source Software, 2021, 6:2484, doi: 10.21105/joss.02484

Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, "Experiences with a Flexible User Research Process to Build Data Change Tools", Journal of Open Research Software, September 1, 2020, doi: 10.5334/jors.284

Scientific software development processes are understood to be distinct from commercial software development practices due to uncertain and evolving states of scientific knowledge. Sustaining these software products is a recognized challenge, but under-examined is the usability and usefulness of such tools to their scientific end users. User research is a well-established set of techniques (e.g., interviews, mockups, usability tests) applied in commercial software projects to develop foundational, generative, and evaluative insights about products and the people who use them. Currently these approaches are not commonly applied and discussed in scientific software development work. The use of user research techniques in scientific environments can be challenging due to the nascent, fluid problem spaces of scientific work, varying scope of projects and their user communities, and funding/economic constraints on projects.

In this paper, we reflect on our experiences undertaking a multi-method user research process in the Deduce project. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists make decisions around data change. There is a lack of common terminology since the concept of systematically measuring and managing data change is under explored in scientific environments. To bridge this gap we conducted user research that focuses on user practices, needs, and motivations to help us design and develop metrics and tools for data change. This paper contributes reflections and the lessons we have learned from our experiences. We offer key takeaways for scientific software project teams to effectively and flexibly incorporate similar processes into their projects.

Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, "Investigating Scientific Data Change with User Research Methods", August 20, 2020, LBNL LBNL-2001347,

Scientific datasets are continually expanding and changing due to fluctuations with instruments, quality assessment and quality control processes, and modifications to software pipelines. Datasets include minimal information about these changes or their effects requiring scientists manually assess modifications through a number of labor intensive and ad-hoc steps. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists systematically identify and make decisions around data changes. Currently, there is a lack of understanding, and common practices, for identifying and evaluating changes in datasets since systematically measuring and managing data change is under explored in scientific work. We are conducting user research to address this need by exploring scientist's conceptualizations, behaviors, needs, and motivations when dealing with changing datasets. Our user research utilizes multiple methods to produce foundational, generative insights and evaluate research products produced by our team. In this paper, we detail our user research process and outline our findings about data change that emerge from our studies. Our work illustrates how scientific software teams can push beyond just usability testing user interfaces or tools to better probe the underlying ideas they are developing solutions to address.

P. Linton, W. Melodia, A. Lazar, D. Agarwal, L. Bianchi, D. Ghoshal, K. Wu, G. Pastorello, L. Ramakrishnan, "Identifying Time Series Similarity in Large-Scale Earth System Datasets", The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19), 2019,

Payton A Linton, William M Melodia, Alina Lazar, Deborah Agarwal, Ludovico Bianchi, Devarshi Ghoshal, Kesheng Wu, Gilberto Pastorello, Lavanya Ramakrishnan, "Identifying Time Series Similarity in Large-Scale Earth System Datasets", 2019,

Cheah You-Wei, Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, Bringing Data Science to Qualitative Analysis, 2018 IEEE 14th International Conference on e-Science, Pages: 325-326 2018, doi: 10.1109/eScience.2018.00076

Devarshi Ghoshal, "Deduce: Managing Data Change Pipelines", Conference on Data Analysis (CoDA 2018), 2018,

D Ghoshal, L Ramakrishnan, D Agarwal, "Dac-Man: Data Change Management for Scientific Datasets on HPC Systems", SC ’18, Piscataway, NJ, USA, IEEE Press, 2018, 72:1--72:1,

S Swaid, M Maat, H Krishnan, D Ghoshal, L Ramakrishnan, "Usability heuristic evaluation of scientific data analysis and visualization tools", Advances in Intelligent Systems and Computing, 2018, 607:471--482, doi: 10.1007/978-3-319-60492-3_45

Devarshi Ghoshal, Lavanya Ramakrishnan, "MaDaTS: Managing Data on Tiered Storage for Scientific Workflows", Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing (HPDC '17), ACM, 2017, 41--52, doi: 10.1145/3078597.3078611

D Ghoshal, V Hendrix, W Fox, S Balasubhramanian, L Ramakrishnan, "FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework", The Journal of Open Source Software, 2017, 2:164--164, doi: 10.21105/joss.00164

W Fox, D Ghoshal, A Souza, GP Rodrigo, L Ramakrishnan, "E-HPC: A library for elastic resource management in HPC environments", Proceedings of WORKS 2017: 12th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2017, doi: 10.1145/3150994.3150996

V Hendrix, J Fox, D Ghoshal, L Ramakrishnan, "Tigres Workflow Library: Supporting Scientific Pipelines on HPC Systems", Proceedings - 2016 16th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2016, 2016, 146--155, doi: 10.1109/CCGrid.2016.54

CS Daley, D Ghoshal, GK Lockwood, S Dosanjh, L Ramakrishnan, NJ Wright, "Performance characterization of scientific workflows for the optimal use of Burst Buffers", CEUR Workshop Proceedings, 2016, 1800:69--73,

L Ramakrishnan, D Ghoshal, V Hendrix, E Feller, P Mantha, C Morin, "Storage and Data Life Cycle Management in Cloud Environments with FRIEDA.", Cloud Computing for Data-Intensive Applications, (Springer: 2014) Pages: 357--378

Devarshi Ghoshal, Lavanya Ramakrishnan, "FRIEDA: Flexible Robust Intelligent Elastic Data Management in Cloud Environments", 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), IEEE, 2013, 1096--1105, doi: 10.1109/SC.Companion.2012.132

Lavanya Ramakrishnan, Adam Scovel, Iwona Sakrejda, Susan Coghlan, Shane Canon, Anping Liu, Devarshi Ghoshal, Krishna Muriki, Nicholas J. Wright, "Magellan - A Testbed to Explore Cloud Computing for Science", On the Road to Exascale Computing: Contemporary Architectures in High Performance Computing, (Chapman & Hall/CRC Press: 2013)

Devarshi Ghoshal, Richard Shane Canon, Lavanya Ramakrishnan, "I/O Performance of Virtualized Cloud Environments", Proceedings of the Second International Workshop on Data Intensive Computing in the Clouds (DataCloud-SC '11), ACM, 2011, 71--80, doi: 10.1145/2087522.2087535

Ayaz Akram, Anna Giannakou, Venkatesh Akella, Jason Lowe-Power, Sean Peisert, "Performance Analysis of Scientific Computing Workloads on General Purpose TEEs", Proceedings of the 35th IEEE International Parallel & Distributed Processing Symposium (IPDPS), IEEE, May 2021, doi: 10.1109/IPDPS49936.2021.00115

Ayaz Akram, Anna Giannakou, Venkatesh Akella, Jason Lowe-Power, Sean Peisert, "Performance Analysis of Scientific Computing Workloads on Trusted Execution Environments", arXiv preprint arXiv:2010.13216, October 25, 2020,

Anna Giannakou, Dipankar Dwivedi, Sean Peisert, "A Machine Learning Approach for Packet Loss Prediction in ScienceFlows", Future Generation Computer Systems, January 2020, 102:190-197, doi: 10.1016/j.future.2019.07.053

Anna Giannakou, Daniel Gunter, Sean Peisert, "Flowzilla: A Methodology for Detecting Data Transfer Anomalies in Research Networks", Workshop on Innovating the Network for Data-Intensive Science (INDIS), November 11, 2018, doi: 10.1109/INDIS.2018.00004

Taghrid Samak, Dan Gunter, Monte Goode, Ewa Deelman, Fabio Silva, Karan Vahi, "Failure Analysis of Distributed Scientific Workflows Executing in the Cloud", 8th International Conference on Network and Service Management (CNSM 2012), 2012,

Taghrid Samak, Dan Gunter, Monte Goode, Ewa Deelman, Gaurang Mehta, Fabio Silva, Karan Vahi, "Failure Prediction and Localization in Large Scientific Workflows", The Sixth Workshop on Workflows in Support of Large-Scale Science (WORKS11), 2011,

Dan Gunter, Taghrid Samak, Ewa Deelman, Christopher H. Brooks, Monte Goode, Gideon Juve, Gaurang Mehta, Priscilla Moraes, Fabio Silva, Martin Swany, Karan Vahi, "Online Workflow Management and Performance Analysis with STAMPEDE", 7th International Conference on Network and Service Management (CNSM 2011), Paris, France, 2011,

D. Agarwal, M. Humphrey,N., Beekwilder, K. Jackson, M. Goode, and C. van Ingen, "A Data Centered Collaboration Portal to Support Global Carbon-Flux Analysis", Concurrency and Computation: Practice and Experience - Successes in Furthering Scientific Discovery, June 2010, LBNL 1827E,

DA Agarwal, M Humphrey, NF Beekwilder, KR Jackson, MM Goode, C Van Ingen, "A data-centered collaboration portal to support global carbon-flux analysis", Concurrency Computation Practice and Experience, 2010, 22:2323--2334, doi: 10.1002/cpe.1600

M Humphrey, N Beekwilder, D Agarwal, D Baldocchi, M Goode, C Ingen, D Papale, M Reichstein, M Rodriguez, R Vargas, N Li, J Gupchup, Y Ryu, Creating and Accessing the Global Fluxnet Data Set, 2008,

Lipeng Wan, Axel Huebl, Junmin Gu, Franz Poeschel, Ana Gainaru, Ruonan Wang, Jieyang Chen, Xin Liang, Dmitry Ganyushin, Todd Munson, Ian Foster, Jean-Luc Vay, Norbert Podhorszki, Kesheng Wu, Scott Klasky, "Improving I/O Performance for Exascale Applications Through Online Data Layout Reorganization", IEEE Transactions on Parallel and Distributed Systems, 2022, 33:878-890, doi: 10.1109/TPDS.2021.3100784

E. Wes Bethel, Burlen Loring, Utkarsh Ayachit, P. N. Duque, Nicola Ferrier, Joseph Insley, Junmin Gu, Kress, Patrick O’Leary, Dave Pugmire, Silvio Rizzi, Thompson, Will Usher, Gunther H. Weber, Brad Whitlock, Wolf, Kesheng Wu, "Proximity Portability and In Transit, M-to-N Data Partitioning and Movement in SENSEI", In Situ Visualization for Computational Science, ( 2022) doi: 10.1007/978-3-030-81627-8_20

E. Wes Bethel, Burlen Loring, Utkarsh Ayatchit, David Camp, P. N. Duque, Nicola Ferrier, Joseph Insley, Junmin Gu, Kress, Patrick O’Leary, David Pugmire, Silvio Rizzi, Thompson, Gunther H. Weber, Brad Whitlock, Matthew Wolf, Kesheng Wu, "The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale", In Situ Visualization for Computational Science, ( 2022) doi: 10.1007/978-3-030-81627-8_13

Franz Poeschel, Juncheng E, William F. Godoy, Norbert Podhorszki, Scott Klasky, Greg Eisenhauer, Philip E. Davis, Lipeng Wan, Ana Gainaru, Junmin Gu, Fabian Koller, René Widera, Michael Bussmann, Axel Huebl, "Transitioning from file-based HPC workflows to streaming data pipelines with openPMD and ADIOS2", Smoky Mountains Computational Sciences and Engineering Conference (SMC2021), 2021,

William F.Godoy, Norbert Podhorszki, Ruonan Wang, Chuck Atkins, Greg Eisenhauer, Junmin Gu,Philip Davis,J ong Choi, Kai Germaschewski, Kevin Huck, Axel Huebl, Mark Kim, James Kress, Tahsin Kurc, Qing Liu, Jeremy Logan, Kshitij Mehta, George Ostrouchov, Manish Parashar, Franz Poeschel, David Pugmire, Eric Suchyta, KeichiTakahashi, NickThompson, Seiji Tsutsumi, Lipeng Wan, Matthew Wolf, Kesheng Wu, Scott Klasky, "ADIOS 2: The Adaptable Input Output System. A framework for high-performance data management", SoftwareX, 2020, 12,

Junmin Gu, Burlen Loring, Kesheng Wu, E. Wes Bethel, "HDF5 as a vehicle for in transit data movement", The Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV'19), 2019, doi: 10.1145/3364228.3364237

Junmin Gu, Burlen Loring, Kesheng Wu, E Wes Bethel, "HDF5 as a vehicle for in transit data movement", Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, 2019, 39--43,

Junmin Gu, Scott Klasky, Norbert Podhorszki, Ji Qiang, Kesheng Wu, "Querying large scientific data sets with adaptable IO system ADIOS", Asian Conference on Supercomputing Frontiers, 2018, 51--69,

Burlen Loring, Suren Byna, Prabhat, Junmin Gu, Hari Krishnan, Michael Wehner, and Oliver Ruebel, "TECA an Extreme Event Detection and Climate Analysis Package for High Performance Computing", The AMS (American Meteorological Society) 96th Annual Meeting, January 6, 2016,

David Pugmire, James Kress, Jong Choi, Scott Klasky, Tahsin Kurc, Randy Michael Churchill, Matthew Wolf, Greg Eisenhower, Hank Childs, Kesheng Wu, others, "Visualization and analysis for near-real-time decision making in distributed workflows", 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2016, 1007--1013,

Utkarsh Ayachit, Andrew Bauer, Earl PN Duque, Greg Eisenhauer, Nicola Ferrier, Junmin Gu, Kenneth E Jansen, Burlen Loring, Zarija Lukic, Suresh Menon, others, "Performance analysis, design considerations, and applications of extreme-scale in situ infrastructures", SC 16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2016, 921--932, LBNL 1007264,

D. Pugmire, J. Kress, J. Choi, S. Klasky, Kurc, R. M. Churchill, M. Wolf, G., H. Childs, K. Wu, A. Sim, J. Gu, J. Low, "Visualization and Analysis for Near-Real-Time Decision in Distributed Workflows", 2016 IEEE International Parallel and Distributed Symposium Workshops (IPDPSW), 2016, 1007--1013, doi: 10.1109/IPDPSW.2016.175

A. L. Chervenak, A. Sim, J. Gu, R. Schuler, N. Hirpathak, "Adaptation and Policy-Based Resource Allocation for Efficient Bulk Data Transfers in High Performance Computing Environments", 4th International Workshop on Network-aware Data Management (NDM'14), 2014,

A. L. Chervenak, A. Sim, J. Gu, R. Schuler, N. Hirpathak, "Efficient Data Staging Using Performance-Based Adaptation and Policy-Based Resource Allocation", 22nd Euromicro International Conference on Parallel, Distributed and Network-based Processing, 2014,

Junmin Gu, David Smith, Ann L. Chervenak, Alex Sim, "Adaptive Data Transfers that Utilize Policies for Resource Sharing", The 2nd International Workshop on Network-Aware Data Management Workshop (NDM2012), 2012,

D. Yu, D. Katramatos, A. Shoshani, A. Sim, J. Gu, V. Natarajan, "StorNet: Integrating Storage Resource Management with Dynamic Network Provisioning for Automated Data Transfer", International Committee for Future Accelerators (ICFA) Standing Committee on Inter-Regional Connectivity (SCIC) 2012 Report: Networking for High Energy Physics, 2012,

J. Gu, D. Katramatos, X. Liu, V. Natarajan, A. Shoshani, A. Sim, D. Yu, S. Bradley, S. McKee, "StorNet: Integrated Dynamic Storage and Network Resource Provisioning and Management for Automated Data Transfers", Journal of Physics: Conf. Ser., 2011, 331, doi: 10.1088/1742- 6596/331/1/012002

G. Garzoglio, J. Bester, K. Chadwick, D. Dykstra, D. Groep, J. Gu, T. Hesselroth, O. Koeroo, T. Levshina, S. Martin, M. Salle, N. Sharma, A. Sim, S. Timm, A. Verstegen, "Adoption of a SAML-XACML Profile for Authorization Interoperability across Grid Middleware in OSG and EGEE", Journal of Physics: Conf. Ser., 2011, 331, doi: 10.1088/1742-6596/331/6/062011

Junmin Gu, Dimitrios Katramatos, Xin Liu, Vijaya Natarajan, Arie Shoshani, Alex Sim, Dantong Yu, Scott Bradley, Shawn McKee, "StorNet: Co-Scheduling of End-to-End Bandwidth Reservation on Storage and Network Systems for High Performance Data Transfers", IEEE INFOCOM HSN 2011, 2011,

Dean N. Williams, Ian T. Foster, Don E. Middleton, Rachana Ananthakrishnan, Neill Miller, Mehmet Balman, Junmin Gu, Vijaya Natarajan, Arie Shoshani, Alex Sim, Gavin Bell, Robert Drach, Michael Ganzberger, Jim Ahrens, Phil Jones, Daniel Crichton, Luca Cinquini, David Brown, Danielle Harper, Nathan Hook, Eric Nienhouse, Gary Strand, Hannah Wilcox, Nathan Wilhelmi, Stephan Zednik, Steve Hankin, Roland Schweitzer, John Harney, Ross Miller, Galen Shipman, Feiyi Wang, Peter Fox, Patrick West, Stephan Zednik, Ann Chervenak, Craig Ward, "Earth System Grid Center for Enabling Technologies (ESG-CET): A Data Infrastructure for Data-Intensive Climate Research", SciDAC Conference, 2011,

M. Riedel, E. Laure, Th. Soddemann, L. Field, J. P. Navarro, J. Casey, M. Litmaath, J. Ph. Baud, B. Koblitz, C. Catlett, D. Skow, C. Zheng, P. M. Papadopoulos, M. Katz, N. Sharma, O. Smirnova, B. Kónya, P. Arzberger, F. Würthwein, A. S. Rana, T. Martin, M. Wan, V. Welch, T. Rimovsky, S. Newhouse, A. Vanni, Y. Tanaka, Y. Tanimura, T. Ikegami, D. Abramson, C. Enticott, G. Jenkins, R. Pordes, N. Sharma, S. Timm, N. Sharma, G. Moont, M. Aggarwal, D. Colling, O. van der Aa, A. Sim, V. Natarajan, A. Shoshani, J. Gu, S. Chen, G. Galang, R. Zappi, L. Magnoni, V. Ciaschini, M. Pace, V. Venturi, M. Marzolla, P. Andreetto, B. Cowles, S. Wang, Y. Saeki, H. Sato, S. Matsuoka, P. Uthayopas, S. Sriprayoonsakul, O. Koeroo, M. Viljoen, L. Pearlman, S. Pickles, David Wallom, G. Moloney, J. Lauret, J. Marsteller, P. Sheldon, S. Pathak, S. De Witt, J. Mencák, J. Jensen, M. Hodges, D. Ross, S. Phatanapherom, G. Netzer, A. R. Gregersen, M. Jones, S. Chen, P. Kacsuk, A. Streit, D. Mallmann, F. Wolf, T. Lippert, Th. Delaitre, E. Huedo, N. Geddes, "Interoperation of world-wide production e-Science infrastructures", Concurrency and Computation: Practice and Experience, 2009, 21(8):961-990,

Arie Shoshani, Flavia Donno, Junmin Gu, Jason Hick, Maarten Litmaath, Alex Sim, "Dynamic Storage Management", Scientific Data Management: Challenges, Technology, and Deployment, edited by Arie Shoshani, Doron Rotem, (Chapman & Hall/CRC Computational Science: 2009)

K Wu, S Ahern, EW Bethel, J Chen, H Childs, C Geddes, J Gu, H Hagen, B Hamann, J Lauret, others, "FastBit: Interactively Searching Massive Data", Proc. of SciDAC 2009, 2009, LBNL 2164E,

Download File: LBNL-2164E.pdf (pdf: 3.2 MB)

P. Jakl, J. Lauret, A. Hanushevsky, A. Shoshani, A. Sim, J. Gu, "Grid data access on widely distributed worker nodes using scalla and SRM", Journal of Physics: Conf. Ser., 2008, 119, doi: 10.1088/1742-6596/119/7/072019

Alex Sim, Arie Shoshani (Editors), Paolo Badino, Olof Barring, Jean‐Philippe Baud, Ezio Corso, Shaun De Witt, Flavia Donno, Junmin Gu, Michael Haddox‐Schatz, Bryan Hess, Jens Jensen, Andy Kowalski, Maarten Litmaath, Luca Magnoni, Timur Perelmutov, Don Petravick, Chip Watson, The Storage Resource Manager Interface Specification Version 2.2, Open Grid Forum, Document in Full Recommendation, GFD.129, 2008,

L. Abadie, P. Badino, J. Baud, E. Corso, M. Crawford, S. De Witt, F. Donno, A. Forti, P. Fuhrmann,
G. Grosdidier, J. Gu , J. Jensen, S. Lemaitre, M. Litmaath, D. Litvinsev, G. Lo Presti, L. Magnoni, T. Mkrtchan, A. Moibenko, V. Natarajan, G. Oleynik, T. Perelmutov, D. Petravick, A. Shoshani, A. Sim, M. Sponza, R. Zappi, "Storage Resource Managers: Recent International Experience on Requirements and Multiple Co-Operating Implementations", the 24th IEEE Conference on Mass Storage Systems and Technologies, 2007,

F. Donno, L. Abadie, P. Badino, J. Baud, E. Corso, M. Crawford, S. De Witt, A. Forti, P. Fuhrmann, G. Grosdidier, J. Gu , J. Jensen, S. Lemaitre, M. Litmaath, D. Litvinsev, G. Lo Presti, L. Magnoni, T. Mkrtchan, A. Moibenko, V. Natarajan, G. Oleynik, T. Perelmutov, D. Petravick, A. Shoshani, A. Sim, M. Sponza, R. Zappi, "Storage Resource Manager version 2.2: design, implementation, and testing experience", Journal of Physics: Conf. Ser., 2007, 119, doi: 10.1088/1742-6596/119/6/062028

Kesheng Wu, Junmin Gu, Jerome Lauret, Arthur Poskanzer, Arie Shoshani, Alexander Sim, Zhang, "Grid Collector: Facilitating Efficient Selective from Data Grids", International Supercomputer Conference 2005, 2005,

Alex Sim, Junmin Gu, Arie Shoshani, Vijaya Natarajan, "DataMover: Robust Terabytes-Scale Multi-file Replication over Wide-Area Networks", the 16th International Conference on Scientific and Statistical Database Management (SSDBM 2004), 2004,

Arie Shoshani, Alexander Sim, Junmin Gu, "Storage Resource Managers: Essential Components for the Grid", Grid Resource Management: State of the Art and Future Trends, edited by Jarek Nabrzyski, Jennifer M. Schopf, Jan Weglarz, (Kluwer Academic Publishers: 2003)

A. Sim, J. Gu, A. Shoshani, E. Hjort, D. Olson, "Experience with Deploying Storage Resource Managers to Achieve Robust File Replication", Computing in High Energy Physics, 2003,

Arie Shoshani, Alex Sim, Junmin Gu, Storage Resource Managers: Essential Components for Grid Applications, Globus World, 2003,

Kesheng Wu, Wei-Ming Zhang, Alexander Sim, Gu, Arie Shoshani, "Grid Collector: An Event Catalog With Automated File", Proceedings of IEEE Nuclear Science Symposium 2003, 2003, doi: 10.1109/NSSMIC.2003.1351830

Kesheng Wu, Wei-Ming Zlang, Alexander Sim, Junmin Gu, Arie Shoshani, "Grid collector: An event catalog with automated file management", 2003 IEEE Nuclear Science Symposium. Conference Record (IEEE Cat. No. 03CH37515), 2003, LBNL 55563,

A. Shoshani, A. Sim, J. Gu, "Storage Resource Managers: Middleware components for Grid Storage", the 19th IEEE Symposium on Mass Storage Systems, 2002,

Mohammed A. Alhussaini, Zachary M. Binger, Bianca M. Souza-Chaves, Oluwamayowa O. Amusat, Jangho Park, Timothy V. Bartholomew, Dan Gunter, Andrea Achilli, "Analysis of backwash settings to maximize net water production in an engineering-scale ultrafiltration system for water reuse", Journal of Water Process Engineering, 2023, 53, doi: 10.1016/j.jwpe.2023.103761

Andrew Adams, Emily K. Adams, Dan Gunter, Ryan Kiser, Mark Krenz, Sean Peisert, John Zage, "Roadmap for Securing Operational Technology in NSF Scientific Research", Trusted CI Report, November 16, 2022, doi: 10.5281/zenodo.7327987

Emily K. Adams, Daniel Gunter, Ryan Kiser, Mark Krenz, Sean Peisert, Susan Sons, John Zage, "Findings of the 2022 Trusted CI Study on the Security of Operational Technology in NSF Scientific Research", Trusted CI Report, July 15, 2022, doi: doi.org/10.5281/zenodo.6828675

Dan Gunter, Oluwamayowa Amusat, Tim Bartholomew, Markus Drouven, "Santa Barbara Desalination Digital Twin Technical Report", LBNL Technical Report, 2021, LBNL LBNL-2001437,

Devarshi Ghoshal, Drew Paine, Gilberto Pastorello, Abdelrahman Elbashandy, Dan Gunter, Oluwamayowa Amusat, Lavanya Ramakrishnan, "Experiences with Reproducibility: Case Studies from Scientific Workflows", (P-RECS'21) Proceedings of the 4th International Workshop on Practical Reproducible Evaluation of Computer Systems, ACM, June 21, 2021, doi: 10.1145/3456287.3465478

Reproducible research is becoming essential for science to ensure transparency and for building trust. Additionally, reproducibility provides the cornerstone for sharing of methodology that can improve efficiency. Although several tools and studies focus on computational reproducibility, we need a better understanding about the gaps, issues, and challenges for enabling reproducibility of scientific results beyond the computational stages of a scientific pipeline. In this paper, we present five different case studies that highlight the reproducibility needs and challenges under various system and environmental conditions. Through the case studies, we present our experiences in reproducing different types of data and methods that exist in an experimental or analysis pipeline. We examine the human aspects of reproducibility while highlighting the things that worked, that did not work, and that could have worked better for each of the cases. Our experiences capture a wide range of scenarios and are applicable to a much broader audience who aim to integrate reproducibility in their everyday pipelines.

Reinhard Gentz, Sean Peisert, Joshua Boverhof, Daniel Gunter, "SPARCS: Stream-Processing Architecture applied in Real-time Cyber-physical Security", Proceedings of the 15th IEEE International Conference on e-Science (eScience), San Diego, CA, IEEE, September 2019, doi: 10.1109/eScience.2019.00028

Anna Giannakou, Daniel Gunter, Sean Peisert, "Flowzilla: A Methodology for Detecting Data Transfer Anomalies in Research Networks", Workshop on Innovating the Network for Data-Intensive Science (INDIS), November 11, 2018, doi: 10.1109/INDIS.2018.00004

AP Arkin,RW Cottingham,CS Henry,NL Harris,RL Stevens,S Maslov,P Dehal,D Ware,F Perez,S Canon,MW Sneddon,ML Henderson,WJ Riehl,D Murphy-Olson,SY Chan,RT Kamimura,S Kumari,MM Drake,TS Brettin,EM Glass,D Chivian,D Gunter,DJ Weston,BH Allen,J Baumohl,AA Best,B Bowen,SE Brenner,CC Bun,JM Chandonia,JM Chia,R Colasanti,N Conrad,JJ Davis,BH Davison,M Dejongh,S Devoid,E Dietrich,I Dubchak,JN Edirisinghe,G Fang,JP Faria,PM Frybarger,W Gerlach,M Gerstein,A Greiner,J Gurtowski,HL Haun,F He,R Jain,MP Joachimiak,KP Keegan,S Kondo,V Kumar,ML Land,F Meyer,M Mills,PS Novichkov,T Oh,GJ Olsen,R Olson,B Parrello,S Pasternak,E Pearson,SS Poon,GA Price,S Ramakrishnan,P Ranjan,PC Ronald,MC Schatz,SMD Seaver,M Shukla,RA Sutormin,MH Syed,J Thomason,NL Tintle,D Wang,F Xia,H Yoo,S Yoo,D Yu, "KBase: The United States department of energy systems biology knowledgebase", Nature Biotechnology, July 2018, 36:566--569, doi: 10.1038/nbt.4163

DC Miller, JD Siirola, D Agarwal, AP Burgard, A Lee, JC Eslick, B Nicholson, C Laird, LT Biegler, D Bhattacharyya, NV Sahinidis, IE Grossmann, CE Gounaris, D Gunter, "Next Generation Multi-Scale Process Systems Engineering Framework", Computer Aided Chemical Engineering, 2018, 44:2209--2214, doi: 10.1016/B978-0-444-64241-7.50363-3

DA Agarwal, C Varadharajan, S Cholia, C Snavely, VC Hendrix, D Gunter, WJ Riley, M jones, AE budden, D Vieglas, Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE)-A New US DOE Data Archive, 2017,

Gilberto Z. Pastorello, Dan K. Gunter, Housen Chu, Danielle S. Christianson, Carlo Trotta, Eleonora Canfora, Boris Faybishenko, You-Wei Cheah, Norm Beekwilder, Stephen W. Chan, Sigrid Dengel, Trevor Keenan, Fianna O Brien, Abderahman Elbashandy, Cristina M. Poindexter, Marty Humphrey, Dario Papale, Deb A. Agarwal, "Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures", Proceedings of the 13th IEEE International Conference on e-Science (e-Science 2017), Auckland, New Zealand, 2017, doi: 10.1109/eScience.2017.64

S Peisert, R Gentz, J Boverhof, C McParland, S Engle, A Elbashandy, D Gunter, "LBNL Open Power Data", January 2017, doi: 10.21990/C21599

L Ramakrishnan, D Gunter, "Ten principles for creating usable software for science", Proceedings - 13th IEEE International Conference on eScience, eScience 2017, 2017, 210--218, doi: 10.1109/eScience.2017.34

Patrick Huck, Dan Gunter, Shreyas Cholia, Donald Winston, AT N Diaye, Kristin Persson, "User applications driven by the community contribution framework MPContribs in the Materials Project", Concurrency and Computation: Practice and Experience, 2016, 28:1982--1993,

P Huck, D Gunter, S Cholia, D Winston, A N Diaye, KA Persson, "User Applications Driven by the Community Contribution Framework MPContribs in the Materials Project.", CoRR, 2015, abs/1510,

SP Ong, S Cholia, A Jain, M Brafman, D Gunter, G Ceder, KA Persson, "The Materials Application Programming Interface (API): A simple, flexible and efficient API for materials data based on REpresentational State Transfer (REST) principles", Computational Materials Science, 2015, 97:209--215, doi: 10.1016/j.commatsci.2014.10.037

Gilberto Z. Pastorello, Deb A. Agarwal, Taghrid Samak, Dario Papale, Trotta, Alessio Ribeca, Cristina M. Poindexter, Boris Faybishenko, Dan K. Gunter, Rachel Hollowgrass, Eleonora Canfora, "Observational data patterns for time series data quality assessment", Proceedings of the 10th IEEE International Conference on e-Science (e-Science 2014), Guaruja, Brazil, 2014, doi: 10.1109/eScience.2014.45

Lavanya Ramakrishnan, Sarah S. Poon, Val C. Hendrix, Dan K. Gunter, Gilberto Z. Pastorello, Deb A. Agarwal, "Experiences with User-Centered Design for the Tigres Workflow API", Proceedings of the 10th IEEE International Conference on e-Science (e-Science 2014), Guaruja, Brazil, 2014, doi: 10.1109/eScience.2014.56

Elif Dede, Madhusudhan Govindaraju, Daniel Gunter, Richard Canon, Lavanya Ramakrishnan, "Semi-Structured Data Analysis using MongoDB and MapReduce: A Performance Evaluation", Proceedings of the 4th international workshop on Scientific cloud computing, 2013,

A Jain, SP Ong, G Hautier, W Chen, WD Richards, S Dacek, S Cholia, D Gunter, D Skinner, G Ceder, KA Persson, "Commentary: The materials project: A materials genome approach to accelerating materials innovation", APL Materials, 2013, 1, doi: 10.1063/1.4812323

SP Ong, WD Richards, A Jain, G Hautier, M Kocher, S Cholia, D Gunter, VL Chevrier, KA Persson, G Ceder, "Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis", Computational Materials Science, 2013, 68:314--319, doi: 10.1016/j.commatsci.2012.10.028

Taghrid Samak, Dan Gunter, Monte Goode, Ewa Deelman, Fabio Silva, Karan Vahi, "Failure Analysis of Distributed Scientific Workflows Executing in the Cloud", 8th International Conference on Network and Service Management (CNSM 2012), 2012,

Karan Vahi, Ian Harvey, Taghrid Samak, Dan Gunter, Kieran Evans, David Rogers, Ian Taylor, Monte Goode, Fabio Silva, Eddie Al-Shakarchi, Gaurang Mehta, Andrew Jones, Ewa Deelman, "A General Approach to Real-time Workflow Monitoring", The Seventh Workshop on Workflows in Support of Large-Scale Science (WORKS12), 2012,

D Gunter, S Cholia, A Jain, M Kocher, K Persson, L Ramakrishnan, SP Ong, G Ceder, "Community accessible datastore of high-throughput calculations: Experiences from the materials project", Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, 2012, 1244--1251, doi: 10.1109/SC.Companion.2012.150

Dan Gunter, Raj Kettimuthu, Ezra Kissel, Martin Swany, Jun Yi, Jason Zurawski, "Exploiting Network Parallelism for Improving Data Transfer Performance", SC12 Companion, 2012,

T Samak, D Gunter, V Hendrix, "Scalable analysis of network measurements with Hadoop and Pig", Proceedings of the 2012 IEEE Network Operations and Management Symposium, NOMS 2012, 2012, 1254--1259, doi: 10.1109/NOMS.2012.6212060

Ezra Kissel, Ahmed El-Hassany, Guilherme Fernandes, Martin Swany, Dan Gunter, Taghrid Samak, Jennifer M. Schopf, "Scalable Integrated Performance Analysis of Multi-Gigabit Networks", Fifth International Workshop on Distributed Autonomous Network Management Systems 2012 (DANMS 12), 2012,

Elif Dede, Zacharia Fadika, Jessica Hartog, Modhusudhan Govindaraju, Lavanya Ramakrishnan, Daniel Gunter, Richard Shane Canon, "MARISSA: MApReduce Implementation for Streaming Science Applications", IEEE eScience Conference, 2012,

A Jain, G Hautier, SP Ong, C Moore, B Kang, H Chen, X Ma, JC Kim, M Kocher, D Gunter, S Cholia, A Greiner, DH Bailey, D Skinner, K Persson, G Ceder, "Materials Project: A public materials database and its application to lithium ion battery cathode design", ABSTRACTS OF PAPERS OF THE AMERICAN CHEMICAL SOCIETY, 2012, 243,

Taghrid Samak, Dan Gunter, Monte Goode, Ewa Deelman, Gaurang Mehta, Fabio Silva, Karan Vahi, "Failure Prediction and Localization in Large Scientific Workflows", The Sixth Workshop on Workflows in Support of Large-Scale Science (WORKS11), 2011,

Dan Gunter, Taghrid Samak, Ewa Deelman, Christopher H. Brooks, Monte Goode, Gideon Juve, Gaurang Mehta, Priscilla Moraes, Fabio Silva, Martin Swany, Karan Vahi, "Online Workflow Management and Performance Analysis with STAMPEDE", 7th International Conference on Network and Service Management (CNSM 2011), Paris, France, 2011,

Taghrid Samak, Dan Gunter, Ewa Deelman, Gideon Juve, Gaurang Mehta, Fabio Silva, Karan Vahi, "Online Fault and Anomaly Detection for Large-Scale Scientific Workflows", 13th IEEE Conference on High Performance Computing and Communications (HPCC-2011), 2011,

Elif Dede, Madhusudan Govindaraju, Daniel Gunter, Lavanya Ramakrishnan, "Riding the Elephant: Managing Ensembles with Hadoop", 4th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS), 2011,

A. Sim, D. Gunter, V. Natarajan, A. Shoshani, D. Williams, J. Long, J. Hick, J. Lee, E. Dart, "Efficient Bulk Data Replication for the Earth System Grid", Data Driven E-science: Use Cases and Successful Applications of Distributed Computing Infrastructures (ISGC 2010), (Springer-Verlag New York Inc: 2010) Pages: 435

Scott Callaghan, Ewa Deelman, Dan Gunter, Gideon Juve, Philip Maechling, Christopher Brooks, Karan Vahi, Kevin Milner, Robert Graves, Edward Field, David Okaya, Thomas Jordan, "Scaling up Workflow-based Applications", Journal of Computer and System Sciences, 2010, 76:428--446,

Raj Kettimuthu, Alex Sim, Dan Gunter, Bill Allcock, Peer T. Bremer, John Bresnahan, Andrew Cherry, Lisa Childers, Eli Dart, Ian Foster, Kevin Harms, Jason Hick, Jason Lee, Michael Link, Jeff Long, Keith Miller, Vijaya Natarajan, Valerio Pascucci, Ken Raffenetti, David Ressman, Dean Williams, Loren Wilson, Linda Winkler, "Lessons learned from moving earth system grid data sets over a 20 Gbps wide-area network", HPDC 10, New York, NY, USA, ACM, 2010, 316--319, doi: 10.1145/1851476.1851519

Shoaib Kamil, Pinar, Gunter, Lijewski, Oliker, John Shalf, "Reconfigurable hybrid interconnection for static and dynamic scientific applications", Conf. Computing Frontiers, 2007, 183-194, LBNL 60060,

Download File: CF07.pdf (pdf: 9.5 MB)

Dan Gunter, Brian L. Tierney, Aaron Brown, Martin Swany, John Bresnahan, Jennifer M. Schopf, "Log summarization and anomaly detection for troubleshooting distributed systems", Grid Computing, 2007 8th IEEE/ACM International Conference on, Washington, DC, USA, IEEE Computer Society, 2007, 226--234, doi: 10.1109/GRID.2007.4354137

C Varadharajan, Z Kakalia, E Alper, EL Brodie, M Burrus, RWH Carroll, D Christianson, W Dong, V Hendrix, M Henderson, S Hubbard, D Johnson, R Versteeg, KH Williams, DA Agarwal, The Colorado East River Community Observatory Data Collection, Hydrological Processes 35(6), 2021, doi: 10.22541/au.161962485.54378235/v1

C Varadharajan, B Faybishenko, A Henderson, M Henderson, VC Hendrix, SS Hubbard, Z Kakalia, A Newman, B Potter, H Steltzer, R Versteeg, DA Agarwal, KH Williams, C Wilmer, Y Wu, W Brown, M Burrus, RWH Carroll, DS Christianson, B Dafflon, D Dwivedi, BJ Enquist, Challenges in Building an End-to-End System for Acquisition, Management, and Integration of Diverse Data from Sensor Networks in Watersheds: Lessons from a Mountainous Community Observatory in East River, Colorado, IEEE Access, Pages: 182796--18 2019, doi: 10.1109/ACCESS.2019.2957793

Wahid Bhimji, Steven Farrell, Oliver Evans, Matthew Henderson, Shreyas Cholia, Aaron Vose, Mr Prabhat, Rollin Thomas, Richard Shane Canon, "Interactive HPC Deep Learning with Jupyter Notebooks", Supercomputing 2018, Dallas, TX, November 2018,

Shreyas Cholia, Matthew Henderson, Oliver Evans, Fernando Perez, "Kale: A System for Enabling Human-in-the-loop Interactivity in HPC Workflows", Science Gateways 2018, figshare, September 26, 2018, doi: 10.6084/m9.figshare.7067075.v3

AP Arkin,RW Cottingham,CS Henry,NL Harris,RL Stevens,S Maslov,P Dehal,D Ware,F Perez,S Canon,MW Sneddon,ML Henderson,WJ Riehl,D Murphy-Olson,SY Chan,RT Kamimura,S Kumari,MM Drake,TS Brettin,EM Glass,D Chivian,D Gunter,DJ Weston,BH Allen,J Baumohl,AA Best,B Bowen,SE Brenner,CC Bun,JM Chandonia,JM Chia,R Colasanti,N Conrad,JJ Davis,BH Davison,M Dejongh,S Devoid,E Dietrich,I Dubchak,JN Edirisinghe,G Fang,JP Faria,PM Frybarger,W Gerlach,M Gerstein,A Greiner,J Gurtowski,HL Haun,F He,R Jain,MP Joachimiak,KP Keegan,S Kondo,V Kumar,ML Land,F Meyer,M Mills,PS Novichkov,T Oh,GJ Olsen,R Olson,B Parrello,S Pasternak,E Pearson,SS Poon,GA Price,S Ramakrishnan,P Ranjan,PC Ronald,MC Schatz,SMD Seaver,M Shukla,RA Sutormin,MH Syed,J Thomason,NL Tintle,D Wang,F Xia,H Yoo,S Yoo,D Yu, "KBase: The United States department of energy systems biology knowledgebase", Nature Biotechnology, July 2018, 36:566--569, doi: 10.1038/nbt.4163

S. Farrell, A. Vose, O. Evans, M. Henderson, S. Cholia, W. Bhimji, R. Thomas, S.
Canon, and Prabhat,, "Interactive Distributed Deep Learning with Jupyter Notebooks", ISC Workshop on Interactive High-Performance Computing, June 28, 2018,

GP Rodrigo, M Henderson, GH Weber, C Ophus, K Antypas, L Ramakrishnan, "ScienceSearch: Enabling Search through Automatic Metadata Generation", 2018 IEEE 14th International Conference on e-Science (e-Science), IEEE, 2018, doi: 10.1109/escience.2018.00025

Shreyas Cholia, Matthew Henderson, Oliver Evans, Demo: Extending Jupyter to Support Interactive High Performance Computing, Science Gateways 2017, October 2017, doi: 10.6084/m9.figshare.5501137.v1

MB Simmonds, WJ Riley, DA Agarwal, X Chen, S Cholia, R Crystal-Ornelas, ET Coon, D Dwivedi, VC Hendrix, M Huang, A Jan, Z Kakalia, J Kumar, CD Koven, L Li, M Melara, L Ramakrishnan, DM Ricciuto, AP Walker, W Zhi, Q Zhu, C Varadharajan, Guidelines for Publicly Archiving Terrestrial Model Data to Enhance Usability, Intercomparison, and Synthesis, Data Science Journal, 2022, doi: 10.5334/dsj-2022-003

C Varadharajan, VC Hendrix, DS Christianson, M Burrus, C Wong, SS Hubbard, DA Agarwal, BASIN-3D: A brokering framework to integrate diverse environmental data, Computers and Geosciences, 2022, doi: 10.1016/j.cageo.2021.105024

C Varadharajan, AP Appling, B Arora, DS Christianson, VC Hendrix, V Kumar, AR Lima, J Müller, S Oliver, M Ombadi, T Perciano, JM Sadler, H Weierbach, JD Willard, Z Xu, J Zwart, "Can machine learning accelerate process understanding and decision-relevant predictions of river water quality?", Hydrological Processes, January 1, 2022, 36, doi: 10.1002/hyp.14565

H Weierbach, AR Lima, JD Willard, VC Hendrix, DS Christianson, M Lubich, C Varadharajan, Stream Temperature Predictions for River Basin Management in the Pacific Northwest and Mid-Atlantic Regions Using Machine Learning, Water (Switzerland), 2022, doi: 10.3390/w14071032

C Varadharajan, Z Kakalia, E Alper, EL Brodie, M Burrus, RWH Carroll, D Christianson, W Dong, V Hendrix, M Henderson, S Hubbard, D Johnson, R Versteeg, KH Williams, DA Agarwal, The Colorado East River Community Observatory Data Collection, Hydrological Processes 35(6), 2021, doi: 10.22541/au.161962485.54378235/v1

JE Damerow, C Varadharajan, K Boye, EL Brodie, M Burrus, KD Chadwick, R Crystal-Ornelas, H Elbashandy, RJ Eloy Alves, KS Ely, AE Goldman, T Haberman, V Hendrix, Z Kakalia, KM Kemner, AB Kersting, N Merino, F O Brien, Z Perzan, E Robles, P Sorensen, JC Stegen, RL Walls, P Weisenhorn, M Zavarin, D Agarwal, Sample identifiers and metadata to support data management and reuse in multidisciplinary ecosystem sciences, Data Science Journal, 2021, doi: 10.5334/dsj-2021-011

R Crystal-Ornelas, C Varadharajan, B Bond-Lamberty, K Boye, M Burrus, S Cholia, M Crow, J Damerow, R Devarakonda, KS Ely, A Goldman, S Heinz, V Hendrix, Z Kakalia, SC Pennington, E Robles, A Rogers, M Simmonds, T Velliquette, H Weierbach, P Weisenhorn, JN Welch, DA Agarwal, A Guide to Using GitHub for Developing and Versioning Data Standards and Reporting Formats, Earth and Space Science, 2021, doi: 10.1029/2021EA001797

C Varadharajan, S Cholia, C Snavely, V Hendrix, C Procopiou, D Swantek, W Riley, D Agarwal, "Launching an Accessible Archive of Environmental Data", Eos, January 1, 2019, 100, doi: 10.1029/2019eo111263

C Varadharajan, B Faybishenko, A Henderson, M Henderson, VC Hendrix, SS Hubbard, Z Kakalia, A Newman, B Potter, H Steltzer, R Versteeg, DA Agarwal, KH Williams, C Wilmer, Y Wu, W Brown, M Burrus, RWH Carroll, DS Christianson, B Dafflon, D Dwivedi, BJ Enquist, Challenges in Building an End-to-End System for Acquisition, Management, and Integration of Diverse Data from Sensor Networks in Watersheds: Lessons from a Mountainous Community Observatory in East River, Colorado, IEEE Access, Pages: 182796--18 2019, doi: 10.1109/ACCESS.2019.2957793

EM Stewart, P Top, M Chertkov, D Deka, S Backhaus, A Lokhov, C Roberts, V Hendrix, S Peisert, A Florita, TJ King, MJ Reno, "Integrated multi-scale data analytics and machine learning for the distribution grid", 2017 IEEE International Conference on Smart Grid Communications, SmartGridComm 2017, 2018, 2018-Jan:423--429, doi: 10.1109/SmartGridComm.2017.8340693

Danielle S. Christianson, Charuleka Varadharajan, Bradley Christoffersen, Matteo Detto, Faybishenko, Bruno O. Gimenez, Val C. Hendrix, Kolby J. Jardine, Robinson Negron-Juarez, Z. Pastorello, Thomas L. Powell, Megha Sandesh, Jeffrey M. Warren, Brett T. Wolfe, Jeffrey Q. Chambers, Lara M. Kueppers, Nathan G. McDowell, Deborah A. Agarwal, "A metadata reporting framework (FRAMES) for synthesis of ecohydrological observations", Ecological Informatics, 2017, 42:148-158, doi: 10.1016/j.ecoinf.2017.06.002

DA Agarwal, C Varadharajan, S Cholia, C Snavely, VC Hendrix, D Gunter, WJ Riley, M jones, AE budden, D Vieglas, Environmental System Science Data Infrastructure for a Virtual Ecosystem (ESS-DIVE)-A New US DOE Data Archive, 2017,

GH Weber, MS Bandstra, DH Chivers, HH Elgammal, V Hendrix, J Kua, JS Maltz, K Muriki, Y Ong, K Song, MJ Quinlan, L Ramakrishnan, BJ Quiter, "Web-based visual data exploration for improved radiological source detection", Concurrency Computation, 2017, 29, doi: 10.1002/cpe.4203

D Ghoshal, V Hendrix, W Fox, S Balasubhramanian, L Ramakrishnan, "FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework", The Journal of Open Source Software, 2017, 2:164--164, doi: 10.21105/joss.00164

E Stewart, V Hendrix, M Chertkov, D Deka, "Integrated Multi-Scale Data Analytics and Machine Learning for the Distribution Grid and Building-to-Grid Interface", 2017,

V Hendrix, J Fox, D Ghoshal, L Ramakrishnan, "Tigres Workflow Library: Supporting Scientific Pipelines on HPC Systems", Proceedings - 2016 16th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2016, 2016, 146--155, doi: 10.1109/CCGrid.2016.54

M Verma, JB Fisher, K Mallick, Y Ryu, H Kobayashi, A Guillaume, G Moore, L Ramakrishnan, V Hendrix, S Wolf, M Sikka, G Kiely, G Wohlfahrt, B Gielen, O Roupsard, P Toscano, A Arain, A Cescatti, "Global surface net-radiation at 5 km from MODIS Terra", Remote Sensing, 2016, 8, doi: 10.3390/rs8090739

V Hendrix, L Ramakrishnan, Y Ryu, C Van Ingen, KR Jackson, D Agarwal, "CAMP: Community access MODIS pipeline", Future Generation Computer Systems, 2014, 36:418--429, doi: 10.1016/j.future.2013.09.023

Lavanya Ramakrishnan, Sarah S. Poon, Val C. Hendrix, Dan K. Gunter, Gilberto Z. Pastorello, Deb A. Agarwal, "Experiences with User-Centered Design for the Tigres Workflow API", Proceedings of the 10th IEEE International Conference on e-Science (e-Science 2014), Guaruja, Brazil, 2014, doi: 10.1109/eScience.2014.56

JR Balderrama, M Simonin, L Ramakrishnan, V Hendrix, C Morin, D Agarwal, C Tedeschi, "Combining workflow templates with a shared space-based execution model", Proceedings of WORKS 2014: The 9th Workshop on Workflows in Support of Large-Scale Science - held in conjunction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2014, 50--58, doi: 10.1109/WORKS.2014.14

S Panitkin, FB Megino, JC Bejar, D Benjamin, AD Girolamo, I Gable, V Hendrix, J Hover, K Kucharczyk, RM Llamas, P Love, H Ohman, M Paterson, R Sobie, R Taylor, R Walker, A Zaytsev, "ATLAS cloud R\&D", Journal of Physics: Conference Series, 2014, 513, doi: 10.1088/1742-6596/513/6/062037

H Öhman, S Panitkin, V Hendrix, "Using Puppet to contextualize computing resources for ATLAS analysis on Google compute engine", Journal of Physics: Conference Series, 2014, 513, doi: 10.1088/1742-6596/513/3/032073

L Ramakrishnan, D Ghoshal, V Hendrix, E Feller, P Mantha, C Morin, "Storage and Data Life Cycle Management in Cloud Environments with FRIEDA.", Cloud Computing for Data-Intensive Applications, (Springer: 2014) Pages: 357--378

E. Masanet, A. Shehabi, L. Ramakrishnan, J. Liang, X. Ma, B. Walker, V. Hendrix, P Mantha, "The Energy Efficiency Potential of Cloud-Based Software: A U.S.Case Study", 2013, LBNL 6298E,

V Hendrix, J Li, K Jackson, L Ramakrishnan, Y Ryu, K Beattie, C Morin, D Skinner, C van Ingen, D Agarwal, "Community Access to MODIS Satellite Reprojection and Reduction Pipeline and Data Sets", AGU Fall Meeting, 2012,

T Samak, D Gunter, V Hendrix, "Scalable analysis of network measurements with Hadoop and Pig", Proceedings of the 2012 IEEE Network Operations and Management Symposium, NOMS 2012, 2012, 1254--1259, doi: 10.1109/NOMS.2012.6212060

F Harald Barreiro Megino, D Benjamin, K De, I Gable, V Hendrix, S Panitkin, M Paterson, A De Silva, D Van Der Ster, R Taylor, RA Vitillo, R Walker, "Exploiting virtualization and cloud computing in ATLAS", Journal of Physics: Conference Series, 2012, 396, doi: 10.1088/1742-6596/396/3/032011

V Hendrix, D Benjamin, Y Yao, "Scientific cluster deployment and recovery - Using puppet to simplify cluster management", Journal of Physics: Conference Series, 2012, 396, doi: 10.1088/1742-6596/396/4/042027

D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, W. Arndt, J. Blaschke, S. Byna, R. Cheema, S. Cholia, M. Day, B. Enders, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, T. Lehman, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, L. Stephey, R. Thomas, G. Torok, "LBNL Superfacility Project Report", Lawrence Berkeley National Laboratory, 2022, doi: 10.48550/arXiv.2206.11992

Qiang Du, Dan Wang, Tong Zhou, Antonio Gilardi, Mariam Kiran, Bashir Mohammed, Derun Li, and Russell Wilcox, "Experimental beam combining stabilization using machine learning trained while phases drift", Advanced Solid State Lasers 2022, © 2022 Optica Publishing Group, June 1, 2022, Vol. 30,:pp. 12639-, doi: https://doi.org/10.1364/OE.450255

Sugeerth Murugesan, Mariam Kiran, Bernd Hamann, Gunther H. Weber, "Netostat: Analyzing Dynamic Flow Patterns in High-Speed Networks", Cluster Computing, 2022, doi: 10.1007/s10586-022-03543-0

Shen Sheng, Mariam Kiran, Bashir Mohammed, "DynamicDeepFlow: An Approach for Identifying Changes in Network Traffic Flow Using Unsupervised Clustering", (BEST PAPER) 4th International Conference on Machine Learning for Networking (MLN'2021), December 6, 2021,

V. Dumont, C. Garner, A. Trivedi, C. Jones, V. Ganapati, J. Mueller, T. Perciano, M. Kiran, and M. Day, "HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization", 2021 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC), November 15, 2021,

Bashir Mohammed, Mariam Kiran, Bjoern Enders, "NetGraf: An End-to-End Learning Network Monitoring Service", 2021 IEEE Workshop on Innovating the Network for Data-Intensive Science (INDIS), November 15, 2021, doi: 10.1109/INDIS54524.2021.00007

B Mohammed, M Kiran; N Krishnaswamy; Keshang, Wu, "Predicting WAN Traffic Volumes using Fourier and Multivariate SARIMA Approach", International Journal of Big Data Intelligence, November 3, 2021, doi: 10.1504/IJBDI.2021.118742

M Kiran, B Mohammed, Q Du, D Wang, S Shen, R Wilcox, "Controlling Laser Beam Combining via an Active Reinforcement Learning Algorithm", Advanced Solid State Lasers 2021, Washington, DC United States, October 4, 2021,

Meriam Gay Bautista, Zhi Jackie Yao, Anastasiia Butko, Mariam Kiran, Mekena Metcalf, "Towards Automated Superconducting Circuit Calibration using Deep Reinforcement Learning", 2021 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), Tampa, FL, USA, IEEE, August 23, 2021, pp. 462-46, doi: 10.1109/ISVLSI51109.2021.00091

Bashir Mohammed, Mariam Kiran, Dan Wang, Qiang Du, Russell Wilcox, "Deep Reinforcement Learning based Control for two-dimensional Coherent Combining", Laser Applications Conference, pp. JTu5A-7. Optical Society of America, 2020., OSA Publishing, December 1, 2020,

Dan Wang, Qiang Du, Tong Zhou, Bashir Mohammed, Mariam Kiran, Derun Li, Russell Wilcox, "Artificial Neural Networks Applied to Stabilization of 81-beam Coherent Combining", Advanced Solid State Lasers, Optical Society of America, December 1, 2020,

Divneet Kaur, Bashir Mohammed, Mariam Kiran, "NetGraf: A Collaborative Network Monitoring Stack for Network Experimental Testbeds", December 1, 2020,

D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, S. Byna, R. Cheema, S. Cholia, M. Day, B. Enders, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, R. Thomas, G. Torok, The Superfacility project: automated pipelines for experiments and HPC, International Conference for High Performance Computing, Networking, Storage, and Analysis (SC20), State of the Practice (SOP), 2020,

B. Enders, D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, S. Byna, R. Cheema, S. Cholia, M. Day, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, R. Thomas, G. Torok, "Cross-facility science with the Superfacility Project at LBNL", 2nd Workshop on Large-scale Experiment-in-the-Loop Computing (XLOOP 2020), in conjunction with the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC 20), 2020, doi: 10.1109/XLOOP51963.2020.00006

N Krishnaswamy; M Kiran; B Mohammed; Singh Kunal, "Data-driven Learning to Predict WAN Network Traffic.", SNTA '20: Proceedings of the 3rd International Workshop on Systems and Network Telemetry and Analytics, November 3, 2020, 11-18, doi: 10.1145/3391812.3396268

T Mallick, M Kiran, B Mohammed, Prasanna Balaprakash, "Dynamic Graph Neural Network for Traffic Forecasting in Wide Area Networks.", Machine Learning Big Data 2020, IEEE, November 2, 2020, 1-10, doi: 10.1109/BigData50022.2020.9512748

M Hocine, M Kiran, A Mercian, and B Mohammed, "Using Machine Learning for Intent-based provisioning in High-Speed Science Networks.", SNTA '20: Proceedings of the 3rd International Workshop on Systems and Network Telemetry and Analytics, November 2, 2020, 27-30, doi: 10.1145/3391812.3396269

M Kiran, B Mohammed and N. Krishnaswamy, "DeepRoute: Herding Elephant and Mice Flows with Reinforcement Learning", 2nd IFIP International Conference on Machine Learning for Networking (MLN'2019), December 2, 2019, doi: 10.1007/978-3-030-45778-5_20

B Mohammed, M Kiran, N Krishnaswamy, "DeepRoute on Chameleon: Experimenting with Large-scale Reinforcement Learning and SDN on Chameleon Testbed", IEEE 27th International Conference on Network Protocols (ICNP), IEEE, November 14, 2019, 1-2, doi: 10.1109/ICNP.2019.8888090

George Papadimitriou, Mariam Kiran, Cong Wang, Anirban Mandal, Ewa Deelman, "Training Classifiers to Identify TCP Signatures in Scientific Workflows", INDIS, SC19, November 14, 2019,

B Mohammed, N Krishnaswamy, M Kiran, "Multivariate Time-Series Prediction for Traffic in Large WAN Topology", ACM/IEEE Symposium on Architectures for Networking and Communications, August 1, 2019, doi: 10.1109/ANCS.2019.8901870

M Kiran, A Chhabra, "Understanding flows in high-speed scientific networks: A Netflow data study", Future Generation Computer Systems, February 1, 2019, 94:72-79,

Mariam Kiran, Anshuman Chhabra, "Understanding flows in high-speed scientific networks: A Netflow data study", Future Generation Computer Science, 2019,

F Alali, N Hanford, E Pouyoul, R Kettimuthu, M Kiran, B Mack-Crane, B Tierney, Y Kumar, D Ghosal, "Calibers: A bandwidth calendaring paradigm for science workflows", Future Generation Computer Systems, 2018, 89:736--745, doi: 10.1016/j.future.2018.07.030

A Bennaceur, A Cano, L Georgieva, M Kiran, M Salama, P Yadav, "Issues in gender diversity and equality in the UK", Proceedings - International Conference on Software Engineering, 2018, 5--9, doi: 10.1145/3195570.3195571

B Mohammed, B Modu, KM Maiyama, H Ugail, I Awan, M Kiran, "Failure Analysis Modelling in an Infrastructure as a Service (Iaas) Environment", Electronic Notes in Theoretical Computer Science, 2018, 340:41--54, doi: 10.1016/j.entcs.2018.09.004

M Gribaudo, M Iacono, M Kiran, "A performance modeling framework for lambda architecture based applications", Future Generation Computer Systems, 2018, 86:1032--1041, doi: 10.1016/j.future.2017.07.033

M Kiran, E Pouyoul, A Mercian, B Tierney, C Guok, I Monga, "Enabling intent to configure scientific networks for high performance demands", Future Generation Computer Systems, 2018, 79:205--214, doi: 10.1016/j.future.2017.04.020

B Mohammed, S Moyo, KM Maiyama, S Kinteh, ANMK Al-Shaidy, MA Kamala, M Kiran, "Technical Report on Deploying a highly secured OpenStack Cloud Infrastructure using BradStack as a Case Study", 2017,

A Mercian, M Kiran, E Pouyoul, B Tierney, I Monga, "INDIRA: Application Intent network assistant to configure SDN-based high performance scientific networks", Optics InfoBase Conference Papers, 2017, Part F40, doi: 10.1364/OFC.2017.Tu3L.6

M Usman, A Iqbal, M Kiran, "A bandwidth friendly architecture for Cloud Gaming", International Conference on Information Networking, 2017, 179--184, doi: 10.1109/ICOIN.2017.7899500

KM Maiyama, D Kouvatsos, B Mohammed, M Kiran, MA Kamala, "Performance modelling and analysis of an OpenStack IaaS cloud computing platform", Proceedings - 2017 IEEE 5th International Conference on Future Internet of Things and Cloud, FiCloud 2017, 2017, 2017-Jan:198--205, doi: 10.1109/FiCloud.2017.54

T Eftonova, M Kiran, M Stannett, "Long-term Macroeconomic Dynamics of Competition in the Russian Economy using Agent- based Modelling.", IJSDA, 2017, 6:1--20,

B Mohammed, M Kiran, KM Maiyama, MM Kamala, IU Awan, "Failover strategy for fault tolerance in cloud computing environment", Journal of Software - Practice and Experience, 2017, 47:1243--1274, doi: https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.2491

B Mohammed, M Kiran, I-U Awan, KM Maiyama, "An Integrated Virtualized Strategy for Fault Tolerance in Cloud Computing Environment", 2016 Intl IEEE Conferences on Ubiquitous Intelligence & Computing, Advanced and Trusted Computing, Scalable Computing and Communications, Cloud and Big Data Computing, Internet of People, and Smart World Congress (UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld), 2016, 542--549, doi: 10.1109/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.0094

B Mohammed, M Kiran, IU Awan, KM Maiyama, "Optimising fault tolerance in real-time cloud computing IaaS environment", Proceedings - 2016 IEEE 4th International Conference on Future Internet of Things and Cloud, FiCloud 2016, 2016, 363--370, doi: 10.1109/FiCloud.2016.58

M Kiran, A Simons, "Testing Software Services in Cloud Ecosystems", International Journal of Cloud Applications and Computing, 2016, 6:42--58, doi: 10.4018/ijcac.2016010103

B Mohammed, M Kiran, "Analysis of Cloud Test Beds Using OpenSource Solutions", Proceedings - 2015 International Conference on Future Internet of Things and Cloud, FiCloud 2015 and 2015 International Conference on Open and Big Data, OBD 2015, 2015, 195--203, doi: 10.1109/FiCloud.2015.106

B Mohammed, M Kiran, "Experimental Report on Setting up a Cloud Computing Environment at the University of Bradford", 2014,

S Swaid, M Maat, H Krishnan, D Ghoshal, L Ramakrishnan, "Usability heuristic evaluation of scientific data analysis and visualization tools", Advances in Intelligent Systems and Computing, 2018, 607:471--482, doi: 10.1007/978-3-319-60492-3_45

Dáithí A Stone, Mark D Risser, Oliver M Angélil, Michael F Wehner, Shreyas Cholia, Noel Keen, Harinarayan Krishnan, Travis A O Brien, William D Collins, "A basis set for exploration of sensitivity to prescribed ocean conditions for estimating human contributions to extreme weather in CAM5. 1-1degree", Weather and climate extremes, 2018, 19:10--19,

P Enfedaque, H Chang, H Krishnan, S Marchesini, "GPU-based implementation of ptycho-ADMM for high performance x-ray imaging", Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018, 10860 LN:540--553, doi: 10.1007/978-3-319-93698-7_41

RJ Pandolfi, DB Allan, E Arenholz, L Barroso-Luque, SI Campbell, TA Caswell, A Blair, F De Carlo, S Fackler, AP Fournier, G Freychet, M Fukuto, D Gürsoy, Z Jiang, H Krishnan, D Kumar, RJ Kline, R Li, C Liman, S Marchesini, A Mehta, AT N Diaye, DY Parkinson, H Parks, LA Pellouchoud, T Perciano, F Ren, S Sahoo, J Strzalka, D Sunday, CJ Tassone, D Ushizima, S Venkatakrishnan, KG Yager, P Zwart, JA Sethian, A Hexemer, "Xi-cam: a versatile interface for data visualization and analysis", Journal of Synchrotron Radiation, 2018, 25:1261--1270, doi: 10.1107/S1600577518005787

C Li, C Michel, L Seland Graff, I Bethke, G Zappa, TJ Bracegirdle, E Fischer, BJ Harvey, T Iversen, MP King, H Krishnan, L Lierhammer, D Mitchell, J Scinocca, H Shiogama, DA Stone, JJ Wettstein, Midlatitude atmospheric circulation responses under 1.5 and 2.0g°C warming and implications for regional impacts, Earth System Dynamics, Pages: 359--382 2018, doi: 10.5194/esd-9-359-2018

M Wehner, D Stone, D Mitchell, H Shiogama, E Fischer, LS Graff, VV Kharin, L Lierhammer, B Sanderson, H Krishnan, Changes in extremely hot days under stabilized 1.5 and 2.0 °c global warming scenarios as simulated by the HAPPI multi-model ensemble, Earth System Dynamics, Pages: 299--311 2018, doi: 10.5194/esd-9-299-2018

MF Wehner, KA Reed, B Loring, D Stone, H Krishnan, "Changes in tropical cyclones under stabilized 1.5 and 2.0°C global warming scenarios as simulated by the Community Atmospheric Model under the HAPPI protocols", Earth System Dynamics, 2018, 9:187--195, doi: 10.5194/esd-9-187-2018

JJ Billings, AR Bennett, J Deyton, K Gammeltoft, J Graham, D Gorin, H Krishnan, M Li, AJ McCaskey, T Patterson, R Smith, GR Watson, A Wojtowicz, The eclipse integrated computational environment, SoftwareX, Pages: 234--244 2018, doi: 10.1016/j.softx.2018.07.004

M Wehner, D Stone, H Shiogama, P Wolski, A Ciavarella, N Christidis, H Krishnan, Early 21st century anthropogenic changes in extremely hot days as simulated by the C20C+ detection and attribution multi-model ensemble, Weather and Climate Extremes, Pages: 1--8 2018, doi: 10.1016/j.wace.2018.03.001

DA Stone, MD Risser, OM Angélil, MF Wehner, S Cholia, N Keen, H Krishnan, TA O Brien, WD Collins, A basis set for exploration of sensitivity to prescribed ocean conditions for estimating human contributions to extreme weather in CAM5.1-1degree, Weather and Climate Extremes, Pages: 10--19 2018, doi: 10.1016/j.wace.2017.12.003

YS Yu, M Farmand, C Kim, Y Liu, CP Grey, FC Strobridge, T Tyliszczak, R Celestre, P Denes, J Joseph, H Krishnan, FRNC Maia, ALD Kilcoyne, S Marchesini, TPC Leite, T Warwick, H Padmore, J Cabana, DA Shapiro, Three-dimensional localization of nanoscale battery reactions using soft X-ray tomography, Nature Communications, 2018, doi: 10.1038/s41467-018-03401-x

DA Shapiro, R Celestre, B Enders, J Joseph, H Krishnan, MA Marcus, K Nowrouzi, H Padmore, J Park, A Warwick, Y-S Yu, The COSMIC Imaging Beamline at the Advanced Light Source: a new facility for spectro-microscopy of nano-materials, Microscopy and Microanalysis, Pages: 8--11 2018, doi: 10.1017/s1431927618012485

B Enders, K Nowrouzi, H Krishnan, S Marchesini, J Park, Y-S Yu, DA Shapiro, Dataflow at the COSMIC Beamline - Stream Processing and Supercomputing, Microscopy and Microanalysis, Pages: 58--59 2018, doi: 10.1017/s1431927618012710

Stone, D. A., H. Krishnan, R. Lance, S. Sippel, and M. F. Wehner, "The First and Second Hackathons of the International CLIVAR C20C+ Detection and Attribution Project", CLIVAR Exchanges, 2017,

Download File: StoneDA-KrishnanH-etalii-2017.pdf (pdf: 636 KB)

Timmermans, B., D. Stone, M. Wehner, and H. Krishnan, "Impact of tropical cyclones on modeled wind-wave climate", Geophysical Research Letters, 2017, 44:1393-1401, doi: 10.1002/2016GL071681

T Perciano, D Ushizima, H Krishnan, D Parkinson, N Larson, DM Pelt, W Bethel, F Zok, J Sethian, "Insight into 3D micro-CT data: Exploring segmentation algorithms through performance metrics", Journal of Synchrotron Radiation, 2017, 24:1065--1077, doi: 10.1107/S1600577517010955

O Angélil, D Stone, M Wehner, CJ Paciorek, H Krishnan, W Collins, "An independent assessment of anthropogenic attribution statements for recent extreme temperature and rainfall events", Journal of Climate, 2017, 30:5--16, doi: 10.1175/JCLI-D-16-0077.1

Benedikt J Daurer, Hari Krishnan, Talita Perciano, Filipe RNC Maia, David A Shapiro, James A Sethian, Stefano Marchesini, "Nanosurveyor: a framework for real-time data processing", Advanced structural and chemical imaging, 2017, 3:7,

T Perciano, D Ushizima, H Krishnan, D Parkinson, J Sethian, "FibriPy: A software environment for fiber analysis from 3D micro-computed tomography data", Advanced Materials - TechConnect Briefs 2017, 2017, 1:25--28,

DY Parkinson, DM Pelt, T Perciano, D Ushizima, H Krishnan, HS Barnard, AA MacDowell, J Sethian, "Machine learning for micro-tomography", Proceedings of SPIE - The International Society for Optical Engineering, 2017, 10391, doi: 10.1117/12.2274731

B Timmermans, D Stone, M Wehner, H Krishnan, Impact of tropical cyclones on modeled extreme wind-wave climate, Geophysical Research Letters, Pages: 1393--1401 2017, doi: 10.1002/2016GL071681

BJ Daurer, H Krishnan, T Perciano, FRNC Maia, DA Shapiro, JA Sethian, S Marchesini, Nanosurveyor: a framework for real-time data processing., Advanced structural and chemical imaging, Pages: 7 2017, doi: 10.1186/s40679-017-0039-0

M Farmand, R Celestre, P Denes, ALD Kilcoyne, S Marchesini, H Padmore, T Tyliszczak, T Warwick, X Shi, J Lee, YS Yu, J Cabana, J Joseph, H Krishnan, T Perciano, FRNC Maia, DA Shapiro, "Near-edge X-ray refraction fine structure microscopy", Applied Physics Letters, 2017, 110, doi: 10.1063/1.4975377

M. Wehner, D. Stone, H. Krishnan, K. AchutaRao, F. Castillo, "The deadly combination of heat and humidity in India and Pakistan in summer 201", Bulletin of the American Meteorological Society, 2016, 97:S81-S86, doi: 10.1175/BAMS-D-16-0145.2

Harinarayan Krishnan, Burlen Loring, Suren Byna, Michael F. Wehner, Travis A. O'Brien, Prabhat, Chris Paciorek, and Daithi Stone, "Enabling End-to-End Climate Science Workflows in High Performance Computing Environments", The AMS (American Meteorological Society) 96th Annual Meeting, January 6, 2016,

Burlen Loring, Suren Byna, Prabhat, Junmin Gu, Hari Krishnan, Michael Wehner, and Oliver Ruebel, "TECA an Extreme Event Detection and Climate Analysis Package for High Performance Computing", The AMS (American Meteorological Society) 96th Annual Meeting, January 6, 2016,

Deborah A Agarwal, Boris Faybishenko, Vicky L Freedman, Harinarayan Krishnan, Gary Kushner, Carina Lansing, Ellen Porter, Alexandru Romosan, Arie Shoshani, Haruko Wainwright, others, "A science data gateway for environmental management", Concurrency and Computation: Practice and Experience, 2016, 28:1994--2004,

DM Ushizima, HA Bale, EW Bethel, P Ercius, BA Helms, H Krishnan, LT Grinberg, M Haranczyk, AA Macdowell, K Odziomek, DY Parkinson, T Perciano, RO Ritchie, C Yang, "IDEAL: Images Across Domains, Experiments, Algorithms and Learning", JOM, 2016, 68:2963--2972, doi: 10.1007/s11837-016-2098-4

SV Venkatakrishnan, KA Mohan, K Beattie, J Correa, E Dart, JR Deslippe, A Hexemer, H Krishnan, AA MacDowell, S Marchesini, SJ Patton, T Perciano, JA Sethian, R Stromsness, BL Tierney, CE Tull, D Ushizima, DY Parkinson, "Making advanced scientific algorithms and big scientific data management more accessible", IS and T International Symposium on Electronic Imaging Science and Technology, 2016, doi: 10.2352/ISSN.2470-1173.2016.19.COIMG-155

DY Parkinson, K Beattie, X Chen, J Correa, E Dart, BJ Daurer, JR Deslippe, A Hexemer, H Krishnan, AA Macdowell, FRNC Maia, S Marchesini, HA Padmore, SJ Patton, T Perciano, JA Sethian, D Shapiro, R Stromsness, N Tamura, BL Tierney, CE Tull, D Ushizima, "Real-time data-intensive computing", AIP Conference Proceedings, 2016, 1741, doi: 10.1063/1.4952921

S Marchesini, H Krishnan, BJ Daurer, DA Shapiro, T Perciano, JA Sethian, FRNC Maia, "SHARP: A distributed GPU-based ptychographic solver", Journal of Applied Crystallography, 2016, 49:1245--1252, doi: 10.1107/S1600576716008074

M Wehner, D Stone, H Krishnan, K Achutarao, F Castillo, 16. The deadly combination of heat and humidity in India and Pakistan in summer 2015, Bulletin of the American Meteorological Society, Pages: S81--S86 2016, doi: 10.1175/BAMS-D-16-0145.1

TA O Brien, WD Collins, K Kashinath, O Rübel, S Byna, J Gu, H Krishnan, PA Ullrich, Resolution dependence of precipitation statistical fidelity in hindcast simulations, Journal of Advances in Modeling Earth Systems, Pages: 976--990 2016, doi: 10.1002/2016MS000671

Stone, D., H. Shiogama, P. Wolski, O. Angélil, S. Cholias, N. Christidis, A. Dittus, C. Folland, A. King, J. Kinter, H. Krishnan, S.-K. Min, M. Wehner, "The C20C+ Detection and Attribution Project", Fall Meeting of the American Geophysical Union, 2015,

Download File: StoneDAGU2015.pdf (pdf: 4.7 MB)

Hari Krishnan, Suren Byna, Michael Wehner, Junmin Gu, Travis O'Brien, Burlen Loring, Daithi Stone, William Collins, Prabhat, Yunjie Liu, Jeffrey Johnson, and Christopher Paciorek, "Enabling Efficient Climate Science Workflows in High Performance Computing Environments", AGU Fall Meeting, 2015, December 13, 2015,

J Donatelli, M Haranczyk, A Hexemer, H Krishnan, X Li, L Lin, F Maia, S Marchesini, D Parkinson, T Perciano, D Shapiro, D Ushizima, C Yang, JA Sethian, "CAMERA: The Center for Advanced Mathematics for Energy Research Applications", Synchrotron Radiation News, 2015, 28:4--9, doi: 10.1080/08940886.2015.1013413

Y Li, S Meyer, J Lim, SC Lee, WE Gent, S Marchesini, H Krishnan, T Tyliszczak, D Shapiro, ALD Kilcoyne, WC Chueh, Effects of Particle Size, Electronic Connectivity, and Incoherent Nanoscale Domains on the Sequence of Lithiation in LiFePO<inf>4</inf> Porous Electrodes, Advanced Materials, Pages: 6591--6597 2015, doi: 10.1002/adma.201502276

Yiyang Li, Sophie Meyer, Jongwoo Lim, Sang Chul Lee, William E Gent, Stefano Marchesini, Harinarayan Krishnan, Tolek Tyliszczak, David Shapiro, Arthur L David Kilcoyne, others, Effects of Particle Size, Electronic Connectivity, and Incoherent Nanoscale Domains on the Sequence of Lithiation in LiFePO4 Porous Electrodes, Advanced Materials, Pages: 6591--6597 2015,

Yiyang Li, Sophie Meyer, Jongwoo Lim, Sang Chul Lee, William E Gent, Stefano Marchesini, Harinarayan Krishnan, Tolek Tyliszczak, David Shapiro, Arthur L David Kilcoyne, others, Electrode Lithiation: Effects of Particle Size, Electronic Connectivity, and Incoherent Nanoscale Domains on the Sequence of Lithiation in LiFePO4 Porous Electrodes (Adv. Mater. 42/2015), Advanced Materials, Pages: 6590--6590 2015,

Dáithí Stone, Michael Wehner, Shreyas Cholia, Harinarayan Krishnan, Piotr Wolski, Mark Tadross, Chris Folland, Nikos Christidis, Hideo Shiogama, "The C20C+ Detection and Attribution Project", Integrated Climate Modeling Principal Investigator Meeting 2014, 2014,

D Ushizima, T Perciano, H Krishnan, B Loring, H Bale, D Parkinson, J Sethian, "Structure recognition from high resolution images of ceramic composites", Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014, 2014, 683--691, doi: 10.1109/BigData.2014.7004292

David Camp, Hari Krishnan, David Pugmire, Christoph Garth, Ian Johnson, E. Wes Bethel, Kenneth I. Joy, and Hank Childs., "GPU Acceleration of Particle Advection Workloads in a Parallel, Distributed Memory Setting", Proceedings of Eurographics Symposium on Parallel Graphics and Visualization (EGPGV), May 5, 2013,

DN Williams, T Bremer, C Doutriaux, J Patchett, S Williams, G Shipman, R Miller, DR Pugmire, B Smith, C Steed, EW Bethel, H Childs, H Krishnan, P Prabhat, M Wehner, CT Silva, E Santos, D Koop, T Ellqvist, J Poco, B Geveci, A Chaudhary, A Bauer, A Pletzer, D Kindig, GL Potter, TP Maxwell, Ultrascale visualization of climate data, Computer, Pages: 68--76 2013, doi: 10.1109/MC.2013.119

Hank Childs, Eric Brugger, Brad Whitlock, Jeremy Meredith, Sean Ahern, David Pugmire, Kathleen Biagas, Mark Miller, Cyrus Harrison, Gunther H. Weber, Hari Krishnan, Thomas Fogal, Allen Sanderson, Christoph Garth, E. Wes Bethel, David Camp, Oliver Rubel, Marc Durant, Jean M. Favre, Paul Navratil, "VisIt: An End-User Tool For Visualizing and Analyzing Very Large Data", High Performance Visualization---Enabling Extreme-Scale Scientific Insight, ( October 2012) Pages: 357--372

Tamay M. Ozgokmen, Andrew C. Poje, Paul F. Fischer, Hank Childs, Harinarayan Krishnan, Christoph Garth, Angelique C. Haza, Edward Ryan, "On multi-scale dispersion under the influence of surface mixed layer instabilities", Ocean Modelling, October 2012, 56:16-30,

E. Wes Bethel, David Camp, Hank Childs, Mark Howison, Hari Krishnan, Burlen Loring, Joerg Meyer, Prabhat, Oliver Ruebel, Daniela Ushizima, Gunther Weber, "Towards Exascale: High Performance Visualization and Analytics – Project Status Report. Technical Report", DOE Exascale Research Conference, April 2012,

J Meyer, EW Bethel, JL Horsman, SS Hubbard, H Krishnan, A Romosan, EH Keating, L Monroe, R Strelitz, P Moore, G Taylor, B Torkian, TC Johnson, I Gorton, Visual data analysis as an integral part of environmental management, IEEE Transactions on Visualization and Computer Graphics, Pages: 2088--2094 2012, doi: 10.1109/TVCG.2012.278

H Krishnan, J Meyer, A Romosan, H Childs, EW Bethel, Enabling advanced environmental management via remote and distributed visual data exploration and analysis, Computing and Visualization in Science, Pages: 123--133 2012, doi: 10.1007/s00791-013-0204-5

TM Özgökmen, AC Poje, PF Fischer, H Childs, H Krishnan, C Garth, AC Haza, E Ryan, On multi-scale dispersion under the influence of surface mixed layer instabilities and deep flows, Ocean Modelling, Pages: 16--30 2012, doi: 10.1016/j.ocemod.2012.07.004

H Krishnan, C Garth, J Gühring, MA Gülsün, A Greiser, KI Joy, Analysis of time-dependent flow-sensitive PC-MRI data, IEEE Transactions on Visualization and Computer Graphics, Pages: 966--977 2012, doi: 10.1109/TVCG.2011.80

H Krishnan, C Garth, KI Joy, Time and streak surfaces for flow visualization in large time-varying data sets, IEEE Transactions on Visualization and Computer Graphics, Pages: 1267--1274 2009, doi: 10.1109/TVCG.2009.190

C Garth, H Krishnan, X Tricoche, T Bobach, KI Joy, Generation of accurate integral surfaces in time-dependent vector fields, IEEE Transactions on Visualization and Computer Graphics, Pages: 1404--1411 2008, doi: 10.1109/TVCG.2008.133

AR Fuller, H Krishnan, K Mahrous, B Hamann, KI Joy, Real-time procedural volumetric fire, Proceedings - I3D 2007, ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Pages: 175--180 2007, doi: 10.1145/1230100.1230131

E. Wes Bethel, Burlen Loring, Utkarsh Ayachit, P. N. Duque, Nicola Ferrier, Joseph Insley, Junmin Gu, Kress, Patrick O’Leary, Dave Pugmire, Silvio Rizzi, Thompson, Will Usher, Gunther H. Weber, Brad Whitlock, Wolf, Kesheng Wu, "Proximity Portability and In Transit, M-to-N Data Partitioning and Movement in SENSEI", In Situ Visualization for Computational Science, ( 2022) doi: 10.1007/978-3-030-81627-8_20

E. Wes Bethel, Burlen Loring, Utkarsh Ayatchit, David Camp, P. N. Duque, Nicola Ferrier, Joseph Insley, Junmin Gu, Kress, Patrick O’Leary, David Pugmire, Silvio Rizzi, Thompson, Gunther H. Weber, Brad Whitlock, Matthew Wolf, Kesheng Wu, "The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale", In Situ Visualization for Computational Science, ( 2022) doi: 10.1007/978-3-030-81627-8_13

Junmin Gu, Burlen Loring, Kesheng Wu, E Wes Bethel, "HDF5 as a vehicle for in transit data movement", Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, 2019, 39--43,

B Loring, A Myers, D Camp, EW Bethel, "Python-based in situ analysis and visualization", Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV 18, ACM Press, 2018, doi: 10.1145/3281464.3281465

MF Wehner, KA Reed, B Loring, D Stone, H Krishnan, "Changes in tropical cyclones under stabilized 1.5 and 2.0°C global warming scenarios as simulated by the Community Atmospheric Model under the HAPPI protocols", Earth System Dynamics, 2018, 9:187--195, doi: 10.5194/esd-9-187-2018

U Ayachit, B Whitlock, M Wolf, B Loring, B Geveci, D Lonie, EW Bethel, "The SENSEI generic in situ interface", Proceedings of ISAV 2016: 2nd Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2017, 40--44, doi: 10.1109/ISAV.2016.13

Harinarayan Krishnan, Burlen Loring, Suren Byna, Michael F. Wehner, Travis A. O'Brien, Prabhat, Chris Paciorek, and Daithi Stone, "Enabling End-to-End Climate Science Workflows in High Performance Computing Environments", The AMS (American Meteorological Society) 96th Annual Meeting, January 6, 2016,

Burlen Loring, Suren Byna, Prabhat, Junmin Gu, Hari Krishnan, Michael Wehner, and Oliver Ruebel, "TECA an Extreme Event Detection and Climate Analysis Package for High Performance Computing", The AMS (American Meteorological Society) 96th Annual Meeting, January 6, 2016,

Utkarsh Ayachit, Andrew Bauer, Earl PN Duque, Greg Eisenhauer, Nicola Ferrier, Junmin Gu, Kenneth E Jansen, Burlen Loring, Zarija Lukic, Suresh Menon, others, "Performance analysis, design considerations, and applications of extreme-scale in situ infrastructures", SC 16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2016, 921--932, LBNL 1007264,

O Rübel, B Loring, JL Vay, DP Grote, R Lehe, S Bulanov, H Vincenti, EW Bethel, "WarpIV: In Situ Visualization and Analysis of Ion Accelerator Simulations", IEEE Computer Graphics and Applications, 2016, 36:22--35, doi: 10.1109/MCG.2016.62

M Alegro, E Amaro, B Loring, H Heinsen, E Alho, L Zollei, D Ushizima, LT Grinberg, "Multimodal Whole Brain Registration: MRI and High Resolution Histology", IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2016, 634--642, doi: 10.1109/CVPRW.2016.85

Hari Krishnan, Suren Byna, Michael Wehner, Junmin Gu, Travis O'Brien, Burlen Loring, Daithi Stone, William Collins, Prabhat, Yunjie Liu, Jeffrey Johnson, and Christopher Paciorek, "Enabling Efficient Climate Science Workflows in High Performance Computing Environments", AGU Fall Meeting, 2015, December 13, 2015,

B Loring, H Karimabadi, V Rortershteyn, "A Screen Space GPGPU Surface LIC Algorithm for Distributed Memory Data Parallel Sort Last Rendering Infrastructures", NUMERICAL MODELING OF SPACE PLASMA FLOWS: ASTRONUM-2014, 2015, 498:231--239,

D Ushizima, T Perciano, H Krishnan, B Loring, H Bale, D Parkinson, J Sethian, "Structure recognition from high resolution images of ceramic composites", Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014, 2014, 683--691, doi: 10.1109/BigData.2014.7004292

H Karimabadi, V Roytershteyn, HX Vu, YA Omelchenko, J Scudder, W Daughton, A Dimmock, K Nykyri, M Wan, D Sibeck, M Tatineni, A Majumdar, B Loring, B Geveci, "The link between shocks, turbulence, and magnetic reconnection in collisionless plasmas", Physics of Plasmas, 2014, 21, doi: 10.1063/1.4882875

W Daughton, TKM Nakamura, H Karimabadi, V Roytershteyn, B Loring, "Computing the reconnection rate in turbulent kinetic layers by using electron mixing to identify topology", Physics of Plasmas, 2014, 21, doi: 10.1063/1.4875730

H Karimabadi, B Loring, P O leary, A Majumdar, M Tatineni, B Geveci, "In-situ visualization for global hybrid simulations", ACM International Conference Proceeding Series, 2013, doi: 10.1145/2484762.2484822

H Karimabadi, V Roytershteyn, M Wan, WH Matthaeus, W Daughton, P Wu, M Shay, B Loring, J Borovsky, E Leonardis, SC Chapman, TKM Nakamura, "Coherent structures, intermittent turbulence, and dissipation in high-temperature plasmas", Physics of Plasmas, 2013, 20, doi: 10.1063/1.4773205

Mehmet Balman, Eric Pouyoul, Yushu Yao, E. Wes Bethel, Burlen Loring, Prabhat, John Shalf, Alex Sim, and Brian L. Tierney, "Experiences with 100G Network Applications", In Proceedings of the Fifth international Workshop on Data-intensive Distributed Computing, in conjunction with ACM High Performance Distributing Computing (HPDC) Conference, 2012, Delft, Netherlands, June 2012, LBNL 5603E, doi: 10.1145/2286996.2287004

Download File: didc-report.pdf (pdf: 11 MB)

100Gbps networking has finally arrived, and many research and educational in- stitutions have begun to deploy 100Gbps routers and services. ESnet and Internet2 worked together to make 100Gbps networks available to researchers at the Super- computing 2011 conference in Seattle Washington. In this paper, we describe two of the first applications to take advantage of this network. We demonstrate a visu- alization application that enables remotely located scientists to gain insights from large datasets. We also demonstrate climate data movement and analysis over the 100Gbps network. We describe a number of application design issues and host tuning strategies necessary for enabling applications to scale to 100Gbps rates.

E. Wes Bethel, David Camp, Hank Childs, Mark Howison, Hari Krishnan, Burlen Loring, Joerg Meyer, Prabhat, Oliver Ruebel, Daniela Ushizima, Gunther Weber, "Towards Exascale: High Performance Visualization and Analytics – Project Status Report. Technical Report", DOE Exascale Research Conference, April 2012,

M Balman, E Pouyoul, Y Yao, EW Bethel, B Loring, Prabhat, J Shalf, A Sim, BL Tierney, "Experiences with 100Gbps network applications", DIDC 12 - 5th International Workshop on Data-Intensive Distributed Computing, 2012, 33--42, doi: 10.1145/2286996.2287004

M Wan, WH Matthaeus, H Karimabadi, V Roytershteyn, M Shay, P Wu, W Daughton, B Loring, SC Chapman, "Intermittent dissipation at kinetic scales in collisionless plasma turbulence", Physical Review Letters, 2012, 109, doi: 10.1103/PhysRevLett.109.195001

H Karimabadi, B Loring, HX Vu, Y Omelchenko, M Tatineni, A Majumdar, U Ayachit, B Geveci, "Petascale Global Kinetic Simulations of The Magnetosphere and Visualization Strategies for Analysis of Very Large Multi-Variate Data Sets", NUMERICAL MODELING OF SPACE PLASMA FLOWS - ASTRONUM 2010, 2011, 444:281--+,

H Karimabadi, J Dorelli, HX Vu, B Loring, Y Omelchenko, Is Quadrupole Structure of Out-of-Plane Magnetic Field Evidence for Hall Reconnection?, MODERN CHALLENGES IN NONLINEAR PLASMA PHYSICS: A FESTSCHRIFT HONORING THE CAREER OF DENNIS PAPADOPOULOS, Pages: 137--+ 2010,

K Beketayev, D Yeliussizov, D Morozov, GH Weber, B Hamann, "Measuring the Error in Approximating the Sub-Level Set Topology of Sampled Scalar Data", International Journal of Computational Geometry and Applications, 2018, 28:57--77, doi: 10.1142/S0218195918500036

Erich Lohrmann, Zarija Lukic, Dmitriy Morozov, Juliane Mueller, "Programmable In Situ System for Iterative Workflows", Lecture Notes in Computer Science, Cham, Springer International Publishing, January 1, 2018, 10773:122--131, doi: 10.1007/978-3-319-77398-8\_7

P Koanantakool, A Ali, A Azad, A Buluç, D Morozov, L Oliker, KA Yelick, S-Y Oh, "Communication-Avoiding Optimization Methods for Distributed Massive-Scale Sparse Inverse Covariance Estimation.", Proceedings of Machine Learning Research, PMLR, 2018, 84:1376--1386,

Andrey Babichev, Dmitriy Morozov, Yuri Dabaghian, "Robust spatial memory maps encoded by networks with transient", PLoS computational biology, 2018, 14:e1006433, doi: https://doi.org/10.1371/journal.pcbi.1006433

P Oesterling, C Heine, GH Weber, D Morozov, G Scheuermann, "Computing and visualizing time-varying merge trees for high-dimensional data", Mathematics and Visualization, ( 2017) Pages: 87--101 doi: 10.1007/978-3-319-44684-4_5

Michael Kerber, Dmitriy Morozov, Arnur Nigmetov, "Geometry Helps to Compare Persistence Diagrams", Journal of Experimental Algorithmics, 2017, 22:1.4:1--1.4, doi: 10.1145/3064175

Dmitriy Smirnov, Dmitriy Morozov, Triplet Merge Trees, 2017,

Herbert Edelsbrunner, Dmitriy Morozov, "Persistent Homology", Handbook of Discrete and Computational Geometry, 3rd ed., (CRC Press: 2017)

Brian Friesen, Ann Almgren, Zarija Lukić, Gunther Weber, Dmitriy Morozov, Vincent Beckner, Marcus Day, "In situ and in-transit analysis of cosmological simulations", Computational Astrophysics and Cosmology, 2016, 3 (4):1-18,

Utkarsh Ayachit, Andrew Bauer, Earl PN Duque, Greg Eisenhauer, Nicola Ferrier, Junmin Gu, Kenneth E Jansen, Burlen Loring, Zarija Lukic, Suresh Menon, others, "Performance analysis, design considerations, and applications of extreme-scale in situ infrastructures", SC 16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2016, 921--932, LBNL 1007264,

P Koanantakool, A Azad, A Buluc, D Morozov, SY Oh, L Oliker, K Yelick, "Communication-Avoiding Parallel Sparse-Dense Matrix-Matrix Multiplication", Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016, January 2016, 842--853, doi: 10.1109/IPDPS.2016.117

Dmitriy Morozov, Tom Peterka, "Block-parallel data analysis with DIY2", Large Data Analysis and Visualization (LDAV), 2016 IEE Symposium on, 2016, 29--36,

D Morozov, Z Lukić, "Master of puppets: Cooperative multitasking for in situ processing", HPDC 2016 - Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing, 2016, 285--288, doi: 10.1145/2907294.2907301

Dmitriy Morozov, Tom Peterka, "Efficient Delaunay tessellation through K-D tree decomposition", SC 16, Piscataway, NJ, USA, IEEE Press, 2016, 62,

Gunnar Carlsson, Vin de Silva, Sara Verovsek, Dmitriy Morozov, Parametrized Homology via Zigzag Persistence, 2016,

R Lewis, D Morozov, "Parallel computation of persistent homology using the blowup complex", Annual ACM Symposium on Parallelism in Algorithms and Architectures, 2015, 2015-Jun:323--331, doi: 10.1145/2755573.2755587

K Beketayev, D Yeliussizov, D Morozov, GH Weber, B Hamann, "Measuring the distance between merge trees", Mathematics and Visualization, ( 2014) Pages: 151--165 doi: 10.1007/978-3-319-04099-8_10

Dmitriy Morozov, Gunther H Weber, "Distributed Contour Trees", Topological Methods in Data Analysis and Visualization III, (Springer International Publishing: 2014) Pages: 89--102 doi: 10.1007/978-3-319-04099-8\_6

T Peterka, D Morozov, C Phillips, "High-Performance Computation of Distributed-Memory Parallel 3D Voronoi and Delaunay Tessellation", International Conference for High Performance Computing, Networking, Storage and Analysis, SC, 2014, 2015-Jan:997--1007, doi: 10.1109/SC.2014.86

Dmitriy Morozov, Gunther Weber, "Distributed Merge Trees", PPoPP 13, New York, NY, USA, ACM, 2013, 93--102, doi: 10.1145/2442516.2442526

Leonidas Guibas, Dmitriy Morozov, Quentin M\ erigot, "Witnessed k-Distance", Discrete \& computational geometry, 2013, 49:22--45, doi: 10.1007/s00454-012-9465-x

Herbert Edelsbrunner, Dmitriy Morozov, "Persistent Homology: Theory and Practice", European Congress of Mathematics Krakow, 2 -- 7 July, 2012, Zuerich, Switzerland, European Mathematical Society Publishing House, 2013, 31--50, doi: 10.4171/120-1/3

Paul Bendich, Herbert Edelsbrunner, Dmitriy Morozov, Amit, "Homology and robustness of level and interlevel sets", Homology, Homotopy and Applications, 2013, 15:51--72,

D Ushizima, D Morozov, GH Weber, AGC Bianchi, JA Sethian, EW Bethel, "Augmented topological descriptors of pore networks for material science", IEEE Transactions on Visualization and Computer Graphics, 2012, 18:2041--2050, LBNL 5964E, doi: 10.1109/TVCG.2012.200

Download File: LBNL-5964E.pdf (pdf: 5.4 MB)

K Beketayev, GH Weber, D Morozov, A Abzhanov, B Hamann, "Geometry-preserving topological landscapes", Proceedings - WASA 2012: Workshop at SIGGRAPH Asia 2012, 2012, 155--160, doi: 10.1145/2425296.2425324

M Aanjaneya, F Chazal, D Chen, M Glisse, L Guibas, D Morozov, "Metric graph reconstruction from noisy data", International Journal of Computational Geometry and Applications, 2012, 22:305--325, doi: 10.1142/S0218195912600072

You-Wei Cheah, Danielle Svehla Christianson, Housen Chu, Gilberto Pastorello, Fianna O’Brien, Yeongshnn Ong, Catharine van Ingen, Margaret Torn, Deb Agarwal, AmeriFlux BADM: Implementing lessons from 12 years of long-tail data management into next generation earth science systems (IN34A-03), 2018 AGU Fall Meeting, Washington, D.C., December 12, 2018,

Gilberto Z. Pastorello, Dan K. Gunter, Housen Chu, Danielle S. Christianson, Carlo Trotta, Eleonora Canfora, Boris Faybishenko, You-Wei Cheah, Norm Beekwilder, Stephen W. Chan, Sigrid Dengel, Trevor Keenan, Fianna O Brien, Abderahman Elbashandy, Cristina M. Poindexter, Marty Humphrey, Dario Papale, Deb A. Agarwal, "Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures", Proceedings of the 13th IEEE International Conference on e-Science (e-Science 2017), Auckland, New Zealand, 2017, doi: 10.1109/eScience.2017.64

P Koanantakool, A Ali, A Azad, A Buluç, D Morozov, L Oliker, KA Yelick, S-Y Oh, "Communication-Avoiding Optimization Methods for Distributed Massive-Scale Sparse Inverse Covariance Estimation.", Proceedings of Machine Learning Research, PMLR, 2018, 84:1376--1386,

P Koanantakool, A Azad, A Buluc, D Morozov, SY Oh, L Oliker, K Yelick, "Communication-Avoiding Parallel Sparse-Dense Matrix-Matrix Multiplication", Proceedings - 2016 IEEE 30th International Parallel and Distributed Processing Symposium, IPDPS 2016, January 2016, 842--853, doi: 10.1109/IPDPS.2016.117

Sang-Yun Oh, Bala Rajaratnam, Joong-Ho Won, "On the Solution Path of Regularized Covariance Estimators", (submitted), 2015,

Sang-Yun Oh, Onkar Dalal, Kshitij Khare, Bala Rajaratnam, "Optimization Methods for Sparse Pseudo-Likelihood Graphical Model Selection", Neural Information Processing Systems, 2014,

Kshitij Khare, Sang-Yun Oh, Bala Rajaratnam, "A convex pseudo-likelihood framework for high dimensional partial correlation estimation with convergence guarantees", Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2014,

Schizophrenia Working Group of the Psychiatric Genomics Consortium, "Biological insights from 108 schizophrenia-associated genetic loci", July 24, 2014, 511:421-427, doi: doi:10.1038/nature13595

Douglas F Levinson, Jianxin Shi, Kai Wang, Sang Oh, Brien Riley, Ann E Pulver, Dieter B Wildenauer, Claudine Laurent, Bryan J Mowry, Pablo V Gejman, Michael J Owen, Kenneth S Kendler, Gerald Nestadt, Sibylle G Schwab, Jacques Mallet, Deborah Nertney, Alan R Sanders, Nigel M Williams, Brandon Wormley, Virginia K Lasseter, Margot Albus, Stephanie Godard-Bauché, Madeline Alexander, Jubao Duan, Michael C O’Donovan, Dermot Walsh, Anthony O’Neill, George N Papadimitriou, Dimitris Dikeos, Wolfgang Maier, Bernard Lerer, Dominique Campion, David Cohen, Maurice Jay, Ayman Fanous, Peter Eichhammer, Jeremy M Silverman, Nadine Norton, Nancy Zhang, Hakon Hakonarson, Cynthia Gao, Ami Citri, Mark Hansen, Stephan Ripke, Frank Dudbridge, Peter A Holmans, "Genome-wide association study of multiplex schizophrenia pedigrees", The American Journal of Psychiatry, September 1, 2012, doi: doi:10.1176/appi.ajp.2012.11091423

Douglas F Levinson, Jubao Duan, Sang Oh, Kai Wang, Alan R Sanders, Jianxin Shi, Nancy Zhang, Bryan J Mowry, Ann Olincy, Farooq Amin, C Robert Cloninger, Jeremy M Silverman, Nancy G Buccola, William F Byerley, Donald W Black, Kenneth S Kendler, Robert Freedman, Frank Dudbridge, Itsik Pe'er, Hakon Hakonarson, Sarah E Bergen, Ayman H Fanous, Peter A Holmans, Pablo V Gejman, "Copy number variants in schizophrenia: confirmation of five previous findings and new evidence for 3q29 microdeletions and VIPR2 duplications", The American Journal of Psychiatry, March 1, 2011, doi: doi:10.1176/appi.ajp.2010.10060876

Bryan W. Reutter, Sang Oh, Grant T. Gullberg, Ronald H. Huesman, "Improved quantitation of dynamic SPECT via fully 4-D joint estimation of compartmental models and blood input function directly from projections", Nuclear Science Symposium Conference Record, 2005 IEEE, October 23, 2005, doi: 10.1109/NSSMIC.2005.1596802

R. Stompor, A. Balbi, J.D. Borrill, P.G. Ferreira, S. Hanany, A.H. Jaffe, A.T. Lee, S. Oh, B. Rabii, P.L. Richards, G.F. Smoot, C.D. Winant, J.-H.P. Wu, "Making maps of the cosmic microwave background: The MAXIMA example", Physical Review D, January 2002, 65:022003, doi: 10.1103/PhysRevD.65.022003

A.H. Jaffe, P.A.R. Ade, A. Balbi, J.J. Bock, J.R. Bond, J. Borrill, A. Boscaleri, K. Coble, B.P. Crill, P. De Bernardis, P. Farese, P.G. Ferreira, K. Ganga, M. Giacometti, S. Hanany, E. Hivon, V.V. Hristov, A. Iacoangeli, A.E. Lange, A.T. Lee, L. Martinis, S. Masi, P.D. Mauskopf, A. Melchiorri, T. Montroy, C.B. Netterfield, S. Oh, E. Pascale, F. Piacentini, D. Pogosyan, S. Prunet, B. Rabii, S. Rao, P.L. Richards, G. Romeo, J.E. Ruhl, F. Scaramuzzi, D. Sforna, G.F. Smoot, R. Stompor, C.D. Winant, J.H.P. Wu, "Cosmology from MAXIMA-1, BOOMERANG, and COBE DMR cosmic microwave background observations", Physical Review Letters, January 2001, 86:3475-3479, doi: 10.1103/PhysRevLett.86.3475

A. Balbi, P. Ade, J. Bock, J. Borrill, A., P. De Bernardis, P. G. Ferreira, S., V. Hristov, A. H. Jaffe, A. T. Lee, S., E. Pascale, B. Rabii, P. L. Richards, G. F., R. Stompor, C. D. Winant, J. H. P. Wu, "Erratum: Constraints on Cosmological Parameters from MAXIMA-1", Astrophysical Journal Letters, January 2001, 558:L145-L145, doi: 10.1086/323608

J.H.P. Wu, A. Balbi, J. Borrill, P.G. Ferreira, S. Hanany, A.H. Jaffe, A.T. Lee, S. Oh, B. Rabii, P.L. Richards, G.F. Smoot, R. Stompor, C.D. Winant, "Asymmetric beams in cosmic microwave background anisotropy experiments", Astrophysical Journal, Supplement Series, January 2001, 132:1-17, doi: 10.1086/318947

S. Hanany, P. Ade, A. Balbi, J. Bock, J., A. Boscaleri, P. de Bernardis, P. G., V. V. Hristov, A. H. Jaffe, A. E., A. T. Lee, P. D. Mauskopf, C. B. Netterfield, S., E. Pascale, B. Rabii, P. L. Richards, G. F., R. Stompor, C. D. Winant, J. H. P. Wu, "MAXIMA-1: A Measurement of the Cosmic Microwave Background Anisotropy on Angular Scales of 10 -5\deg", Astrophysical Journal Letters, December 2000, 545:L5-L9, doi: 10.1086/317322

A. Balbi, P. Ade, J. Bock, J. Borrill, A. Boscaleri, P. De Bernardis, P.G. Ferreira, S. Hanany, V. Hristov, A.H. Jaffe, A.T. Lee, S. Oh, E. Pascale, B. Rabii, P.L. Richards, G.F. Smoot, R. Stompor, C.D. Winant, J.H.P. Wu, "Constraints on cosmological parameters from MAXIMA-1", Astrophysical Journal, January 2000, 545:L1-L4, doi: 10.1086/317323

Bala Rajaratnam, Sang-Yun Oh, Michael T. Tsiang, Richard A. Olshen, "Successive Standardization: Application to Case-Control Studies", Topics in Applied Statistics, ( December 31, 1969) doi: 10.1007/978-1-4614-7846-1_19

Devarshi Ghoshal, Ludovico Bianchi, Abdelilah Essiari, Drew Paine, Sarah Poon, Michael Beach, Alpha N'Diaye, Patrick Huck, Lavanya Ramakrishnan, "Science Capsule: Towards Sharing and Reproducibility of Scientiﬁc Workﬂows", 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS), November 15, 2021, doi: 10.1109/WORKS54523.2021.00014

Workflows are increasingly processing large volumes of data from scientific instruments, experiments and sensors. These workflows often consist of complex data processing and analysis steps that might include a diverse ecosystem of tools and also often involve human-in-the-loop steps. Sharing and reproducing these workflows with collaborators and the larger community is critical but hard to do without the entire context of the workflow including user notes and execution environment. In this paper, we describe Science Capsule, which is a framework to capture, share, and reproduce scientific workflows. Science Capsule captures, manages and represents both computational and human elements of a workflow. It automatically captures and processes events associated with the execution and data life cycle of workflows, and lets users add other types and forms of scientific artifacts. Science Capsule also allows users to create `workflow snapshots' that keep track of the different versions of a workflow and their lineage, allowing scientists to incrementally share and extend workflows between users. Our results show that Science Capsule is capable of processing and organizing events in near real-time for high-throughput experimental and data analysis workflows without incurring any significant performance overheads.

Drew Paine, Sarah Poon, Lavanya Ramakrishnan, "Investigating User Experiences with Data Abstractions on High Performance Computing Systems", June 29, 2021, LBNL LBNL-2001374,

Scientific exploration generates expanding volumes of data that commonly require High Performance Computing (HPC) systems to facilitate research. HPC systems are complex ecosystems of hardware and software that frequently are not user friendly. The Usable Data Abstractions (UDA) project set out to build usable software for scientific workflows in HPC environments by undertaking multiple rounds of qualitative user research. Qualitative research investigates how individuals accomplish their work and our interview-based study surfaced a variety of insights about the experiences of working in and with HPC ecosystems. This report examines multiple facets to the experiences of scientists and developers using and supporting HPC systems. We discuss how stakeholders grasp the design and configuration of these systems, the impacts of abstraction layers on their ability to successfully do work, and the varied perceptions of time that shape this work. Examining the adoption of the Cori HPC at NERSC we explore the anticipations and lived experiences of users interacting with this system's novel storage feature, the Burst Buffer. We present lessons learned from across these insights to illustrate just some of the challenges HPC facilities and their stakeholders need to account for when procuring and supporting these essential scientific resources to ensure their usability and utility to a variety of scientific practices.

Devarshi Ghoshal, Drew Paine, Gilberto Pastorello, Abdelrahman Elbashandy, Dan Gunter, Oluwamayowa Amusat, Lavanya Ramakrishnan, "Experiences with Reproducibility: Case Studies from Scientific Workflows", (P-RECS'21) Proceedings of the 4th International Workshop on Practical Reproducible Evaluation of Computer Systems, ACM, June 21, 2021, doi: 10.1145/3456287.3465478

Reproducible research is becoming essential for science to ensure transparency and for building trust. Additionally, reproducibility provides the cornerstone for sharing of methodology that can improve efficiency. Although several tools and studies focus on computational reproducibility, we need a better understanding about the gaps, issues, and challenges for enabling reproducibility of scientific results beyond the computational stages of a scientific pipeline. In this paper, we present five different case studies that highlight the reproducibility needs and challenges under various system and environmental conditions. Through the case studies, we present our experiences in reproducing different types of data and methods that exist in an experimental or analysis pipeline. We examine the human aspects of reproducibility while highlighting the things that worked, that did not work, and that could have worked better for each of the cases. Our experiences capture a wide range of scenarios and are applicable to a much broader audience who aim to integrate reproducibility in their everyday pipelines.

Devarshi Ghoshal, Ludovico Bianchi, Abdelilah Essiari, Michael Beach, Drew Paine, Lavanya Ramakrishnan, "Science Capsule - Capturing the Data Life Cycle", Journal of Open Source Software, 2021, 6:2484, doi: 10.21105/joss.02484

Marco Pritoni, Drew Paine, Gabriel Fierro, Cory Mosiman, Michael Poplawski, Joel Bender, Jessica Granderson, "Metadata Schemas and Ontologies for Building Energy Applications: A Critical Review and Use Case Analysis", Energies, April 6, 2021, doi: 10.3390/en14072024

Digital and intelligent buildings are critical to realizing efficient building energy operations and a smart grid. With the increasing digitalization of processes throughout the life cycle of buildings, data exchanged between stakeholders and between building systems have grown significantly. However, a lack of semantic interoperability between data in different systems is still prevalent and hinders the development of energy-oriented applications that can be reused across buildings, limiting the scalability of innovative solutions. Addressing this challenge, our review paper systematically reviews metadata schemas and ontologies that are at the foundation of semantic interoperability necessary to move toward improved building energy operations. The review finds 40 schemas that span different phases of the building life cycle, most of which cover commercial building operations and, in particular, control and monitoring systems. The paper’s deeper review and analysis of five popular schemas identify several gaps in their ability to fully facilitate the work of a building modeler attempting to support three use cases: energy audits, automated fault detection and diagnosis, and optimal control. Our findings demonstrate that building modelers focused on energy use cases will find it difficult, labor intensive, and costly to create, sustain, and use semantic models with existing ontologies. This underscores the significant work still to be done to enable interoperable, usable, and maintainable building models. We make three recommendations for future work by the building modeling and energy communities: a centralized repository with a search engine for relevant schemas, the development of more use cases, and better harmonization and standardization of schemas in collaboration with industry to facilitate their adoption by stakeholders addressing varied energy-focused use cases.

Drew Paine, Lavanya Ramakrishnan, "Understanding Interactive and Reproducible Computing With Jupyter Tools at Facilities", LBNL Technical Report, October 31, 2020, LBNL LBNL-2001355,

Increasingly Jupyter tools are being adopted and incorporated into High Performance Computing (HPC) and scientific user facilities. Adopting Jupyter tools enables more interactive and reproducible computational work at facilities across data life cycles. As the volume, variety, and scope of data grow, scientists need to be able to analyze and share results in user friendly ways. Human-centered research highlights design challenges around computational notebooks, and our qualitative user study shifts focus to better characterize how Jupyter tools are being used in HPC and science user facilities today. We conducted twenty-nine interviews, and obtained 103 survey responses from NERSC Jupyter users, to better understand the increasing role of interactive computing tools in DOE sponsored scientific work. We examine a range of issues that emerge using and supporting Jupyter in HPC ecosystems, including: how Jupyter is being used by scientists in HPC and user facility ecosystems; how facilities are purposefully supporting Jupyter in their ecosystems; feedback NERSC users have about the facility’s deployment, and, discuss features NERSC indicated would be helpful. We offer a variety of takeaways for staff supporting Jupyter at facilities, Project Jupyter and related open source communities, and funding agencies supporting interactive computing work.

Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, "Experiences with a Flexible User Research Process to Build Data Change Tools", Journal of Open Research Software, September 1, 2020, doi: 10.5334/jors.284

Scientific software development processes are understood to be distinct from commercial software development practices due to uncertain and evolving states of scientific knowledge. Sustaining these software products is a recognized challenge, but under-examined is the usability and usefulness of such tools to their scientific end users. User research is a well-established set of techniques (e.g., interviews, mockups, usability tests) applied in commercial software projects to develop foundational, generative, and evaluative insights about products and the people who use them. Currently these approaches are not commonly applied and discussed in scientific software development work. The use of user research techniques in scientific environments can be challenging due to the nascent, fluid problem spaces of scientific work, varying scope of projects and their user communities, and funding/economic constraints on projects.

In this paper, we reflect on our experiences undertaking a multi-method user research process in the Deduce project. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists make decisions around data change. There is a lack of common terminology since the concept of systematically measuring and managing data change is under explored in scientific environments. To bridge this gap we conducted user research that focuses on user practices, needs, and motivations to help us design and develop metrics and tools for data change. This paper contributes reflections and the lessons we have learned from our experiences. We offer key takeaways for scientific software project teams to effectively and flexibly incorporate similar processes into their projects.

Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, "Investigating Scientific Data Change with User Research Methods", August 20, 2020, LBNL LBNL-2001347,

Scientific datasets are continually expanding and changing due to fluctuations with instruments, quality assessment and quality control processes, and modifications to software pipelines. Datasets include minimal information about these changes or their effects requiring scientists manually assess modifications through a number of labor intensive and ad-hoc steps. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists systematically identify and make decisions around data changes. Currently, there is a lack of understanding, and common practices, for identifying and evaluating changes in datasets since systematically measuring and managing data change is under explored in scientific work. We are conducting user research to address this need by exploring scientist's conceptualizations, behaviors, needs, and motivations when dealing with changing datasets. Our user research utilizes multiple methods to produce foundational, generative insights and evaluate research products produced by our team. In this paper, we detail our user research process and outline our findings about data change that emerge from our studies. Our work illustrates how scientific software teams can push beyond just usability testing user interfaces or tools to better probe the underlying ideas they are developing solutions to address.

Christine T. Wolf, Drew Paine, "Sensemaking Practices in the Everyday Work of AI/ML Software Engineering", IEEE/ACM 42nd International Conference on Software Engineering Workshops (ICSEW’20), ACM, April 5, 2020, doi: 10.1145/3387940.3391496

Drew Paine, Charlotte P. Lee, "Coordinative Entities: Forms of Organizing in Data Intensive Science", Journal of Computer Supported Cooperative Work, February 11, 2020, doi: 10.1007/s10606-020-09372-2

Christine T. Wolf, Julia Bullard, Stacy Wood, Amelia Acker, Drew Paine, Charlotte P. Lee, "Mapping the “How” of Collaborative Action: Research Methods for Studying Contemporary Sociotechnical Processes", CSCW '19: Conference Companion Publication of the 2019 on Computer Supported Cooperative Work and Social Computing, November 10, 2019, doi: 10.1145/3311957.3359441

Process has been a concern since the beginning of CSCW. Developments in sociotechnical landscapes raise new challenges for studying processes (e.g., massive online communities bringing together vast crowds; Big Data technologies connecting many through the flow of data). This re-opens questions about how we study, document, conceptualize, and design to support processes in complex, contemporary sociotechnical systems. This one-day workshop will bring together researchers to discuss the CSCW community’s unique focus and methodological toolkit for studying process and workflow; provide a collaborative space for the improvement and extension of research projects within this space; and catalyze a network of scholars with expertise and interest in addressing challenging methodological questions around studying process in contemporary, sociotechnical systems.

Drew Paine, Lavanya Ramakrishnan, "Surfacing Data Change in Scientiﬁc Work", iConference 2019, Springer Verlag, March 19, 2019, 15-26, doi: 10.1007/978-3-030-15742-5_2

Cheah You-Wei, Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, Bringing Data Science to Qualitative Analysis, 2018 IEEE 14th International Conference on e-Science, Pages: 325-326 2018, doi: 10.1109/eScience.2018.00076

David P. Randall, Drew Paine, Charlotte P. Lee, "Educational Outreach & Stakeholder Role Evolution in a Cyberinfrastructure Project", 2018 IEEE 14th International Conference on e-Science, IEEE Computer Society, 2018, 201-211, doi: 10.1109/eScience.2018.00035

B Faybishenko, R Versteeg, G Pastorello, D Dwivedi, C Varadharajan, D Agarwal, Challenging problems of quality assurance and quality control (QA/QC) of meteorological time series data, Stochastic Environmental Research and Risk Assessment, Pages: 1049--1062 2022, doi: 10.1007/s00477-021-02106-w

Devarshi Ghoshal, Drew Paine, Gilberto Pastorello, Abdelrahman Elbashandy, Dan Gunter, Oluwamayowa Amusat, Lavanya Ramakrishnan, "Experiences with Reproducibility: Case Studies from Scientific Workflows", (P-RECS'21) Proceedings of the 4th International Workshop on Practical Reproducible Evaluation of Computer Systems, ACM, June 21, 2021, doi: 10.1145/3456287.3465478

Reproducible research is becoming essential for science to ensure transparency and for building trust. Additionally, reproducibility provides the cornerstone for sharing of methodology that can improve efficiency. Although several tools and studies focus on computational reproducibility, we need a better understanding about the gaps, issues, and challenges for enabling reproducibility of scientific results beyond the computational stages of a scientific pipeline. In this paper, we present five different case studies that highlight the reproducibility needs and challenges under various system and environmental conditions. Through the case studies, we present our experiences in reproducing different types of data and methods that exist in an experimental or analysis pipeline. We examine the human aspects of reproducibility while highlighting the things that worked, that did not work, and that could have worked better for each of the cases. Our experiences capture a wide range of scenarios and are applicable to a much broader audience who aim to integrate reproducibility in their everyday pipelines.

D. A. Agarwal, J. Damerow, C. Varadharajan, D. S. Christianson, G. Z. Pastorello, Y.-W. Cheah, L. Ramakrishnan, "Balancing the needs of consumers and producers for scientific data collections", Ecological Informatics, 2021, 62:101251, doi: 10.1016/j.ecoinf.2021.101251

G. Z. Pastorello, C. Trotta, E. Canfora, H. Chu, D. Christianson, Y.-W. Cheah, C. Poindexter, J. Chen, A. Elbashandy, M. Humphrey, P. Isaac, D. Polidori, M. Reichstein, A. Ribeca, C. van Ingen, N. Vuichard, L. Zhang, B. Amiro, C. Ammann, M. A. Arain, J. Ardö, T. Arkebauer, S. K. Arndt, N. Arriga, M. Aubinet, M. Aurela, D. Baldocchi, A. Barr, E. Beamesderfer, L. B. Marchesini, O. Bergeron, J. Beringer, C. Bernhofer, D. Berveiller, D. Billesbach, T. A. Black, P. D. Blanken, G. Bohrer, J. Boike, P. V. Bol stad, D. Bonal, J.-M. Bonnefond, D. R. Bowling, R. Bracho, J. Brodeur, C. Brümmer, N. Buchmann, B. Burban, S. P. Burns, P. Buysse, P. Cale, M. Cavagna, P. Cellier, S. Chen, I. Chini, T. R. Chris tensen, J. Cleverly, A. Collalti, C. Consalvo, B. D. Cook, D. Cook, C. Coursolle, E. Cremonese, P. S. Curtis, E. D’Andrea, H. da Rocha, X. Dai, K. J. Davis, B. D. Cinti, A. de Grandcourt, A. D. Ligne, R. C. D. Oliveira, N. Delpierre, A. R. Desai, C. M. D. Bella, P. di Tommasi, H. Dolman, F. Domingo, G. Dong, S. Dore, P. Duce, E. Dufrêne, A. Dunn, J. Dušek, D. Eamus, U. Eichelmann, H. A. M. ElKhidir, W. Eugster, C. M. Ewenz, B. Ewers, D. Famulari, S. Fares, I. Feigenwinter, A. Feitz, R. Fensholt, G. Fil ippa, M. Fischer, J. Frank, M. Galvagno, M. Gharun, D. Gianelle, B. Gielen, B. Gioli, A. Gitelson, I. Goded, M. Goeckede, A. H. Goldstein, C. M. Gough, M. L. Goulden, A. Graf, A. Griebel, C. Gruening, T. Grünwald, A. Hammerle, S. Han, X. Han, B. U. Hansen, C. Hanson, J. Hatakka, Y. He, M. Hehn, B. Heinesch, N. Hinko-Najera, L. Hörtnagl, L. Hutley, A. Ibrom, H. Ikawa, M. Jackowicz-Korczynski, D. Janouš, W. Jans, R. Jassal, S. Jiang, T. Kato, M. Khomik, J. Klatt, A. Knohl, S. Knox, H. Kobayashi, G. Koerber, O. Kolle, Y. Kosugi, A. Kotani, A. Kowalski, B. Kruijt, J. Kurbatova, W. L. Kutsch, H. Kwon, S. Launiainen, T. Laurila, B. Law, R. Leuning, Y. Li, M. Liddell, J.-M. Limousin, M. Lion, A. J. Liska, A. Lohila, A. López-Ballesteros, E. López-Blanco, B. Loubet, D. Loustau, A. Lucas-Moffat, J. Lüers, S. Ma, C. Macfarlane, V. Magliulo, R. Maier, I. Mammarella, G. Manca, B. Marcolla, H. A. Margolis, S. Mar ras, W. Massman, M. Mastepanov, R. Matamala, J. H. Matthes, F. Mazzenga, H. McCaughey, I. McHugh, A. M. S. McMillan, L. Merbold, W. Meyer, T. Meyers, S. D. Miller, S. Minerbi, U. Moderow, R. K. Monson, L. Montagnani, C. E. Moore, E. Moors, V. Moreaux, C. Moureaux, J. W. Munger, T. Nakai, J. Neirynck, Z. Nesic, G. Nicolini, A. Noormets, M. Northwood, M. Nosetto, Y. Nouvellon, K. Novick, W. Oechel, J. E. Olesen, J.-M. Ourcival, S. A. Papuga, F.-J. Parmentier, E. Paul-Limoges, M. Pavelka, M. Peichl, E. Pendall, R. P. Phillips, K. Pilegaard, N. Pirk, G. Posse, T. Powell, H. Prasse, S. M. Prober, S. Ram bal, U. Rannik, N. Raz-Yaseef, D. Reed, V. R. de Dios, N. Restrepo-Coupe, B. R. Reverter, M. Roland, S. Sabbatini, T. Sachs, S. R. Saleska, E. P. S.-C. nete, Z. M. Sanchez-Mejia, H. P. Schmid, M. Schmidt, K. Schneider, F. Schrader, I. Schroder, R. L. Scott, P. Sedlák, P. Serrano-Ortíz, C. Shao, P. Shi, I. Shironya, L. Siebicke, L. Šigut, R. Silberstein, C. Sirca, D. Spano, R. Steinbrecher, R. M. Stevens, C. Sturtevant, A. Suyker, T. Tagesson, S. Takanashi, Y. Tang, N. Tapper, J. Thom, F. Tiedemann, M. Tomassucci, J.-P. Tuovinen, S. Urbanski, R. Valentini, M. van der Molen, E. van Gorsel, K. van Huissteden, A. Varlagin, J. Verfaillie, T. Vesala, C. Vincke, D. Vitale, N. Vygodskaya, J. P. Walker, E. Walter-Shea, H. Wang, R. Weber, S. Westermann, C. Wille, S. Wofsy, G. Wohlfahrt, S. Wolf, W. Woodgate, Y. Li, R. Zampedri, J. Zhang, G. Zhou, D. Zona, D. Agarwal, S. Biraud, M. Torn, D. Papale, "The FLUXNET2015 dataset and the ONEFlux processing pipeline for eddy covariance data", Scientific Data, 2020, 7:225, doi: 10.1038/s41597-020-0534-3

P. Linton, W. Melodia, A. Lazar, D. Agarwal, L. Bianchi, D. Ghoshal, K. Wu, G. Pastorello, L. Ramakrishnan, "Identifying Time Series Similarity in Large-Scale Earth System Datasets", The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19), 2019,

Bing Xu, M. Altaf Arain, Beverly E. Law, Gilberto Z. Pastorello, "Seasonal variability of forest sensitivity to heat and drought stresses: A synthesis based on carbon fluxes from North American forest ecosystems", Global Change Biology, September 2019, doi: 10.1111/gcb.14843

Payton A Linton, William M Melodia, Alina Lazar, Deborah Agarwal, Ludovico Bianchi, Devarshi Ghoshal, Kesheng Wu, Gilberto Pastorello, Lavanya Ramakrishnan, "Identifying Time Series Similarity in Large-Scale Earth System Datasets", 2019,

Payton Linton, William Melodia, Alina Lazar, Deborah Agarwal, Ludovico Bianchi, Devarshi Ghoshal, Gilberto Pastorello, Lavanya Ramakrishnan, Kesheng Wu, Understanding Data Similarity in Large-Scale Scientific Datasets, 2019 IEEE International Conference on Big Data (Big Data), Pages: 4525--4531 2019,

Gilberto Z. Pastorello, Dario Papale, Housen Chu, Carlo Trotta, Deb A. Agarwal, Eleonora Canfora, Dennis D. Baldocchi, M. S. Torn, "A new data set to keep a sharper eye on land-air exchanges", Eos, 2017, 98:28-32, doi: 10.1029/2017EO071597

Danielle S. Christianson, Charuleka Varadharajan, Bradley Christoffersen, Matteo Detto, Faybishenko, Bruno O. Gimenez, Val C. Hendrix, Kolby J. Jardine, Robinson Negron-Juarez, Z. Pastorello, Thomas L. Powell, Megha Sandesh, Jeffrey M. Warren, Brett T. Wolfe, Jeffrey Q. Chambers, Lara M. Kueppers, Nathan G. McDowell, Deborah A. Agarwal, "A metadata reporting framework (FRAMES) for synthesis of ecohydrological observations", Ecological Informatics, 2017, 42:148-158, doi: 10.1016/j.ecoinf.2017.06.002

Gilberto Z. Pastorello, Dan K. Gunter, Housen Chu, Danielle S. Christianson, Carlo Trotta, Eleonora Canfora, Boris Faybishenko, You-Wei Cheah, Norm Beekwilder, Stephen W. Chan, Sigrid Dengel, Trevor Keenan, Fianna O Brien, Abderahman Elbashandy, Cristina M. Poindexter, Marty Humphrey, Dario Papale, Deb A. Agarwal, "Hunting Data Rogues at Scale: Data Quality Control for Observational Data in Research Infrastructures", Proceedings of the 13th IEEE International Conference on e-Science (e-Science 2017), Auckland, New Zealand, 2017, doi: 10.1109/eScience.2017.64

Gilberto Z. Pastorello, "Generation of Uniform Data Products for AmeriFlux and FLUXNET", The practice of reproducible research: case studies and lessons from the data-intensive sciences, (University of California Press: 2017) Pages: 305-309

Craig A. Emmerton, Vincent L. St. Louis, Elyn R. Humphreys, John A. Gamon, Joel D. Barker, Gilberto Z. Pastorello, "Net ecosystem exchange of CO2 with rapidly changing high Arctic landscapes", Global Change Biology, 2016, 22:1185-1200, doi: 10.1111/gcb.13064

Ran Wang, John A. Gamon, Craig A. Emmerton, Li Haitao, Enrica Nestola, Gilberto Z. Pastorello, Olaf Menzer, "Integrated Analysis of Productivity and Biodiversity in a Southern Alberta Prairie", Remote Sensing, 2016, 8:2014, doi: 10.3390/rs8030214

Gilberto Z. Pastorello, Deb A. Agarwal, Taghrid Samak, Dario Papale, Trotta, Alessio Ribeca, Cristina M. Poindexter, Boris Faybishenko, Dan K. Gunter, Rachel Hollowgrass, Eleonora Canfora, "Observational data patterns for time series data quality assessment", Proceedings of the 10th IEEE International Conference on e-Science (e-Science 2014), Guaruja, Brazil, 2014, doi: 10.1109/eScience.2014.45

Lavanya Ramakrishnan, Sarah S. Poon, Val C. Hendrix, Dan K. Gunter, Gilberto Z. Pastorello, Deb A. Agarwal, "Experiences with User-Centered Design for the Tigres Workflow API", Proceedings of the 10th IEEE International Conference on e-Science (e-Science 2014), Guaruja, Brazil, 2014, doi: 10.1109/eScience.2014.56

Angela Harris, John A. Gamon, Gilberto Z. Pastorello, Christopher Y. S. Wong, "Retrieval of the photochemical reflectance index for assessing xanthophyll cycle activity: a comparison of near-surface optical sensors", Biogeosciences, 2014, 11:6277-6292, doi: 10.5194/bg-11-6277-2014

Gilberto Z. Pastorello, G. Arturo Sanchez-Azofeifa, Mario A. Nascimento, "Enviro-Net: from networks of ground-based sensor systems to a Web platform for sensor data management", Sensors, 2011, 11:6454-6479, doi: 10.3390/s110606454

Gilberto Z. Pastorello, G. Arturo Sanchez-Azofeifa, Mario A. Nascimento, "Enviro-Net: A Network of Ground-based Sensors for Tropical Dry Forests in the Americas", Proceedings of the 34th International Symposium on Remote Sensing of Environment (ISRSE 2011), Sydney, Australia, 2011, 4p,

Gilberto Z. Pastorello, Jaudete Daltio, Claudia M. B. Medeiros, "A Mechanism for Propagation of Semantic Annotations of Multimedia Content", Journal of Multimedia, 2010, 5:332-342, doi: 10.4304/jmm.5.4.332-342

John A. Gamon, Craig Coburn, Lawrence B. Flanagan, Karl F. Huemmrich, Kiddle, G. Arturo Sanchez-Azofeifa, Donnette R. Thayer, Loris Vescovo, Damiano Gianelli, Daniel A. Sims, Abdullah Faiz Rahman, Gilberto Z. Pastorello, "SpecNet Revisited: Bridging Flux and Remote Sensing Communities", Canadian Journal of Remote Sensing, 2010, 36:S376-S390, doi: 10.5589/m10-067

Roger Curry, Cameron Kiddle, Rob Simmonds, Gilberto Z. Pastorello, "An On-line Collaborative Data Management System", Proceedings of the 6th Gateway Computing Environments Workshop (GCE 2010), New Orleans, LA, USA, 2010, doi: 10.1109/GCE.2010.5676120

Gilberto Z. Pastorello, Rodrigo D. A. Senra, Claudia M. B. Medeiros, "A standards-based framework to foster geospatial data and process interoperability", Journal of the Brazilian Computer Society, 2009, 15:13-26, doi: 10.1007/BF03192574

Gilberto Z. Pastorello, Rodrigo D. A. Senra, Claudia M. B. Medeiros, "Bridging the gap between geospatial resource providers and model developers", Proceedings of the 16th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (GIS 2008), Irvine, CA, USA, 2008, 379-382, doi: 10.1145/1463434.1463489

Gilberto Z. Pastorello, Luiz C. Gomes Jr, Claudia M. B. Medeiros, Andre Santanche, "Sensor Data Publication on the Web for Scientific Applications", Proceedings of the 4th International Conference on Web Information Systems and Technologies (WEBIST 2008), Funchal, Madeira, Portugal, 2008, 137-142, doi: 10.5220/0001515301370142

Gilberto Z. Pastorello, Claudia M. B. Medeiros, Andre Santanche, "Accessing and Processing Sensing Data", Proceedings of the 11th IEEE International Conference on Computational Science and Engineering (CSE 2008), Sao Paulo, Brazil, 2008, 353--360, doi: 10.1109/CSE.2008.23

Gilberto Z. Pastorello, Jaudete Daltio, Claudia M. B. Medeiros, "Multimedia Semantic Annotation Propagation", Proceedings of the 1st IEEE International Workshop on Data Semantics for Multimedia Systems and Applications (DSMSA 2008), 10th IEEE International Symposium on Multimedia (ISM 2008), Berkeley, CA, USA, 2008, 509--514, doi: 10.1109/ISM.2008.77

Andre Santanche, Claudia M. B. Medeiros, Gilberto Z. Pastorello, "User-author centered multimedia building blocks", Multimedia Systems Journal, 2007, 12:403-421, doi: 10.1007/s00530-006-0050-0

Gilberto Z. Pastorello, C. M. B. Medeiros, Andre Santanche, "Applying Scientific Workflows to Manage Sensor Data", Proceedings of the 1st Brazilian e-Science Workshop (BreSci 2007), 22nd Brazilian Symposium on Databases (SBBD 2007), Joao Pessoa, Brazil, 2007, 9-18,

Claudia M. B. Medeiros, Jose Perez-Alcazar, Luciano Digiampietri, Gilberto Z. Pastorello, Andre Santanche, Ricardo S. Torres, Edmundo Madeira, Evandro Bacarin, "WOODSS and the Web: Annotating and Reusing Scientific Workflows", SIGMOD Record, 2005, 34:18-23, doi: 10.1145/1084805.1084810

Tong Wu, Anna Scaglione, Adrian Petru Surani, Daniel Arnold, Sean Peisert, "Network-Constrained Reinforcement Learning for Optimal EV Charging Control", Proceedings of the IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), October 2023,

Robert Currie, Sean Peisert, Anna Scaglione, Aram Shumavon, Nikhil Ravi, "Data Privacy for the Grid: Toward a Data Privacy Standard for Inverter-Based and Distributed Energy Resources", IEEE Power & Energy Magazine, October 1, 2023,

Jim Basney, Sean Peisert, Scott Russell, Kelli Shute, Bart Miller, Kathy Benninger, "A Vision for Securing NSF's Essential Scientific Cyberinfrastructure - Trusted CI Five-Year Strategic Plan (2024-2029)", Trusted CI Report, August 1, 2023, doi: 10.5281/zenodo.8193607

Sachin Kadam, Anna Scaglione, Nikhil Ravi, Sean Peisert, Brent Lunghino, Aram Shumavon, "Optimum Noise Mechanism for Differentially Private Queries in Discrete Finite Sets", Proceedings of the 2023 IEEE International Conference on Smart Applications, Communications and Networking (SmartNets), Istanbul, Turkey, July 25, 2023,

Sean Peisert, "On Software Infrastructure: Develop, Prove, Profit? [From the Editors]", IEEE Security & Privacy, July 2023, doi: 10.1109/MSEC.2023.3273492

Raksha Ramakrishna, Anna Scaglione, Tong Wu, Nikhil Ravi, Sean Peisert, "Differential Privacy for Class-based Data: A Practical Gaussian Mechanism", June 23, 2023, doi: 10.1109/TIFS.2023.3289128

Nikhil Ravi, Anna Scaglione, Julieta Giraldez, Parth Pradhan, Chuck Moran, Sean Peisert, "Solar Photovoltaic Systems Metadata Inference and Differentially Private Publication", arXiv preprint arXiv:2304.03749, April 7, 2023, doi: 10.48550/arXiv.2304.03749

George Cybenko, Carl Landwehr, Shari Lawrence Pfleeger, Sean Peisert, A 20th Anniversary Episode Chat With S&P Editors, IEEE Security & Privacy, Pages: 9-16 April 2023, doi: 10.1109/MSEC.2023.3239179

Sean Peisert, "The First 20 Years of IEEE Security & Privacy [From the Editors]", IEEE Security & Privacy, April 1, 2023, 21(2):4-6, doi: 10.1109/MSEC.2023.3236420

Hector G. Martin, Tijana Radivojevic, Jeremy Zucker, Kristofer Bouchard, Jess Sustarich, Sean Peisert, Dan Arnold, Nathan Hillson, Gyorgy Babnigg, Jose M. Marti, Christopher J. Mungall, Gregg T. Beckham, Lucas Waldburger, James Carothers, ShivShankar Sundaram, Deb Agarwal, Blake A. Simmons, Tyler Backman, Deepanwita Banerjee, Deepti Tanjore, Lavanya Ramakrishnan, Anup Singh, "Perspectives for Self-Driving Labs in Synthetic Biology", Current Opinion in Biotechnology, February 2023, doi: 10.1016/j.copbio.2022.102881

Ammar Haydari, Chen-Nee Chuah, Michael Zhang, Jane Macfarlane, Sean Peisert, "Differentially Private Map Matching for Mobility Trajectories", Proceedings of the 2022 Annual Computer Security Applications Conference (ACSAC), Austin, TX, ACM, December 2022, doi: 0.1145/3564625.3567974

Andrew Adams, Emily K. Adams, Dan Gunter, Ryan Kiser, Mark Krenz, Sean Peisert, John Zage, "Roadmap for Securing Operational Technology in NSF Scientific Research", Trusted CI Report, November 16, 2022, doi: 10.5281/zenodo.7327987

Ayaz Akram, Venkatesh Akella, Sean Peisert, Jason Lowe-Power, "SoK: Limitations of Confidential Computing via TEEs for High-Performance Compute Systems", Proceedings of the 2022 IEEE International Symposium on Secure and Private Execution Environment Design (SEED), September 2022,

Yize Chen, Yuanyuan Shi, Daniel Arnold, Sean Peisert, "SAVER: Safe Learning-Based Controller for Real-Time Voltage Regulation", Proceedings of the 2022 IEEE Power Engineering Society (PES) General Meeting, Denver, CO, July 2022,

Emily K. Adams, Daniel Gunter, Ryan Kiser, Mark Krenz, Sean Peisert, Susan Sons, John Zage, "Findings of the 2022 Trusted CI Study on the Security of Operational Technology in NSF Scientific Research", Trusted CI Report, July 15, 2022, doi: doi.org/10.5281/zenodo.6828675

Daniel Arnold, Sy-Toan Ngo, Ciaran Roberts, Yize Chen, Anna Scaglione, Sean Peisert, "Adam-based Augmented Random Search for Control Policies for Distributed Energy Resource Cyber Attack Mitigation", Proceedings of the 2022 American Control Conference (ACC), June 2022,

Sean Peisert, Unsafe at Any Clock Speed: the Insecurity of Computer System Design, Implementation, and Operation [From the Editors], IEEE Security & Privacy, Pages: 4-9 January 2022, doi: 10.0.4.85/MSEC.2021.3127086

James R. Clavin, Yue Huang, Xin Wang, Pradeep M. Prakash, Sisi Duan, Jianwu Wang, Sean Peisert, "A Framework for Evaluating BFT", Proceedings of the IEEE International Conference on Parallel and Distributed Systems (ICPADS), IEEE, December 2021,

Andrew Adams, Kay Avila, Elisa Heymann, Mark Krenz, Jason R. Lee, Barton Miller, Sean Peisert, "Guide to Securing Scientific Software", Trusted CI Report, December 14, 2021, doi: 10.5281/zenodo.5777646

Ammar Haydari, Michael Zhang, Chen-Nee Chuah, Jane Macfarlane, Sean Peisert, Adaptive Differential Privacy Mechanism for Aggregated Mobility Dataset, arXiv preprint arXiv:2112.08487, December 10, 2021,

Yize Chen, Yuanyuan Shi, Daniel Arnold, Sean Peisert, SAVER: Safe Learning-Based Controller for Real-Time Voltage Regulation, arXiv preprint arXiv:2111.15152,, November 30, 2021,

Luca Pion-Tonachini, Kristofer Bouchard, Hector Garcia Martin, Sean Peisert, W. Bradley Holtz, Anil Aswani, Dipankar Dwivedi, Haruko Wainwright, Ghanshyam Pilania, Benjamin Nachman, Babetta L. Marrone, Nicola Falco, Prabhat, Daniel Arnold, Alejandro Wolf-Yadlin, Sarah Powers, Sharlee Climer, Quinn Jackson, Ty Carlson, Michael Sohn, Petrus Zwart, Neeraj Kumar, Amy Justice, Claire Tomlin, Daniel Jacobson, Gos Micklem, Georgios V. Gkoutos, Peter J. Bickel, Jean-Baptiste Cazier, Juliane Müller, Bobbie-Jo Webb-Robertson, Rick Stevens, Mark Anderson, Ken Kreutz-Delgado, Michael W. Mahoney, James B. Brown,, Learning from Learning Machines: a New Generation of AI Technology to Meet the Needs of Science, arXiv preprint arXiv:2111.13786, November 27, 2021,

Nikhil Ravi, Anna Scaglione, Sachin Kadam, Reinhard Gentz, Sean Peisert, Brent Lunghino, Emmanuel Levijarvi, Aram Shumavon, Differentially Private K-means Clustering Applied to Meter Data Analysis and Synthesis, arXiv preprint arXiv:2112.03801, November 23, 2021,

Sachin Kadam, Anna Scaglione, Nikhil Ravi, Sean Peisert, Brent Lunghino, Aram Shumavon, Optimum Noise Mechanism for Differentially Private Queries in Discrete Finite Sets, arXiv preprint arXiv:2111.11661, November 23, 2021,

Nikhil Ravi, Anna Scaglione, Sean Peisert, Colored Noise Mechanism for Differentially Private Clustering, arXiv preprint arXiv:2111.07850, November 15, 2021,

Yize Chen, Daniel Arnold, Yuanyuan Shi, Sean Peisert, Understanding the Safety Requirements for Learning-based Power Systems Operations, arXiv preprint arXiv:2110.04983, October 11, 2021,

Andrew Adams, Kay Avila, Elisa Heymann, Mark Krenz, Jason R. Lee, Barton Miller, Sean Peisert, "The State of the Scientific Software World: Findings of the 2021 Trusted CI Software Assurance Annual Challenge Interviews", Trusted CI Report, September 29, 2021,

Ayaz Akram, Venkatesh Akella, Sean Peisert, Jason Lowe-Power,, "Enabling Design Space Exploration for RISC-V Secure Compute Environments", Proceedings of the Fifth Workshop on Computer Architecture Research with RISC-V (CARRV), (co-located with ISCA 2021), June 17, 2021,

Ciaran Roberts, Sy-Toan Ngo, Alexandre Milesi, Anna Scaglione, Sean Peisert, Daniel Arnold, "Deep Reinforcement Learning for Mitigating Cyber-Physical DER Voltage Unbalance Attacks”", Proceedings of the 2021 American Control Conference (ACC), May 2021, doi: 10.23919/ACC50511.2021.9482815

Ayaz Akram, Anna Giannakou, Venkatesh Akella, Jason Lowe-Power, Sean Peisert, "Performance Analysis of Scientific Computing Workloads on General Purpose TEEs", Proceedings of the 35th IEEE International Parallel & Distributed Processing Symposium (IPDPS), IEEE, May 2021, doi: 10.1109/IPDPS49936.2021.00115

Sean Peisert, "Trustworthy Scientific Computing", Communications of the ACM (CACM), May 2021, doi: 10.1145/3457191

Sean Peisert, Bruce Schneier, Hamed Okhravi, Fabio Massacci, Terry Benzel, Carl Landwehr, Mohammad Mannan, Jelena Mirkovic, Atul Prakash, James Bret Michael, "Perspectives on the SolarWinds Incident", IEEE Security & Privacy, April 2021, 7-13, doi: 10.1109/MSEC.2021.3051235

Fabio Massacci, Trent Jaeger, Sean Peisert, "SolarWinds and the Challenges of Patching: Can We Ever Stop Dancing With the Devil?", IEEE Security & Privacy, April 2021, 14-19, doi: 10.1109/MSEC.2021.3050433

Sean Peisert, Reflections on the Past, Perspectives on the Future [From the Editors], IEEE Security & Privacy, January 2021, doi: 10.1109/MSEC.2020.3034670

Chris Lawson, Jose Manuel Martí, Tijana Radivojevic, Sai Vamshi R. Jonnalagadda, Reinhard Gentz, Nathan J. Hillson, Sean Peisert, Joonhoon Kim, Blake A. Simons, Christopher J. Petzold, Steven W. Singer, Aindrila Mukhopadhyay, Deepti Tanjore, Josh Dunn, Héctor García Martín,, "Machine Learning for Metabolic Engineering: A Review", Metabolic Engineering, November 19, 2020,

Ignacio Losada Carreño, Raksha Ramakrishna, Anna Scaglione, Daniel Arnold, Ciaran Roberts, Sy-Toan Ngo, Sean Peisert, David Pinney, "SODA: An Irradiance-Based Tool to Generate Sub-Minute Solar Power Stochastic Time Series", Proceedings of the IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), IEEE, November 2020,

Ciaran Roberts Sy-Toan Ngo, Alexandre Milesi, Sean Peisert, Daniel Arnold, Shammya Saha, Anna Scaglione, Nathan Johnson, Anton Kocheturov, Dmitriy Fradkin, "Deep Reinforcement Learning for DER Cyber-Attack Mitigation", Proceedings of the IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), IEEE, November 2020,

Ayaz Akram, Anna Giannakou, Venkatesh Akella, Jason Lowe-Power, Sean Peisert, "Performance Analysis of Scientific Computing Workloads on Trusted Execution Environments", arXiv preprint arXiv:2010.13216, October 25, 2020,

Sean Peisert, "An Examination and Survey of Data Confidentiality Issues and Solutions in Academic Research Computing", Trusted CI Report, September 8, 2020,

Sean Peisert, Isolating Insecurely: A Call to Arms for the Security and Privacy Community During the Time of COVID-19 [From the Editors], IEEE Security & Privacy, Pages: 4-7 August 2020, doi: 10.1109/MSEC.2020.2992316

Bogdan Copos, Sean Peisert, "Catch Me If You Can: Using Power Analysis to Identify HPC Activity", arXiv:2005.03135 [cs.CR], May 6, 2020,

Ross Gegan, Christina Mao, Dipak Ghosal, Matt Bishop, Sean Peisert, "Anomaly Detection for Science DMZ Using System Performance Data", Proceedings of the 2020 IEEE International Conference on Computing, Networking and Communications (ICNC 2020), Big Island, HI, February 2020, doi: 10.1109/ICNC47757.2020.9049695

Anna Giannakou, Dipankar Dwivedi, Sean Peisert, "A Machine Learning Approach for Packet Loss Prediction in ScienceFlows", Future Generation Computer Systems, January 2020, 102:190-197, doi: 10.1016/j.future.2019.07.053

Reinhard Gentz, Sean Peisert, "An Examination and Survey of Random Bit Flips and Scientific Computing", Trusted CI Report, December 20, 2019,

Amir Teshome Wonjiga, Louis Rilling, Christine Morin, Sean Peisert, "Blockchain as a Trusted Component in Cloud SLA Verification", Proceedings of the International Workshop on Cloud, IoT and Fog Security (CIFS), co-located with the 12th IEEE/ACM International Conference on Utility and Cloud Computing (UCC), Auckland, New Zealand, December 2019, 93-100, doi: 10.1145/3368235.3368872

Mahdi Jamei, Raksha Ramakrishna, Teklemariam Tesfay, Reinhard Gentz, Ciaran Roberts, Anna Scaglione, Sean Peisert, "Phasor Measurement Units Optimal Placement and Performance Limits for Fault Localization", IEEE Journal on Selected Areas in Communications (J-SAC), Special Issue on Communications and Data Analytics in Smart Grid, November 6, 2019, 38(1):180-192, doi: 10.1109/jsac.2019.2951971

Thomas W. Edgar, Aditya Ashok, Garret E. Seppala, K.M. Arthur-Durrett, M. Engels, Reinhard Gentz, Sean Peisert, "An Automated Disruption-Tolerant Key Management Framework for Critical Systems", Journal of Information Warfare, October 8, 2019, 18(4):85-124, doi: https://www.jinfowar.com/journal/volume-18-issue-4/automated-disruption-tolerant-device-authentication-key-management-framework-critical-systems

Reinhard Gentz, Sean Peisert, Joshua Boverhof, Daniel Gunter, "SPARCS: Stream-Processing Architecture applied in Real-time Cyber-physical Security", Proceedings of the 15th IEEE International Conference on e-Science (eScience), San Diego, CA, IEEE, September 2019, doi: 10.1109/eScience.2019.00028

Reinhard Gentz, Héctor García Martin, Edward Baidoo, Sean Peisert, "Workflow Automation in Liquid Chromatography Mass Spectrometry", Proceedings of the 15th IEEE International Conference on e-Science (eScience), San Diego, CA, IEEE, September 2019, doi: 10.1109/eScience.2019.00095

Melissa Stockman, Dipankar Dwivedi, Reinhard Gentz, Sean Peisert, "Detecting Programmable Logic Controller Code Using Machine Learning", International Journal of Critical Infrastructure Protection, September 2019, vol. 26,, doi: 10.1016/j.ijcip.2019.100306

Ciaran Roberts, Anna Scaglione, Mahdi Jamei, Reinhard Gentz, Sean Peisert, Emma M. Stewart, Chuck McParland, Alex McEachern, Daniel Arnold, "Learning Behavior of Distribution System Discrete Control Devices for Cyber-Physical Security", IEEE Transaction on Smart Grid, August 1, 2019, 11(1):749-761, doi: 0.1109/TSG.2019.2936016

Andrew Adams, Kay Avila, Jim Basney, Dana Brunson, Robert Cowles, Jeannette Dopheide, Terry Fleury, Elisa Heymann, Florence Hudson, Craig Jackson, Ryan Kiser, Mark Krenz, Jim Marsteller, Barton P. Miller, Sean Peisert, Scott Russell, Susan Sons, Von Welch, John Zage, "Trusted CI Experiences in Cybersecurity and Service to Open Science", Proceedings of the Conference on Practice and Experience in Advanced Research Computing (PEARC), ACM, July 2019, doi: 10.1145/3332186.3340601

Sean Peisert, "Some Experiences in Developing Security Technology That Actually Get Used", IEEE Security & Privacy, April 2019, 17(2):4–7, doi: 10.1109/MSEC.2019.2899711

Sean Peisert, Brooks Evans, Michael Liang, Barclay Osborn, David Rusting, David Thurston, Security Without Moats and Walls: Zero-Trust Networking for Enhancing Security in R&E Environments, CENIC Annual Conference, March 19, 2019,

Sean Peisert, Experiences in Building a Mission-Driven Security R&D Program for Science and Energy, Computer Science Colloquium Seminar, University of California, Davis, February 7, 2019,

Sean Peisert, Daniel Arnold, Using Physics to Improve Cybersecurity for the Distribution Grid and Distributed Energy Resources, Naval Postgraduate School, February 5, 2019,

Sean Peisert, Building a Mission-Driven, Applied Cybersecurity R&D Program from Scratch, VISA Research, January 23, 2019,

Anna Giannakou, Daniel Gunter, Sean Peisert, "Flowzilla: A Methodology for Detecting Data Transfer Anomalies in Research Networks", Workshop on Innovating the Network for Data-Intensive Science (INDIS), November 11, 2018, doi: 10.1109/INDIS.2018.00004

Sean Peisert, Usable Computer Security and Privacy to Enable and Encourage Data Sharing for Scientific Research, National Academies of Sciences, Engineering, and Medicine Committee on Science, Engineering, Medicine, and Public Policy (COSEMPUP) Meeting, November 8, 2018,

Mahdi Jamei, Anna Scaglione, Sean Peisert, "Low-Resolution Fault Localization Using Phasor Measurement Units with Community Detection", Proceedings of the 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), Allborg, Denmark, IEEE, October 29, 2018, doi: 10.1109/SmartGridComm.2018.8587461

Sean Peisert, Ciaran Roberts, Anna Scaglione, Mahdi Jamei, Reinhard Gentz, Charles McParland, Alex McEachren, Galen Rasche, Aaron Snyder, "Supporting Cyber Security of Power Distribution Systems by Detecting Differences Between Real-time Micro-Synchrophasor Measurements and Cyber-Reported SCADA - Final Report", October 15, 2018,

Sean Peisert, Security Concerns of an NRP, Second National Research Platform (NRP) Workshop, August 6, 2018,

"A Holistic Approach to Distribution Grid Intrusion Detection Systems", Ciaran Roberts, Anna Scaglione Sean Peisert,, EnergyCentral, July 18, 2018,

Sean Peisert, Cyber Security Challenges and Opportunities in High-Performance Computing Environments, International Supercomputing Conference, June 26, 2018,

Sean Peisert, Keynote: Cybersecurity for HPC Systems: State of the Art and Looking to the Future, High-Performance Computing Security Workshop, National Institute of Standards and Technology (NIST), March 28, 2018,

Sean Peisert, Eli Dart, William K. Barnett, James Cuff, Robert L. Grossman, Edward Balas, Ari Berman, Anurag Shankar, Brian Tierney, "The Medical Science DMZ: An Network Design Pattern for Data-Intensive Medical Science", Journal of the American Medical Informatics Association (JAMIA), March 2018, 25(3):267-274, doi: 10.1093/jamia/ocx104

Sean Peisert, Ciaran Roberts, Cyber Security of Power Distribution Systems Using Micro-Synchrophasor Measurements and Cyber-Reported SCADA, EPRI Power Delivery & Utilization Winter 2018 Program Advisory & Sector Council Meeting, February 7, 2018,

EM Stewart, P Top, M Chertkov, D Deka, S Backhaus, A Lokhov, C Roberts, V Hendrix, S Peisert, A Florita, TJ King, MJ Reno, "Integrated multi-scale data analytics and machine learning for the distribution grid", 2017 IEEE International Conference on Smart Grid Communications, SmartGridComm 2017, 2018, 2018-Jan:423--429, doi: 10.1109/SmartGridComm.2017.8340693

Terry Benzel and Sean Peisert, Selected Papers from the 2017 IEEE Symposium on Security and Privacy [Guest editors' introduction], IEEE Security & Privacy, Pages: 10-11 January 2018, doi: 10.1109/MSP.2018.1331038

David Hatchell, Patrick Miller, Michael Coleman, Sean Peisert, Cybersecurity for the Electricity Grid", Bits & Watts Annual Conference, November 6, 2017,

Sean Peisert, Security in High Performance Computing Environments, Computing Sciences/NERSC Security Seminar, October 5, 2017,

Sean Peisert, Matt Bishop, Ed Talbot,, "A Model of Owner Controlled, Full-Provenance, Non-Persistent, High-Availability Information Sharing", Proceedings of the 2017 New Security Paradigms Workshop (NSPW), Santa Cruz, CA, October 2017, 80-89, doi: 10.1145/3171533.3171536

Sean Peisert, Security and Privacy in Data-Intensive, High-Performance Computing Contexts, Berkeley Institute for Data Science (BIDS), October 2, 2017,

Jonathan Ganz, Sean Peisert, "ASLR: How Robust is the Randomness", Proceedings of the IEEE Secure Development Conference (SecDev), Cambridge, MA, IEEE Computer Society, September 24, 2017, doi: 10.1109/SecDev.2017.19

Mahdi Jamei, Anna Scaglione, Ciaran Roberts, Emma Stewart, Sean Peisert, Chuck McParland, Alex McEachern, "Anomaly Detection Using μPMU Measurements in Distribution Grids", IEEE Transactions on Power Systems, 2017, doi: 10.1109/TPWRS.2017.2764882

Sean Peisert, "Security in High-Performance Computing Environments", Communications of the ACM (CACM), September 2017, 60(9):72-80, doi: 10.1145/3096742

Mahdi Jamei, Anna Scaglione, Ciaran Roberts, Alex McEachern, Emma Stewart, Sean Peisert, Chuck McParland, "Online Thevenin Parameter Tracking Using Synchrophasor Data", Proceedings of the 2017 IEEE Power Engineering Society (PES) General Meeting (GM), Chicago, IL, IEEE, July 2017, doi: 10.1109/PESGM.2017.8273818

Alberto Gonzalez, Jason Leigh, Sean Peisert, Brian Tierney, Edward Balas, Predrag Radulovic, Jennifer M. Schopf, "Big Data and Analysis of Data Transfers for International Research Networks Using NetSage", Proceedings of IEEE BigData Congress 2017, Honolulu, Hawaii, June 2017, doi: 10.1109/BigDataCongress.2017.51

Sean Peisert, Mike Corn, Dewight Kramer, David Rusting, Tye Stallard, The Role of the WAN and the Community to Improve Security, 2017 UC Information Security Symposium,, June 21, 2017,

Galen Rasche, Jenna Goodward, Sheeraz Haji, Gabriel Paun, Sean Peisert, Managing Energy: Role of Data and Security, Prospect Silicon Valley 2017 Innovation and Impact Symposium, June 14, 2017,

Sean Peisert, Greg Bell, Anita Nikolich, Von Welch, Cybersecurity: New Directions for Research and Education - Your own safety is at stake when your neighbor's wall is ablaze. (—Horace), CENIC Annual Conference — The Right Connection ¦ CENIC 2.0, March 22, 2017,

Leon J. Osterweil, Matt Bishop, Heather M. Conboy, Huong Phan, Borislava I. Simidchieva, George S. Avrunin, Lori A. Clarke, Sean Peisert, "A Comprehensive Framework for Using Iterative Analysis to Improve Human-Intensive Process Security: An Election Example", ACM Transactions on Privacy and Security (TOPS), 2017, 20(2), doi: https://doi.org/10.1145/3041041

Sean Peisert, Von Welch, Andrew Adams, Michael Dopheide, Susan Sons, RuthAnne Bevier, Rich LeDuc, Pascal Meunier, Stephen Schwab, and Karen Stocks, Ilkay Altintas, James Cuff, Reagan Moore, Warren Raquel, "Open Science Cyber Risk Profile", February 10, 2017, doi: 2022/21259

Richard LeDuc, Sean Peisert, Karen Stocks, Von Welch, Open Science Cyber Risk Profile (OSCRP), National Science Foundation Cybersecurity Center of Excellence (CCoE) Webinar Series, January 23, 2017,

Mahdi Jamei, Anna Scaglione, Ciaran Roberts, Emma Stewart, Sean Peisert, Chuck McParland, Alex McEachern, "Automated Anomaly Detection in Distribution Grids Using µPMU Measurements", Proceedings of the 50th Hawaii International Conference on System Sciences (HICSS), Electric Energy Systems Track, Resilient Networks Minitrack, IEEE, January 2017, doi: http://hdl.handle.net/10125/41543

S Peisert, R Gentz, J Boverhof, C McParland, S Engle, A Elbashandy, D Gunter, "LBNL Open Power Data", January 2017, doi: 10.21990/C21599

Sean Peisert, Overcoming Security and Privacy Challenges in Computing and Networking in Medical Research Environments, Department of Public Health Sciences, University of California, Davis School of Medicine,, December 14, 2016,

Sean Peisert, Emma Stewart, Alex McEachern, Supporting Cyber Security of Power Distribution Systems by Detecting Differences Between Real-time Micro-synchrophasor Measurements and Cyber-Reported SCADA, 2016 DOE CEDS Peer Review, December 8, 2016,

Sean Peisert (moderator), Jill Gemmill, Michael Sinatra, Von Welch, National Cybersecurity Panel, NSF Campus Cyberinfrastructure/ESCC/The Quilt Colocated Meeting, October 20, 2016,

Lee Beausoleil, David Lombard, Angelos Keromytis, Sean Peisert, Panel: HPC Monitoring, NSCI: High-Performance Computing Security Workshop, September 30, 2016,

Sean Peisert, Security Expert on Why HPC Matters - Cybersecurity for HPC Systems: Challenges and Opportunities, NSCI: High-Performance Computing Security Workshop, September 29, 2016,

Mahdi Jamei, Emma Stewart, Sean Peisert, Anna Scaglione, Chuck McParland, Ciaran Roberts, Alex McEachern, "Micro Synchrophasor-Based Intrusion Detection in Automated Distribution Systems: Towards Critical Infrastructure Security", IEEE Internet Computing, September 2016, 20(5):18-27, doi: 10.1109/MIC.2016.102

Jonathan Ganz, Matt Bishop, and Sean Peisert, "Security Analysis of Scantegrity, an Electronic Voting System", University of California, Davis, Department of Computer Science Technical Report, June 2016,

Alberto Gonzalez, Jason Leigh, Sean Peisert, Brian Tierney, Andrew Lee, Jennifer M. Schopf, "NetSage: Open Privacy-Aware Network Measurement, Analysis, And Visualization Service", Proceedings of TNC16 Networking Conference, Prague, Czech Republic, June 2016,

Sean Peisert, William K. Barnett, Eli Dart, James Cuff, Robert L. Grossman, Edward Balas, Ari Berman, Anurag Shankar, Brian Tierney, "The Medical Science DMZ", Journal of the American Medical Informatics Association (JAMIA), May 2, 2016, 23(6):1199-1201, doi: 10.1093/jamia/ocw032

Sean Peisert, CENIC 2016 Conference Panel: Security in R&E Networks and Campus Environments, 2016 CENIC Annual Conference, March 22, 2016,

Sean Peisert, Computer Security & the Electric Power Grid, 15th Annual ON*VECTOR Photonics Workshop, March 1, 2016,

Jason Adams, Monica Lieng, Brooks Kuhn, Edward Guo, Edik Simonian, Sean Peisert, JP Delplanque, Nick Anderson, "Automated Mechanical Ventilator Waveform Analysis of Patient-Ventilator Asynchrony", CHEST Journal, Pages: 175A October 2015, doi: 10.1378/chest.2281731

PURPOSE: Mechanical ventilation is a life-saving intervention but is associated with adverse effects including ventilator-induced lung injury (VILI). Patient-ventilator asynchrony (PVA) is thought to contribute to VILI, but the study of PVA has been hampered by limited access to the high frequency, large volume data streams produced by modern ventilators and a lack of robust analytics. To address these limitations, we developed an automated pipeline for breath-by-breath analysis of ventilator waveform data.

METHODS: Simulated pressure and flow time series data representing normal breaths and common forms of PVA were generated on PB840 ventilators, collected unobtrusively using small, customized wireless peripheral devices, and transmitted to a networked server for storage and analysis. Two critical care physicians reviewed all waveforms to generate gold standards. Rule-based algorithms were developed to quantify inspiratory and expiratory tidal volumes (TV) and identify PVA subtypes including double trigger and delayed termination asynchrony. Data were split randomly into derivation and validation sets. Algorithm performance was compared to ventilator reported values and clinician annotation.

RESULTS: The mean difference between algorithm-determined and ventilator-reported TVs was 3.1% (99% CI ± 1.36%). Algorithm agreement with clinician annotation was excellent for double trigger PVA and moderate for delayed termination PVA, with Kappa statistics of 0.85 and 0.58, respectively. In the validation data set (n = 492 breaths), double trigger asynchrony was detected with an overall accuracy of 94.1%, sensitivity of 100%, and specificity of 92.8%.

CONCLUSIONS: A pipeline combining wireless ventilator data acquisition and rule-based analytic algorithms informed by the principles of bedside ventilator waveform analysis allows for automated, quantitative breath-by-breath analysis of patient-ventilator interactions.

CLINICAL IMPLICATIONS: We have recently deployed this system in the medical intensive care unit of the UC Davis Medical Center, which will enable further development of mechanical ventilation analytics. We have begun to explore the use of supervised machine learning and dynamic time series modeling to improve the classification of other common types of PVA and of clinical phenotypes associated with respiratory failure. This system will help to better define the epidemiology and clinical impact of PVA and other forms of off-target mechanical ventilation, and may lead to improved decision support and patient outcomes.

Daniel Chung, Matt Bishop, and Sean Peisert, "Distributed Helios - Mitigating Denial of Service Attacks in Online Voting", University of California, Davis, Department of Computer Science Technical Report, October 16, 2015,

Adrian Chavez, William M.S. Stout, and Sean Peisert, "Techniques for the Dynamic Randomization of Network Attributes", Proceedings of the 49th Annual International Carnahan Conference on Security Technology, Taipei, Taiwan, Republic of China, IEEE Press, September 2015, doi: 10.1109/CCST.2015.7389661

Sisi Duan, Jingtao Sun, Sean Peisert, "Towards a Self-Adaptive Middleware for Building Reliable Publish/Subscribe Systems", Proceedings of the 8th International Conference on Internet and Distributed Computing Systems (IDCS), Berkshire, United Kingdom, Springer, September 2015, 157-168, doi: 10.1007/978-3-319-23237-9_14

Sean Peisert, et al., "ASCR Cybersecurity for Scientific Computing Integrity - Research Pathways and Ideas", U.S. Department of Energy Office of Science report, September 2015, LBNL 191105, doi: 10.2172/1236181

Sean Peisert, Security Research Using Cyber-Physical Systems, IT Security Symposium, June 16, 2015,

Sean Peisert, Models of Secure and Private Information Sharing, University of California, San Diego School of Medicine, Division of Biomedical Informatics Seminar Series, April 10, 2015,

Sean Peisert, et al., "ASCR Cybersecurity for Scientific Computing Integrity", U.S. Department of Energy Office of Science report, February 27, 2015, LBNL 6953E, doi: 10.2172/1223021

Georgia Koutsandria, Reinhard Gentz, Mahdi Jamei, Anna Scaglione, Sean Peisert, Chuck McParland, "A real-time testbed environment for cyber-physical security on the power grid", Proceedings of the First ACM Workshop on Cyber-Physical Systems-Security and/or PrivaCy, January 1, 2015, 67--78, doi: 10.1145/2808705.2808707

Sisi Duan, Hein Meling, Sean Peisert, Haibin Zhang,, "BChain: Byzantine Replication with High Throughput and Embedded Reconfiguration", Proceedings of the 18th International Conference on Principles of Distributed Systems (OPODIS), Cortina, Italy, Springer, December 2014, 91-106, doi: 10.1007/978-3-319-14472-6_7

Chuck McParland, Sean Peisert, Anna Scaglione, "Monitoring Security of Networked Control Systems: It's the Physics", IEEE Security and Privacy, November 2014, 12(6):32-39, doi: 10.1109/MSP.2014.122

Sean Peisert, Jonathan Margulies, Closing the Gap on Securing Energy Sector Control Systems [Guest editors' introduction], IEEE Security & Privacy, Pages: 13-14 November 2014, doi: 10.1109/MSP.2014.110

Sean Peisert, Jonathan Margulies, Eric Byres, Paul Dorey, Dale Peterson, Zach Tudor, "Control System Security from the Front Lines (Roundtable)", IEEE Security and Privacy, November 2014, 12(6):55-58, doi: 10.1109/MSP.2014.112

Sisi Duan, Karl Levitt, Hein Meling, Sean Peisert, Haibin Zhang, "Byzantine Fault Tolerance from Intrusion Detection", Proceedings of the 33rd IEEE International Symposium on Reliable Distributed Systems (SRDS), Nara, Japan, October 2014, 253-264, doi: 10.1109/SRDS.2014.28

Sean Peisert, Security for Computational Infrastructure for Financial Technology, DataLead 2014: Leading the Way in Big Data, Haas School of Business, UC Berkeley, September 30, 2014,

Sean Peisert, Jonathan Margulies, David M. Nicol, Himanshu Khurana, Chris Sawall,, "Designed-in Security for Cyber-Physical Systems (Roundtable)", IEEE Security and Privacy, September 2014, 12(5):9-12, doi: 10.1109/MSP.2014.90

Tiancheng Chang, Sisi Duan, Hein Meling, Sean Peisert, Haibin Zhang, "P2S: A Fault-Tolerant Publish/Subscribe Infrastructure", Proceedings of the 8th ACM International Conference on Distributed Event Based Systems (DEBS), Mumbai, India, ACM Press, May 2014, 189-197, doi: 10.1145/2611286.2611305

Matt Bishop, Heather Conboy, Huong Phan, Borislava I. Simidchieva, George Avrunin, Lori Clarke, Lee Osterweil, Sean Peisert,, "Insider Detection by Process Analysis", Proceedings of the 2014 Workshop on Research for Insider Threat (WRIT), IEEE Computer Society Security and Privacy Workshops, San Jose, CA, IEEE Computer Society, May 18, 2014, doi: 10.1109/SPW.2014.40

Sean Peisert, Challenges in Insider Threat Research, Workshop on Research for Insider Threat (WRIT), IEEE Security and Privacy Workshops (SPW), May 18, 2014,

Peter G. Neumann, Sean Peisert, Marvin Schaefer, "The IEEE Symposium on Security and Privacy, in Retrospect", IEEE Security and Privacy, May 2014, 12(3):15-17, doi: 10.1109/MSP.2014.59

Sisi Duan, Sean Peisert, and Karl Levitt, "hBFT: Speculative Byzantine Fault Tolerance With Minimum Cost", IEEE Transactions on Dependable and Secure Computing (TDSC), March 19, 2014, 12(1):58-70, doi: 10.1109/TDSC.2014.2312331

Masood Parvania, Georgia Koutsandria, Vishak Muthukumary, Sean Peisert, Chuck McParland, Anna Scaglione, "Hybrid Control Network Intrusion Detection Systems for Automated Power Distribution Systems", Proc. of the 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), January 1, 2014, 774--779, doi: 10.1109/DSN.2014.81

Georgia Koutsandria, Vishak Muthukumar, Masood Parvania, Sean Peisert, Chuck McParland, Anna Scaglione, "A hybrid network IDS for protective digital relays in the power transmission grid", Smart Grid Communications (SmartGridComm), 2014 IEEE International Conference on, January 2014, 908--913, doi: 10.1109/SmartGridComm.2014.7007764

Lizzie Coles-Kemp, Carrie Gates, Dieter Gollmann, Sean Peisert, Christian Probst, "Organizational Processes for Supporting Sustainable Security", Report from Dagstuhl Seminar 120501, November 4, 2013, doi: 10.4230/DagRep.2.12.37

Sean Peisert, Ed Talbot, Tom Kroeger, "Principles of Authentication", Proceedings of the 2013 New Security Paradigms Workshop (NSPW), Banff, Canada, ACM, September 2013, 47-56, doi: 10.1145/2535813.2535819

Sean Peisert and Steven Templeton, "The Hive Mind: Applying a Distributed Security Sensor Network to GENI- GENI Spiral 2 Final Project Report", UC Davis Technical Report, September 4, 2013,

Sean Peisert, Cyber Resilience Metrics, First International Symposium on Resilient Cyber Systems, Resilience Week 2013, August 13, 2013,

Sean Peisert, Health Informatics Minute: Aligning Organizational and Employee Computer Security Goals for Health Informatics, Seventh Annual Health Informatics Graduate Program Conference, March 22, 2013,

Sean Whalen, Sean Peisert, Matt Bishop, "Multiclass Classification of Distributed Memory Parallel Computations", Pattern Recognition Letters (PRL), February 2013, 34(3):322-329, doi: 10.1016/j.patrec.2012.10.007

Sean Peisert, Matt Bishop, "Dynamic, Flexible, and Optimistic Access Control", Working Paper, January 2013,

Xiao Li, Zhifang Wang, Vishak Muthukumar, Anna Scaglione, Chuck McParland, Sean Peisert, "Networked Loads in the Distribution Grid", Proceedings of the 2012 APSIPA Annual Summit and Conference, Hollywood, CA, December 3, 2012,

Sean Peisert, Institute for Information Infrastructure Protection (I3P), 10th Anniversary Event, The National Press Club, October 10, 2012,

Sean Peisert, Ed Talbot, Matt Bishop, "Turtles All the Way Down: A Clean-Slate, Ground-Up, First-Principles Approach to Secure Systems", Proceedings of the 2012 New Security Paradigms Workshop (NSPW), ACM, September 2012, doi: 10.1145/2413296.2413299

Matt Bishop, Sean Peisert, "Security and Elections", IEEE Security & Privacy, September 2012, 10(5):64-67, doi: 10.1109/MSP.2012.127

Sean Whalen, Sophie Engle, Sean Peisert, Matt Bishop, "Network-Theoretic Classification of Parallel Computation Patterns", International Journal of High Performance Computing Applications (IJHPCA), May 2012, 26(2):159-169, doi: 10.1177/1094342012436618

"4th Workshop on Cyber Security Experimentation and Test (CSET ’11) Conference Report Summary", Sean Peisert, USENIX ;login:, November 2011,

"The Hive Mind Project - Digital Ants for Intrusion Detection", Sean Peisert, DETERlab Testbed Quarterly Newsletter, 2011,

Sean Whalen, Sean Peisert, Matt Bishop, "Network-Theoretic Classification of Parallel Computation Patterns", Proceedings of the First International Workshop on Characterizing Applications for Heterogeneous Exascale Systems (CACHES), Tucson, AZ, IEEE Computer Society, June 4, 2011,

Matt Bishop, Sophie Engle, Deborah A. Frincke, Carrie Gates, Frank L. Greitzer, Sean Peisert, Sean Whalen, "A Risk Management Approach to the "Insider Threat"", Insider Threats in Cyber Security, "Advances in Information Security" Series, edited by Christian W. Probst, Jeffrey Hunker, Matt Bishop, (Springer: September 2010) Pages: 115-138 doi: 10.1007/978-1-4419-7133-3_6

Borislava I. Simidchieva, Sophie J. Engle, Michael Clifford, Alicia Clay Jones, Sean Peisert, Matt Bishop, Lori A. Clarke, Leon J. Osterweil,, "Modeling Faults to Improve Election Process Robustness", Proceedings of the 2010 Electronic Voting Technology Workshop/ Workshop on Trustworthy Elections (EVT/WOTE), Washinton, D.C., USENIX, August 2010,

Sean Peisert, "Fingerprinting Communication and Computation on HPC Machines", Lawrence Berkeley National Laboratory Technical Report, June 2010, LBNL LBNL-3483E,

Peter G. Neumann, Matt Bishop, Sean Peisert, Marv Schaefer, "Reflections on the 30th Anniversary of the IEEE Symposium on Security and Privacy", Proceedings of the 31st IEEE Symposium on Security and Privacy, Oakland/Berkeley, CA, IEEE Computer Society, May 2010, 3-13, doi: 10.1109/SP.2010.43

Sean Peisert, Matt Bishop, Keith Marzullo, "What Do Firewalls Protect? An Empirical Study of Firewalls, Vulnerabilities, and Attacks", UC Davis CS Technical Report CSE-2010-8, March 2010,

M Bishop, J Cummins, S Peisert, A Singh, B Bhumiratana, D Agarwal, D Frincke, M Hogarth, "Relationships and data sanitization: A study in scarlet", Proceedings New Security Paradigms Workshop, January 2010, 151--163, doi: 10.1145/1900546.1900567

Sean Peisert, Matt Bishop, Laura Corriss, Steven J. Greenwald, "Quis Custodiet ipsos Custodes? A New Paradigm for Analyzing Security Paradigms", Proceedings of the 2009 New Security Paradigms Workshop (NSPW), The Queen's College, Oxford, United Kingdom, ACM, September 2009, 133-144, doi: 10.1145/1719030.1719041

Matt Bishop, Sean Peisert, Candice Hoke, Mark Graff, David Jefferson, "E-Voting and Forensics: Prying Open the Black Box", Proceedings of the 2009 Electronic Voting Technology Workshop/Workshop on Trustworthy Elections (EVT/WOTE), Montreal, Canada, USENIX, August 2009,

Matt Bishop, Mark Graff, Candice Hoke, David Jefferson, Sean Peisert, "Resolving the Unexpected in Elections: Election Officials' Options", October 8, 2008,

Sean Peisert, Keynote Address: Computer Forensics In Forensis, Third International IEEE Workshop on Systematic Approaches to Digital Forensic Engineering (IEEE/SADFE-2008) (held in conjunction with the 2008 IEEE Symposium on Security and Privacy), May 22, 2008,

Sean Peisert, Matt Bishop, Keith Marzullo, "Computer Forensics In Forensis", ACM Operating Systems Review (OSR), April 2008, 42:112-122, doi: 10.1109/TDSC.2007.1003

Sean Peisert, Matt Bishop, "I'm a Scientist, Not a Philosopher!", IEEE Security and Privacy, July 2007, 5(4):48-51, doi: 10.1109/MSP.2007.84

Sean Peisert, Matt Bishop, Sidney Karin, Keith Marzullo,, "Analysis of Computer Intrusions Using Sequences of Function Calls", IEEE Transactions on Dependable and Secure Computing (TDSC), April 2007, 4(2):137-150, doi: 10.1109/TDSC.2007.1003

Sean Peisert, Matt Bishop, Sidney Karin, Keith Marzullo,, "Toward Models for Forensic Analysis", Proceedings of the Second International Workshop on Systematic Approaches to Digital Forensic Engineering (SADFE), IEEE, April 1, 2007, 3-15, doi: 10.1109/SADFE.2007.23

A Model of Forensic Analysis Using Goal-Oriented Logging, Sean P. Peisert, Ph.D. Dissertation, Dept. of Computer Science and Engineering, University of California, San Diego, March 2007,

Matt Bishop, Sean Peisert, "Your Security Policy is What???", UC Davis CS Technical Report CSE-2006-20, March 2006,

Sean Peisert, Matt Bishop, Sidney Karin, Keith Marzullo,, "Principles-Driven Forensic Analysis", Proceedings of the 2005 New Security Paradigms Workshop (NSPW), Lake Arrowhead, CA, ACM, September 1, 2005, 85-93, doi: 10.1145/1146269.1146291

Sean Peisert, "Forensics for System Administrators", ;login:, August 2005, 30(4):34-42,

Jan Balewski, Mercy G Amankwah, Roel Van Beeumen, E Wes Bethel, Talita Perciano, Daan Camps, "Quantum-parallel vectorized data encodings and computations on trapped-ion and transmon QPUs", Journal, February 10, 2024, 14, doi: https://doi.org/10.1038/s41598-024-53720-x

Zhe Bai, Abdelilah Essiari, Talita Perciano, Kristofer E Bouchard, "AutoCT: Automated CT registration, segmentation, and quantification", Software X, January 5, 2024, 26, doi: https://doi.org/10.1016/j.softx.2024.101673

E Wes Bethel, Mercy G Amankwah, Jan Balewski, Roel Van Beeumen, Daan Camps, Daniel Huang, Talita Perciano, "Quantum computing and visualization: A disruptive technological change ahead", Journal, November 6, 2023, 43, doi: https://doi.org/10.1109/MCG.2023.3316932

GM Wallace, Z Bai, N Bertelli, EW Bethel, T Perciano, S Shiraiwa, JC Wright, "Towards Fast, Accurate Predictions of RF Simulations via Data-driven Modeling: Forward and Lateral Models", Conference, AIP Publishing, August 1, 2023, 2984, doi: https://doi.org/10.1063/5.0162422

Gregory Wallace, Zhe Bai, Robbie Sadre, Talita Perciano, Nicola Bertelli, Syun'ichi Shiraiwa, Wes Bethel, John Wright, "Towards fast and accurate predictions of radio frequency power deposition and current profile via data-driven modelling: applications to lower hybrid current drive", Journal of Plasma Physics, August 18, 2022, 88:895880401, doi: 10.1017/S0022377822000708

M. G. Amankwah, D. Camps, E. W. Bethel, R. Van Beeumen, T. Perciano, "Quantum pixel representations and compression for N-dimensional images", Nature Scientific Reports, May 11, 2022, 12:7712, doi: 10.1038/s41598-022-11024-y

S. Zhang, R. Sadre, B. A. Legg, H. Pyles, T. Perciano, E. W. Bethel, D. Baker, O. Rübel, J. J. D. Yoreo, "Rotational dynamics and transition mechanisms of surface-adsorbed proteins", Proceedings of the National Academy of Sciences, April 11, 2022, 119:e202024211, doi: 10.1073/pnas.2020242119

M. Avaylon, R. Sadre, Z. Bai, T. Perciano, "Adaptable Deep Learning and Probabilistic Graphical Model System for Semantic Segmentation", Advances in Artificial Intelligence and Machine Learnin, March 31, 2022, 2:288--302, doi: 10.54364/AAIML.2022.1119

C Varadharajan, AP Appling, B Arora, DS Christianson, VC Hendrix, V Kumar, AR Lima, J Müller, S Oliver, M Ombadi, T Perciano, JM Sadler, H Weierbach, JD Willard, Z Xu, J Zwart, "Can machine learning accelerate process understanding and decision-relevant predictions of river water quality?", Hydrological Processes, January 1, 2022, 36, doi: 10.1002/hyp.14565

V. Dumont, C. Garner, A. Trivedi, C. Jones, V. Ganapati, J. Mueller, T. Perciano, M. Kiran, and M. Day, "HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization", 2021 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC), November 15, 2021,

E. W. Bethel, C. Heinemann, and T. Perciano, "Performance Tradeoffs in Shared-memory Platform Portable Implementations of a Stencil Kernel", Eurographics Symposium on Parallel Graphics and Visualization, June 14, 2021,

Li Zhou, Chao Yang, Weiguo Gao, Talita Perciano, Karen M. Davies, Nicholas K. Sauter, "Subcellular structure segmentation from cryo-electron tomograms via machine learning", PLOS Journal of Computational Biology, April 2, 2020, submitte, doi: doi: https://doi.org/10.1101/2020.04.09.034025

Stefano Marchesini, Anuradha Trivedi, Pablo Enfedaque, Talita Perciano, Dilworth Parkinson, "Sparse Matrix-Based HPC Tomography", Computational Science -- ICCS 2020, Cham, Springer International Publishing, 2020, 248--261, doi: 10.1007/978-3-030-50371-0_18

E. Wes Bethel, David Camp, Talita Perciano, Colleen Heinemann, "Performance Analysis of Traditional and Data-Parallel Primitive Implementations of Visualization and Analysis Kernels", Berkeley, CA, USA, 94720, 2020,

H Chang, J J Donatelli, P Enfedaque, G Freychet, M Haranczyk, A Hexemer, Z Hu, O Jain, H Krishnan, D Kumar, X Li, L Lin, M MacNeil, S Marchesini, X Mo, M Noack, K Pande, R Pandolfi, D Parkinson, D M Pelt, T Perciano, D A Shapiro, D Ushizima, C Yang, P H Zwart, J A Sethian, "Building Mathematics, Algorithms, and Software for Experimental Facilities", Handbook on Big Data and Machine Learning in the Physical Sciences, ( 2020) Pages: 189--240 doi: 10.1142/9789811204579_0012

Talita Perciano, Colleen Heinemann, David Camp, Brenton Lessley, E Wes Bethel, "Shared-Memory Parallel Probabilistic Graphical Modeling Optimization: Comparison of Threads, OpenMP, and Data-Parallel Primitives", High Performance Computing, Cham, Springer International Publishing, 2020, 127--145, doi: 10.1007/978-3-030-50743-5_7

B Lessley, T Perciano, C Heinemann, D Camp, H Childs, EW Bethel, "DPP-PMRF: Rethinking Optimization for a Probabilistic Graphical Model Using Data-Parallel Primitives", The 8th IEEE Symposium on Large Data Analysis and Visualization - LDAV 2018, 2018,

C Heinemann, T Perciano, D Ushizima, EW Bethel, "Distributed memory parallel Markov random fields using graph partitioning", Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017, 2018, 2018-Jan:3332--3341, doi: 10.1109/BigData.2017.8258318

RJ Pandolfi, DB Allan, E Arenholz, L Barroso-Luque, SI Campbell, TA Caswell, A Blair, F De Carlo, S Fackler, AP Fournier, G Freychet, M Fukuto, D Gürsoy, Z Jiang, H Krishnan, D Kumar, RJ Kline, R Li, C Liman, S Marchesini, A Mehta, AT N Diaye, DY Parkinson, H Parks, LA Pellouchoud, T Perciano, F Ren, S Sahoo, J Strzalka, D Sunday, CJ Tassone, D Ushizima, S Venkatakrishnan, KG Yager, P Zwart, JA Sethian, A Hexemer, "Xi-cam: a versatile interface for data visualization and analysis", Journal of Synchrotron Radiation, 2018, 25:1261--1270, doi: 10.1107/S1600577518005787

B Lessley, T Perciano, M Mathai, H Childs, EW Bethel, "Maximal clique enumeration with data-parallel primitives", 2017 IEEE 7th Symposium on Large Data Analysis and Visualization, LDAV 2017, 2017, 2017-Dec:16--25, doi: 10.1109/LDAV.2017.8231847

T Perciano, D Ushizima, H Krishnan, D Parkinson, N Larson, DM Pelt, W Bethel, F Zok, J Sethian, "Insight into 3D micro-CT data: Exploring segmentation algorithms through performance metrics", Journal of Synchrotron Radiation, 2017, 24:1065--1077, doi: 10.1107/S1600577517010955

Benedikt J Daurer, Hari Krishnan, Talita Perciano, Filipe RNC Maia, David A Shapiro, James A Sethian, Stefano Marchesini, "Nanosurveyor: a framework for real-time data processing", Advanced structural and chemical imaging, 2017, 3:7,

T Perciano, D Ushizima, H Krishnan, D Parkinson, J Sethian, "FibriPy: A software environment for fiber analysis from 3D micro-computed tomography data", Advanced Materials - TechConnect Briefs 2017, 2017, 1:25--28,

DY Parkinson, DM Pelt, T Perciano, D Ushizima, H Krishnan, HS Barnard, AA MacDowell, J Sethian, "Machine learning for micro-tomography", Proceedings of SPIE - The International Society for Optical Engineering, 2017, 10391, doi: 10.1117/12.2274731

M Farmand, R Celestre, P Denes, ALD Kilcoyne, S Marchesini, H Padmore, T Tyliszczak, T Warwick, X Shi, J Lee, YS Yu, J Cabana, J Joseph, H Krishnan, T Perciano, FRNC Maia, DA Shapiro, "Near-edge X-ray refraction fine structure microscopy", Applied Physics Letters, 2017, 110, doi: 10.1063/1.4975377

T Perciano, DM Ushizima, EW Bethel, YD Mizrahi, D Parkinson, JA Sethian, "Reduced-complexity image segmentation under parallel Markov Random Field formulation using graph partitioning", Proceedings - International Conference on Image Processing, ICIP, 2016, 2016-Aug:1259--1263, doi: 10.1109/ICIP.2016.7532560

DM Ushizima, HA Bale, EW Bethel, P Ercius, BA Helms, H Krishnan, LT Grinberg, M Haranczyk, AA Macdowell, K Odziomek, DY Parkinson, T Perciano, RO Ritchie, C Yang, "IDEAL: Images Across Domains, Experiments, Algorithms and Learning", JOM, 2016, 68:2963--2972, doi: 10.1007/s11837-016-2098-4

SV Venkatakrishnan, KA Mohan, K Beattie, J Correa, E Dart, JR Deslippe, A Hexemer, H Krishnan, AA MacDowell, S Marchesini, SJ Patton, T Perciano, JA Sethian, R Stromsness, BL Tierney, CE Tull, D Ushizima, DY Parkinson, "Making advanced scientific algorithms and big scientific data management more accessible", IS and T International Symposium on Electronic Imaging Science and Technology, 2016, doi: 10.2352/ISSN.2470-1173.2016.19.COIMG-155

DY Parkinson, K Beattie, X Chen, J Correa, E Dart, BJ Daurer, JR Deslippe, A Hexemer, H Krishnan, AA Macdowell, FRNC Maia, S Marchesini, HA Padmore, SJ Patton, T Perciano, JA Sethian, D Shapiro, R Stromsness, N Tamura, BL Tierney, CE Tull, D Ushizima, "Real-time data-intensive computing", AIP Conference Proceedings, 2016, 1741, doi: 10.1063/1.4952921

S Marchesini, H Krishnan, BJ Daurer, DA Shapiro, T Perciano, JA Sethian, FRNC Maia, "SHARP: A distributed GPU-based ptychographic solver", Journal of Applied Crystallography, 2016, 49:1245--1252, doi: 10.1107/S1600576716008074

T Perciano, F Tupin, R Hirata, RM Cesar, "A two-level Markov random field for road network extraction and its application with optical, SAR, and multitemporal data", International Journal of Remote Sensing, 2016, 37:3584--3610, doi: 10.1080/01431161.2016.1201227

J Donatelli, M Haranczyk, A Hexemer, H Krishnan, X Li, L Lin, F Maia, S Marchesini, D Parkinson, T Perciano, D Shapiro, D Ushizima, C Yang, JA Sethian, "CAMERA: The Center for Advanced Mathematics for Energy Research Applications", Synchrotron Radiation News, 2015, 28:4--9, doi: 10.1080/08940886.2015.1013413

AW Wills, DJ Michalak, P Ercius, ER Rosenberg, T Perciano, D Ushizima, R Runser, BA Helms, "Block Copolymer Packing Limits and Interfacial Reconfigurability in the Assembly of Periodic Mesoporous Organosilicas", Advanced Functional Materials, 2015, 25:4120--4128, doi: 10.1002/adfm.201501059

D Ushizima, T Perciano, D Parkinson, "Fast detection of material deformation through structural dissimilarity", Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015, 2015, 2775--2781, doi: 10.1109/BigData.2015.7364080

D Ushizima, T Perciano, H Krishnan, B Loring, H Bale, D Parkinson, J Sethian, "Structure recognition from high resolution images of ceramic composites", Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014, 2014, 683--691, doi: 10.1109/BigData.2014.7004292

T Perciano, MW Urban, NDA Mascarenhas, M Fatemi, AC Frery, GT Silva, "Deconvolution of vibroacoustic images using a simulation model based on a three dimensional point spread function", Ultrasonics, 2013, 53:36--44, doi: 10.1016/j.ultras.2012.03.011

A. C. Frery, T. Perciano, Introduction to Image Processing Using R: Learning by Examples, (2013)

Hector G. Martin, Tijana Radivojevic, Jeremy Zucker, Kristofer Bouchard, Jess Sustarich, Sean Peisert, Dan Arnold, Nathan Hillson, Gyorgy Babnigg, Jose M. Marti, Christopher J. Mungall, Gregg T. Beckham, Lucas Waldburger, James Carothers, ShivShankar Sundaram, Deb Agarwal, Blake A. Simmons, Tyler Backman, Deepanwita Banerjee, Deepti Tanjore, Lavanya Ramakrishnan, Anup Singh, "Perspectives for Self-Driving Labs in Synthetic Biology", Current Opinion in Biotechnology, February 2023, doi: 10.1016/j.copbio.2022.102881

MB Simmonds, WJ Riley, DA Agarwal, X Chen, S Cholia, R Crystal-Ornelas, ET Coon, D Dwivedi, VC Hendrix, M Huang, A Jan, Z Kakalia, J Kumar, CD Koven, L Li, M Melara, L Ramakrishnan, DM Ricciuto, AP Walker, W Zhi, Q Zhu, C Varadharajan, Guidelines for Publicly Archiving Terrestrial Model Data to Enhance Usability, Intercomparison, and Synthesis, Data Science Journal, 2022, doi: 10.5334/dsj-2022-003

Devarshi Ghoshal, Ludovico Bianchi, Abdelilah Essiari, Drew Paine, Sarah Poon, Michael Beach, Alpha N'Diaye, Patrick Huck, Lavanya Ramakrishnan, "Science Capsule: Towards Sharing and Reproducibility of Scientiﬁc Workﬂows", 2021 IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS), November 15, 2021, doi: 10.1109/WORKS54523.2021.00014

Workflows are increasingly processing large volumes of data from scientific instruments, experiments and sensors. These workflows often consist of complex data processing and analysis steps that might include a diverse ecosystem of tools and also often involve human-in-the-loop steps. Sharing and reproducing these workflows with collaborators and the larger community is critical but hard to do without the entire context of the workflow including user notes and execution environment. In this paper, we describe Science Capsule, which is a framework to capture, share, and reproduce scientific workflows. Science Capsule captures, manages and represents both computational and human elements of a workflow. It automatically captures and processes events associated with the execution and data life cycle of workflows, and lets users add other types and forms of scientific artifacts. Science Capsule also allows users to create `workflow snapshots' that keep track of the different versions of a workflow and their lineage, allowing scientists to incrementally share and extend workflows between users. Our results show that Science Capsule is capable of processing and organizing events in near real-time for high-throughput experimental and data analysis workflows without incurring any significant performance overheads.

Drew Paine, Sarah Poon, Lavanya Ramakrishnan, "Investigating User Experiences with Data Abstractions on High Performance Computing Systems", June 29, 2021, LBNL LBNL-2001374,

Scientific exploration generates expanding volumes of data that commonly require High Performance Computing (HPC) systems to facilitate research. HPC systems are complex ecosystems of hardware and software that frequently are not user friendly. The Usable Data Abstractions (UDA) project set out to build usable software for scientific workflows in HPC environments by undertaking multiple rounds of qualitative user research. Qualitative research investigates how individuals accomplish their work and our interview-based study surfaced a variety of insights about the experiences of working in and with HPC ecosystems. This report examines multiple facets to the experiences of scientists and developers using and supporting HPC systems. We discuss how stakeholders grasp the design and configuration of these systems, the impacts of abstraction layers on their ability to successfully do work, and the varied perceptions of time that shape this work. Examining the adoption of the Cori HPC at NERSC we explore the anticipations and lived experiences of users interacting with this system's novel storage feature, the Burst Buffer. We present lessons learned from across these insights to illustrate just some of the challenges HPC facilities and their stakeholders need to account for when procuring and supporting these essential scientific resources to ensure their usability and utility to a variety of scientific practices.

Devarshi Ghoshal, Drew Paine, Gilberto Pastorello, Abdelrahman Elbashandy, Dan Gunter, Oluwamayowa Amusat, Lavanya Ramakrishnan, "Experiences with Reproducibility: Case Studies from Scientific Workflows", (P-RECS'21) Proceedings of the 4th International Workshop on Practical Reproducible Evaluation of Computer Systems, ACM, June 21, 2021, doi: 10.1145/3456287.3465478

Reproducible research is becoming essential for science to ensure transparency and for building trust. Additionally, reproducibility provides the cornerstone for sharing of methodology that can improve efficiency. Although several tools and studies focus on computational reproducibility, we need a better understanding about the gaps, issues, and challenges for enabling reproducibility of scientific results beyond the computational stages of a scientific pipeline. In this paper, we present five different case studies that highlight the reproducibility needs and challenges under various system and environmental conditions. Through the case studies, we present our experiences in reproducing different types of data and methods that exist in an experimental or analysis pipeline. We examine the human aspects of reproducibility while highlighting the things that worked, that did not work, and that could have worked better for each of the cases. Our experiences capture a wide range of scenarios and are applicable to a much broader audience who aim to integrate reproducibility in their everyday pipelines.

Devarshi Ghoshal, Ludovico Bianchi, Abdelilah Essiari, Michael Beach, Drew Paine, Lavanya Ramakrishnan, "Science Capsule - Capturing the Data Life Cycle", Journal of Open Source Software, 2021, 6:2484, doi: 10.21105/joss.02484

D. A. Agarwal, J. Damerow, C. Varadharajan, D. S. Christianson, G. Z. Pastorello, Y.-W. Cheah, L. Ramakrishnan, "Balancing the needs of consumers and producers for scientific data collections", Ecological Informatics, 2021, 62:101251, doi: 10.1016/j.ecoinf.2021.101251

J Müller, B Faybishenko, D Agarwal, S Bailey, C Jiang, Y Ryu, C Tull, L Ramakrishnan, Assessing data change in scientific datasets, Concurrency and Computation: Practice and Experience, 2021, doi: 10.1002/cpe.6245

Drew Paine, Lavanya Ramakrishnan, "Understanding Interactive and Reproducible Computing With Jupyter Tools at Facilities", LBNL Technical Report, October 31, 2020, LBNL LBNL-2001355,

Increasingly Jupyter tools are being adopted and incorporated into High Performance Computing (HPC) and scientific user facilities. Adopting Jupyter tools enables more interactive and reproducible computational work at facilities across data life cycles. As the volume, variety, and scope of data grow, scientists need to be able to analyze and share results in user friendly ways. Human-centered research highlights design challenges around computational notebooks, and our qualitative user study shifts focus to better characterize how Jupyter tools are being used in HPC and science user facilities today. We conducted twenty-nine interviews, and obtained 103 survey responses from NERSC Jupyter users, to better understand the increasing role of interactive computing tools in DOE sponsored scientific work. We examine a range of issues that emerge using and supporting Jupyter in HPC ecosystems, including: how Jupyter is being used by scientists in HPC and user facility ecosystems; how facilities are purposefully supporting Jupyter in their ecosystems; feedback NERSC users have about the facility’s deployment, and, discuss features NERSC indicated would be helpful. We offer a variety of takeaways for staff supporting Jupyter at facilities, Project Jupyter and related open source communities, and funding agencies supporting interactive computing work.

Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, "Experiences with a Flexible User Research Process to Build Data Change Tools", Journal of Open Research Software, September 1, 2020, doi: 10.5334/jors.284

Scientific software development processes are understood to be distinct from commercial software development practices due to uncertain and evolving states of scientific knowledge. Sustaining these software products is a recognized challenge, but under-examined is the usability and usefulness of such tools to their scientific end users. User research is a well-established set of techniques (e.g., interviews, mockups, usability tests) applied in commercial software projects to develop foundational, generative, and evaluative insights about products and the people who use them. Currently these approaches are not commonly applied and discussed in scientific software development work. The use of user research techniques in scientific environments can be challenging due to the nascent, fluid problem spaces of scientific work, varying scope of projects and their user communities, and funding/economic constraints on projects.

In this paper, we reflect on our experiences undertaking a multi-method user research process in the Deduce project. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists make decisions around data change. There is a lack of common terminology since the concept of systematically measuring and managing data change is under explored in scientific environments. To bridge this gap we conducted user research that focuses on user practices, needs, and motivations to help us design and develop metrics and tools for data change. This paper contributes reflections and the lessons we have learned from our experiences. We offer key takeaways for scientific software project teams to effectively and flexibly incorporate similar processes into their projects.

Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, "Investigating Scientific Data Change with User Research Methods", August 20, 2020, LBNL LBNL-2001347,

Scientific datasets are continually expanding and changing due to fluctuations with instruments, quality assessment and quality control processes, and modifications to software pipelines. Datasets include minimal information about these changes or their effects requiring scientists manually assess modifications through a number of labor intensive and ad-hoc steps. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists systematically identify and make decisions around data changes. Currently, there is a lack of understanding, and common practices, for identifying and evaluating changes in datasets since systematically measuring and managing data change is under explored in scientific work. We are conducting user research to address this need by exploring scientist's conceptualizations, behaviors, needs, and motivations when dealing with changing datasets. Our user research utilizes multiple methods to produce foundational, generative insights and evaluate research products produced by our team. In this paper, we detail our user research process and outline our findings about data change that emerge from our studies. Our work illustrates how scientific software teams can push beyond just usability testing user interfaces or tools to better probe the underlying ideas they are developing solutions to address.

P. Linton, W. Melodia, A. Lazar, D. Agarwal, L. Bianchi, D. Ghoshal, K. Wu, G. Pastorello, L. Ramakrishnan, "Identifying Time Series Similarity in Large-Scale Earth System Datasets", The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19), 2019,

Drew Paine, Lavanya Ramakrishnan, "Surfacing Data Change in Scientiﬁc Work", iConference 2019, Springer Verlag, March 19, 2019, 15-26, doi: 10.1007/978-3-030-15742-5_2

Payton A Linton, William M Melodia, Alina Lazar, Deborah Agarwal, Ludovico Bianchi, Devarshi Ghoshal, Kesheng Wu, Gilberto Pastorello, Lavanya Ramakrishnan, "Identifying Time Series Similarity in Large-Scale Earth System Datasets", 2019,

Payton Linton, William Melodia, Alina Lazar, Deborah Agarwal, Ludovico Bianchi, Devarshi Ghoshal, Gilberto Pastorello, Lavanya Ramakrishnan, Kesheng Wu, Understanding Data Similarity in Large-Scale Scientific Datasets, 2019 IEEE International Conference on Big Data (Big Data), Pages: 4525--4531 2019,

Cheah You-Wei, Drew Paine, Devarshi Ghoshal, Lavanya Ramakrishnan, Bringing Data Science to Qualitative Analysis, 2018 IEEE 14th International Conference on e-Science, Pages: 325-326 2018, doi: 10.1109/eScience.2018.00076

D Ghoshal, L Ramakrishnan, D Agarwal, "Dac-Man: Data Change Management for Scientific Datasets on HPC Systems", SC ’18, Piscataway, NJ, USA, IEEE Press, 2018, 72:1--72:1,

GP Rodrigo, M Henderson, GH Weber, C Ophus, K Antypas, L Ramakrishnan, "ScienceSearch: Enabling Search through Automatic Metadata Generation", 2018 IEEE 14th International Conference on e-Science (e-Science), IEEE, 2018, doi: 10.1109/escience.2018.00025

Gunther H. Weber, Colin Ophus, Lavanya Ramakrishnan, "Automated Labeling of Electron Microscopy Images Using Deep Learning", Proc. IEEE/ACM Machine Learning in HPC Environments (MLHPC), 2018, 26--36, doi: 10.1109/MLHPC.2018.8638633

S Swaid, M Maat, H Krishnan, D Ghoshal, L Ramakrishnan, "Usability heuristic evaluation of scientific data analysis and visualization tools", Advances in Intelligent Systems and Computing, 2018, 607:471--482, doi: 10.1007/978-3-319-60492-3_45

E Deelman, T Peterka, I Altintas, CD Carothers, KK van Dam, K Moreland, M Parashar, L Ramakrishnan, M Taufer, J Vetter, "The future of scientific workflows", International Journal of High Performance Computing Applications, 2018, 32:159--175, doi: 10.1177/1094342017704893

GP Rodrigo, PO Östberg, E Elmroth, K Antypas, R Gerber, L Ramakrishnan, "Towards understanding HPC users and systems: A NERSC case study", Journal of Parallel and Distributed Computing, 2018, 111:206--221, doi: 10.1016/j.jpdc.2017.09.002

GP Rodrigo, E Elmroth, P-O Ostberg, L Ramakrishnan, "ScSF: A Scheduling Simulation Framework", Job Scheduling Strategies for Parallel Processing, Cham, Springer International Publishing, 2018, 152--173,

Devarshi Ghoshal, Lavanya Ramakrishnan, "MaDaTS: Managing Data on Tiered Storage for Scientific Workflows", Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing (HPDC '17), ACM, 2017, 41--52, doi: 10.1145/3078597.3078611

L Ramakrishnan, D Gunter, "Ten principles for creating usable software for science", Proceedings - 13th IEEE International Conference on eScience, eScience 2017, 2017, 210--218, doi: 10.1109/eScience.2017.34

GH Weber, MS Bandstra, DH Chivers, HH Elgammal, V Hendrix, J Kua, JS Maltz, K Muriki, Y Ong, K Song, MJ Quinlan, L Ramakrishnan, BJ Quiter, "Web-based visual data exploration for improved radiological source detection", Concurrency Computation, 2017, 29, doi: 10.1002/cpe.4203

D Ghoshal, V Hendrix, W Fox, S Balasubhramanian, L Ramakrishnan, "FRIEDA: Flexible Robust Intelligent Elastic Data Management Framework", The Journal of Open Source Software, 2017, 2:164--164, doi: 10.21105/joss.00164

W Fox, D Ghoshal, A Souza, GP Rodrigo, L Ramakrishnan, "E-HPC: A library for elastic resource management in HPC environments", Proceedings of WORKS 2017: 12th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2017, doi: 10.1145/3150994.3150996

GP Rodrigo, E Elmroth, P-O Östberg, L Ramakrishnan, "Enabling Workflow-Aware Scheduling on HPC Systems", Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing - HPDC 17, ACM Press, 2017, doi: 10.1145/3078597.3078604

V Hendrix, J Fox, D Ghoshal, L Ramakrishnan, "Tigres Workflow Library: Supporting Scientific Pipelines on HPC Systems", Proceedings - 2016 16th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2016, 2016, 146--155, doi: 10.1109/CCGrid.2016.54

M Verma, JB Fisher, K Mallick, Y Ryu, H Kobayashi, A Guillaume, G Moore, L Ramakrishnan, V Hendrix, S Wolf, M Sikka, G Kiely, G Wohlfahrt, B Gielen, O Roupsard, P Toscano, A Arain, A Cescatti, "Global surface net-radiation at 5 km from MODIS Terra", Remote Sensing, 2016, 8, doi: 10.3390/rs8090739

CS Daley, D Ghoshal, GK Lockwood, S Dosanjh, L Ramakrishnan, NJ Wright, "Performance characterization of scientific workflows for the optimal use of Burst Buffers", CEUR Workshop Proceedings, 2016, 1800:69--73,

NC Chen, SS Poon, L Ramakrishnan, CR Aragon, "Considering time in designing Large-Scale systems for scientific computing", Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW, 2016, 27:1535--1547, doi: 10.1145/2818048.2819988

GP Rodrigo, P-O Ostberg, E Elmroth, K Antypas, R Gerber, L Ramakrishnan, IEEE, "Towards Understanding Job Heterogeneity in HPC: A NERSC Case Study", 2016 16TH IEEE/ACM INTERNATIONAL SYMPOSIUM ON CLUSTER, CLOUD AND GRID COMPUTING (CCGRID), 2016, 521--526, doi: 10.1109/CCGrid.2016.32

E Dede, B Sendir, P Kuzlu, J Weachock, M Govindaraju, L Ramakrishnan, "Processing Cassandra Datasets with Hadoop-Streaming Based Approaches", IEEE Transactions on Services Computing, 2016, 9:46--58, doi: 10.1109/tsc.2015.2444838

GP Rodrigo Álvarez, P-O Östberg, E Elmroth, L Ramakrishnan, A2L2, Proceedings of the 8th International Workshop on Virtualization Technologies in Distributed Computing - VTDC 15, 2015, doi: 10.1145/2755979.2755983

C Daley, L Ramakrishnan, S Dosanjh, N Wright, "Analyses of Scientific Workflows for Effective Use of Future Architectures", The 6th International Workshop on Big Data Analytics: Challenges, and Opportunities (BDAC-15), 2015 at SC, 2015,

E Feller, L Ramakrishnan, C Morin, "Performance and energy efficiency of big data applications in cloud environments: A Hadoop case study", Journal of Parallel and Distributed Computing, 2015, 79-80:80--89, doi: 10.1016/j.jpdc.2015.01.001

GP Rodrigo Álvarez, P-O Östberg, E Elmroth, K Antypas, R Gerber, L Ramakrishnan, "HPC System Lifetime Story: Workload Characterization and Evolutionary Analyses on NERSC Systems", Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing, 2015, 57--60,

H Sim, Y Kim, SS Vazhkudai, D Tiwari, A Anwar, AR Butt, L Ramakrishnan, ACM, "AnalyzeThis: An Analysis Workflow-Aware Storage System", PROCEEDINGS OF SC15: THE INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS, 2015, doi: 10.1145/2807591.2807622

V Hendrix, L Ramakrishnan, Y Ryu, C Van Ingen, KR Jackson, D Agarwal, "CAMP: Community access MODIS pipeline", Future Generation Computer Systems, 2014, 36:418--429, doi: 10.1016/j.future.2013.09.023

Lavanya Ramakrishnan, Sarah S. Poon, Val C. Hendrix, Dan K. Gunter, Gilberto Z. Pastorello, Deb A. Agarwal, "Experiences with User-Centered Design for the Tigres Workflow API", Proceedings of the 10th IEEE International Conference on e-Science (e-Science 2014), Guaruja, Brazil, 2014, doi: 10.1109/eScience.2014.56

JR Balderrama, M Simonin, L Ramakrishnan, V Hendrix, C Morin, D Agarwal, C Tedeschi, "Combining workflow templates with a shared space-based execution model", Proceedings of WORKS 2014: The 9th Workshop on Workflows in Support of Large-Scale Science - held in conjunction with SC 2014: The International Conference for High Performance Computing, Networking, Storage and Analysis, 2014, 50--58, doi: 10.1109/WORKS.2014.14

L Ramakrishnan, D Ghoshal, V Hendrix, E Feller, P Mantha, C Morin, "Storage and Data Life Cycle Management in Cloud Environments with FRIEDA.", Cloud Computing for Data-Intensive Applications, (Springer: 2014) Pages: 357--378

Elif Dede, Zacharia Fadika, Madhusudhan Govindaraju, Lavanya Ramakrishnan, "Benchmarking MapReduce Implementations Under Different Application Scenarios", Future Generation Computer Systems, 2014,

Elif Dede, Zacharia Fadika, Madhusudhan Govindaraju, Lavanya Ramakrishnan, "MARIANE: Using MApReduce In HPC Environments", Future Generation Computer Systems, 2014,

Elif Dede, Bedri Sendir, Pinar Kuzlu, Madhusudhan Govindaraju, Lavanya Ramakrishnan, "A Processing Pipeline for Cassandra Datasets Based on Hadoop Streaming", IEEE International Congress on Big Data, 2014,

Tonglin Hawk, Ioan Raicu, Lavanya Ramakrishnan, "Scalable State Management for Scientific Applications in the Cloud", IEEE International Congress on Big Data, 2014,

Devarshi Ghoshal, Lavanya Ramakrishnan, "FRIEDA: Flexible Robust Intelligent Elastic Data Management in Cloud Environments", 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), IEEE, 2013, 1096--1105, doi: 10.1109/SC.Companion.2012.132

Elif Dede, Madhusudhan Govindaraju, Daniel Gunter, Richard Canon, Lavanya Ramakrishnan, "Semi-Structured Data Analysis using MongoDB and MapReduce: A Performance Evaluation", Proceedings of the 4th international workshop on Scientific cloud computing, 2013,

E. Masanet, A. Shehabi, L. Ramakrishnan, J. Liang, X. Ma, B. Walker, V. Hendrix, P Mantha, "The Energy Efficiency Potential of Cloud-Based Software: A U.S.Case Study", 2013, LBNL 6298E,

Lavanya Ramakrishnan, Adam Scovel, Iwona Sakrejda, Susan Coghlan, Shane Canon, Anping Liu, Devarshi Ghoshal, Krishna Muriki, Nicholas J. Wright, "Magellan - A Testbed to Explore Cloud Computing for Science", On the Road to Exascale Computing: Contemporary Architectures in High Performance Computing, (Chapman & Hall/CRC Press: 2013)

Eugen Feller, Lavanya Ramakrishnan, Christine Morin, "On the Performance and Energy Efficiency of Hadoop Deployment Models", The IEEE International Conference on Big Data 2013 (IEEE BigData 2013), Santa Clara, U.S.A, 2013,

Lavanya Ramakrishnan, Iwona Sakrejda, Richard Shane Canon and Nicholas Wright, "CAMP", On the Road to Exascale Computing: Contemporary Architectures in High Performance Computing, (Chapman & Hall/CRC Press: 2013)

You-Wei Cheah, Richard Canon, Beth Plale, Lavanya Ramakrishnan, "Milieu: Provenance Collection and Query Framework for High Performance Computing Systems", IEEE Big Data Congress, 2013,

Zacharia Fadika, Madhusudhan Govindaraju, Shane Canon, Lavanya Ramakrishnan, "Evaluting Hadoop for Data-Intensive Scientific Operations", IEEE Cloud Computing, 2012,

V Hendrix, J Li, K Jackson, L Ramakrishnan, Y Ryu, K Beattie, C Morin, D Skinner, C van Ingen, D Agarwal, "Community Access to MODIS Satellite Reprojection and Reduction Pipeline and Data Sets", AGU Fall Meeting, 2012,

D Gunter, S Cholia, A Jain, M Kocher, K Persson, L Ramakrishnan, SP Ong, G Ceder, "Community accessible datastore of high-throughput calculations: Experiences from the materials project", Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, 2012, 1244--1251, doi: 10.1109/SC.Companion.2012.150

Elif Dede, Zacharia Fadika, Jessica Hartog, Modhusudhan Govindaraju, Lavanya Ramakrishnan, Daniel Gunter, Richard Shane Canon, "MARISSA: MApReduce Implementation for Streaming Science Applications", IEEE eScience Conference, 2012,

Lavanya Ramakrishnan, Richard Shane Canon, Krishna Muriki, Iwona Sakrejda, Nicholas J. Wright, "Evaluating Interconnect and Virtualization Performance for High Performance Computing", Special Issue of ACM Performance Evaluation Review, 2012, 40(2),

Devarshi Ghoshal, Richard Shane Canon, Lavanya Ramakrishnan, "I/O Performance of Virtualized Cloud Environments", Proceedings of the Second International Workshop on Data Intensive Computing in the Clouds (DataCloud-SC '11), ACM, 2011, 71--80, doi: 10.1145/2087522.2087535

Elif Dede, Madhusudan Govindaraju, Daniel Gunter, Lavanya Ramakrishnan, "Riding the Elephant: Managing Ensembles with Hadoop", 4th Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS), 2011,

Zacharia Fadika, Elif Dede, Madhusudhan Govindaraju, Lavanya Ramakrishnan, "Benchmarking MapReduce Implementations for Application Usage Scenarios", Grid 2011: 12th IEEE/ACM International Conference on Grid Computing, Lyon Conference Centre, France, IEEE Computer Society, 2011, 1-8, doi: http://grid2011.mnm-team.org/?page_id138

Zacharia Fadika, Elif Dede, Madhusudhan Govindaraju, Lavanya Ramakrishnan, "MARIANE: MApReduce Implementation Adapted for HPC Environments", Grid 2011: 12th IEEE/ACM International Conference on Grid Computing, Lyon Conference Centre, France, IEEE Computer Society, 2011, 1-8, doi: http://grid2011.mnm-team.org/?page_id138

You-Wei Cheah, Beth Plale, Joey Kendall-Morwick, David Leake, Lavanya Ramakrishnan, "A Noisy 10GB Provenance Database", Second International Workshop on Traceability and Compliance of Semi-Structured Processes (TC4SP2011), 2011,

Lavanya Ramakrishnan, Piotr T. Zbiegel, Scott Campbell, Rick Bradshaw, Richard Shane Canon, Susan Coghlan, Iwona Sakrejda, Narayan Desai, Tina Declerck, Anping Liu, "Magellan: Experiences from a Science Cloud", ScienceCloud 11, New York, NY, USA, ACM, 2011, 49--58, doi: http://doi.acm.org/10.1145/1996109.1996119

Lavanya Ramakrishnan, Dennis Gannon, Jeffrey Chase, Daniel Nurmi, Rich Wolski, "Deadline-Sensitive Workflow Orchestration Without Explicit Resource Control", Journal of Parallel and Distributed Computing, 2011,

Keith R. Jackson, Krishna Muriki, Lavanya Ramakrishnan, Karl J. Runge, Rollin C. Thomas, "Performance and cost analysis of the Supernova factory on the Amazon AWS cloud", Sci. Program., 2011, 19:107--119,

Keith Jackson, Lavanya Ramakrishnan, Rollin Thomas, Karl J. Runge, "Seeking Supernovae in the Clouds: A Performance Study", 1st Workshop on Scientific Cloud Computing, co-located with ACM HPDC 2010 (High Performance Distributed Computing), Chicago, IL, 2010,

Lavanya Ramakrishnan, Keith Jackson, Shane Canon, Shreyas Cholia, John Shalf, "Defining Future Platform Requirements for e-Science Cloud (Position paper)", ACM Symposium on Cloud Computing 2010 (ACM SOCC 2010), Indianapolis, Indiana, 2010,

Lavanya Ramakrishnan, Dennis Gannon, Beth Plale, "WORKEM: Representing and Emulating Distributed Scientific Workflow Execution State", 10th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2010), Melbourne Australia, 2010,

Y Simmhan, E Soroush, C Van Ingen, D Agarwal, L Ramakrishnan, "BReW: Blackbox resource selection for e-Science workflows", 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010, 2010, doi: 10.1109/WORKS.2010.5671857

L Ramakrishnan, C Guok, K Jackson, E Kissel, DM Swany, D Agarwal, "On-demand overlay networks for large scientific data transfers", CCGrid 2010 - 10th IEEE/ACM International Conference on Cluster, Cloud, and Grid Computing, 2010, 359--367, doi: 10.1109/CCGRID.2010.82

Lavanya Ramakrishnan, Beth Plale, "A Multi-Dimensional Classification Model for Scientific Workflow Characteristics", Workshop on Workflow Approaches to New Data-centric Science, Indianapolis, Indiana, 2010,

Yogesh Simmhan, Lavanya Ramakrishnan, "Comparison of Resource Platform Selection Approaches for Scientific Workflows", 1st Workshop on Scientific Cloud Computing, co-located with ACM HPDC 2010 (High Performance Distributed Computing), Chicago, Illinois, 2010,

KR Jackson, L Ramakrishnan, K Muriki, S Canon, S Cholia, J Shalf, HJ Wasserman, NJ Wright, "Performance analysis of high performance computing applications on the Amazon Web Services cloud", Proceedings - 2nd IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2010, 2010, 159--168, doi: 10.1109/CloudCom.2010.69

Lavanya Ramakrishnan, Daniel Nurmi, Anirban Mandal, Charles Koelbel, Dennis Gannon, T. Mark Huang, Yang-Suk Kee, Graziano Oberteli, Kiran Thyagaraja, Rich Wolski, Asim Yarkhan, Dmitrii Zagorodnov, "VGrADS: Enabling e-Science Workflows on Grids and Clouds with Fault Tolerance", Proceedings of the ACM/IEEE SC2009 Conference on High Performance Computing, Networking, Storage and Analysis, Portland, Oregon, Portland, Oregon, 2009,

Lavanya Ramakrishnan, Daniel A. Reed, "Predictable Quality of Service Atop Degradable Distributed Systems", Journal of Cluster Computing, 2009,

Lavanya Ramakrishnan, Daniel A. Reed, "Performability Modeling for Scheduling and Fault Tolerance Strategies for Scientific Workflows", ACM/IEEE International Symposium on High Performance Distributed Computing, Boston Massachusetts, 2008,

Robert J. Fowler, Todd Gamblin, Gopi Kandaswamy, Anirban Mandal, Allan K. Porterfield, Lavanya Ramakrishnan, Daniel A. Reed, "Challenges of Scale: When All Computing Becomes Grid Computing", High Performance Computing and Grids in Action, (IOS Press: 2008)

Lavanya Ramakrishnan, Yogesh Simmhan, Beth Plale, "Realization of Dynamically Adaptive Weather Analysis and Forecasting in LEAD", In Dynamic Data Driven Applications Systems Workshop (DDDAS) in conjunction with ICCS (Invited), Beijing, China, 2007,

Lavanya Ramakrishnan, Laura Grit, Adriana Iamnitchi, David Irwin, Aydan Yumerefendi, Jeff Chase, "Toward a Doctrine of Containment: Grid Hosting with Adaptive Resource Control", Proceedings of the ACM/IEEE SC2006 Conference on High Performance Computing, Networking, Storage and Analysis, Tampa, Florida, Tampa, Florida, 2006,

Lavanya Ramakrishnan, Mark S.C Reed, Jeffrey L. Tilson, Daniel A. Reed, "Grid Portals for Bioinformatics", Second International Workshop on Grid Computing Environments (GCE), Held in conjunction with ACM/IEEE Conference for High Performance Computing, Networking, Storage and Analysis, Tampa, Florida, 2006,

Lavanya Ramakrishnan, Brian O. Blanton, Howard M. Lander, Richard A. Luettich, Jr, Daniel A. Reed, Steven R. Thorpe, "Real-time Storm Surge Ensemble Modeling in a Grid Environment", Second International Workshop on Grid Computing Environments (GCE), Held in conjunction ACM/IEEE Conference for High Performance Computing, Networking, Storage and Analysis, Tampa, Florida, 2006,

Kelvin K. Droegemeier, Dennis Gannon, Daniel Reed, Beth Plale, Jay Alameda, Tom Baltzer, Keith Brewster, Richard Clark, Ben Domenico, Sara Graves, Everette Joseph, Donald Murray, Rahul Ramachandran, Mohan Ramamurthy, Lavanya Ramakrishnan, John A. Rushing, Daniel Weber, Robert Wilhelmson, Anne Wilson, Ming Xue, Sepideh Yalda, "Service-Oriented Environments for Dynamically Interacting with Mesoscale Weather", Computing in Science and Engg., 2005, 7:12--29, doi: http://dx.doi.org/10.1109/MCSE.2005.124

Lavanya Ramakrishnan, "Securing Next-Generation Grids", IT Professional., 2004, 6:34-39,

Timothy J. Smith, Lavanya Ramakrishnan, "Joint Policy Management and Auditing in Virtual Organizations.", Grid Computing, Phoenix, Arizona, 2003, 117-124,

Dennis Gannon, Rachana Ananthakrishnan, Sriram, Madhusudhan Govindaraju, Lavanya, Aleksander Slominski, "Grid Web Services and Application Factories", Grid Computing: Making the Global Infrastructure a Reality, (Wiley: 2003)

Lavanya Ramakrishnan, Helen Nell Rehn, Jay, Rachana Ananthakrishnan, Madhusudhan, Aleksander Slominski, Kay Connelly Von Welch, Dennis Gannon, Randall Bramley, Hampton, "An Authorization Framework for a Grid Based Common Architecture", Proceedings of the 3rd International Workshop on Grid Baltimore, Maryland, Springer Press, 2002, 169--180,

Dennis Gannon, Randall Bramley, Geoffrey Fox, Shava Smallen, Al Rossi, Rachana Ananthakrishnan, Felipe Bertrand, Ken Chiu, Matt Farrellee, Madhu Govindaraju, Sriram Krishnan, Lavanya Ramakrishnan, Yogesh Simmhan, Alek Slominski, Yu Ma, Caroline Olariu, Nicolas Rey-Cenvaz, "Programming the Grid: Distributed Software Components, P2P and Grid Web Services for Scientific Applications", Journal of Cluster Computing, 2002, 5:325--336,

S. Zhang, R. Sadre, B. A. Legg, H. Pyles, T. Perciano, E. W. Bethel, D. Baker, O. Rübel, J. J. D. Yoreo, "Rotational dynamics and transition mechanisms of surface-adsorbed proteins", Proceedings of the National Academy of Sciences, April 11, 2022, 119:e202024211, doi: 10.1073/pnas.2020242119

Hamish A. Carr, Gunther H. Weber, Christopher M. Sewell, Oliver R\ ubel, Patricia Fasel, James P. Ahrens, "Scalable Contour Tree Computation by Data Parallel Peak Pruning", Transactions on Visualization and Computer Graphics, 2021, 27:2437--2454, doi: 10.1109/TVCG.2019.2948616

Hamish Carr, Oliver Rübel, Gunther H. Weber, James Ahrens, "Optimization and Augmentation for Data Parallel Contour Trees", IEEE Transactions on Visualization and Computer Graphics, 2021, doi: 10.1109/TVCG.2021.3064385

Petar Hristov, Gunther H. Weber, Hamish A. Carr, Oliver R\ ubel, James P. Ahrens, "Data Parallel Hypersweeps for In Situ Topological Analysis", Proceedings of the 10th IEEE Symposium on Large Data Analysis and Visualization (LDAV), 2020, 12--21, doi: 10.1109/LDAV51489.2020.00008

Donghe Kang, Oliver Rübel, Suren Byna, Spyros Blanas, "Comparison of Array Management Library Performance - A Neuroscience Use Case", SC19 Poster, November 20, 2019,

Oliver Rübel, Andrew Tritt, Benjamin Dichter, Thomas Braun, Nicholas Cain, Nathan Clack, Thomas J. Davidson, Max Dougherty, Jean-Christophe Fillion-Robin, Nile Graddis, Michael Grauer, Justin T. Kiggins, Lawrence Niu, Doruk Ozturk, William Schroeder, Ivan Soltesz, Friedrich T. Sommer, Karel Svoboda, Lydia Ng, Loren M. Frank, Kristofer Bouchard, "NWB:N 2.0: An Accessible Data Standard for Neurophysiology", bioRxiv, January 17, 2019, doi: https://doi.org/10.1101/523035

O. Erbilgin, O. Rübel, K. B. Louie, M. Trinh, M. de Raad, T. Wildish, D. W. Udwary, C. A. Hoover,, "MAGI: A Bayesian-like method for metabolite, annotation, and gene integration", bioRxiv, December 19, 2018, doi: 10.1101/204362

K. E. Bouchard, J.B. Aimone, M. Chun, T. Dean, M. Denker, M. Diesmann, D. Donofrio, L.M. Frank, N. Kasthuri, C. Koch, O. Rübel, H. Simon, F. T. Sommer, Prabhat, "International Neuroscience Initiatives Through the Lens of High-Performance Computing", IEEE Computer, April 12, 2018, 51(4):50-59, doi: doi 10.1109/MC.2018.2141039

O. Rübel, B. P. Bowen, "BASTet: Shareable and reproducible analysis and visualization of mass spectrometry imaging data via OpenMSI", IEEE Transactions on Visualization and Computer Graphics, January 12, 2018, 24,no. 1, doi: 10.1109/TVCG.2017.2744479

M. de Raad, T. de Rond, O. Rübel, J. D. Keasling, T. R. Northen, B. P. Bowen, "OpenMSI Arrayed Analysis Toolkit: Analyzing Spatially Defined Samples Using Mass Spectrometry Imaging", ACS Analytical Chemistry, May 3, 2017, doi: 10.1021/acs.analchem.6b05004

O. Rübel, M.Dougherty, Prabhat, P. Denes, D. Conant, E. F. Chang, and K. Bouchard, "Methods for Specifying Scientic Data Standards and Modeling Relationships with Applications to Neuroscience," Frontiers in Neuroinformatics", Frontiers in Neuroinformatics, November 4, 2016, 10, doi: 10.3389/fninf.2016.00048

K. E. Bouchard, J, B. Aimone, M. Chun, T. Dean, M. Denker, M. Diesmann, D. D. Donofrio, L. M. Frank, N. Kasthuri, C. Koch, O. Rübel, H. D. Simon, F. T. Sommer, Prabhat, "High-Performance Computing in Neuroscience for Data-Driven Discovery, Integration, and Dissemination", Neuron, November 2, 2016, 92(3):628-631, doi: http://dx.doi.org/10.1016/j.neuron. 2016.10.035

Burlen Loring, Suren Byna, Prabhat, Junmin Gu, Hari Krishnan, Michael Wehner, and Oliver Ruebel, "TECA an Extreme Event Detection and Climate Analysis Package for High Performance Computing", The AMS (American Meteorological Society) 96th Annual Meeting, January 6, 2016,

O Rübel, B Loring, JL Vay, DP Grote, R Lehe, S Bulanov, H Vincenti, EW Bethel, "WarpIV: In Situ Visualization and Analysis of Ion Accelerator Simulations", IEEE Computer Graphics and Applications, 2016, 36:22--35, doi: 10.1109/MCG.2016.62

TA O Brien, WD Collins, K Kashinath, O Rübel, S Byna, J Gu, H Krishnan, PA Ullrich, Resolution dependence of precipitation statistical fidelity in hindcast simulations, Journal of Advances in Modeling Earth Systems, Pages: 976--990 2016, doi: 10.1002/2016MS000671

J. Yang, O. Rübel, M. Mahoney, Prabhat, B.P. Bowen, "Identifying important ions and positions in mass spectrometry imaging data using CUR matrix decompositions", ACS Analytical Chemistry, March 2015, doi: 10.1021/ac5040264

E Wes Bethel, Prabhat Prabhat, Suren Byna, Oliver R\ ubel, K John Wu, Michael Wehner, "Why high performance visual data analytics is both relevant and difficult", Visualization and Data Analysis 2013, January 2013, 8654:86540B, LBNL LBNL-6063E,

Download File: LBNL-6063E.pdf (pdf: 3.6 MB)

Kesheng Wu, E Bethel, Ming Gu, David Leinweber, Oliver R\ ubel, "A big data approach to analyzing market volatility", Algorithmic Finance, 2013, 2:241--267, LBNL LBNL-6382E,

Understanding the microstructure of the financial market requires the processing of a vast amount of data related to individual trades, and sometimes even multiple levels of quotes. Analyzing such a large volume of data requires tremendous computing power that is not easily available to financial academics and regulators. Fortunately, public funded High Performance Computing (HPC) power is widely available at the National Laboratories in the US. In this paper we demonstrate that the HPC resource and the techniques for data-intensive sciences can be used to greatly accelerate the computation of an early warning indicator called Volume-synchronized Probability of Informed trading (VPIN). The test data used in this study contains five and a half year's worth of trading data for about 100 most liquid futures contracts, includes about 3 billion trades, and takes 140GB as text files. By using (1) a more efficient file format for storing the trading records, (2) more effective data structures and algorithms, and (3) parallelizing the computations, we are able to explore 16,000 different ways of computing VPIN in less than 20 hours on a 32-core IBM DataPlex machine. Our test demonstrates that a modest computer is sufficient to monitor a vast number of trading activities in real-time -- an ability that could be valuable to regulators.

Our test results also confirm that VPIN is a strong predictor of liquidity-induced volatility. With appropriate parameter choices, the false positive rates are about 7% averaged over all the futures contracts in the test data set. More specifically, when VPIN values rise above a threshold (CDF > 0.99), the volatility in the subsequent time windows is higher than the average in 93% of the cases.

Kesheng Wu, Wes Bethel, Ming Gu, David, Oliver R\ ubel, Testing VPIN on Big Data, Available at SSRN 2318259, 2013,

O Rübel, A Greiner, S Cholia, K Louie, EW Bethel, TR Northen, BP Bowen, "OpenMSI: A high-performance web-based platform for mass spectrometry imaging", Analytical Chemistry, 2013, 85:10354--103, doi: 10.1021/ac402540a

Hank Childs, Eric Brugger, Brad Whitlock, Jeremy Meredith, Sean Ahern, David Pugmire, Kathleen Biagas, Mark Miller, Cyrus Harrison, Gunther H. Weber, Hari Krishnan, Thomas Fogal, Allen Sanderson, Christoph Garth, E. Wes Bethel, David Camp, Oliver Rubel, Marc Durant, Jean M. Favre, Paul Navratil, "VisIt: An End-User Tool For Visualizing and Analyzing Very Large Data", High Performance Visualization---Enabling Extreme-Scale Scientific Insight, ( October 2012) Pages: 357--372

Oliver Rübel, Cameron, G. R. Geddes, Min Chen, Estelle Cormier-Michel, and E. Wes Bethel, "Query-driven Analysis of Plasma-based Particel Acceleration Data", Poster Abstracts of IEEE VisWeek, October 2012,

E. Wes Bethel, David Camp, Hank Childs, Mark Howison, Hari Krishnan, Burlen Loring, Joerg Meyer, Prabhat, Oliver Ruebel, Daniela Ushizima, Gunther Weber, "Towards Exascale: High Performance Visualization and Analytics – Project Status Report. Technical Report", DOE Exascale Research Conference, April 2012,

Jihan Kim, Richard L. Martin, Oliver Rübel, Maciej Haranczyk & Berend Smit, "High-throughput Characterization of Porous Materials Using Graphics Processing Units", Journal of Chemical Theory and Computation, March 16, 2012, 8:1684–1693, LBNL 5409E, doi: 10.1021/ct200787v

We have developed a high-throughput graphics processing unit (GPU) code that can characterize a large database of crystalline porous materials. In our algorithm, the GPU is utilized to accelerate energy grid calculations, where the grid values represent interactions (i.e., Lennard-Jones + Coulomb potentials) between gas molecules (i.e., CH₄ and CO₂) and materials’ framework atoms. Using a parallel flood fill central processing unit (CPU) algorithm, inaccessible regions inside the framework structures are identified and blocked, based on their energy profiles. Finally, we compute the Henry coefficients and heats of adsorption through statistical Widom insertion Monte Carlo moves in the domain restricted to the accessible space. The code offers significant speedup over a single core CPU code and allows us to characterize a set of porous materials at least an order of magnitude larger than those considered in earlier studies. For structures selected from such a prescreening algorithm, full adsorption isotherms can be calculated by conducting multiple Grand Canonical Monte Carlo (GCMC) simulations concurrently within the GPU.

Soile V.E. Keränen, Oliver Rübel, David W. Knowles and Mark D. Biggin, "Computational modeling of cis-regulatory modules from 3D exprression data in Drosophila blastoderm atlas", Drosophila Genetics, March 2012,

Surendra Byna, Jerry Chou, Oliver Rubel, Homa Karimabadi, William S Daughter, Vadim Roytershteyn, E Wes Bethel, Mark Howison, Ke-Jou Hsu, Kuan-Wu Lin, others, "Parallel I/O, analysis, and visualization of a trillion particle simulation", SC 12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, January 2012, 1--12,

Oliver R\ ubel, Surendra Byna, Kesheng Wu, Fuyu Li, Michael Wehner, Wes Bethel, others, "Teca: A parallel toolkit for extreme climate analysis", Procedia Computer Science, Elsevier, January 2012, 9:866--876, LBNL 5352E,

We present TECA, a parallel toolkit for detecting extreme events in large climate datasets. Modern climate datasets expose parallelism across a number of dimensions: spatial locations, timesteps and ensemble members. We design TECA to exploit these modes of parallelism and demonstrate a prototype implementation for detecting and tracking three classes of extreme events: tropical cyclones, extra-tropical cyclones and atmospheric rivers. We process a modern TB-sized CAM5 simulation dataset with TECA, and demonstrate good runtime performance for the three case studies.

E. W. Bethel, Surendra Byna, Jerry Chou, Cormier-Michel, Cameron G. R. Geddes, Howison, Fuyu Li, Prabhat, Ji Qiang, R\ ubel, Rob D. Ryne, Michael Wehner, Wu, "Big Data Analysis and Visualization: What Do LINACS Tropical Storms Have In Common?", 11th International Computational Accelerator Physics ICAP 2012, Germany, 2012,

Allen R Sanderson, Brad Whitlock, H Childs, GH Weber, K Wu, others, "A system for query based analysis and visualization", January 2012, LBNL 5507E,

Download File: LBNL-5507E.pdf (pdf: 562 KB)

E. W. Bethel and D. Leinweber and O. Rubel and K. Wu, "Federal Market Information Technology in the Post Flash Crash Era: Roles of Supercomputing", The Journal of Trading, 2012, 7:9-24, LBNL 5263E, doi: 10.3905/jot.2012.7.2.009

E. Wes Bethel, David Leinweber, Oliver Rübel Kesheng Wu, Federal Market Information Technology in the Crash Era: Roles for Supercomputing, The Journal of Trading, Pages: 9--25 2012, doi: 10.3905/jot.2012.7.2.009

O. Rübel, S.V.E. Keränen, M.D. Biggin, D.W. Knowles, G.H. Weber, H. Hagen, B. Hamann, and E.W. Bethel, "Linking Advanced Visualization and MATLAB for the Analysis of 3D Gene Expression Data", Mathematics and Visualization, Visualization in Medicine and Life Sciences II, Progress and New Challenges, edited by L. Linsen and B. Hamann and H. Hagen and H.-C. Hege, (Springer Verlag: 2012) Pages: 267-285, LBNL 4891E,

Download File: LBNL-4891E.pdf (pdf: 15 MB)

Samuel Gerber, Oliver Rübel, Peer-Timo Bremer, Valerio Pascucci and Ross T. Whitaker, "Morse-Smale Regression", Journal of Computational and Graphical Statistics, January 2012, doi: 10.1080/10618600.2012.657132

Download File: MSR.pdf (pdf: 292 KB)

E. Wes Bethel, David Leinweber, Oliver Rübel, Kesheng Wu, "Federal Market Information Technology in the Post Flash Crash Era: Roles of Supercomputing", Workshop on High Performance Computational Finance at SC11, Seattle, WA, USA, November 2011, LBNL 5263E,

R. Ryne, B. Austin, J. Byrd, J. Corlett, E. Esarey, C. G. R. Geddes, W. Leemans, X. Li, Prabhat, J. Qiang, O. Rübel, J.-L. Vay, M. Venturini, K. Wu, B. Carlsten, D. Higdon and N. Yampolsky, "High Performance Computing in Accelerator Science: Past Successes, Future Challenges", Workshop on Data and Communications in Basic Energy Sciences: Creating a Pathway for Scientific Discovery, October 2011,

Jerry Chou, Mark Howison, Brian Austin, Kesheng Wu, Ji Qiang, E Wes Bethel, Arie Shoshani, Oliver R\ ubel, Rob D Ryne, "Parallel index and query for large scale data analysis", Proceedings of 2011 international conference for high performance computing, networking, storage and analysis, 2011, 1--11, LBNL 5317E,

J. Chou, K. Wu, O. R\ ubel, M. Howison, Qiang, Prabhat, B. Austin, E. W. Bethel, D. Ryne, A. Shoshani, "Parallel Index and Query for Large Scale Data", SC11, 2011, doi: 10.1145/2063384.2063424

Mark Howison, Mike McGreevy, Bruce Palmer, Oliver Ruebel, Kesheng Wu, "ExaHDF5: An I/O Platform for Exascale Data Models, Analysis and Performance", 2011,

Download File: Prabhat.pdf (pdf: 784 KB)

E. W. Bethel, D. Leinweber, O. Rübel, K., Federal Market Information Technology in the Post Crash Era: Roles for Supercomputing, WHPCF, Pages: 23--30 2011, doi: 10.1145/2088256.2088267

O. Rübel, Linking Automated Data Analysis and Visualization with Applications in Developmental Biology and High-energy Physics, Schriftenreihe Informatik, (Der Dekan (hrsg), Fachbereich Informatik, Technische Universität Kaiserslautert: December 2010) LBNL 3155E,

O. Rübel, G. H. Weber, M-Y Huang, E. W. Bethel, M. D. Biggin, C. C. Fowlkes, C. Luengo Hendriks, S. V. E. Keränen, M. Eisen, D. Knowles, J. Malik, H. Hagen and B. Hamann, "Integrating Data Clustering and Visualization for the Analysis of 3D Gene Expression Data", IEEE Transactions on Computational Biology and Bioinformatics, March 2010, 7:64-79, LBNL 382E,

Download File: LBNL-382E-TCBB.pdf (pdf: 11 MB)

Daniela Ushizima, Cameron Geddes, Estelle Cormier-Michel, E. Wes Bethel, Janet Jacobsen, Prabhat, Oliver Rubel, Gunther Weber, Bernard Hamann, Peter Messmer, Hans Hagen, "Automated detection and analysis of particle beams in laser-plasma accelerator simulations", Machine Learning, edited by Yagang Zhang, (In-Teh: February 2010) Pages: 367-389, LBNL 3845E,

Oliver R\ ubel, Sean Ahern, E Wes Bethel, Mark D Biggin, Hank Childs, Estelle Cormier-Michel, Angela DePace, Michael B Eisen, Charless C Fowlkes, Cameron GR Geddes, others, "Coupling visualization and data analysis for knowledge discovery from multi-dimensional scientific data", Procedia computer science, Elsevier, January 2010, 1:1757--1764, LBNL 3669E,

Download File: ICCS2010Ruebel.pdf (pdf: 3.7 MB)

Gunther Weber, "Recent advances in visit: Amr streamlines and query-driven visualization", 2010,

Download File: LBNL-3185E.pdf (pdf: 2.1 MB)

Oliver Rübel, Sean Ahern, E. Wes Bethel, D. Biggin, Hank Childs, Estelle, Angela DePace, Michael B. Eisen Charless C. Fowlkes, Cameron G. R. Geddes, Hagen, Bernd Hamann, Min-Yu Huang, Soile E. Keränen, David W. Knowles, Cris L. Hendriks, Jitendra Malik, Jeremy Meredith Peter Messmer, Prabhat, Daniela Ushizima, H. Weber, Kesheng Wu, "Coupling visualization and data analysis for knowledge from multi-dimensional scientific data", Procedia Computer Science, 2010, 1:1751--1758, doi: 10.1016/j.procs.2010.04.197

E. W. Bethel, C. Johnson, S. Ahern, J. Bell, P.-T. Bremer, H. Childs, E. Cormier-Michel, M. Day, E. Deines, T. Fogal, C. Garth, C. G. R. Geddes, H. Hagen, B. Hamann, C. Hansen, J. Jacobsen, K. Joy, J. Kruger, J. Meredith, P. Messmer, G. Ostrouchov, V. Pascucci, K. Potter, Prabhat, D. Pugmire, O. Rubel, A. Sanderson, C. Silva, D. Ushizima, G. Weber, B. Whitlock, K. Wu, "Occam's Razor and Petascale Visual Data Analysis", SciDAC 2009, J. of Physics: Conference Series, San Diego, California, July 2009, LBNL 2210E,

Download File: BethelETALSciDAC2009.pdf (pdf: 23 MB)

K Wu, S Ahern, EW Bethel, J Chen, H Childs, C Geddes, J Gu, H Hagen, B Hamann, J Lauret, others, "FastBit: Interactively Searching Massive Data", Proc. of SciDAC 2009, 2009, LBNL 2164E,

Download File: LBNL-2164E.pdf (pdf: 3.2 MB)

Oliver R\ ubel, Cameron GR Geddes, Estelle Cormier-Michel, Kesheng Wu, Gunther H Weber, Daniela M Ushizima, Peter Messmer, Hans Hagen, Bernd Hamann, Wes Bethel, others, "Automatic beam path analysis of laser wakefield particle acceleration data", Computational Science \& Discovery, January 2009, 2:015005, LBNL 2734E,

Download File: LBNL-2734E.pdf (pdf: 15 MB)

C. G. R. Geddes, E Cormier-Michel, E. H. Esarey, C. B. Schroeder, J.-L. Vay, W. P. Leemans, D. L.. Bruhwiler, J. R. Cary, B. Cowan, M. Durant, P. Hamill, P. Messmer, P. Mullowney, C. Nieter, K. Paul, S. Shasharina, S. Veitzer, G. Weber, O. Rübel, D. Ushizima, Prabhat, E. W.Bethel, K. Wu, Large Fields for Smaller Facility Sources, SciDAC Review, Pages: 13-21, 2009,

Download File: geddes.pdf (pdf: 344 KB)

E Bethel, "Modern Scientific Visualization is More than Just Pretty Pictures", January 2009, LBNL 1450E,

Download File: LBNL-1450E.pdf (pdf: 9.8 MB)

E Wes Bethel, Chris Johnson, Sean Ahern, John Bell, Peer-Timo Bremer, Hank Childs, Estelle Cormier-Michel, Marc Day, Eduard Deines, Tom Fogal, others, "Occam s razor and petascale visual data analysis", Journal of Physics: Conference Series, 2009, 180:012084,

Oliver R\ ubel, Cameron G R Geddes, Estelle, Kesheng Wu, Prabhat, Gunther H, Daniela M Ushizima, Peter Messmer, Hans, Bernd Hamann, Wes Bethel, "Automatic beam path analysis of laser wakefield acceleration data", Computational Science \& Discovery, 2009, 2:015005,

G. H. Weber, O. Rübel, M.-Y. Huang, A. H. DePace, C. C. Fowlkes, S. V. E. Keränen, C. L. Luengo Hendriks, H. Hagen, D. W. Knowles, J. Malik, M. D. Biggin and B. Hamann, "Visual exploration of three-dimensional gene expression using physical views and linked abstract views", IEEE Transactions on Computational Biology and Bioinformatics, 2009, 6:296-309, LBNL 63776, doi: 10.1109/TCBB.2007.70249

Download File: LBNL-63776.pdf (pdf: 11 MB)

O. Rübel, Prabhat, K. Wu, H. Childs, J. Meredith, C.G.R. Geddes, E. Cormier-Michel, S. Ahern, G.H. Weber, P. Messmer, H. Hagen, B. Hamann and E.W. Bethel, "High Performance Multivariate Visual Data Exploration for Extemely Large Data", Supercomputing (SC), Austin, Texas, USA, November 2008, LBNL 716E,

Download File: LBNL-716E.pdf (pdf: 8.7 MB)

O. Rübel, Prabhat, K. Wu, H. Childs, J. Meredith, C.G.R. Geddes, E. Cormier-Michel, S. Ahern, G.H. Weber, P. Messmer, H. Hagen, B. Hamann and E.W. Bethel, "Application of High-performance Visual Analysis Methods to Laser Wakefield Particle Acceleration Data", IEEE Visualization 2008, October 2008,

Download File: LBNL-952E.pdf (pdf: 1.3 MB)

C. C. Fowlkes, C. L. Luengo Hendriks, S. V. E. Keränen, G. H. Weber, O. Rübel, M.-Y. Huang, S. Chatoor, A. H. DePace, L. Simirenko, C. Henriquez, A. Beaton, R. Weiszmann, S. Celniker, B. Hamann, D. W. Knowles, M. D. Biggin, M. B. Eisen, J. Malik, "A Quantitative Spatio-temporal Atlas of Gene Expression in the Drosophila Blastoderm", Cell, April 18, 2008, 133:364-374,

E. Wes Bethel, Oliver Rübel, Prabhat, Wu, Gunther H. Weber, Valerio Pascucci Hank Childs, Ajith Mascarenhas, Jeremy, Sean Ahern, "Modern Scientific Visualization is More than Just Pictures", Numerical Modeling of Space Plasma Flows: (Astronomical Society of the Pacific Series), St. Thomas, USVI, 2008, 301--317,

Oliver R\ ubel, Prabhat, Kesheng Wu, Hank, Jeremy Meredith, Cameron G. R. Geddes, Cormier-Michel, Sean Ahern, Gunther H., Peter Messmer, Hans Hagen, Bernd Hamann E. Wes Bethel, Application of High-performance Visual Analysis to Laser Wakefield Particle Acceleration Data, IEEE Visualization 2008, 2008,

Oliver R\ ubel, Prabhat, Kesheng Wu, Hank, Jeremy Meredith, Cameron G. R. Geddes, Cormier-Michel, Sean Ahern, Gunther H., Peter Messmer, Hans Hagen, Bernd Hamann E. Wes Bethel, High Performance Multivariate Visual Data Exploration Extemely Large Data, SuperComputing 2008 (SC08), Pages: 51 2008,

Daniela Ushizima, Oliver Rübel, Prabhat, Gunther Weber, E. Wes Bethel, Cecilia Aragon, Cameron Geddes, Estelle Cormier-Michel, Bernd Hamann, Peter Messmer, Hans Hagen, "Automated Analysis for Detecting Beams in Laser Wakefield Simulations", 2008 Seventh International Conference on Machine Learning and Applications, Proceedings of IEEE ICMLA'08, 2008, 382-387, LBNL 960E,

Download File: LBNL-960E.pdf (pdf: 528 KB)

O. Rübel, G. H. Weber, M-Y Huang, E. W. Bethel, S. V. E. Keränen, C. C. Fowlkes, C. L. Luengo Hendriks, A. H. DePace, L. Simirenko, M. B. Eisen, M. D. Biggin, H. Hagen, J. Malik, D. W. Knowles and B. Hamann, "PointCloudXplore 2: Visual Exploration of 3D Gene Expression", Visualization of Large and Unstructured Data Sets, edited by C. Garth, H. Hagen, M. Hering-Bertram, (Gesellschaft fuer Informatik (GI): 2008) LBNL 249E,

Download File: LBNL-249E.pdf (pdf: 2.9 MB)

M.-Y. Huang, O. Rübel, G.H. Weber, C.L. Luengo Hendriks, M.D. Biggin, H. Hagen, B. Hamann, "Segmenting Gene Expression Patterns of Early-stage Drosophila Embryos.", Mathematical Methods for Visualization in Medicine and Life Sciences, edited by L. Linsen, H. Hagen, B. Hamann, (Springer-Verlag: January 2008) Pages: 313--327, LBNL 62450,

Download File: LBNL-62450.pdf (pdf: 3.7 MB)

Cris L. Luengo Hendriks, Soile V. E. Keränen, C. Fowlkes, Lisa Simirenko, Gunther H. Weber, H. DePace, Clara Henriquez, David W. Kaszuba, Hamann, Michael B. Eisen, Jitendra Malik, Damir Sudar, D. Biggin, David W. Knowles, "Three-dimensional Morphology and Gene Expression in the Drosophila Blastoderm at Cellular Resolution I: Data Acquisition Pipeline", Genome Biology, 2006, 7:R123, doi: 10.1186/gb-2006-7-12-r123

Beytullah Yildiz, Kesheng Wu, Suren Byna, Arie Shoshani, "Parallel membership queries on very large scientific data sets using bitmap indexes", Concurrency and Computation: Practice and Experience, January 1, 2019, 31:e5157,

Many scientific applications produce very large amounts of data as advances in hardware fuel computing and experimental facilities. Managing and analyzing massive quantities of scientific data is challenging as data are often stored in specific formatted files, such as HDF5 and NetCDF, which do not offer appropriate search capabilities. In this research, we investigated a special class of search capability, called membership query, to identify whether queried elements of a set are members of an attribute. Attributes that naturally have classification values appear frequently in scientific domains such as category and object type as well as in daily life such as zip code and occupation. Because classification attribute values are discrete and require random data access, performing a membership query on a large scientific data set creates challenges. We applied bitmap indexing and parallelization to membership queries to overcome these challenges. Bitmap indexing provides high performance not only for low cardinality attributes but also for high cardinality attributes, such as floating‐point variables, electric charge, or momentum in a particle physics data set, due to compression algorithms such as Word‐Aligned Hybrid. We conducted experiments, in a highly parallelized environment, on data obtained from a particle accelerator model and a synthetic data set.

Deborah A Agarwal, Boris Faybishenko, Vicky L Freedman, Harinarayan Krishnan, Gary Kushner, Carina Lansing, Ellen Porter, Alexandru Romosan, Arie Shoshani, Haruko Wainwright, others, "A science data gateway for environmental management", Concurrency and Computation: Practice and Experience, 2016, 28:1994--2004,

Xiaocheng (Chris) Zou, Suren Byna, Hans Johansen, Daniel Martin, Nagiza F. Samatova, Arie Shoshani, John Wu, "Six-fold Speedup of Ice Calving Detection Achieved by AMR-aware Parallel Connected Component Labeling", SciDAC PI Meeting, July 2015, 2015,

Elaheh Pourabbas, Arie Shoshani, "The Composite Data Model: A Unified Approach for Combining and Querying Multiple Data Models", IEEE Trans. Knowl. Data Eng, 2015, 27(5):1424-1437,

Spyros Blanas, Kesheng Wu, Surendra Byna, Bin Dong, Arie Shoshani, "Parallel Data Analysis Directly on Scientific File", SIGMOD 14, 2014, 385--396, doi: 10.1145/2588555.2612185

D. Yu, D. Katramatos, A. Sim, A. Shoshani, Co-scheduling of network resource provisioning and host-to-host bandwidth reservation on high-performance network and storage systems, US Patent No. 8,705,342, 2014,

US Patent 8,705,342 B2. “Co-scheduling of network resource provisioning and host-to-host bandwidth reservation on high-performance network and storage systems”, D. Yu, D. Katramatos, A. Sim, and A. Shoshani, Apr. 22, 2014, LBNL IB-3152, BNL BSA 11-02.

Spyros Blanas, Kesheng Wu, Surendra Byna, Bin Dong, Arie Shoshani, "Parallel Data Analysis Directly on Scientific File Formats", SIGMOD 14, 2014, 385--396, doi: 10.1145/2588555.2612185

DP Schissel, Gheni Abla, SM Flanagan, M Greenwald, X Lee, A Romosan, A Shoshani, J Stillerman, J Wright, "Automated metadata, provenance cataloging and navigable interfaces: Ensuring the usefulness of extreme-scale data", Fusion Engineering and Design, North-Holland, 2014,

John C Wright, Martin Greenwald, Joshua Stillerman, Gheni Abla, Bobby Chanthavong, Sean Flanagan, David Schissel, Xia Lee, Alex Romosan, Arie Shoshani, The MPO API: A tool for recording scientific workflows, Fusion Engineering and Design, 2014,

Qian Sun, Fan Zhang, Tong Jin, Hoang Bui, Kesheng Wu, Arie Shoshani, Hemanth Kolla, Scott Klasky, Jacqueline Chen, Manish Parashar, "Scalable run-time data indexing and querying for scientific simulations", Big Data Analytics: Challenges and Opportunities (BDAC-14) Workshop at Supercomputing Conference, 2014,

Spyros Blanas, Kesheng Wu, Surendra Byna, Bin Dong, Arie Shoshani, "Parallel data analysis directly on scientific file formats", Proceedings of the 2014 ACM SIGMOD international conference on Management of data, January 1, 2014, 385--396,

Alex Romosan, Arie Shoshani, Kesheng Wu, Victor Markowitz, Kostas Mavrommatis, "Accelerating gene context analysis using bitmaps", Proceedings of the 25th International Conference on Scientific and Statistical Database Management, 2013, 1--12, LBNL 6397E,

Karen L. Schuchardt, Deborah A. Agarwal, Stefan A. Finsterle, Carl W. Gable, Ian Gorton, Luke J. Gosink, Elizabeth H. Keating, Carina S. Lansing, Joerg Meyer, William A.M. Moeglein, George S.H. Pau, Ellen A. Porter, Sumit Purohit, Mark L. Rockhold, Arie Shoshani, and Chandrika Sivaramakrishnan, Akuna, "Integrated Toolsets Supporting Advanced Subsurface Flow and Transport Simulations for Environmental Management", XIX International Conference on Computational Methods in Water Resources (CMWR 2012), University of Illinois at Urbana-Champaign, June 2012,

Download File: CMWR-2012.pdf (pdf: 1.9 MB)

D. Yu, D. Katramatos, A. Shoshani, A. Sim, J. Gu, V. Natarajan, "StorNet: Integrating Storage Resource Management with Dynamic Network Provisioning for Automated Data Transfer", International Committee for Future Accelerators (ICFA) Standing Committee on Inter-Regional Connectivity (SCIC) 2012 Report: Networking for High Energy Physics, 2012,

Karen L. Schuchardt, Deborah A. Agarwal, Stefan A. Finsterle, Carl W. Gable, Ian Gorton, Luke J. Gosink, Elizabeth H. Keating, Carina S. Lansing, Joerg Meyer, William A.M. Moeglein, George S.H. Pau, Ellen A. Porter, Sumit Purohit, Mark L. Rockhold, Arie Shoshani, Chandrika Sivaramakrishnan, "Akuna-Integrated Toolsets Supporting Advanced Subsurface Flow and Transport Simulations for Environmental Management", XIX International Conference on Computational Methods in Water Resources (CMWR 2012), University of Illinois at Urbana-Champaign, June 17-22, 2012, 2012,

Surendra Byna, Jerry Chou, Oliver Rubel, Homa Karimabadi, William S Daughter, Vadim Roytershteyn, E Wes Bethel, Mark Howison, Ke-Jou Hsu, Kuan-Wu Lin, others, "Parallel I/O, analysis, and visualization of a trillion particle simulation", SC 12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, January 2012, 1--12,

Elaheh Pourabbas, Arie Shoshani, Kesheng Wu, "Minimizing index size by reordering rows and columns", International Conference on Scientific and Statistical Database Management, January 2012, 467--484,

Benson Ma, Arie Shoshani, Alex Sim, Kesheng, Yong-Ik Byun, Jaegyoon Hahm, Min-Su Shin, "Efficient Attribute-Based Data Access in Astronomy", The 2nd International Workshop on Network-Aware Data Workshop (NDM2012), 2012, 562--571,

G. F. Lofstead, Q. Liu, J. Logan, Y. Tian, Abbasi, N. Podhorszki, J. Y. Choi, S., R. Tchoua, R. A. Oldfield, others, "Hello ADIOS: The Challenges and Lessons of Leadership Class I/O Frameworks", 2012,

J. Gu, D. Katramatos, X. Liu, V. Natarajan, A. Shoshani, A. Sim, D. Yu, S. Bradley, S. McKee, "StorNet: Integrated Dynamic Storage and Network Resource Provisioning and Management for Automated Data Transfers", Journal of Physics: Conf. Ser., 2011, 331, doi: 10.1088/1742- 6596/331/1/012002

A. Shoshani, I. Altintas, J. Chen, G. Chin, A. Choudhary, D. Crawl, T. Critchlow, K. Gao, B. Grimm, H. Iyer, C. Kamath, A. Khan, S. Klasky, S. Koehler, S. Lang, R. Latham, J. W. Li, W. Liao, J. Ligon, Q. Liu, B. Ludaescher, P. Mouallem, M. Nagappan, N. Podhorszki, R. Ross, D. Rotem, N. Samatova, C. Silva, A. Sim, R. Tchoua, R. Thakur, M. Vouk, K. Wu, W. Yu, "The Scientific Data Management Center: Available Technologies and Highlights", SciDAC Conference, 2011,

Junmin Gu, Dimitrios Katramatos, Xin Liu, Vijaya Natarajan, Arie Shoshani, Alex Sim, Dantong Yu, Scott Bradley, Shawn McKee, "StorNet: Co-Scheduling of End-to-End Bandwidth Reservation on Storage and Network Systems for High Performance Data Transfers", IEEE INFOCOM HSN 2011, 2011,

Jerry Chou, Mark Howison, Brian Austin, Kesheng Wu, Ji Qiang, E Wes Bethel, Arie Shoshani, Oliver R\ ubel, Rob D Ryne, "Parallel index and query for large scale data analysis", Proceedings of 2011 international conference for high performance computing, networking, storage and analysis, 2011, 1--11, LBNL 5317E,

Kesheng Wu, Rishi R Sinha, Chad Jones, Stephane Ethier, Scott Klasky, Kwan-Liu Ma, Arie Shoshani, Marianne Winslett, "Finding regions of interest on toroidal meshes", Computational Science \& Discovery, 2011, 4:015003,

Kesheng Wu, Surendra Byna, Doron Rotem, Arie, "Scientific Data Services -- A High-Performance I/O with Array Semantics", HPCDB, IEEE, 2011, doi: 10.11v45/2125636.2125640

J. Chou, K. Wu, O. R\ ubel, M. Howison, Qiang, Prabhat, B. Austin, E. W. Bethel, D. Ryne, A. Shoshani, "Parallel Index and Query for Large Scale Data", SC11, 2011, doi: 10.1145/2063384.2063424

Jinoh Kim, Hasan Abbasi, Luis Chac\ on, Docan, Scott Klasky, Qing Liu, Norbert, Arie Shoshani, Kesheng Wu, "Parallel In Situ Indexing for Data-intensive", LDAV, 2011, 65--72, doi: 10.1109/LDAV.2011.6092319

Dean N. Williams, Ian T. Foster, Don E. Middleton, Rachana Ananthakrishnan, Neill Miller, Mehmet Balman, Junmin Gu, Vijaya Natarajan, Arie Shoshani, Alex Sim, Gavin Bell, Robert Drach, Michael Ganzberger, Jim Ahrens, Phil Jones, Daniel Crichton, Luca Cinquini, David Brown, Danielle Harper, Nathan Hook, Eric Nienhouse, Gary Strand, Hannah Wilcox, Nathan Wilhelmi, Stephan Zednik, Steve Hankin, Roland Schweitzer, John Harney, Ross Miller, Galen Shipman, Feiyi Wang, Peter Fox, Patrick West, Stephan Zednik, Ann Chervenak, Craig Ward, "Earth System Grid Center for Enabling Technologies (ESG-CET): A Data Infrastructure for Data-Intensive Climate Research", SciDAC Conference, 2011,

Alex Sim, Mehmet Balman, Dean N. Williams, Arie Shoshani, Vijaya Natarajan, "Adaptive Transfer Adjustment in Efficient Bulk Data Transfer Management for Climate Datasets", The 22nd IASTED International Conference on Parallel and Distributed Computing and System, Marina Del Rey, CA, November 20, 2010, LBNL 3985E,

Download File: PDCS2010-balman.pdf (pdf: 1.6 MB)

Many scientific applications and experiments, such as high energy and nuclear physics, astrophysics, climate observation and modeling, combustion, nano-scale material sciences, and computational biology, generate extreme volumes of data with a large number of files. These data sources are distributed among national and international data repositories, and are shared by large numbers of geographically distributed scientists. A large portion of the data is frequently accessed, and a large volume of data is moved from one place to another for analysis and storage. A challenging issue in such efforts is the limited network capacity for moving large datasets. A tool that addresses this challenge is the Bulk Data Mover (BDM), a data transfer management tool used in the Earth System Grid (ESG) community. It has been managing massive dataset transfers efficiently in the environment where the network bandwidth is limited. Adaptive transfer adjustment was studied to enhance the BDM to handle significant end-to-end performance changes in the dynamic network environments as well as to control the data transfers for the desired transfer performance. We describe the results from our hands-on data transfer management experience in the climate research community. We study a practical transfer estimation model and state our initial results from the adaptive transfer adjustment methodology.

Mehmet Balman, Evangelos Chaniotakis, Arie Shoshani, Alex Sim, "A Flexible Reservation Algorithm for Advance Network Provisioning", ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, New Orleans, LA, November 2010 (SC'10)., New Orleans, LA, IEEE Computer Society Washington, DC, USA ISBN: 978-1-4244-7559-, November 14, 2010, LBNL 4017E, doi: http://dx.doi.org/10.1109/SC.2010.4

Download File: rpt79306.PDF (PDF: 2.8 MB)

Many scientific applications need support from a communication infrastructure that provides predictable performance, which requires effective algorithms for bandwidth reservations. Network reservation sys- tems such as ESnet’s OSCARS, establish guaranteed bandwidth of secure virtual circuits for a certain bandwidth and length of time. However, users currently cannot inquire about bandwidth availability, nor have alternative suggestions when reservation requests fail. In general, the number of reservation options is exponential with the number of nodes n, and current reservation commitments. We present a novel approach for path finding in time-dependent networks taking advantage of user-provided parameters of total volume and time constraints, which produces options for earliest completion and shortest duration. The theoretical complexity is only O(n2r2) in the worst-case, where r is the number of reservations in the desired time interval. We have implemented our algorithm and developed efficient methodologies for incorporation into network reservation frameworks. Performance measurements confirm the theoretical predictions.

M. Balman, E. Chaniotakis, A. Shoshani, A. Sim, "A New Approach in Advance Network Reservation and Provisioning for High-Performance Scientific Data Transfers", 2010, LBNL 4091E,

Julian Cummings, Jay Lofstead, Karsten Schwan, Alexander Sim, Arie Shoshani, Ciprian Docan, Manish Parashar, Scott Klasky, Norbert Podhorszki, Roselyne Barreto, "EFFIS: An End-to-end Framework for Fusion Integrated Simulation", 18th Euromicro Conference on Parallel, Distributed and Network-based Processing, 2010,

A. Sim, D. Gunter, V. Natarajan, A. Shoshani, D. Williams, J. Long, J. Hick, J. Lee, E. Dart, "Efficient Bulk Data Replication for the Earth System Grid", Data Driven E-science: Use Cases and Successful Applications of Distributed Computing Infrastructures (ISGC 2010), (Springer-Verlag New York Inc: 2010) Pages: 435

Kesheng Wu, Arie Shoshani, Kurt Stockinger, "Analyses of multi-level and multi-component compressed indexes", ACM Transactions on Database Systems, ACM, 2010, 35:1--52, doi: 10.1145/1670243.1670245

E. Pourabbas, A. Shoshani, "Improving Estimation Accuracy of Aggregate Queries on Data Cubes", Data & Knowledge Engineering 69 (2010), January 1, 2010, 69:50-72,

Scientific Data Management: Challenges, Technology, and Deployment, edited by Arie Shoshani and Doron Rotem, (Chapman & Hall/CRC Computational Science: December 2009)

A. Sim, A. Shoshani, F. Donno, J. Jensen, Storage Resource Manager Interface Specification V2.2 Implementations Experience Report, Open Grid Forum, GFD.154, 2009,

D. N. Williams, R. Ananthakrishnan, D. E. Bernholdt, S. Bharathi, D. Brown, M. Chen, A. L. Chervenak, L. Cinquini, R. Drach, I. T. Foster, P. Fox, D. Fraser, J. Garcia, S. Hankin, P. Jones, D. E. Middleton, J. Schwidder, R. Schweitzer, R. Schuler, A. Shoshani, F. Siebenlist, A. Sim, W. G. Strand, M. Su, N. Wilhelmi, "The Earth System Grid: Enabling Access to Multimodel Climate Simulation Data", American Meteorological Society, 2009, 90(2):195-205,

M. Riedel, E. Laure, Th. Soddemann, L. Field, J. P. Navarro, J. Casey, M. Litmaath, J. Ph. Baud, B. Koblitz, C. Catlett, D. Skow, C. Zheng, P. M. Papadopoulos, M. Katz, N. Sharma, O. Smirnova, B. Kónya, P. Arzberger, F. Würthwein, A. S. Rana, T. Martin, M. Wan, V. Welch, T. Rimovsky, S. Newhouse, A. Vanni, Y. Tanaka, Y. Tanimura, T. Ikegami, D. Abramson, C. Enticott, G. Jenkins, R. Pordes, N. Sharma, S. Timm, N. Sharma, G. Moont, M. Aggarwal, D. Colling, O. van der Aa, A. Sim, V. Natarajan, A. Shoshani, J. Gu, S. Chen, G. Galang, R. Zappi, L. Magnoni, V. Ciaschini, M. Pace, V. Venturi, M. Marzolla, P. Andreetto, B. Cowles, S. Wang, Y. Saeki, H. Sato, S. Matsuoka, P. Uthayopas, S. Sriprayoonsakul, O. Koeroo, M. Viljoen, L. Pearlman, S. Pickles, David Wallom, G. Moloney, J. Lauret, J. Marsteller, P. Sheldon, S. Pathak, S. De Witt, J. Mencák, J. Jensen, M. Hodges, D. Ross, S. Phatanapherom, G. Netzer, A. R. Gregersen, M. Jones, S. Chen, P. Kacsuk, A. Streit, D. Mallmann, F. Wolf, T. Lippert, Th. Delaitre, E. Huedo, N. Geddes, "Interoperation of world-wide production e-Science infrastructures", Concurrency and Computation: Practice and Experience, 2009, 21(8):961-990,

Arie Shoshani, Flavia Donno, Junmin Gu, Jason Hick, Maarten Litmaath, Alex Sim, "Dynamic Storage Management", Scientific Data Management: Challenges, Technology, and Deployment, edited by Arie Shoshani, Doron Rotem, (Chapman & Hall/CRC Computational Science: 2009)

John Shalf and Jason Hick (Arie Shoshani and Doron Rotem), "Storage Technology Fundamentals", Scientific Data Management: Challenges, Technology, and Deployment, Volume . Chapman & Hall/CRC, 2009,

Download File: iobook08.pdf (pdf: 1.2 MB)

K Wu, S Ahern, EW Bethel, J Chen, H Childs, C Geddes, J Gu, H Hagen, B Hamann, J Lauret, others, "FastBit: Interactively Searching Massive Data", Proc. of SciDAC 2009, 2009, LBNL 2164E,

Download File: LBNL-2164E.pdf (pdf: 3.2 MB)

P. Jakl, J. Lauret, A. Hanushevsky, A. Shoshani, A. Sim, J. Gu, "Grid data access on widely distributed worker nodes using scalla and SRM", Journal of Physics: Conf. Ser., 2008, 119, doi: 10.1088/1742-6596/119/7/072019

D. N. Williams, R. Ananthakrishnan, D. E. Bernholdt, S. Bharathi, D. Brown, M. Chen, A. L. Chervenak, L. Cinquini, R. Drach, I. T. Foster, P. Fox, S. Hankin, V. E. Henson, P. Jones, D. E. Middleton, J. Schwidder, R. Schweitzer, R. Schuler, A Shoshani, F. Siebenlist, A. Sim, W. G. Strand, N. Wilhelmi, M. Su, "Data Management and Analysis for the Earth System Grid", SciDAC Conference, 2008,

Alex Sim, Arie Shoshani (Editors), Paolo Badino, Olof Barring, Jean‐Philippe Baud, Ezio Corso, Shaun De Witt, Flavia Donno, Junmin Gu, Michael Haddox‐Schatz, Bryan Hess, Jens Jensen, Andy Kowalski, Maarten Litmaath, Luca Magnoni, Timur Perelmutov, Don Petravick, Chip Watson, The Storage Resource Manager Interface Specification Version 2.2, Open Grid Forum, Document in Full Recommendation, GFD.129, 2008,

C S Chang, S Klasky, J Cummings, R. Samtaney, A Shoshani, L Sugiyama, D Keyes, S Ku, G Park, S Parker, N Podhorszki, H. Strauss, H Abbasi, M Adams, R Barreto, G Bateman, K Bennett, Y Chen, E D’Azevedo, C Docan, S Ethier, E Feibush, L Greengard, T Hahm, F Hinton, C Jin, A. Khan, A Kritz, P Krsti, T Lao, W Lee, Z Lin, J Lofstead, P Mouallem, M Nagappan, A Pankin, M Parashar, M Pindzola, C Reinhold, D Schultz, K Schwan, D. Silver, A Sim, D Stotler, M Vouk, M Wolf, H Weitzner, P Worley, Y Xiao, E Yoon, D Zorin, "Toward a first- principles integrated simulation of tokamak edge plasmas", Journal of Physics: Conf. Ser., 2008, 125, doi: 10.1088/1742-6596/125/1/012042

R Ananthakrishnan, D E Bernholdt, S Bharathi, D Brown, M Chen, A L Chervenak, L Cinquini, R Drach, I T Foster, P Fox, D Fraser, K Halliday, S Hankin, P Jones, C Kesselman, D E Middleton, J Schwidder, R Schweitzer, R Schuler, A Shoshani, F Siebenlist, A Sim, W G Strand, N Wilhelmi, M Su, D N Williams, "Building a global federation system for climate change research: the earth system grid center for enabling technologies (ESG-CET)", Journal of Physics: Conf. Ser., 2008, 78, doi: 10.1088/1742-6596/78/1/012050

Kurt Stockinger, John Cieslewicz, Kesheng Wu, Rotem, Arie Shoshani, "Using Bitmap Indexing Technology for Combined and Text Queries", Annals of Information Systems, (Springer: 2008) Pages: 1--23

Rishi Rakesh Sinha, Marianne Winslett, Kesheng, Kurt Stockinger, Arie Shoshani, "Adaptive Bitmap Indexes for Space-Constrained", ICDE 2008, 2008, 1418--1420,

Kesheng Wu, Kurt Stockinger, Arie Shoshani, "Breaking the curse of cardinality on bitmap indexes", International Conference on Scientific and Statistical Database Management, 2008, 348--365,

Meiyappan Nagappan, Mladen A. Vouk, Kesheng Wu Alex Sim, Arie Shoshani, "Efficient Operational Profiling of Systems Using Arrays on Execution Logs", ISSRE, 2008, 313--314, doi: 10.1109/ISSRE.2008.45

L. Abadie, P. Badino, J. Baud, E. Corso, M. Crawford, S. De Witt, F. Donno, A. Forti, P. Fuhrmann,
G. Grosdidier, J. Gu , J. Jensen, S. Lemaitre, M. Litmaath, D. Litvinsev, G. Lo Presti, L. Magnoni, T. Mkrtchan, A. Moibenko, V. Natarajan, G. Oleynik, T. Perelmutov, D. Petravick, A. Shoshani, A. Sim, M. Sponza, R. Zappi, "Storage Resource Managers: Recent International Experience on Requirements and Multiple Co-Operating Implementations", the 24th IEEE Conference on Mass Storage Systems and Technologies, 2007,

F. Donno, L. Abadie, P. Badino, J. Baud, E. Corso, M. Crawford, S. De Witt, A. Forti, P. Fuhrmann, G. Grosdidier, J. Gu , J. Jensen, S. Lemaitre, M. Litmaath, D. Litvinsev, G. Lo Presti, L. Magnoni, T. Mkrtchan, A. Moibenko, V. Natarajan, G. Oleynik, T. Perelmutov, D. Petravick, A. Shoshani, A. Sim, M. Sponza, R. Zappi, "Storage Resource Manager version 2.2: design, implementation, and testing experience", Journal of Physics: Conf. Ser., 2007, 119, doi: 10.1088/1742-6596/119/6/062028

Elaheh Pourabbas, Arie Shoshani, "Efficient Estimation of Joint Queries from Multiple OLAP Databases", ACM Transactions on Database Systems (TODS), March 1, 2007, Volume 3,

Kesheng Wu, Kurt Stockinger, Arie Shoshani, Performance of Multi-Level and Multi-Component Bitmap Indexes, 2007, doi: 10.1145/1670243.1670245

Frederick Reiss, Kurt Stockinger, Kesheng Wu, Shoshani, Joseph M. Hellerstein, "Enabling Real-Time Querying of Live and Historical Data", SSDBM 2007, 2007,

Elaheh Pourabbas, Arie Shoshani, "The Composite OLAP-Object Data Model: Removing an Unnecessary Barrier", International Conference on Scientific and Statistical Database Management (SSDBM) 2006, July 3, 2006, 291-300,

A. Shoshani, A. Sim, K. Stockinger, "RRS: Replica Registration Service for Data Grids", Lecture Notes in Computer Science, edited by Jean-Marc Pierson, (Springer-Verlag GmbH Publisher: 2006) Pages: 100-112

D. E. Middleton, D. E. Bernholdt, D. Brown, M. Chen, A. L. Chervenak, L. Cinquini, R. Drach, P. Fox, P. Jones, C. Kesselman, I. T. Foster, V. Nefedova, A. Shoshani, A. Sim, W. G. Strand, D. Williams, "Enabling worldwide access to climate simulation data: the earth system grid (ESG)", SciDAV Conference, 2006,

P. Jakl, J. Lauret, A. Hanushevky, A. Shoshani, A. Sim, "From rootd to Xrootd, from physical to logical files: experience on accessing and managing distributed data", Computing in High Energy Physics (CHEP), 2006,

E. Hjort, L. Hajdu, J. Lauret, D. Olson, A. Sim, A. Shoshani, "Data and Computational Grid Coupling in RHIC/STAR – An Analysis Scenario using SRM Technology", Computing in High Energy Physics (CHEP), 2006,

Kesheng Wu, Ekow J Otoo, Arie Shoshani, "Optimizing bitmap indices with efficient compression", ACM Transactions on Database Systems (TODS), 2006, 31:1--38,

K. Wu, K. Stockinger, A. Shoshani, Wes, "FastBit--Helps Finding the Proverbial Needle in a", 2006, LBNL LBNL-PUB/963,

F. Reiss, K. Stockinger, K. Wu, A. Shoshani J. M. Hellerstein, "Efficient analysis of live and historical streaming and its application to cybersecurity", 2006,

D. Bernholdt, S. Bharathi, D. Brown, K. Chanchio, M. Chen, A. Chervenak, L. Cinquini, B. Zrach, I. Foster, P. Fox, J. Garcia, C. Kesselman, R. Markel, D. Middleton, V. Nefedova, L. Pouchard, A. Shoshani, A. Sim, G. Strand, D. Williams, "The Earth System Grid: Supporting the Next Generation of Climate Modeling Research", IEEE, 2005, 93(3):485-495,

A. Shoshani, A. Sim, K. Stockinger, "RRS: Replica Registration Service for Data Grids", International Workshop on Data Management in Grids, 2005,

Arie Shoshani, Alex Sim, Kurt Stockinger, "Replica Registration Service Functional Interface Specification 1.0", 2005, LBNL 57520,

Kesheng Wu, Junmin Gu, Jerome Lauret, Arthur Poskanzer, Arie Shoshani, Alexander Sim, Zhang, "Grid Collector: Facilitating Efficient Selective from Data Grids", International Supercomputer Conference 2005, 2005,

Eric Hjort, Doug Olson, Jerome Lauret, Arie Shoshani, Alex Sim, "Production mode Data- Replication framework in STAR using the HRM Grid middleware", Computing in High Energy Physics, 2004,

Alex Sim, Junmin Gu, Arie Shoshani, Vijaya Natarajan, "DataMover: Robust Terabytes-Scale Multi-file Replication over Wide-Area Networks", the 16th International Conference on Scientific and Statistical Database Management (SSDBM 2004), 2004,

K. Wu, A. Shoshani, E. J. Otoo, Word aligned bitmap compression method, data and apparatus, US Patent 6,831,575, 2004,

Kesheng Wu, Ekow J Otoo, Arie Shoshani, "An efficient compression scheme for bitmap indices", 2004,

Kesheng Wu, Wei-Ming Zhang, Victor, Jerome Lauret, Arie Shoshani, "The Grid Collector: Using an Event Catalog to Speed up Analysis in Distributed Environment", Proceedings of Computing in High Energy and Nuclear (CHEP) 2004, 2004,

Kurt Stockinger, Kesheng Wu, Arie Shoshani, Evaluation Strategies for Bitmap Indices with, International Conference on Database and Expert Applications (DEXA 2004), Zaragoza, Spain, 2004,

Elaheh Pourabbas, Arie Shoshani, "Answering Joint Queries from Multiple Aggregate OLAP Databases", Data Warehousing and Knowledge Discovery, 5th International Conference, DaWaK 2003, September 3, 2003, 24-34,

Arie Shoshani, Alexander Sim, Junmin Gu, "Storage Resource Managers: Essential Components for the Grid", Grid Resource Management: State of the Art and Future Trends, edited by Jarek Nabrzyski, Jennifer M. Schopf, Jan Weglarz, (Kluwer Academic Publishers: 2003)

Ann L. Chervenak, Ewa Deelman, Carl Kesselman, William E. Allcock, Ian T. Foster, Veronika Nefedova, Jason Lee, Alex Sim, Arie Shoshani, Bob Drach, Dean Williams, Don Middleton, "High-performance remote access to climate simulation data: a challenge problem for data grid technologies", Parallel Computing, 2003, 29(10):1335-1356,

A. Sim, J. Gu, A. Shoshani, E. Hjort, D. Olson, "Experience with Deploying Storage Resource Managers to Achieve Robust File Replication", Computing in High Energy Physics, 2003,

D. Yu, J. Lauret, A. Shoshani, D. Oldon, E. Hjort, A. Sim, "The Design of High Performance Data Replication in the Grid Environment for the STAR Collaboration", Computing in High Energy Physics, 2003,

L. Pouchard, L. Cinquini, B. Drach, D. Middleton, D. Bernholdt, K. Chanchio, I. Foster, V. Nefedova, D. Brown, P. Fox, J. Garcia, G. Strand, D. Williams, A. Chervenak, C. Kesselman, A. Shoshani, A. Sim, "An Ontology for Scientific Information in a Grid Environment: the Earth System Grid", the Symposium on Cluster Computing and the Grid (CCGrid), 2003,

Arie Shoshani, Alex Sim, Junmin Gu, Storage Resource Managers: Essential Components for Grid Applications, Globus World, 2003,

Kesheng Wu, Wei-Ming Zhang, Alexander Sim, Gu, Arie Shoshani, "Grid Collector: An Event Catalog With Automated File", Proceedings of IEEE Nuclear Science Symposium 2003, 2003, doi: 10.1109/NSSMIC.2003.1351830

Kesheng Wu, Wei-Ming Zlang, Alexander Sim, Junmin Gu, Arie Shoshani, "Grid collector: An event catalog with automated file management", 2003 IEEE Nuclear Science Symposium. Conference Record (IEEE Cat. No. 03CH37515), 2003, LBNL 55563,

Elaheh Pourabbas, Arie Shoshani, "Joint Queries Estimation from Multiple OLAP Databases", International Conference on Scientific and Statistical Database Management, 2002 (SSDBM’02), July 24, 2002,

A. Shoshani, A. Sim, J. Gu, "Storage Resource Managers: Middleware components for Grid Storage", the 19th IEEE Symposium on Mass Storage Systems, 2002,

Kesheng Wu, Ekow Otoo, Arie Shoshani, Compressing Bitmap Indexes for Faster Search, Proceedings of SSDBM 02, Pages: 99--108 2002,

Kurt Stockinger, Kesheng Wu, Arie Shoshani, Strategies for processing ad hoc queries on large data, Proceedings of DOLAP 02, Pages: 72--79 2002,

B. Allcock, I. Foster, V. Nefedova, A. Chervenak, E. Deelman, C. Kesselman, J. Lee, A. Sim, A. Shoshani, B. Drach, D. Williams, "High-Performance Remote Access to Climate Simulation Data: A Challenge Problem for Data Grid Technologies", Super Computing 2001, 2001,

A. Sim, H. Nordberg, L.M. Bernardo, A. Shoshani, D. Rotem, "Experience with using CORBA to implement a file caching coordination system", Concurrency and Computation: Practice and Experience, 2001, 13:1-15,

D. Olson, E. Hjort, J. Lauret, M. Messer, A. Shoshani, A. Sim, "Non-shared Disk Cluster - A Fault Tolerant, Commodity Approach to Hi-Bandwidth Data Analysis", Computing in High Energy Physics, 2001,

L Bernardo, H Nordberg, D Olson, A Shoshani, A Sim, A Vaniachine, D Zimmerman, B Gibbard, R Porter, T Wenaus, others, "New capabilities in the HENP grand challenge storage access system and its application at RHIC", Computer physics communications, 2001, 140:179--188,

Kesheng Wu, Ekow J Otoo, Arie Shoshani, A performance comparison of bitmap indexes, Proceedings of the tenth international conference on Information and knowledge management, Pages: 559--561 2001,

A. Shoshani, A. Sim, L.M. Bernerdo, H. Nordberg, "Coordinating Simultaneous Caching of File Bundles from Tertiary Storage", International Conference on Scientific and Statistical Database Management (SSDBM), 2000,

L. M. Bernardo, B. Gibbard, D. Malon, H. Nordberg, D. Olson, R. Porter, A. Shoshani, A. Sim, A. Vaniachine, T. Wenaus, K. Wu, D. Zimmerman, "New Capabilities in the HENP Grand Challenge Storage Access System and its Application at RHIC", Computing in High Energy Physics, 2000,

L. M. Bernardo, A. Shoshani, A. Sim, H. Nordberg, "Access Coordination Of Tertiary Storage For High Energy Physics Applications", the 17th IEEE Symposium on Mass Storage Systems, 2000,

A. Sim, A. Shoshani, HRM: Hierarchical Resource Manager, Globus World, 2000,

A. Sim, A. Shoshani, L. M. Bernardo, H. Nordberg, A Storage Access Coordination System for Perabyte Scale Scientific Data, IONA World, 2000,

A. Sim, H. Nordberg, L. M. Bernardo, A. Shoshani, D. Rotem, "Storage Access Coordination Using CORBA", Distributed Objects and Application, 1999, 168-175,

A. Shoshani, L.M. Bernardo, H. Nordberg, D. Rotem and A. Sim, "Multidimensional Indexing and Query Coordination for Tertiary Storage Management", International Conference on Scientific and Statistical Database Management, 1999, 214-225,

L.M. Bernardo, D. Rotem, A. Shoshani, H. Nordberg, A. Sim, "Using Access Patterns to Partition Large Datasets on Tertiary Storage in Order to Minimize Retrieval Costs", 1998, LBNL 41504,

A. Shoshani, L.M. Bernardo, H. Nordberg, D. Rotem, A. Sim, "Storage Management for High Energy Physics Applications", Computing in High Energy Physics, 1998,

D.K. Sung, Y. Son, A. Sim, K. Wu, S. Byna, H. Tang, H. Eom, C. Kim, S. Kim, "A2FL: Autonomous and Adaptive File Layout in HPC through Real-time Access Pattern Analysis", 38th IEEE International Parallel & Distributed Processing Symposium (IPDPS2024), 2024,

L. Zhou, Q. Lin, K. Chowdhury, S. Masood, A. Eichenberger, H. Min, A. Sim, J. Wang, Y. Wang, K. Wu, B. Yuan, J. Zou, "Serving Deep Learning Model in Relational Databases", 27th International Conference on Extending Database Technology (EDBT2024), 2024,

R. Frehner, K. Wu, A. Sim, J. Kim, K. Stockinger, "Detecting Anomalies in Time Series Using Kernel Density Approaches", IEEE Access, 2024, doi: 10.1109/ACCESS.2024.3371891

A, Sharma, X. Li, H. Guan, G. Sun, L. Zhang, L. Wang, K. Wu, L. Cao, E. Zhu, A. Sim, T. Wu, J. Zou, "Automatic Data Transformation Using Large Language Model – An Experimental Study on Building Energy Data", IEEE International Conference on Big Data (BigData), 2023,

C. M. Oguchi, D. Ghosal, A. Sim, K. Wu, "Counterfactual Analysis: A Case Study on Impact of External Events on Building Energy Consumption", International Workshop on Big Data Analytics for Sustainability (BDA4S), 2023,

A. Sim, E. Kissel, D. Hazen, C. Guok, "Experiences in deploying in-network data caches", 26th International Conference on Computing in High Energy & Nuclear Physics (CHEP2023), 2023, doi: 10.1051/epjconf/202429507018

J. Bellavita, C. Sim, K. Wu, A. Sim, S. Yoo, H. Ito, V. Garonne, E. Lancon, "Understanding Data Access Patterns for dCache System", 26th International Conference on Computing in High Energy & Nuclear Physics (CHEP2023), 2023, doi: 10.1051/epjconf/202429501053

C. Sim, K. Wu, A. Sim, I. Monga, C. Guok, D. Hazen, F. Würthwein, D. Davila, H. Newman, J. Balcas, "Predicting Resource Utilization Trends with Southern California Petabyte Scale Cache", 26th International Conference on Computing in High Energy & Nuclear Physics (CHEP2023), 2023, doi: 10.1051/epjconf/202429501044

J. W. Chung, A. Sim, B. Quiter, Y. Wu, W. Zhao, K. Wu, "Preparing Spectral Data for Machine Learning: A Study of Geological Classification from Aerial Surveys", Machine Learning and the Physical Sciences Workshop (ML4PS), 2023,

R. Monga, A. Sim (advisor), K. Wu (advisor), "Comparative Study of the Cache Utilization Trends for Regional Scientific Data Caches", ACM/IEEE The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’23), ACM Student Research Competition (SRC), First place winner, 2023,

H-C. Yang, L. Jin, A. Lazar, A. Todd-Blick, A. Sim, K. Wu, Q. Chen, C. A. Spurlock, "Gender Gaps in Mode Usage, Vehicle Ownership, and Spatial Mobility When Entering Parenthood: A Life Course Perspective", Systems, 2023, 11(6):314, doi: 10.3390/systems11060314

R. Shao, A. Sim, K. Wu, J. Kim, "Leveraging History to Predict Abnormal Transfers in Distributed Workflows", Sensors, 2023, 23(12):5485, doi: 10.3390/s23125485

Z. Deng, A. Sim, K. Wu, C. Guok, I. Monga, F. Andrijauskas, F. Wuerthwein, D. Weitzel, "Analyzing Transatlantic Network Traffic Patterns with Scientific Data Caches", 6th ACM International Workshop on System and Network Telemetry and Analysis (SNTA 2023), 2023, doi: 10.1145/3589012.3594897

C. Guok, E. Kissel, A. Sim, ESnet's In-Network Caching Pilot, The Network Conference 2023 (TNC'23), 2023,

C. Sim, K. Wu, A. Sim, I. Monga, C. Guok, F. Wurthwein, D. Davila, H. Newman, J. Balcas, Predicting Resource Usage Trends with Southern California Petabyte Scale Cache, 26th International Conference on Computing in High Energy & Nuclear Physics (CHEP 2023), 2023,

J. Bellavita, C. Sim, K. Wu, A. Sim, S. Yoo, H. Ito, V. Garonne, E. Lancon, Understanding Data Access Patterns for dCache System, 26th International Conference on Computing in High Energy & Nuclear Physics (CHEP 2023), 2023,

E. Kissel, A. Sim, C. Guok, Experiences in deploying in-network data caches, 26th International Conference on Computing in High Energy & Nuclear Physics (CHEP 2023), 2023,

S. Kim, A. Sim, K. Wu, S. Byna, Y. Son, H. Eom, "Design and Implementation of I/O Performance Prediction Scheme on HPC Systems through Large-scale Log Analysis", Journal of Big Data, 2023, 10(65), doi: 10.1186/s40537-023-00741-4

C. Sim, K. Wu, A. Sim, I. Monga, C. Guok, F. Wurthwein, D. Davila, H. Newman, J. Balcas, "Effectiveness and predictability of in-network storage cache for Scientific Workflows", International Conference on Computing, Networking and Communication (ICNC 2023), 2023, doi: 10.1109/ICNC57223.2023.10074058

J. Wang, K. Wu, A. Sim, S. Hwangbo, "Locating Partial Discharges in Power Transformers with Convolutional Iterative Filtering", Sensors, 2023, 23, doi: 10.3390/s23041789

H-C. Yang, L. Jin, A. Lazar, A. Todd-Blick, A. Sim, K. Wu, Q. Chen, C. A. Spurlock, Gender Gaps in Mode Usage, Vehicle Ownership, and Spatial Mobility When Entering Parenthood: A Life Course Perspective, Transportation Research Board 102nd Annual Meeting,, 2023,

J. Bang, A. Sim, G. Lockwood, H. Eom, H. Sung, "Design and Implementation of Burst Buffer Over-Subscription Scheme for HPC Storage Systems", IEEE Access, 2023, doi: 10.1109/ACCESS.2022.3233829

Julian Bellavita, Alex Sim (advisor), John Wu (advisor), "Predicting Scientific Dataset Popularity Using dCache Logs", ACM/IEEE The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’22), ACM Student Research Competition (SRC), Second place winner, 2022,

Poster (PDF)

The dCache installation is a storage management system that acts as a disk cache for high-energy physics (HEP) data. Storagespace on dCache is limited relative to persistent storage devices, therefore, a heuristic is needed to determine what data should be kept in the cache. A good cache policy would keep frequently accessed data in the cache, but this requires knowledge of future dataset popularity. We present methods for forecasting the number of times a dataset stored on dCache will be accessed in the future. We present a deep neural network that can predict future dataset accesses accurately, reporting a final normalized loss of 4.6e-8. We present a set of algorithms that can forecast future dataset accesses given an access sequence. Included are two novel algorithms, Backup Predictor and Last N Successors, that outperform other file prediction algorithms. Findings suggest that it is possible to anticipate dataset popularity in advance.

C. Sim, C. Guok (advisor), A. Sim (advisor), K. Wu (advisor), "Data Throughput Performance Trends of Regional Scientific Data Cache", ACM/IEEE The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC’22), ACM Student Research Competition (SRC), 2022,

J. Wang, K. Wu, A. Sim, S. Hwangbo, "Feature Engineering and Classification Models for Partial Discharge in Power Transformers", arXiv, 2022, doi: 10.48550/arXiv.2210.12216

Sunggon Kim, Alex Sim, Kesheng Wu, Suren Byna, Yongseok Son, "Design and implementation of dynamic I/O control scheme for large scale distributed file systems", Cluster Computing, 2022, 25(6):1--16, doi: 10.1007/s10586-022-03640-0

Download File: wu2022.bib (bib: 22 KB)

L. Jin, A. Lazar, C. Brown, V. Garikapati, B. Sun, S. Ravulaparthy, Q. Chen, A. Sim, K. Wu, T. Wenzel, T. Ho, C. A. Spurlock, "What Makes You Hold onto That Old Car? Joint Insights from Machine Learning and Multinomial Logit on Vehicle-level Transaction Decisions", Frontiers in Future Transportation, Connected Mobility and Automation, 2022, 3:894654, doi: 10.3389/ffutr.2022.894654

R. Shao, J. Kim A. Sim, K. Wu, "Predicting Slow Connections in Scientific Computing", 5th ACM International Workshop on System and Network Telemetry and Analysis (SNTA) 2022, in conjunction with The 31st ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2022, doi: 10.1145/3526064.3534112

J. Bellavita, A. Sim, K. Wu, I. Monga, C. Guok, F. Würthwein, D. Davila, "Studying Scientific Data Lifecycle in On-demand Distributed Storage Caches", 5th ACM International Workshop on System and Network Telemetry and Analysis (SNTA) 2022, in conjunction with The 31st ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2022, doi: 10.1145/3526064.3534111

R. Han, A. Sim, K. Wu, I. Monga, C. Guok, F. Würthwein, D. Davila, J. Balcas, H. Newman, "Access Trends of In-network Cache for Scientific Data", 5th ACM International Workshop on System and Network Telemetry and Analysis (SNTA), in conjunction with The 31st ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2022, doi: 10.1145/3526064.3534110

J. Kim, M. Cafaro, J. Chou, A. Sim, "SNTA’22: The 5th Workshop on Systems and Network Telemetry and Analytics", In the proceedings of The 31st ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC'22), 2022, doi: 10.1145/3502181.3535108

D. Bard, C. Snavely, L. Gerhardt, J. Lee, B. Totzke, K. Antypas, W. Arndt, J. Blaschke, S. Byna, R. Cheema, S. Cholia, M. Day, B. Enders, A. Gaur, A. Greiner, T. Groves, M. Kiran, Q. Koziol, T. Lehman, K. Rowland, C. Samuel, A. Selvarajan, A. Sim, D. Skinner, L. Stephey, R. Thomas, G. Torok, "LBNL Superfacility Project Report", Lawrence Berkeley National Laboratory, 2022, doi: 10.48550/arXiv.2206.11992

Yujing Ma, Florin Rusu, Kesheng Wu, Alexander Sim, 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Pages: 1088--1097 2022, doi: 10.1109/IPDPSW55747.2022.00177

Download File: wu2022.bib (bib: 22 KB)

J. Kim, M. Jin, Y. Homma, A. Sim, W. Kroeger, K. Wu, "Extract Dynamic Information To Improve Time Series Modeling: a Case Study with Scientific Workflow", arXiv, 2022, doi: 10.48550/arXiv.2205.09703

K. Wang, S. Lee, J. Balewski, A. Sim, P. Nugent, A. Agrawal, A. Choudhary, K. Wu, W-K. Liao, "Using Multi-resolution Data to Accelerate Neural Network Training in Scientific Applications", 22nd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGrid 2022), 2022, doi: 10.1109/CCGrid54584.2022.00050

B. Weinger, J. Kim, A. Sim, M. Nakashima, N. Moustafa, K. Wu, "Enhancing IoT Anomaly Detection Performance for Federated Learning", Digital Communications and Networks, Special Issue on Edge Computation and Intelligence, 2022, doi: 10.1016/j.dcan.2022.02.007

A. Sim, E. Kissel, C. Guok, "Deploying in-network caches in support of distributed scientific data sharing", arXiv whitepaper, 2022, doi: /10.48550/arXiv.2203.06843

John Wu, Bin Dong, Alex Sim, Automating Data Management Through Unified Runtime Systems, DOE ASCR Workshop on the Management and Storage of Scientific Data, 2022, doi: 10.2172/1843500

John Wu, Ben Brown, Paolo Calafiura, Quincey Koziol, Dongeun Lee, Alex Sim, Devesh Tiwari, Support for In-Flight Data Analyses in Scientific Workflows, DOE ASCR Workshop on the Management and Storage of Scientific Data, 2022, doi: 10.2172/1843500

A. Pereira, A. Sim, K. Wu, S. Yoo, H. Ito, "Data access pattern analysis for dCache storage system", International Conference on High Performance Computing in Asia-Pacific Region (HPC Asia 2022), 2022,

Ling Jin, Alina Lazar, Caitlin Brown, Bingrong Sun, Venu Garikapati, Srinath Ravulaparthy, Qianmiao Chen, Alexander Sim, Kesheng Wu, Tin Ho, Thomas Wenzel, C. Anna Spurlock, What Makes You Hold on to That Old Car? Joint Insights from Machine Learning and Multinomial Logit on Vehicle-level Transaction Decisions, Transportation Research Board 101st Annual Meeting, 2022,

Download File: wu2022.bib (bib: 22 KB)

J. Bang, C. Kim, K. Wu, A. Sim, S. Byna, H. Sung, H. Eom, "An In-Depth I/O Pattern Analysis in HPC Systems", IEEE International Conference on High Performance Computing, Data & Analytics (HiPC2021), 2021, doi: 10.1109/HiPC53243.2021.00056

S. Lee, Q. Kang, K. Wang, J. Balewski, A. Sim, A. Agrawal, A. Choudhary, P. Nugent, K. Wu, W-K. Liao, "Asynchronous I/O Strategy for Large-Scale Deep Learning Applications", IEEE International Conference on High Performance Computing, Data & Analytics (HiPC2021), 2021, doi: 10.1109/HiPC53243.2021.00046

A. Lazar, L. Jin, C. Brown, C. A. Spurlock, A. Sim, K. Wu, "Performance of the Gold Standard and Machine Learning in Predicting Vehicle Transactions", the 3rd International Workshop on Big Data Tools, Methods, and Use Cases for Innovative Scientific Discovery (BTSD 2021), 2021, doi: 10.1109/BigData52589.2021.9671286

J. Cheung, A. Sim, J. Kim, K. Wu, "Performance Prediction of Large Data Transfers", ACM/IEEE The International Conference for High Performanc