The Pegasus Team
On February 23 and 25, USC’s Information Sciences Institute will be hosting the first ever Pegasus Users Group (PUG) Meeting. The virtual workshop will provide a space for Pegasus users and collaborators to interact with developers and share their feedback about the software.
Along with user experience talks, members from the Pegasus development team will present technical talks and tutorials. PUG 2021 comes on the heels of the Pegasus 5.0 release last year, which included numerous key updates.
A Constellation of Computers
The Pegasus system is comprised of a network of computers that communicate with each other to support researchers in designing workflows, which are step-by-step methods for efficiently analyzing large datasets to help streamline scientific research. Ewa Deelman, research professor of Computer Science at USC Viterbi and research director of the Science Automation Technologies division at ISI, is the PI of the Pegasus Software Project. Researching scientific workflows for over 20 years, Deelman has led the design and development of Pegasus since its inception in 2001.
“Pegasus bridges the scientific domain and the compute environment by allowing scientists to construct workflows in abstract terms without worrying about the details of the underlying execution environment or the particulars of the low-level system specifications,” she said. “[It] can run workflows ranging from just a few computational tasks up to 1 million.”
Funded by numerous US federal agencies over the years such as the NSF, NIH, and DoE, Pegasus was notably used by the Laser Interferometer Gravitational-Wave Observatory (LIGO) collaboration for the analysis that confirmed the existence of gravitational waves. It was also used by the Southern California Earthquake Center (SCEC) to generate the first ever physics-based probabilistic seismic hazard map of Southern California. Today, Pegasus supports workflows on numerous platforms, including Amazon EC2, ExoGeni, Open Science Grid, XSEDE, DOE leadership class systems, and many campus clusters.
New Features for New Needs
The widely applicable and user-friendly nature of Pegasus makes it appealing to an interdisciplinary scientific community. “Pegasus users are domain scientists in fields such as earthquake engineering, geo sciences, astronomy, gravitational wave scientists, bioinformatics, and natural language processing,” said Rafael Ferreira da Silva, research lead at ISI. “They use Pegasus to efficiently execute their science pipelines and scale up the computations to process large data sets, to increase the complexity of the simulation, or to increase the number of resources used to improve the time to solution.”
The 2020 Pegasus 5.0 release included multiple key updates. Pegasus is middleware, also called “software glue” for its ability to bridge an operating system with the applications that are running on it. As the scientific computation landscape advances, user applications have become more complex and require more refined capabilities. This is why it’s important for middleware such as Pegasus to evolve as well so that it can continue serving the science community.
“The Pegasus 5.0 release addresses key improvements of the system usability, the development of core functionalities for improving the management and processing of large, distributed data sets, and the management of experiment campaigns defined as ensembles,” Karan Vahi, lead architect for Pegasus, said. “Specifically, Pegasus 5.0 provides a new Python3 based Pegasus API that allows users to compose workflows and to control their execution programmatically.”
This feature, Vahi explained, is key for seamlessly integrating Pegasus with the JupyterLab framework, a popular open-source web app that allows for easy code development and sharing. This app is frequently used by the science community as a tool for reproducible research. “As data and computational methods are the foundation of scientific progress, we’ve strengthened Pegasus’ support for the management of large datasets,” he continued. “We’ve also developed a novel component for managing experiment campaigns, named Ensemble Manager, in which users can automate/orchestrate collections of workflows within an ensemble via a set of rules defined as triggers (e.g., time interval, data production, etc.).”
The continued development of the Ensemble Manager is a key future direction of the system. “We plan to improve support for experiment campaigns (central to in-situ processing and ML studies) and thus allow scientists to design and fine tune their experiments at runtime,” explained Pegasus developer Ryan Tanaka. “To this end, we’ll extend current monitoring capabilities for allowing fine-grained interactive oversight of the experiment execution.”
Pegasus 5.0 includes preliminary support for CWL (Common Workflow Language), a widely adopted format for describing workflows within the bioinformatics community. The updates also modernize and standardize Pegasus internal formats and interfaces.
As the popularity of Pegasus continues to increase, the development team decided that a workshop was in order. Since its inception 20 years ago, the team mostly interacted with Pegasus users in a more informal, ad hoc manner, usually through tutorials, events, conferences and project meetings.
“As Pegasus usage and its communities have steadily grown, we felt the need for bringing the science and research communities together to share their experiences as well as underline pressing needs for current and emerging challenges with regards to their applications, computing environments, etc.,” explained Mats Rynge, lead for Pegasus user support.
Pegasus shows no signs of slowing down. With the continued evolution and adoption of Machine Learning (ML) in the sciences, researchers are increasingly dealing with larger and larger volumes of data and in need of efficient, automated workflows that can handle large-scale scientific simulations on computing infrastructures.
“ML workflows are structured differently from traditional workflow applications, in the sense that they feature feedback loops and usually exhibit more dynamic behavior,” said Loïc Pottier, Pegasus ML group lead. “We’re in the midst of exploring how we can improve Pegasus to better support such ML-empowered scientific workflows. We’re also conducting research into how ML methods help us predict and categorize anomalies and failures during workflow execution, so we can enhance Pegasus failure detection and recovery mechanisms.”
The inaugural PUG Meeting will serve as a collaboration between users and developers to ensure Pegasus continues to be the standard for workflow management systems. “[Our aim] is to foster a sense of community amongst our users, where they can interact with each other in an open and positive space,” said Rynge. “We expect this meeting will help distill the foremost research and development directions for future Pegasus releases.”
For more information, including registration and agenda details, click here.
Published on February 4th, 2021
Last updated on July 1st, 2021
