B4: Proactive Network, I/O, and Storage Steering for Multiple DAWs on Shared Infrastructures - FONDA – Foundations of Workflows for Large-Scale Scientific Data Analysis

Continued from B4: Exploiting Software-Defined Networks for Efficient Data Management in Next-Generation Data Analysis Workflows

Description

During data analysis workflow (DAW) execution, energy-intensive components such as CPUs, GPUs, and main memory often remain idle when a system is waiting for distributed data access or network data transfers. Therefore, optimizing data access and network usage is essential not only for fast DAW execution, but also for achieving higher efficiency, e.g., in terms of energy, and hence for better sustainability – not least as power and cooling costs continue to rise.

The subproject focuses on the data transport between tasks, including scheduling of transmissions, network capacity management and data placement. The carbon-aware processing depends heavily on the efficient network transfer and data placement, as the network cost can – in the worst case – eliminate the optimization gains.

Scientists

Ansgar Lößer
Tobias Wies
Sami Kharma
Joel Witzke

PIs

Publications

16 entries « ‹ 1 of 2 › »

2025

Lehmann, Fabian; Bader, Jonathan; Tschirpke, Friedrich; Mecquenem, Ninon De; Lößer, Ansgar; Becker, Sören; Lewińska, Katarzyna Ewa; Thamsen, Lauritz; Leser, Ulf

WOW: Workflow-Aware Data Movement and Task Scheduling for Dynamic Scientific Workflows Proceedings Article

In: 2025 IEEE 25th International Symposium on Cluster, Cloud and Internet Computing (CCGrid), Tromsø, Norway, 2025.

BibTeX

Lößer, Ansgar; Witzke, Joel; Schintke, Florian; Scheuermann, Björn

BottleMod: Modeling Data Flows and Tasks for Fast Bottleneck Analysis Proceedings Article

In: Proceedings of the ICPE 2025, May 05–09, 2025, Toronto, Canada, pp. 1–8, 2025.

BibTeX

2024

Witzke, Joel; Lößer, Ansgar; Bountris, Vasilis; Schintke, Florian; Scheuermann, Björn

I/O of Scientific Workflows Monitored in Detail Proceedings Article

In: 2024 IEEE 20th International Conference on e-Science (e-Science), pp. 1-2, 2024.

Links | BibTeX

Bader, Jonathan; Lößer, Ansgar; Thamsen, Lauritz; Scheuermann, Björn; Kao, Odej

KS+: Predicting Workflow Task Memory Usage Over Time Proceedings Article

In: 2024 IEEE 20th International Conference on e-Science (e-Science), 2024.

Links | BibTeX

Witzke, Joel; Lößer, Ansgar; Bountris, Vasilis; Schintke, Florian; Scheuermann, Björn

Low-level I/O Monitoring for Scientific Workflows Journal Article

In: CoRR, vol. abs/2408.00411, 2024.

Links | BibTeX

Jami, Masoud

Optimizing Checkpoint/Restart and Input/Output for Large Scale Applications PhD Thesis

Humboldt-Universität zu Berlin, 2024.

Links | BibTeX

Schintke, Florian; Belhajjame, Khalid; Mecquenem, Ninon De; Frantz, David; Guarino, Vanessa Emanuela; Hilbrich, Marcus; Lehmann, Fabian; Missier, Paolo; Sattler, Rebecca; Sparka, Jan Arne; Speckhard, Daniel T.; Stolte, Hermann; Vu, Anh Duc; Leser, Ulf

Validity constraints for data analysis workflows Journal Article

In: Future Generation Computer Systems, vol. 157, pp. 82–97, 2024, ISSN: 0167-739X.

Abstract | Links | BibTeX

@article{SCHINTKE2024,

title = {Validity constraints for data analysis workflows},

author = {Florian Schintke and Khalid Belhajjame and Ninon De Mecquenem and David Frantz and Vanessa Emanuela Guarino and Marcus Hilbrich and Fabian Lehmann and Paolo Missier and Rebecca Sattler and Jan Arne Sparka and Daniel T. Speckhard and Hermann Stolte and Anh Duc Vu and Ulf Leser},

url = {https://www.sciencedirect.com/science/article/pii/S0167739X24001079},

doi = {https://doi.org/10.1016/j.future.2024.03.037},

issn = {0167-739X},

year  = {2024},

date = {2024-01-01},

urldate = {2024-01-01},

journal = {Future Generation Computer Systems},

volume = {157},

pages = {82--97},

abstract = {Porting a scientific data analysis workflow (DAW) to a cluster infrastructure, a new software stack, or even only a new dataset with some notably different properties is often challenging. Despite the structured definition of the steps (tasks) and their interdependencies during a complex data analysis in the DAW specification, relevant assumptions may remain unspecified and implicit. Such hidden assumptions often lead to crashing tasks without a reasonable error message, poor performance in general, non-terminating executions, or silent wrong results of the DAW, to name only a few possible consequences. Searching for the causes of such errors and drawbacks in a distributed compute cluster managed by a complex infrastructure stack, where DAWs for large datasets typically are executed, can be tedious and time-consuming. We propose validity constraints (VCs) as a new concept for DAW languages to alleviate this situation. A VC is a constraint specifying logical conditions that must be fulfilled at certain times for DAW executions to be valid. When defined together with a DAW, VCs help to improve the portability, adaptability, and reusability of DAWs by making implicit assumptions explicit. Once specified, VCs can be controlled automatically by the DAW infrastructure, and violations can lead to meaningful error messages and graceful behaviour (e.g., termination or invocation of repair mechanisms). We provide a broad list of possible VCs, classify them along multiple dimensions, and compare them to similar concepts one can find in related fields. We also provide a proof-of-concept implementation for the workflow system Nextflow.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

2023

Lößer, Ansgar; Raskob, Florian; Scheuermann, Björn

Lazy Read: Asynchronous Execution of Synchronous File I/O Proceedings Article

In: IEEE International Conference on Big Data, BigData 2023, Sorrento, Italy, December 15-18, 2023, pp. 2311-2318, IEEE, 2023.

Links | BibTeX

Witzke, Joel; Schintke, Florian; Lößer, Ansgar; Scheuermann, Björn

Proactive Resource Management to Optimize Distributed Workflow Executions Proceedings Article

In: He, Jingrui; Palpanas, Themis; Hu, Xiaohua; Cuzzocrea, Alfredo; Dou, Dejing; Slezak, Dominik; Wang, Wei; Gruca, Aleksandra; Lin, Jerry Chun-Wei; Agrawal, Rakesh (Ed.): IEEE International Conference on Big Data, BigData 2023, Sorrento, Italy, December 15-18, 2023, pp. 6305–6307, IEEE, 2023.

Links | BibTeX

Kondrateva, Olga; Dietzel, Stefan; Lößer, Ansgar; Scheuermann, Björn

Parameter Prioritization for Efficient Transmission of Neural Networks in Small Satellite Applications Proceedings Article

In: 2023 21st Mediterranean Communication and Computer Networking Conference (MedComNet), pp. 39-42, 2023.

Links | BibTeX

16 entries « ‹ 1 of 2 › »