B4: Proactive Network, I/O, and Storage Steering for Multiple DAWs on Shared Infrastructures

Continued from B4: Exploiting Software-Defined Networks for Efficient Data Management in Next-Generation Data Analysis Workflows

Description

During data analysis workflow (DAW) execution, energy-intensive components such as CPUs, GPUs, and main memory often remain idle when a system is waiting for distributed data access or network data transfers. Therefore, optimizing data access and network usage is essential not only for fast DAW execution, but also for achieving higher efficiency, e.g., in terms of energy, and hence for better sustainability – not least as power and cooling costs continue to rise.

The subproject focuses on the data transport between tasks, including scheduling of transmissions, network capacity management and data placement. The carbon-aware processing depends heavily on the efficient network transfer and data placement, as the network cost can – in the worst case – eliminate the optimization gains.