Execution of long-running DAWs on distributed resources leaves the authors typically uncertain and uncomfortable about whether the DAW processing runs as planned. A continuous provision of execution-related information is thus decisive for the efficient development and adoption of DAWs.
T4 will thus create a framework for visually monitoring DAW executions and aims at improving the user experience and resource management during the DAW runtime, providing runtime feedback to developers during DAW creation, and supporting DAW debugging. It also illustrates the current DAW execution state, the infrastructure details such as underlying topology, the dependencies between DAW components, the files read and created, including their location and partitioning, and other attributes helping to oversee the execution process and to react early to upcoming problems.
The provided information can be used by the workflow authors to react quickly and deploy additional measures such as task migration, provisioning of additional resources, or an improved load balancing. T4 builds upon existing visualization tools for distributed processes to exploit synergies and focuses the efforts on developing high-quality methods for a holistic view on the current DAW state, despite the distributed placement of the DAW components.
- A2 on data access patterns from deployed platforms
- B1, B2 and B4 on visualization of available components in infrastructure, interconnecting topology, DAW placement
- B3 on resource utilization
- B5 on runtime estimation to determine the current DAW state and expected progress.
- B6 on malfunctions, providing additional debugging information
Towards Advanced Monitoring for Scientific Workflows Proceedings Article
In: 2022 IEEE International Conference on Big Data (IEEE BigData 2022), IEEE, 2022.