T5: DAW Benchmarking

The main goal of team T5 is to publish a comprehensive benchmark for DAW processing systems and apply it to FONDA’s subprojects. As far as our research shows that the scientific community is currently missing a benchmark specification including a representative set of real-life DAWs. T5 will fill this gap such that approaches that target DAWs as a use-case can be compared in an objective way. The foundation of the benchmark will be DAWs collected and developed by the subprojects of FONDA. As a final verification step all subprojects apply the benchmark to compare there approaches with the state-of-the-art.

Collaborations

  • S1 provides the infrastructures and links existing publicly available sets of DAWs.
  • A2 provides DAWs for genome data analyzes.
  • A3 provides DAWs from the computational material science domain and empirical evaluation results with focus on verification and validation.
  • A5 provides DAWs for biomedical image analyzes.
  • A6 provides DAWs that analyse large sets of genomes.
  • B1 uses the developed micro benchmarks.
  • B3 contributes experimental use-cases for the benchmark and develops a benchmark set mainly for error detection.
  • B4 provides a monitoring API for the visualization of DAW executions and the utilized infrastructure.
  • B5 provides large-scale, self-adapting analyzes of satellite data and contributes to the benchmark design.
  • B6 initially supposed to only apply the benchmark for there experiments. Nowadays B6 took over the coordination of the benchmark development.
  • T1 applies the benchmark for DAW language models.
  • T2 provides provenance traces for the benchmark.
  • T3 tests there models and provides soft and hard validity constraints.

Related Publications

2022

Lawrence Benson; Leon Papke; Tilmann Rabl

PerMA-Bench: Benchmarking Persistent Memory Access Proceedings Article

In: Proceedings of the VLDB Endowment, pp. 2463-2476, 2022.

Abstract | Links | BibTeX

Nina Ihde; Paula Marten; Ahmed Eleliemy; Gabrielle Poerwawinata; Pedro Silva; Ilin Tolovski; Florina M. Ciorba; Tilmann Rabl

A Survey of Big Data, High Performance Computing, and Machine Learning Benchmarks Proceedings Article

In: Nambiar, Raghunath; Poess, Meikel (Ed.): Performance Evaluation and Benchmarking, pp. 98–118, Springer International Publishing, Cham, 2022, ISBN: 978-3-030-94437-7.

Abstract | BibTeX