B5: Adaptive, Distributed and Scalable Analysis of Massive Satellite Data – FONDA – Foundations of Workflows for Large-Scale Scientific Data Analysis

Description

Subproject B5 investigates means to improve adaptability and portability of DAWs for large-scale analysis of satellite data. DAWs for such problems are often long and involved and include tasks with very heterogeneous resource requirements (in terms of memory, runtime, and bandwidth). As the manner in which the concrete requirements of tools depend on the input data is a-priori unknown, these DAWs are very difficult to schedule. Today, this problem typically is solved by hardwiring data- and infrastructure dependent scheduling decisions into the DAW. B5 will research new methods for adaptive scheduling of DAWs that adapt to the concrete remote sensing scenario. B5 is an interdisciplinary subproject, led by Prof. Hostert, expert in large-scale satellite data analysis, and Prof. Leser, an expert in workflow management systems for large-scale scientific data analysis.

PIs

Hostert
Leser

Publications

2022

Jonathan Bader; Joel Witzke; Soeren Becker; Ansgar Lößer; Fabian Lehmann; Leon Doehler; Anh Duc Vu; Odej Kao

Towards Advanced Monitoring for Scientific Workflows Inproceedings

In: 2022 IEEE International Conference on Big Data (IEEE BigData 2022), IEEE, 2022.

Links | BibTeX

Jonathan Bader; Fabian Lehmann; Alexander Groth; Lauritz Thamsen; Dominik Scheinert; Jonathan Will; Ulf Leser; Odej Kao

Reshi: Recommending Resources for Scientific Workflow Tasks on Heterogeneous Infrastructures Inproceedings

In: 41th International Performance Computing and Communications Conference 2022, IEEE, 2022.

Links | BibTeX

Jonathan Bader; Fabian Lehmann; Lauritz Thamsen; Jonathan Will; Ulf Leser; Odej Kao

Lotaru: Locally Estimating Runtimes of Scientific Workflow Tasks in Heterogeneous Clusters Inproceedings

In: 34th International Conference on Scientific and Statistical Database Management (SSDBM 2022), pp. 1–12, ACM, 2022.

Links | BibTeX

Marcus Hilbrich; Fabian Lehmann

ßMACH — A Software Management Guidance Inproceedings

In: Reichelt, David Georg; Müller, Richard; Becker, Steffen; Hasselbring, Wilhelm; Hoorn, André; Kounev, Samuel; Koziolek, Anne; Reussner, Ralf (Ed.): Symposium on Software Performance 2021, CEUR-WS, Leipzig, Germany, 2022.

Abstract | Links | BibTeX

David Frantz; Patrick Hostert; Philippe Rufin; Stefan Ernst; Achim Röder; Sebastian Linden

Revisiting the Past: Replicability of a Historic Long-Term Vegetation Dynamics Assessment in the Era of Big Data Analytics Journal Article

In: Remote Sensing, vol. 14, no. 3, 2022, ISSN: 2072-4292.

Abstract | Links | BibTeX

@article{rs14030597,

title = {Revisiting the Past: Replicability of a Historic Long-Term Vegetation Dynamics Assessment in the Era of Big Data Analytics},

author = {David Frantz and Patrick Hostert and Philippe Rufin and Stefan Ernst and Achim Röder and Sebastian Linden},

url = {https://www.mdpi.com/2072-4292/14/3/597},

doi = {10.3390/rs14030597},

issn = {2072-4292},

year  = {2022},

date = {2022-01-01},

urldate = {2022-01-01},

journal = {Remote Sensing},

volume = {14},

number = {3},

abstract = {Open and analysis-ready data, as well as methodological and technical advancements have resulted in an unprecedented capability for observing the Earth’s land surfaces. Over 10 years ago, Landsat time series analyses were inevitably limited to a few expensive images from carefully selected acquisition dates. Yet, such a static selection may have introduced uncertainties when spatial or inter-annual variability in seasonal vegetation growth were large. As seminal pre-open-data-era papers are still heavily cited, variations of their workflows are still widely used, too. Thus, here we quantitatively assessed the level of agreement between an approach using carefully selected images and a state-of-the-art analysis that uses all available images. We reproduced a representative case study from the year 2003 that for the first time used annual Landsat time series to assess long-term vegetation dynamics in a semi-arid Mediterranean ecosystem in Crete, Greece. We replicated this assessment using all available data paired with a time series method based on land surface phenology metrics. Results differed fundamentally because the volatile timing of statically selected images relative to the phenological cycle introduced systematic uncertainty. We further applied lessons learned to arrive at a more nuanced and information-enriched vegetation dynamics description by decomposing vegetation cover into woody and herbaceous components, followed by a syndrome-based classification of change and trend parameters. This allowed for a more reliable interpretation of vegetation changes and even permitted us to disentangle certain land-use change processes with opposite trajectories in the vegetation components that were not observable when solely analyzing total vegetation cover. The long-term budget of net cover change revealed that vegetation cover of both components has increased at large and that this process was mainly driven by gradual processes. We conclude that study designs based on static image selection strategies should be critically evaluated in the light of current data availability, analytical capabilities, and with regards to the ecosystem under investigation. We recommend using all available data and taking advantage of phenology-based approaches that remove the selection bias and hence reduce uncertainties in results.},

keywords = {},

pubstate = {published},

tppubtype = {article}

}

Open and analysis-ready data, as well as methodological and technical advancements have resulted in an unprecedented capability for observing the Earth’s land surfaces. Over 10 years ago, Landsat time series analyses were inevitably limited to a few expensive images from carefully selected acquisition dates. Yet, such a static selection may have introduced uncertainties when spatial or inter-annual variability in seasonal vegetation growth were large. As seminal pre-open-data-era papers are still heavily cited, variations of their workflows are still widely used, too. Thus, here we quantitatively assessed the level of agreement between an approach using carefully selected images and a state-of-the-art analysis that uses all available images. We reproduced a representative case study from the year 2003 that for the first time used annual Landsat time series to assess long-term vegetation dynamics in a semi-arid Mediterranean ecosystem in Crete, Greece. We replicated this assessment using all available data paired with a time series method based on land surface phenology metrics. Results differed fundamentally because the volatile timing of statically selected images relative to the phenological cycle introduced systematic uncertainty. We further applied lessons learned to arrive at a more nuanced and information-enriched vegetation dynamics description by decomposing vegetation cover into woody and herbaceous components, followed by a syndrome-based classification of change and trend parameters. This allowed for a more reliable interpretation of vegetation changes and even permitted us to disentangle certain land-use change processes with opposite trajectories in the vegetation components that were not observable when solely analyzing total vegetation cover. The long-term budget of net cover change revealed that vegetation cover of both components has increased at large and that this process was mainly driven by gradual processes. We conclude that study designs based on static image selection strategies should be critically evaluated in the light of current data availability, analytical capabilities, and with regards to the ecosystem under investigation. We recommend using all available data and taking advantage of phenology-based approaches that remove the selection bias and hence reduce uncertainties in results.

Marcus Hilbrich; Fabian Lehmann

Discussing Microservices: Definitions, Pitfalls, and their Relations Inproceedings

In: 2022 IEEE International Conference on Services Computing (SCC), pp. 39-44, 2022.

Links | BibTeX

2021

Fabian Lehmann; David Frantz; Soeren Becker; Ulf Leser; Patrick Hostert

FORCE on Nextflow: Scalable Analysis of Earth Observation Data on Commodity Clusters Inproceedings

In: CIKM Workshops, 2021.

BibTeX