Asynchronous Scientific Workflows for Simulation-Driven Inference Across the Edge–HPC Computing Continuum

Scientific workflows are increasingly executed across heterogeneous computing continuum spanning edge devices, dedicated clusters, cloud resources, and shared high-performance computing (HPC) systems. Many emerging applications require continuous low-latency inference from streaming observations while periodically incorporating improved models generated by computationally intensive simulations. These competing requirements make workflow orchestration challenging: inference must proceed continuously even though simulation and model generation are delayed by resource availability, workflow scheduling, and batch queue dynamics.

We present RBF (Reverse Backfill), an asynchronous scientific workflow architecture for simulation-driven inference across the edge--HPC computing continuum. RBF decouples workflow stages through the Workflow Artifact Manager (WAM), which maintains persistent workflow state and versioned workflow artifacts, enabling resilient asynchronous coordination across heterogeneous resources. Dedicated computing resources provide predictable model regeneration while shared HPC systems opportunistically augment workflow throughput without modifying existing batch schedulers. To exploit this opportunistic computation effectively, RBF introduces a decision-time scheduling heuristic that determines whether a queued HPC workflow stage is expected to improve the currently deployed surrogate model, allowing shared resources to be consumed only when the anticipated benefit justifies the allocation.

We instantiate RBF using a digital agriculture application that couples edge sensing with computational fluid dynamics simulations to infer spatial airflow within a large citrus screenhouse. Experimental results characterize end-to-end workflow behavior, including simulation latency, model regeneration cadence, workflow staleness, opportunistic execution, and edge deployment overhead. Our results show that asynchronous workflow execution maintains continuous low-latency inference while opportunistic HPC execution significantly reduces model staleness and selectively improves model quality through decision-time adaptation. More broadly, RBF demonstrates how scientific workflows can effectively integrate shared HPC resources into operational cyber-physical systems across the computing continuum without requiring changes to existing scheduling policies.