Traditionally, HPC jobs are submitted as bash scripts to a batch scheduler. These scripts invariably involve specifics of the HPC system, and are not reproducible. This approach also requires micro-management and manual bookkeeping of input, output and temporary files—it is not at the right level of abstraction for scientists thinking about their science.

Workflow languages such as the Common Workflow Language (CWL) abstract away the manual bookkeeping, and provide a description of the workflow that is easily reproduced on other systems. In addition, CWL being a specification, it is easy to implement runners that are standards compliant but run on different execution environments—single machine, HPC cluster and cloud alike.

In this talk, I will demo ravanan—a new CWL implementation that is powered by GNU Guix to provide strong reproducibility guarantees and bullet-proof caching. Thanks to GNU Guix's robust hashing of the entire dependency tree, ravanan's cache is guaranteed to never go stale: thus you never have to recompute any step and never fear that your cache may be stale. ravanan also provides high cluster utilization thanks to its use of propagator networks for scheduling. Intermediate computational steps are run as early as possible without blocking, even when only partial results are available from previous steps.