CIQ

Pipeline

January 19, 2024

The primary processing component of a workflow, i.e., the main sequence of computational jobs some HPC-based task will perform as a part of execution. Generally refers to execution outside of a workflow engine’s control–so, for example, if you have a bunch of scripts you’re running that are coordinated by some master script, you have and are running a pipeline; but if all those scripts, their execution order, necessary resources, container image, etc., are codified with some kind of workflow engine, maybe with a YAML-based DSL, and the execution of this codification is then done by the workflow engine, then you have and are running a workflow.