Pipeline

Demo: pipeline

Source: demo/pipeline/app.py in the repository.

prep → train ×3 (in parallel) → evaluate → package. There is no DAG object: dependencies are plain Python. .remote() blocks until that step’s container returns, .map() fans out one container per input, and a shared volume carries the files between stages.

cd demo/pipeline

# run the whole pipeline (about a minute on a warm cluster)
python app.py
# [1/4] prep ...  [2/4] train x3 in parallel ...  [3/4] evaluate ...  [4/4] package ...
# PIPELINE DONE: {"version": "0.1.0", "elapsed_s": 41.2, "artifact": "artifacts/release-0.1.0.tar.gz", ...}

# what it produced
levers9 ls pipeline-artifacts
levers9 cp pipeline-artifacts/metrics.json ./metrics.json && cat metrics.json
levers9 cp pipeline-artifacts/release-0.1.0.tar.gz ./release.tar.gz

Notes

• The quality gate in run_pipeline aborts if the best model’s MSE is above 5.0: put your own checks there.
• To run it nightly, wrap run_pipeline with @cron(when="0 2 * * *") (see ../cron).
• The dashboard’s Apps page lists the four functions in execution order; Containers shows the fan-out.