dagster-ray
dagster-ray allows easily running Ray computations in Dagster pipelines. It provides various Dagster abstractions, the most important being Resource
, and helper @op
s and @schedule
s, for multiple backends.
Essentially, it enables writing very simple Python code, similar to:
return
Behind the scenes, dagster-ray
will manage the KubeRay
’s RayCluster
lifecycle, spinning it up when the pipeline starts, and tearing it down after the pipeline finishes. The function body can then execute ray
code in a distributed cluster without any additional setup.