dagster-ray

dagster-ray allows easily running Ray computations in Dagster pipelines. It provides various Dagster abstractions, the most important being Resource, and helper @ops and @schedules, for multiple backends.

Essentially, it enables writing very simple Python code, similar to:

from dagster import asset, Definitions
from dagster_ray.kuberay import KubeRayCluster
import ray


@asset
def my_asset(
    ray_cluster: KubeRayCluster,
):
    return ray.get(ray.put(42))

Behind the scenes, dagster-ray will manage the KubeRay’s RayCluster lifecycle, spinning it up when the pipeline starts, and tearing it down after the pipeline finishes. The function body can then execute ray code in a distributed cluster without any additional setup.