Metaxy¶
Metaxy is a metadata layer for multimodal data and ML pipelines. It tracks lineage and versioning across complex computational graphs, down to individual samples, and scales to millions of them. The data itself stays where it is (for example in S3), while Metaxy manages references to it.
Its distinguishing feature is tracking and caching partial data dependencies, which leads to huge time, compute and of course money savings.
Learn more in the docs.
See also:
- Announcement post by Anam
- Dagster + Metaxy by Dagster Labs
- Docling + Slurm + Metaxy by ASCII