Sammi Chen created HDDS-16503:
---------------------------------
Summary: Ozone as LanceDB storage backend
Key: HDDS-16503
URL: https://issues.apache.org/jira/browse/HDDS-16503
Project: Apache Ozone
Issue Type: New Feature
Reporter: Sammi Chen
Assignee: Sammi Chen
Motivation
As AI and ML workloads grow, organizations running on-premises Hadoop clusters
need a local, high-performance multimodal data store. LanceDB's Lance format is
emerging as the dominant open format for this use case, analogous to what
Parquet is for analytics. Supporting Ozone as a LanceDB backend would:
Give Ozone users a production-grade vector and multimodal data layer without
migrating to cloud object storage.
Position Ozone as an AI-ready storage system alongside S3, GCS, and Azure Blob.
Enable co-location of training data (HDFS/Ozone), vector indexes (LanceDB on
Ozone), and compute (YARN/Spark) within a single on-premises cluster.
Technical Approach
LanceDB communicates with object storage through the object_store Rust crate
abstraction. Any backend that correctly implements the required S3 operations
is usable. Ozone's S3 Gateway already covers the core surface area, a POC has
proved that.
This is an umbrella Jira for further optimization and documentation tasks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
