Yep., While the HDFS is a little bit "dated" - I think having direct Ozone support - especially backed by the follow ASF community members, is a good idea.
On Thu, Nov 12, 2026 at 11:10 PM v.chaplygin.arenadata.io via dev < [email protected]> wrote: > Hi Airflow community, > > I'd like to propose adding a new community provider for Apache Ozone. > > It would enter at the incubation stage. In provider.yaml the state > > stays not-ready until the first release. It targets Airflow 2.11+ > > and Python 3.11+. > > Integration overview > > -------------------- > > Apache Ozone (https://ozone.apache.org/) is an ASF top-level project: a > > distributed object store from the Hadoop ecosystem, often deployed next > > to HDFS. I work at Arenadata, and we use this integration in our > > Airflow fork. > > The Amazon provider can talk to Ozone only through its S3 gateway. It > > does not handle ofs:// and o3fs:// paths, Ozone volumes, quotas, or > > snapshots. This provider does. Filesystem and admin commands go through > > the ozone CLI, and Kerberos uses kinit from the connection. It adds an > > ozone connection type, hooks, operators, the OzoneKeySensor, > > HdfsToOzoneOperator (hadoop distcp) and OzoneBackupOperator for bucket > > snapshots. The only Python dependency is apache-airflow. Workers need > > the Ozone client, and the DistCp transfer also needs a Hadoop client. > > System tests > > ------------ > > Unit tests in the PR mock the ozone and hadoop CLIs and run without a > > cluster. Apache Airflow CI has no Ozone service. We already run > > integration tests against the apache/ozone Docker image in our fork. > > They need a new Breeze integration, so they are not part of this PR. > > We plan to propose that Breeze integration in a follow-up PR. > > We are not publishing a public system-test dashboard with this proposal. > > Proposed stewards > > ----------------- > > - Vyacheslav Chaplygin, @slavachy, [email protected] > > - Mikhail Zaplatin, @ZMichael3012, [email protected] > > - Aleksey Gladkih, @gaxeliy, [email protected] > > - Dmitry Bezlutskiy, @Dimitrionian, [email protected] > > Sponsoring committer(s) > > ----------------------- > > - Amogh Desai, @amoghrajesh > > Working implementation > > ---------------------- > > https://github.com/apache/airflow/pull/75978 > > Incubation commitment > > --------------------- > > We commit to: > > - Maintaining the provider and responding to issues within a reasonable > time > > - Meeting the incubation health metrics within 6 months > > - Participating in quarterly governance updates > > Best regards, > > Vyacheslav Chaplygin > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
