+1 too.

Agreed with Jarek, this integration will help on-prem users who are moving
off HDFS/Oozie with native ofs:// support in Airflow.

Thanks & Regards,
Amogh Desai


On Sat, Nov 14, 2026 at 6:20 AM Jarek Potiuk <[email protected]> wrote:

> Yep., While the HDFS is a little bit "dated" - I think having direct Ozone
> support - especially backed by the follow ASF community members, is a good
> idea.
>
> On Thu, Nov 12, 2026 at 11:10 PM v.chaplygin.arenadata.io via dev <
> [email protected]> wrote:
>
> > Hi Airflow community,
> >
> > I'd like to propose adding a new community provider for Apache Ozone.
> >
> > It would enter at the incubation stage. In provider.yaml the state
> >
> > stays not-ready until the first release. It targets Airflow 2.11+
> >
> > and Python 3.11+.
> >
> > Integration overview
> >
> > --------------------
> >
> > Apache Ozone (https://ozone.apache.org/) is an ASF top-level project: a
> >
> > distributed object store from the Hadoop ecosystem, often deployed next
> >
> > to HDFS. I work at Arenadata, and we use this integration in our
> >
> > Airflow fork.
> >
> > The Amazon provider can talk to Ozone only through its S3 gateway. It
> >
> > does not handle ofs:// and o3fs:// paths, Ozone volumes, quotas, or
> >
> > snapshots. This provider does. Filesystem and admin commands go through
> >
> > the ozone CLI, and Kerberos uses kinit from the connection. It adds an
> >
> > ozone connection type, hooks, operators, the OzoneKeySensor,
> >
> > HdfsToOzoneOperator (hadoop distcp) and OzoneBackupOperator for bucket
> >
> > snapshots. The only Python dependency is apache-airflow. Workers need
> >
> > the Ozone client, and the DistCp transfer also needs a Hadoop client.
> >
> > System tests
> >
> > ------------
> >
> > Unit tests in the PR mock the ozone and hadoop CLIs and run without a
> >
> > cluster. Apache Airflow CI has no Ozone service. We already run
> >
> > integration tests against the apache/ozone Docker image in our fork.
> >
> > They need a new Breeze integration, so they are not part of this PR.
> >
> > We plan to propose that Breeze integration in a follow-up PR.
> >
> > We are not publishing a public system-test dashboard with this proposal.
> >
> > Proposed stewards
> >
> > -----------------
> >
> > - Vyacheslav Chaplygin, @slavachy, [email protected]
> >
> > - Mikhail Zaplatin, @ZMichael3012, [email protected]
> >
> > - Aleksey Gladkih, @gaxeliy, [email protected]
> >
> > - Dmitry Bezlutskiy, @Dimitrionian, [email protected]
> >
> > Sponsoring committer(s)
> >
> > -----------------------
> >
> > - Amogh Desai, @amoghrajesh
> >
> > Working implementation
> >
> > ----------------------
> >
> > https://github.com/apache/airflow/pull/75978
> >
> > Incubation commitment
> >
> > ---------------------
> >
> > We commit to:
> >
> > - Maintaining the provider and responding to issues within a reasonable
> > time
> >
> > - Meeting the incubation health metrics within 6 months
> >
> > - Participating in quarterly governance updates
> >
> > Best regards,
> >
> > Vyacheslav Chaplygin
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [email protected]
> > For additional commands, e-mail: [email protected]
> >
> >
>

Reply via email to