Hi Airflow community, I'd like to propose adding a new community provider for Apache Ozone.
It would enter at the incubation stage. In provider.yaml the state stays not-ready until the first release. It targets Airflow 2.11+ and Python 3.11+. Integration overview -------------------- Apache Ozone (https://ozone.apache.org/) is an ASF top-level project: a distributed object store from the Hadoop ecosystem, often deployed next to HDFS. I work at Arenadata, and we use this integration in our Airflow fork. The Amazon provider can talk to Ozone only through its S3 gateway. It does not handle ofs:// and o3fs:// paths, Ozone volumes, quotas, or snapshots. This provider does. Filesystem and admin commands go through the ozone CLI, and Kerberos uses kinit from the connection. It adds an ozone connection type, hooks, operators, the OzoneKeySensor, HdfsToOzoneOperator (hadoop distcp) and OzoneBackupOperator for bucket snapshots. The only Python dependency is apache-airflow. Workers need the Ozone client, and the DistCp transfer also needs a Hadoop client. System tests ------------ Unit tests in the PR mock the ozone and hadoop CLIs and run without a cluster. Apache Airflow CI has no Ozone service. We already run integration tests against the apache/ozone Docker image in our fork. They need a new Breeze integration, so they are not part of this PR. We plan to propose that Breeze integration in a follow-up PR. We are not publishing a public system-test dashboard with this proposal. Proposed stewards ----------------- - Vyacheslav Chaplygin, @slavachy, [email protected] - Mikhail Zaplatin, @ZMichael3012, [email protected] - Aleksey Gladkih, @gaxeliy, [email protected] - Dmitry Bezlutskiy, @Dimitrionian, [email protected] Sponsoring committer(s) ----------------------- - Amogh Desai, @amoghrajesh Working implementation ---------------------- https://github.com/apache/airflow/pull/75978 Incubation commitment --------------------- We commit to: - Maintaining the provider and responding to issues within a reasonable time - Meeting the incubation health metrics within 6 months - Participating in quarterly governance updates Best regards, Vyacheslav Chaplygin --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
