Hi Airflow community,

We'd like to propose adding a new community provider for Apache Polaris:
apache-airflow-providers-apache-polaris.

Integration overview
---------------------------
Apache Polaris (https://polaris.apache.org/) is an open-source catalog for
Apache Iceberg. It implements the Iceberg REST catalog API and adds
centralized access control, credential vending and catalog-level policies.
Many teams running Iceberg already orchestrate their data platform with
Airflow. Today, managing Polaris from a Dag means hand-written REST calls
or shelling out to the Polaris CLI.

The proposed provider would add:

- A "polaris" connection type. It supports Polaris's internal OAuth tokens,
and it masks every token it fetches.
- Hooks for the Polaris Management API and the catalog APIs. They can also
hand ready-made catalog properties to pyiceberg or to Spark and other
engines.
- Operators for catalogs, namespaces, tables, principals, principal and
catalog roles, grants, and policies. Typical uses include provisioning a
team's catalog with its roles and grants, onboarding a pipeline principal,
and managing table lifecycle.
- A helper that turns Polaris table-maintenance policies (compaction,
snapshot expiry, orphan-file removal) into Iceberg Spark procedure calls,
so a Dag can execute the maintenance a policy defines.

It is built on the official Apache Polaris Python SDK
(apache-polaris>=1.9.0). It needs Airflow 3.1.1+ and Python 3.11+, and
pyiceberg is an optional extra. It has been tested against Polaris server
1.8 and 1.9, and installs cleanly against Airflow's constraints for every
3.1.1–3.4.0 release.

System tests
-----------------
Polaris is open source and runs as a single container, so no external paid
service is involved. The PR adds:

- a Breeze integration (breeze --integration polaris) that starts Polaris
1.9.0 in CI, with integration tests against it;
- system-test example Dags for provisioning, tables and policies, which run
against the same server.

All tests can run in Airflow's own CI, so no external dashboard is needed.

Proposed stewards
-------------------------
- Yufei Gu, "flyrain", [email protected] (Apache Polaris PMC/Committer)
- Yong Zheng, "MonkeyCanCode", [email protected] (Apache Polaris
PMC/Committer)
- Xiaodong Deng, "XD-DENG", [email protected] (Apache Airflow PMC/Committer)

Sponsoring committer
-----------------------------
- Xiaodong Deng, "XD-DENG", [email protected]

Working implementation
--------------------------------
Draft PR: https://github.com/apache/airflow/pull/77188. It includes unit,
integration and system tests, and how-to docs for every operator.

Incubation commitment
-------------------------------
We commit to:
- maintaining the provider and responding to issues within a reasonable
time;
- meeting the incubation health metrics within 6 months;
- participating in quarterly governance updates.


Regards,

XD

Reply via email to