kaxil opened a new pull request, #73553: URL: https://github.com/apache/airflow/pull/73553
Follow-up to #73508, which added `ChainRetryPolicy` to the Task SDK in Airflow 3.4. Retry policies themselves arrived in 3.3, so a Dag on 3.3 can run a chain as long as the class comes from somewhere. This makes `airflow.providers.common.compat.sdk` that place: where the SDK has the class (3.4 and later) the compat module returns the SDK class, on 3.3 a copy in this provider stands in for it, and on 3.2 and below the name is absent, as retry policies are. ```python from airflow.providers.common.compat.sdk import ChainRetryPolicy, ExceptionRetryPolicy, RetryAction, RetryRule ``` The other retry policy names (`RetryPolicy`, `RetryDecision`, `RetryAction`, `RetryRule`, `ExceptionRetryPolicy`) are exported from the same module, so a chain is one import line. Those map straight to the SDK on every version that has them. The tests ran on `main` (Airflow 3.4.0), where compat resolves to the SDK class, and against a released Airflow 3.3.2 with Task SDK 1.3.2 and this provider installed from a wheel, where compat resolves to the copy. The behaviour tests import the copy directly and one drives it through the worker's own `_evaluate_retry_policy`, so both run on both versions. ## Design rationale **A copy in compat, rather than raising the Common AI floor to 3.4.** Common AI floors `apache-airflow>=3.0.0`; requiring 3.4 for one class would drop 3.0 to 3.3 for the whole provider, and it would still leave every other Dag author on 3.3 without the chain. Vendoring the class privately inside Common AI would serve one consumer. The compat provider exists for this gap and already carries class copies (the lineage entities) and polyfills, so a copy here serves any Dag on 3.3 through one import line. The cost is that the copy stays in the package until 3.3 leaves compat's support window, which is why the tests compare the source of both classes and fail on drift. **Resolved through the existing lazy import map, not a version gate at import time.** The compat sdk module resolves names on first access. Adding map entries means importing `airflow.providers.common.compat.sdk` gains no new eager import, and the SDK path is listed first, so wherever the SDK has the class the copy is never imported. The copy imports `RetryPolicy`, `RetryDecision` and `RetryAction` from the SDK rather than duplicating them, so it produces SDK decisions and is a subclass of the SDK base; the worker only calls `evaluate` and only a `has_retry_policy` flag is serialized, so nothing can tell the two classes apart. **Gated on 3.3 like the other 3-only names.** On Airflow 2 and on 3.0 to 3.2 the import raises `ImportError`, the way `DownstreamTasksSkipped` does there. A provider that supports those versions and imports any of the six names at module level has to guard the import, as Common AI already does for its SDK retry imports. The Common AI retry guide will point 3.3 users at this import once it is in a compat release. --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
