amoghrajesh opened a new pull request, #76575:
URL: https://github.com/apache/airflow/pull/76575
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
### What
The Registry marks an operator as supporting durable execution, and stops
there. A Dag author still cannot tell which provider version they need, or
whether the operator ships with durable execution switched off. Both facts are
already derivable from the sources the extraction pipeline reads, so nobody
should have to open a changelog for them.
### Current behaviour
`listModules` carries one durable field, `supports_durable_execution`. The
`/durable/` page shows a badge and a hardcoded sentence in the legend saying
durability is "on by default for most, opt-in for the AI agent operators",
which has to be edited by hand every time that stops being true.
### Proposed change
Five derived fields on each module entry:
| Field | |
|---|---|
| `durable_mechanism` | `marker` or `resumable_mixin` |
| `durable_gating_provider` | the provider whose version gates the
capability |
| `durable_gating_class` | the class the capability comes from |
| `durable_since_version` | earliest release of that provider that has it |
| `durable_default` | `false` where `durable` is opt-in, null where
`__init__` computes it |
Nothing here is declared by a provider maintainer. It is all read from the
classes at extraction time.
The `/durable/` page shows the version and the opt-in state as badges beside
the existing Durable one, and the hardcoded legend sentence is gone.
### Changes of Note
**Why a gating provider.** Four operators inherit durability from a base
class in another provider. `EksPodOperator` ships in `amazon` and gets
durability from `KubernetesPodOperator` in `cncf-kubernetes`, so an amazon
version number says nothing about it. Its badge reads `cncf-kubernetes
10.20.0+`, and the provider name appears only when it differs from the one the
module ships in.
**How the version is found.** One `git grep` across a provider's release
tags returns every release whose copy of the gating class's file contains the
mechanism, and the lowest wins. I tried two other alternatives and dropped
them.
a) Parsing changelogs gets 6 of 9 providers right: Livy's entry reads
"Reconnect to the running Livy batch on retry instead of resubmitting" and
never says durable
b) Broader keyword matching hits 2021 entries about `reattach_on_restart`
and dates Kubernetes to 2.0.3. Bisecting with `git show` per release works but
costs seven calls per operator and assumes the capability was added once and
kept.
**Why `task_state_store` and not the marker.** Dating a marker operator by
its `__supports_durable_execution` declaration reports when it was advertised,
not when it worked. `KubernetesPodOperator` reconnected through
`task_state_store` in 10.20.0 and only declared the marker in 10.21.0.
**Why the version lookup runs on the host.** Breeze does not mount `.git`,
so a git command inside the container sees nothing. `fill_durable_versions.py`
runs after the container step and `breeze registry extract-data` drives both. A
checkout with no `providers-*` tags leaves `durable_since_version` null and
says so. Other git failures are silent, which is worth knowing when reading a
run that dated zero modules.
### User implications / backcompat
All five fields default to null or false, so previously generated
`modules.json` from S3 still validates against the contract and the incremental
build path is unaffected. No provider API changes.
### Testing and some screenshots
Extraction run through Breeze across `cncf-kubernetes`, `amazon`,
`common-ai`, `apache-livy`
and `microsoft-azure` (654 modules, 12 durable). Every version below was
then checked
against `git show <tag>:<file>` independently of the new code:
| | gating provider | since |
|---|---|---|
| KubernetesPodOperator | cncf-kubernetes | 10.20.0 |
| EksPodOperator | cncf-kubernetes | 10.20.0 |
| GlueJobOperator | amazon | 9.35.0 |
| RedshiftDataOperator | amazon | 9.33.0 |
| LivyOperator | apache-livy | 4.6.0 |
| AgentOperator | common-ai | 0.7.0 |
| MSGraphAsyncOperator | microsoft-azure | 15.3.0 |
<img width="1466" height="1049" alt="image"
src="https://github.com/user-attachments/assets/27b3fc5f-be40-41c7-9473-363f23b6f3b0";
/>
<img width="1466" height="1049" alt="image"
src="https://github.com/user-attachments/assets/ac1fa6a9-cb70-4764-b3cc-7759813a3641";
/>
<img width="1139" height="174" alt="image"
src="https://github.com/user-attachments/assets/d8b88a2e-e565-47ab-b336-67b25c378005";
/>
<img width="1139" height="174" alt="image"
src="https://github.com/user-attachments/assets/5af29327-177c-4bec-bd9b-40eb4a0db5a2";
/>
related: #67236
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
