ashb opened a new pull request, #73916: URL: https://github.com/apache/airflow/pull/73916
The executor event sub-system in Airflow is a little bit confusing. This hopefully tidies things up a little bit. Previously events were keyed/stored by their "coordinates", the 4-tuple (dag_id, run_id, task_id, map_index) (and in some places a 5-tuple also including try_number). That is unwieldy, and since 3.0.0 a better option is available: a single UUID that uniquely identifies the TI try. This changes the event system to use that, with the net effect being that a deleyed event from a previous try is no longer able to to update the state for a newly scheduled replacement try. I have no evidence this has _ever_ happened, but it is conceptually cleaner, and lets us make some important assumptions going forward with the introduction of Task Loops. Since `apache-airflow-core` can be upgrade by itself and executors from not-yet-updated providers might be in use, we preserve their coordinate-keyed interfaces by capturing UUIDs before submission or adoption and translate legacy events from those snapshots without looking up the current row by coordinates. LocalExecutor opts into UUID keys end to end in this change. Provider-based executors will be their own PRs. ##### Was generative AI tooling used to co-author this PR? <!-- If generative AI tooling has been used in the process of authoring this PR, please change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by". --> - [ ] Yes (please specify the tool below) <!-- Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --> --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. --- <sub>Stack created with <a href="https://github.com/github/gh-stack";>GitHub Stacks CLI</a> • <a href="https://gh.io/stacks-feedback";>Give Feedback 💬</a></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
