dheerajturaga opened a new pull request, #75577: URL: https://github.com/apache/airflow/pull/75577
Part of [AIP-125 (Inbound Webhooks)](https://lists.apache.org/thread/p3dpnor4zwko8tldpncj6ftj1bpp18q7), which lets external services such as GitHub trigger Dags by sending webhooks to Airflow. This PR shows and documents `cancel_on`, which the previous PR in this stack adds. The deliveries API and UI list the cancel requests that each delivery recorded and what the scheduler did with them. The Dag and task instance pages show the conditions, `airflow webhooks check` lists them, and a new page tells Dag authors how to declare a condition and what a cancel does. It has one commit per part and is best reviewed commit by commit. Cancelling runs from webhooks is Phase 3 of AIP-125. It is roadmap, outside the vote on the earlier phases. Here is how the previous PR works. A Dag or a task declares `cancel_on=[WebhookMatch(...)]`. When a delivery has a correlation key, such as `my-org/web#42` for a GitHub pull request, and matches a condition, it records one cancel request per Dag in the delivery's transaction. The scheduler then applies the request, within a second or two, to the Dag's queued and running runs about that key. A Dag's `cancel_on` fails those runs. A task's `cancel_on` sets that task to its `cancel_state`, which is `failed` by default or `skipped`. The endpoint only records the request, for two reasons. It must answer within the sender's time budget, which is 10 seconds for GitHub, and the work of a cancel grows with the size of the Dags and the number of runs. Also, the internet-facing webhooks process should load no Dag and write no run or task instance state. Because of that split, each request has a status (`pending`, `applied` or `failed`) and a result, which this PR shows. ### Commit 1: AIP-125: Show cancel conditions and cancel requests This commit shows what deliveries can cancel and what they did cancel. Each delivery in `GET /api/v2/webhooks/{webhook_name}/deliveries` gains `cancel_requests`, read from `webhook_cancel_request`, with each request's targets, status, `applied_at` and result. The delivery details panel lists them under **Cancel Requests**, links the runs and task instances that each request cancelled, and explains the new **Cancel Requested** outcome. The Dag details and task responses gain `cancel_on`. The Dag details page shows it in a **Cancel On** row, and the task instance page in a **Cancel On** section. `airflow webhooks check` prints one line per condition, such as `Webhook github cancels the runs of Dag deploy_preview on pull_request deliveries with action closed, repository my-org/web, and manual runs by conf key pr`. Points to look at: - The request rows are authoritative. A delivery's `actions.cancel_requests` is only a summary written with the delivery. It can miss a request that a concurrent redelivery recorded, and it still names a request whose row `airflow db clean` removed or that was deleted with its Dag. The panel lists the rows first, then each `actions` entry that has no row, shown as gone. - Loading the rows costs one query per page, by delivery record id. A page where no delivery has a correlation key costs none, since only such a delivery records requests. `test_cancel_requests_of_a_page_cost_one_query_and_none_without_any` pins this. A failed request whose stored result holds only the error is returned with empty lists, so it does not fail the page. - The outcome texts follow the precedence that the previous PR sets: `resumed_wait` > `accepted` > `cancel_requested` > `early` > `dag_paused` > `no_subscribers`. This order differs from the first design, where `cancel_requested` ranked above `accepted`, was never kept for waits, and a redelivery was a 200 duplicate. That order would have broken Phase 2 behaviours: a wait registered later could not claim such a delivery, and a redelivery after unpausing a Dag would not start it. - A `cancel_requested` delivery is kept for webhook waits as an `early` one is, and counts toward `[webhooks] early_delivery_max_per_webhook`. A wait registered later can claim it. A redelivery of it is processed again in full. It records no second request for a Dag that already has one, while a Dag that declared a matching `cancel_on` since then gets its first. Only `accepted` and `resumed_wait` redeliveries are duplicates. The `early`, `expired`, `accepted` and `resumed_wait` explanations and the webhook wait statuses are changed to match. - Dag versions: the task instance page requests the task without `version_number`. It therefore shows the conditions of the Dag's latest version, which are the ones a delivery is matched against. A run's own version decides which tasks the run has, which of them are teardowns, and whether the Dag's `on_failure_callback` runs for the cancel. If the latest version has no such task, the section is not shown. `cancel_state` in the task response is the task's own value, or `failed` when the task has `cancel_on` but no `cancel_state`. Without `cancel_on` it is null, whatever the task inherits. - Teams: with `[core] multi_team` on, no request is recorded for a Dag that has a team, and the previous PR adds no Dag warning for this. It shows up in `airflow webhooks check` instead: such a Dag whose `cancel_on` names a webhook is reported as a problem with that webhook, and the command exits 1. When no names are given, the command now also checks every webhook that a `cancel_on` names. A condition on a webhook that `[webhooks] names` does not list is therefore a problem too. When none of the latest 100 deliveries of a webhook with conditions has a correlation key, the command prints a note, which does not change the exit code. - The OpenAPI spec, the UI client and the airflowctl datamodels are regenerated. The new UI strings are English only, like the rest of the webhook UI so far. `WebhookMatchList` shows the conditions on both pages. The match value formatting moved out of the webhook wait section so that both can use it, so `taskInstance.webhookWait.oneOf` becomes `webhookMatch.oneOf`. The scheduler code in the previous PR writes the run note, `Cancelled by webhook <name> delivery <id>`. The run page shows it like any other note, with no change in this PR. ### Commit 2: AIP-125: Document cancelling runs and tasks from webhooks This commit documents `cancel_on` for Dag authors and Deployment Managers. Dag authors get a new page, `authoring-and-scheduling/webhook-cancel.rst` ("Cancelling Runs from Inbound Webhooks"). It is built around a preview environment example: each labeled pull request starts a run, which fails when the pull request is closed, and a new commit skips that run's end-to-end tests. The page marks the feature as experimental. For Deployment Managers, `webhooks.rst` adds: - the `cancel_requested` outcome and the `cancel_requests` field; - `[webhooks] cancel_requests_per_loop`; - cleaning `webhook_cancel_request`; - the `webhook_cancel` audit entries and the `webhooks.cancels` metric; - the least-privilege grants; - a new section, "Why a run was or was not cancelled". Points to look at: - The new page should state the behaviours that surprise anyone who has not read the code: - Only a delivery that has a correlation key cancels. - A run is about a key only through an event that a delivery to the same webhook recorded, or through the conf key that a condition names. - A delivery cancels only the runs whose event, or whose queueing with the key in their conf, came before it. - Requests also apply to runs that the scheduler creates later from earlier events. That is why requests are kept until `airflow db clean` removes them. - The conditions come from the Dag's latest version, while each run's own version decides which tasks the run has. - A cancel is a failure, because Airflow has no cancelled state. A cancelled run is `failed`, and its Dag's `on_failure_callback` runs with `context["reason"]` set to `cancelled`, just as it is set to `timed_out` for a run that timed out. A cancelled run also counts toward `max_consecutive_failed_dag_runs` when a later run fails. A run that still has a teardown to run is the exception. It ends as its leaf tasks make it end, either `failed` with `task_failure` or `success`, and gets no `cancelled` callback. `callbacks.rst` lists the reasons that a Dag callback can get. - The page documents these limits: - With more than one scheduler, a request can miss a new run. This happens when the request is committed after the creating scheduler read the requests, and another scheduler applies it before the new run commits. - A run triggered with the key in its conf never gets a request that was applied while the run's API transaction was still open. - `airflow dags test` cancels nothing, since only a running scheduler applies requests. - `security_model.rst` says that a leaked secret can now also fail the active runs, or end tasks, of the Dags whose `cancel_on` names the webhook, for any correlation key. A new "Cancel by delivery" residual risk covers: - what a delivery can cancel; - that a generic source's `correlation_key_field` lets the sender choose the key; - that a delivery replayed under a new id records new requests; - that the webhooks process writes no run or task instance state and loads no Dag, and that the scheduler runs no Dag code to apply a request. The security team should review this bullet. - A `cancel_requested` delivery is kept for webhook waits and processed again when redelivered. The waits and topics pages, and the GitHub and standard provider docs, said that only `early` deliveries are kept, so they now cover `cancel_requested` deliveries too. `public-airflow-interface.rst` lists these as public and experimental: `WebhookMatch`, `cancel_on`, `cancel_state`, the outcome, the `cancel_requests` field, the `cancelled` callback reason, the `webhook_cancel` audit event and the `webhooks.cancels` metric. It lists the two tables and the note text as internal. - Release: the new page has `versionadded:: 3.6.0` and the interface entry `versionchanged:: 3.6.0`. These match the two migrations in the previous PR, which are stamped 3.6.0. The release note is a separate significant newsfragment, `airflow-core/newsfragments/aip125-cancel-on.significant.rst`, and `75167.significant.rst` is left unchanged. The note is significant for two reasons: upgrading adds the `dag_cancel_condition` and `webhook_cancel_request` tables, and a Dag that uses `cancel_on` needs the Task SDK on the Dag processor and the workers upgraded together with core. It will be renamed to this PR's number once the PR exists. ### Checks and the live run These checks passed on the branch with both PRs and the demo kit: the SQLite, Postgres and MySQL test sweeps, mypy, prek, the docs build, and 1,087 UI tests. `demo.sh 1a 1b 2 3` also passed on Postgres and MySQL. The demo kit (#75168) gains `demo.sh 3` for this phase. One existing UI test, "shows a 404 for a linked delivery in its panel" in `WebhookDeliveries.test.tsx`, is flaky. It is already flaky on the commit this PR builds on, and the previous PR in this stack changes no UI code. The stack was also run against a real GitHub pull request, through an ngrok tunnel to the webhooks process: - Adding the label started a `preview_environment` run. - A new commit skipped its `run_e2e_tests`. - Merging the pull request resumed the `wait_for_merge` task of a `merge_gate_operator` run, which was waiting for the merge, and cancelled the preview run. The run got the note, `on_kill` ran, and `on_failure_callback` got the reason `cancelled`. ### Merging This PR merges after the previous PR in this stack, which holds the cancel unit. Its SDK, serialization, request and scheduler parts merge together there, so that no release accepts a `cancel_on` that does nothing. This PR holds the release note and the docs for the previous PR's `cancel_on`, including its two migrations and the Task SDK upgrade it needs, so it has to merge in the same release as the previous PR. Before this PR merges, the newsfragment is renamed to this PR's number. Both new PRs sit above #75347 and below the demo kit #75168, which stays on top and is not merged. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
