dheerajturaga opened a new pull request, #75047: URL: https://github.com/apache/airflow/pull/75047
### Stack Stacks on #75046. Stack index: 15 of 20. A PR from a fork cannot target a fork branch, so each PR in the stack is based on `main` and contains every commit below it; only the top commit is new. Review only the commit "AIP-125: Record webhook topic events for subscribed Dags"; the commits below it belong to earlier PRs in the stack. ### Status Draft. [AIP-125 (Inbound Webhooks)](https://lists.apache.org/thread/p3dpnor4zwko8tldpncj6ftj1bpp18q7) is under discussion on dev@ and not yet voted; this stack is its proof of concept so that voters can run it, and nothing should merge before the vote. The feature is opt-in (only `airflow api-server --apps webhooks` serves deliveries) and experimental; no unauthenticated route is reachable until stack PR 18, which merges last. ### Why Dags subscribe to inbound webhooks by scheduling on topic names, so a verified delivery records an event on each candidate name a live Dag schedules on, with its record and an audit entry in one transaction. Paused, draining, stale, partition-aware and (under multi-team) team-owned Dags do not count, so every event has a Dag to run, and a retry can act once one does. Only assets whose URI is their name count, so a sender cannot record events on a data asset that shares a topic name. ### What to look at - Outcomes `accepted`, `dag_paused` and `no_subscribers`. A redelivery of one that recorded nothing is processed again; unpausing a Dag never replays one. - The race: a compare-and-set on `outcome`. A copy that could act but lost to one that recorded nothing tries once more, then answers 503 so the sender retries. - `from_webhook` in the event extra is a claim, not proof: anyone who may create asset events can set it. - `[webhooks] store_payload` defaults to `full`, so every user who may read assets sees up to 64 KiB (`max_event_payload_bytes`) of each payload in event extras. Is `summary` a better default? ### Testing - `test_topics.py`, `test_app.py` and `test_pipeline_concurrency.py` passed under breeze on SQLite, Postgres and MySQL. After the last fixes, the tests were rerun on SQLite and Postgres, and the final head passed again on MySQL. - The live demo checks the event, its extra and the run it starts, on Postgres and MySQL. ### Known limitations - No per-Dag cap yet; the AIP reserves the outcome `dropped` for it. The audit entry's team waits for 3.6. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
