dheerajturaga opened a new pull request, #75052: URL: https://github.com/apache/airflow/pull/75052
### Stack Stacks on #75051. Stack index: 20 of 20. A PR from a fork cannot target a fork branch, so each PR in the stack is based on `main` and contains every commit below it; only the top commit is new. Review only the commit "[DO NOT MERGE] AIP-125 proof of concept: demo kit"; the commits below it belong to earlier PRs in the stack. ### Status **Do not merge, ever.** This PR exists so that voters on [AIP-125 (Inbound Webhooks)](https://lists.apache.org/thread/p3dpnor4zwko8tldpncj6ftj1bpp18q7), which is under discussion on dev@ and has not been voted on, can run the stack. `dev/aip125_demo/` is dropped at the post-vote split, when the stack is opened as PRs of their own. ### Why Voters should be able to run AIP section 9's Phase 1a and 1b checks on core rather than take the unit tests' word for them. `demo.sh` starts the core and webhooks API servers, the scheduler and the Dag processor in breeze, and asserts each check through `send-test`, the CLI and the REST API. `bench_payload.py` (spike S10) and `bench_latency.py` measure what payload copies and deliveries cost. ### What to look at - What `demo.sh` checks, and what it leaves to unit tests (a table in the README). - Whether the numbers settle the payload question. PostgreSQL scans `asset_event` for `GET /api/v2/assets/events?extra=` rather than use the GIN index, as its cost model ignores TOAST. ### Testing - `demo.sh 1a 1b` passed on Postgres and MySQL, and again (53 checks) on Postgres after the last fixes. - The benchmarks ran on Postgres in breeze over loopback, without network, TLS or a proxy: a lower bound for real senders. ### Run it ```bash breeze run --backend postgres bash /opt/airflow/dev/aip125_demo/demo.sh 1a 1b breeze run --backend mysql bash /opt/airflow/dev/aip125_demo/demo.sh 1a 1b breeze run --backend postgres bash /opt/airflow/dev/aip125_demo/demo.sh bench ``` ### Benchmarks Latency of 40 signed deliveries, four at a time, each recording a topic event and queueing a run: p50 31 ms, p99 41 to 44 ms over two runs. The first four, which pay one-time costs, took up to 583 to 705 ms and are left out. Payload cost, three deliveries per size, nine topic events each, no cap: | Payload | Insert p50 | `asset_event` growth | GET latest 50 | GET by extra (scan) | Forced GIN | |---|---|---|---|---|---| | 1 KiB | 28.6 ms | 168 KiB | 18 ms | 16 ms | 0.1 ms | | 64 KiB | 46.8 ms | 2.6 MiB | 185 ms | 104 ms | 2.8 ms | | 1 MiB | 447.6 ms | 14.6 MiB | 3.4 s | 1.7 s | 62.3 ms | With the default 64 KiB cap, an earlier run (ten deliveries per size) left the 1 MiB payloads out of the events: insert p50 31.4 ms, GET latest 50 in 18 ms. ### Known limitations - The scan behind `GET /api/v2/assets/events?extra=` is upstream filter and index behaviour, to raise with the assets owners rather than in this stack. - The payloads are synthetic; recorded deliveries come with the proof of concept's 0.2 release. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
