hussein-awala opened a new pull request, #76214: URL: https://github.com/apache/airflow/pull/76214
The triggerer drains fired events one at a time in its main loop, so per-event statements bound how fast it resumes deferred tasks during a burst. In the common case, one deferred task with no asset watchers or callback, `Trigger.submit_event` issues six statements, two just to load the `Trigger` entity: its row and an unused task-instance query (joined to `dag_run`) from its eager `task_instance` relationship. Look up the trigger's assets and callback by `trigger_id` instead, bringing the common case to four. related: #76209 `Trigger.task_instance` is [`lazy="selectin"`](https://github.com/apache/airflow/blob/b95498cd56/airflow-core/src/airflow/models/trigger.py#L119), so loading the trigger also selected task instances by `trigger_id`. By then `handle_event_submit` has [cleared `trigger_id` on the resumed task instances and flushed](https://github.com/apache/airflow/blob/b95498cd56/airflow-core/src/airflow/models/trigger.py#L669-L675), so that query normally matches nothing. Assets now come from a join on `asset_watcher`, which can use the existing `idx_awm_trigger_id` index, and the callback from a `TriggererCallback` select on `callback.trigger_id` with `LIMIT 1`. Selecting the subclass adds `callback.type IN ('triggerer')`; only `TriggererCallback.queue` attaches a trigger, so this excludes no row a trigger can reference, and a stray non-triggerer row with a `trigger_id` would be skipped instead of raising `AttributeError`. Task resumption, the notification order (tasks, assets, callback) and the transaction are unchanged; there is no API, configuration or schema change. An event for an already-deleted trigger stays a no-op without the existence check: `asset_watcher.trigger_id` cascades on delete and `callback.trigger_id` is a foreign key, so both lookups come back empty. On a callback's terminal event, `TriggererCallback.handle_event` sets `self.trigger = None` on a relationship no longer pre-loaded; the trigger is still detached in the same flush recording the callback's state and output. `Trigger.task_instance` stays `lazy="selectin"` for its other users. `callback.trigger_id` is unindexed on PostgreSQL and SQLite, as it was for the old lazy load; open #75228 would add that index. #72849 independently rewrites the deferred task-instance select in `submit_event`; a trivial rebase at most. A later PR planned in #76209, memoizing asset fan-out, needs a rework: it extends the `Trigger` probe removed here to eager-load `active`, `aliases` and `scheduled_dags.dag` and pass the loaded `AssetModel` on; whichever lands later moves those eager loads onto the asset query. Local benchmark of the supervisor's `handle_events` drain, one deferred task per trigger, no assets or callback: | Scenario | Before | After | |---|---:|---:| | Statements per fired event | 6 | 4 | | 50-event drain, SQLite | 73.0 ms | 42.8 ms | | 200-event drain, SQLite | 258.2 ms | 158.0 ms | | 50-event drain, PostgreSQL 16 | 371.5 ms | 228.8 ms | | 200-event drain, PostgreSQL 16 | 2729.8 ms | 1805.7 ms | An engine-wide SQLAlchemy listener counted statements, excluding COMMIT; PostgreSQL 16 counts match. Times are medians of 5 rounds, base/head order alternated, on a SQLite file and PostgreSQL 16 in Docker (2-CPU, 8 GiB VM). An unchanged control bench moved -0.5 and +1.7 % on SQLite, +2.5 and +3.1 % on PostgreSQL. No before/after ranges overlapped, but PostgreSQL times are noisy: the 200-event base runs spanned 2158 to 2903 ms. The asset-notification query budget drops from 6 to 3, and a new budget test holds the plain deferred-task case at 4; both fail on `main`, which issues 6. Two further tests pass on `main` and pin invariants the new lookups rely on: `test_submit_event_trigger_already_deleted` pins the `asset_watcher` ON DELETE CASCADE that replaces the removed existence check, and `test_submit_event_terminal_callback_event_detaches_trigger` covers `handle_event` detaching the trigger from a callback loaded on its own, which no other test runs because `test_submit_event` mocks `handle_event`. Validation: - Triggerer-job and trigger-model tests: 208 passed, 1 skipped on SQLite. - Same tests through Breeze with the CI image: 209 passed on PostgreSQL 14; 208 passed, 1 skipped on MySQL 8.0 (the skip is a PostgreSQL-only test). - Regular and manual static checks passed on the full diff. - Selected core suites and the documentation build were not run locally; CI validation is pending. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
