shivaam opened a new pull request, #70961: URL: https://github.com/apache/airflow/pull/70961
`Trigger.clean_unused()` runs synchronously in every triggerer supervisor loop. It currently removes every eligible trigger in one transaction, so busy metadata databases can hold many row locks and delay the triggerer heartbeat and concurrent writers. This change selects and deletes unused trigger IDs in bounded transactions, committing between batches. It keeps the current `NOT EXISTS` predicates and `SKIP LOCKED` candidate locking from #68244, uses a portable bounded select-ID/delete-ID shape on every dialect, and retries a failed batch in a fresh transaction. A new positive `[triggerer] unreferenced_triggers_cleanup_batch_size` option defaults to 500; non-positive values fail fast rather than restoring an unbounded path. Focused coverage exercises empty, below/exactly/above-threshold and multi-batch cleanup, protected deferred-task, asset, and callback references, invalid configuration, committed progress across a later failed batch, fresh retry transactions, and eventual drain. Prior work: #68269 proposed the original batching direction. This implementation is a clean current-main rewrite because that draft predates #68244 and committed inside a caller-injected session. Scope note: this PR leaves the pre-existing unbounded TaskInstance reference update and portable `callback.trigger_id` indexing unchanged; those require separate batching/index and migration designs. Validation: - Focused PostgreSQL and MySQL tests through Breeze: 9 passed on each backend. - Real scheduler/triggerer PostgreSQL validation with 1,005 eligible rows, bounded `[500, 500]` deletes while 5 rows were concurrently locked, eventual drain after lock release, and protected asset/callback rows preserved. - Ruff, formatting, YAML, spelling, mypy, security, and changed-file repository checks passed. - Rebased cleanly onto `upstream/main` at `66444504c4`; the rebased patch ID is identical to the runtime-tested patch. - The Docker-backed default-configuration hook passed before rebase. Its post-rebase rerun is currently blocked before validation by a local Colima file bind-mount error after a successful CI-image rebuild; CI can independently confirm it. closes: #68243 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-5) Generated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
