hussein-awala opened a new pull request, #76014: URL: https://github.com/apache/airflow/pull/76014
Dag processor callback polling can scan completed executor callbacks and unrelated bundles before finding its next batch. Add an index on bundle name, callback type and descending priority so polling can locate callbacks for ready bundles directly. The query and callback behavior are unchanged. The migration adds the model index and removes it on downgrade. Warm synthetic measurements of the actual callback-fetching helper, with a batch limit of 16: | Fixture | PostgreSQL before / after | MySQL before / after | |---|---:|---:| | 101,000 callbacks, 1,000 processor callbacks, one ready bundle | 7.647 / 1.289 ms | 44.474 / 2.275 ms | | 101,000 callbacks, 100,000 processor callbacks, one ready bundle | 22.794 / 1.282 ms | 55.836 / 2.378 ms | | 101,000 callbacks, no processor callbacks | 5.150 / 0.484 ms | 41.528 / 0.471 ms | | 100 callbacks, 20 processor callbacks, two ready bundles | 1.108 / 1.108 ms | 1.171 / 1.206 ms | These values are medians of 21 samples per configuration after three warmups, with baseline and indexed configurations interleaved. Unchanged control medians differed by less than 10% from baseline, and ordered callback requests were verified equal across the captured configurations. The fixtures retain baseline indexes, including MySQL's foreign-key-supporting index. The helper measurements include selection, decoding, and deletion within a transaction that is rolled back; foreign-key enforcement, durable commit, callback dispatch, and the complete processor loop are outside these timings. The tested broad backlog with two ready bundles still required sorting and showed no consistent meaningful improvement. Executor callback polling was checked without a material regression in the measured fixtures. For the mixed 101,000-row fixture, the index adds about 7.4 MB on PostgreSQL and 6.9 MB on MySQL. MySQL batches of 1,000 inserts took 26.42→28.65 ms and 100 deletes took 1.97→2.39 ms. A separate PostgreSQL batch of 100 executor state updates took 1.08→1.32 ms. These are write medians from 21 samples per configuration after two warmups, excluding durable commit. Related: [#76010](https://github.com/apache/airflow/pull/76010) adds a separate index for the scheduler’s executor-callback queue. The two indexes were also measured together in the callback fixtures. Focused validation passed on SQLite (90 tests), PostgreSQL (89 tests and one SQLite-only skip), and MySQL (89 tests and one SQLite-only skip). The new real-helper plan regression fails on the predecessor schema and passes after migration. Two upgrade/downgrade cycles preserve rows and foreign keys on each backend. Broader verification covered all six selected core groups: 12,134 passed and 78 skipped on SQLite; the non-DB selection had 5,585 passed, 11 skipped and 2 expected failures. The suites ran in fresh processes to fit local memory, with exact collected-test and JUnit coverage checks. Core documentation and required regular/manual checks, including the migration round trip, passed. The branch is rebased onto current upstream main. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
