hussein-awala opened a new pull request, #76010: URL: https://github.com/apache/airflow/pull/76010
Scheduler callback polling currently scans completed callbacks while looking for pending work in priority order. Add an index on callback state, type and descending priority so the scheduler can retrieve its next batch directly, including when no pending callbacks remain. The polling query and callback behavior are unchanged. The migration adds the same index declared by the model and removes it on downgrade. Warm synthetic measurements of the scheduler's selection and ORM-loading step, with a batch limit of 16: | Fixture | PostgreSQL 14 before / after | MySQL before / after | |---|---:|---:| | 101,000 callbacks, 99 pending executor callbacks | 9.404 / 0.749 ms | 43.222 / 0.804 ms | | 101,000 callbacks, none pending | 8.336 / 0.539 ms | 44.203 / 0.530 ms | | 101,000 callbacks, 49,500 pending executor callbacks | 9.812 / 0.770 ms | 49.776 / 0.846 ms | | 100 callbacks, 9 pending executor callbacks | 0.682 / 0.733 ms | 0.645 / 0.634 ms | These values are medians of 21 samples per configuration after three warmups, with baseline and indexed configurations interleaved. Unchanged control medians differed by less than 3.5%, and captured returned rows matched. The fixtures retain baseline indexes, including MySQL's foreign-key-supporting index; foreign-key enforcement and callback dispatch are outside these timings. They do not represent complete scheduler-loop latency. Dag processor callback polling was also checked without a material regression in the measured fixtures. The additional index uses about 7.4 MB on PostgreSQL and 6.9 MB on MySQL for the 101,000-row fixture. MySQL write medians from seven samples per configuration were 24.98→26.49 ms for 1,000 inserts and 4.84→5.57 ms for 100 state updates. Indexing state prevents PostgreSQL HOT updates for state changes; a separate 21-sample probe using fillfactor 70 measured 100 updates at 0.96→1.32 ms. Write timings exclude durable commit. Focused validation against complete, migrated databases passed on SQLite (46 tests), PostgreSQL (45 tests and one SQLite-only skip), and MySQL (45 tests and one SQLite-only skip). The new scheduler-plan regression fails on the predecessor schema and passes after migration. Two upgrade/downgrade cycles preserve rows and foreign keys on each backend. The complete selected core database scope passed on SQLite (12,134 passed, 78 skipped). The selected non-database scope passed with 5,585 passes, 11 skips and two expected failures. These broader suites ran in separate processes to fit the local VM, with exact collected-test coverage checked. The core documentation build and regular/manual checks passed, including the SQLite migration round-trip check. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
