hussein-awala opened a new pull request, #76463: URL: https://github.com/apache/airflow/pull/76463
When the oldest 500 pending partitioned runs cannot satisfy their asset conditions, every scheduler pass selects them again and newer ready partitions can wait indefinitely. This affects ready partitions in the same Dag and in other Dags. Track the next evaluation order in the database and move each checked row behind rows that have waited longer. The scheduler keeps the existing batch limit and row locks, preserves partially satisfied partitions, and continues making progress after a scheduler restart. Rows skipped because their serialized Dag is unavailable also rotate. The migration initializes evaluation order from `created_at` and adds an index for the pending-row query. Regression coverage includes starvation within and across Dags, missing serialized Dags, scheduler restarts, and late asset arrivals. Migration tests verify existing rows and ordering survive upgrade and downgrade on SQLite and MySQL, including preservation of the MySQL foreign key's cascade behavior. Validation: - Full scheduler test file on PostgreSQL 14: **605 passed, 2 long-running cases skipped**. - Partition scheduler tests on SQLite: **56 passed**. - Migration upgrade/downgrade on SQLite (foreign keys on/off) and MySQL 8.0: **3 passed**. The MySQL downgrade regression first reproduced error 1553 before its correction. - Schema/model consistency and single migration-head checks passed; the manual SQLite round-trip passed all **155 revisions**. - Selected non-database suite on native Python 3.12: **5,601 passed, 11 skipped, 2 xfailed**. - Precommit checks including mypy, manual hooks, and `breeze build-docs apache-airflow` passed. The wider database matrix is **not fully verified locally**. Attempts with cached CI images hit Cadwyn/FastAPI incompatibility and provider-entrypoint discovery failures; the later broad reruns were stopped. After aligning Cadwyn, API initialization and the complete scheduler suite passed. A complete database matrix still needs CI. Fresh full-image builds were blocked by local Docker disk space. Local scheduler benchmark (501 pending rows: 500 unsatisfied partitions followed by one ready partition; batch limit 500): | Measurement | Base | Fix | | --- | ---: | ---: | | Median evaluation, flush, and rollback | 8.942 ms | 14.185 ms | | Unchanged `COUNT` control | 0.05539 ms | 0.05412 ms | | Database calls | 3 reads | 3 reads + 1 batched update | | Ready partition reached on second pass | No | Yes | Seven alternating base/fix pairs show an added 5.242 ms (+58.6%) per full blocked batch; control drift was -2.3%. The bounded update persists fairness for all 500 selected rows. This overhead buys progress across scheduler restarts without discarding pending asset conditions. The regression pins evaluation plus flush to four database calls. These are local SQLite measurements of the base and patched methods using the same post-migration schema, with five evaluations per sample. They exclude commit/fsync and the full scheduler loop and are not production latency estimates. Tests used cached Breeze images; the final native validation image aligned FastAPI, Starlette, and Cadwyn to `uv.lock`. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
