hussein-awala opened a new pull request, #76003: URL: https://github.com/apache/airflow/pull/76003
The Dag list can spend most of its database time ranking old runs that will never appear on the page. This changes PostgreSQL's recent-run lookup to read the newest requested runs for each visible Dag through the existing `(dag_id, run_after)` index. Page IDs are passed as one bound array, so query construction stays inexpensive for larger pages. MySQL and SQLite retain their existing query. There is no new index or migration. Run selection still uses `run_after` followed by ID to break cutoff ties, and the response remains ordered by `run_after`. In a warm PostgreSQL 14 benchmark with 300,006 synthetic runs, the measured recent-run block included query construction, SQLAlchemy execution and fetching: | Request | Before | After | |---|---:|---:| | Ten Dags with 10,000 old runs each; ten runs per Dag | 65.83 ms | 0.91 ms | | 100-Dag page including 30 long histories | 222.48 ms | 1.72 ms | | 100-Dag page with short or empty histories | 1.00 ms | 0.75 ms | | 501-Dag page with short or empty histories | 2.91 ms | 1.31 ms | These are medians from three alternating before/after runs, with an unchanged control staying within 8.8%. They measure this query block, not complete HTTP latency. Single-Dag sparse, short and empty cases added about 0.05–0.11 ms. Large groups sharing the newest timestamp may still need sorting by ID; the index does not contain that tie-break column. The focused Dag-list suite passed on PostgreSQL (72 tests), MySQL (71 tests, one PostgreSQL-only skip), and SQLite (71 tests, one PostgreSQL-only skip). The new PostgreSQL regression checks that adding old history does not cause the lookup to scan it; it fails on the original query and passes with this change. Existing tests cover filters, authorization, pagination, and the highest-ID winner when a limit cuts through a timestamp tie. Validation also passed for the selected API/Always and common.compat/FAB database and non-database suites, the apache-airflow/common.compat/FAB documentation builds, and the required regular and manual static checks. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
