vatsrahul1001 commented on PR #54103: URL: https://github.com/apache/airflow/pull/54103#issuecomment-3812745284
> Hi @vatsrahul1001, has there been any update on your testing? @xBis7 - Apologies for the delay. Completed testing now - this is a significant improvement! I used 10 DAGs with 100 parallel tasks each, `max_active_tasks=10` | Metric | main | PR | |--------|------|-----| | Starved DAG events | 141 | 0 | With this PR, the scheduler no longer fetches tasks it cannot queue. On main, tasks get fetched and then rejected when the DAG's `max_active_tasks` limit is already reached - with the PR, that check happens in the SQL query itself. Looking at the logs: - **Before:** Query pulls tasks mostly from same DAG, most get rejected - **After:** Query spreads task selection across multiple DAGs, all get queued Looks like a big win for scheduler efficiency. Also I do not see any regression from this PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
