GitHub user potiuk added a comment to the discussion: How to use sharding for schedulers?
I think you are mixing Parsing with Scheduling. Parsing is done by dag file processor (which might be embedded in Scheduler or standalone). For parsing - you can use standalone dag processor (the only option with Airflow 3 actually) and see if it helps, I believe 2.11 version had option to have separate dag file processor per directory. Also 6000 Dags does not seem like many, and it simply migh tbe that you have slow parsing. Any network traffic, database access at the top-level parsing of your Dag files is no-no. Optimising parsing time is a good idea and best practice for Airflow. Look at "best practices" in Airflow docs for various optimisation techniques. If it is about scheduling then multiple schedulers are doing auto-sharding internally (using SKIP LOCKED) - so I am not sure if that is your issue. In either case - after you explore standalone dag file processors., you might want to look at recent devlist discussions (you will find links in "community" tab of airflow website) - there are some starvation scenarios being discussed there, it might be one of those - so you might chime in in the discussion. GitHub link: https://github.com/apache/airflow/discussions/56294#discussioncomment-14580757 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
