GitHub user salimuddin07 deleted a comment on the discussion: How to use sharding for schedulers?
You don’t actually need to shard your schedulers manually. Airflow already does auto-sharding of scheduling work when you run multiple schedulers, so splitting DAG folders across them won’t give you much benefit. What usually becomes the bottleneck at scale (like your 6000 DAGs) is parsing, not scheduling. Every scheduler loop has to re-parse DAGs, and if parsing is heavy (e.g., DAG files import big libs, hit APIs, or query a database at the top-level), things slow down a lot. Here’s what you should focus on instead: - Optimize DAG parsing Make sure your DAG files are just DAG definitions — no heavy imports, no DB queries, no API calls. Move heavy logic into operators, not into DAG definitions. Cache static data instead of reloading it inside DAGs. - Use DAG file processors In Airflow 2.11 you can run standalone DAG processors per directory, so you can split your DAGs into subfolders and let processors handle them in parallel. This will reduce parsing pressure on the main scheduler loop. In Airflow 3, this separation becomes the only way parsing works. - Scale schedulers if needed Multiple schedulers already shard scheduling work internally with SKIP LOCKED. So you don’t need to shard manually across DAG folders for scheduling. Just add more schedulers if scheduling is still slow after parsing optimizations. - Check community discussions There are known “scheduler starvation” discussions in the Airflow devlist. If your issue looks more like tasks not being picked up even when slots are free, it could be one of those. Worth keeping an eye on. - Don’t shard schedulers by DAG folders — it won’t help. Instead: Optimize parsing (that’s probably your bottleneck). Use DAG file processors per directory. Add more schedulers if needed (they shard work automatically). GitHub link: https://github.com/apache/airflow/discussions/56294#discussioncomment-14582424 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
