GitHub user potiuk added a comment to the discussion: Less large DAGs or more small DAGs?
It's complicated and there is no single answer or document to describe it. There are many moving parts there and dependencies - a lot depends on your I/O sped and the way how your files are shared, a lot depends on how complex your DAGs are, a lot depends whether you follow best practices (look for best practices in our docs) when you design your DAGs or not, number of tasks you have, number of dynamic dags you have, and plenty of other things There are multiple factors - CPU, I/O, memory, networking, whether you use bare-metal or virtual machines or k8s on top, etc. etc. Generally you should rather apply "modify and observe the result" strategy to assess the impact you might have. Apply a change and monitor your system and see how it impacts your case - for that you need to have good monitoring and understand where your resource usage is. You can read more about it here. Simply airflow is rather complicated beast with far too many knobs to turn to be able to foresee which knobs you have to turn to make things faster - because they are interconnected and depend on many external factors. The only way to see what can help is to turn the knobs and see what happens: https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#notes-about-minimum-requirements GitHub link: https://github.com/apache/airflow/discussions/47132#discussioncomment-12336335 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
