GitHub user potiuk added a comment to the discussion: Less large DAGs or more 
small DAGs?

It's complicated and there is no single answer or document to describe it. 
There are many moving parts there and dependencies - a lot depends on your I/O 
sped and the way how your files are shared, a lot depends on how complex your 
DAGs are, a lot depends whether you follow best practices (look for best 
practices in our docs) when you design your DAGs or not, number of tasks you 
have, number of dynamic dags you have, and plenty of other things

There are multiple factors - CPU, I/O, memory, networking, whether you use 
bare-metal or virtual machines or k8s on top, etc. etc. 

Generally you should rather apply "modify and observe the result" strategy to 
assess the impact you might have. Apply a change and monitor your system and 
see how it impacts your case  - for that you need to have good monitoring and 
understand where your resource usage is.

You can read more about it here. Simply airflow is rather complicated beast 
with far too many knobs to turn to be able to foresee which knobs you have to 
turn to make things faster - because they are interconnected and depend on many 
external factors. The only way to see what can help is to turn the knobs and 
see what happens:  
https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html#notes-about-minimum-requirements


GitHub link: 
https://github.com/apache/airflow/discussions/47132#discussioncomment-12336335

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to