pavelpi commented on issue #34835: URL: https://github.com/apache/airflow/issues/34835#issuecomment-1755074051
Hello @jens-scheffler-bosch thanks for quick reply! I scaled up the environment to have 3 DAG processors, each one having 4 CPUs and 2GB memory. 70% of the time all CPUs are almost fully allocated. There are periods when I can see the same issue. It's better than when I had only 2 processors, 1 CPU each. I also tried 1 processor running on 8CPU/8GB RAM machine, but this did not help much. Memory does not seem to be a problem - processors need about 400MB each regardless of number of processors or #CPUs. Is there any recommended setup or environment setting that would help us? We have 137 DAGs, DAG processor processes them between 0.5s and 2.0s each. When we encounter the issue, although Airflow reports missing DAG, it is still available in the list of DAGs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
