Taragolis commented on issue #34339: URL: https://github.com/apache/airflow/issues/34339#issuecomment-1718328867
> Ah, yeah, in that case, then no - it's run inside a docker container, so effectively completely independent from the airflow process. About timeouts, it seems a bit suspicious when 200 dags can't be processed in reasonable time, so I guess this some dynamic DAGs, which created from single python file. The might be a good reason why some bad things happen, just because Airflow could decide that DAG might be deleted. I'm not sure in this case it is a bug or it's "by design". Some links which might improve your DAG parsing time - [Fine tuning your scheduler performance](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/scheduler.html#fine-tuning-your-scheduler-performance) - [How to control DAG file parsing timeout for different DAG files?](https://airflow.apache.org/docs/apache-airflow/stable/faq.html#how-to-control-dag-file-parsing-timeout-for-different-dag-files) - [Dynamic DAG Generation](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#dynamic-dag-generation) - [Optimizing DAG parsing delays during execution](https://airflow.apache.org/docs/apache-airflow/stable/howto/dynamic-dag-generation.html#optimizing-dag-parsing-delays-during-execution) - [DAG File Processing](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dagfile-processing.html) see standalone DAG Processor and [`airflow dag-processor` cli](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#dag-processor) - [.airflowignore](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#airflowignore) <details> <summary>Just out of my curiosity</summary> The flow something like: AIrflow -> CeleryExecutor -> BashOperator -> Docker -> Prefect ? </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
