ksidata opened a new issue, #44372: URL: https://github.com/apache/airflow/issues/44372
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.9.3 ### What happened? We're using cloud composer (2.9.5) and airflow (2.9.3) on python(3.11) and, after making an upgrade from composer(2.5.4) airflow(2.5.3) python(3.8), we started to face an issue concerning our Dagbag import process. We use python scripts to generate dags. one of those script, generate a specific type of dags called "global", this dags are made and scheduled to control/orchestrate all the other subdags using TriggerDagRunOperators. When executing the triggering tasks in the workers, after the worker process fork, it started to self._process_file() and that starts by making the import of the corresponding python file. Well when the worker forked process starts "Filling up the dagbag [dag_file_folder]" as the logs say, it lasts more than 60 secs and it gets terminated with an error Process timeout (PID:xxxxx). In our configs we have dagbag_import_timeout at 60 and dag_file_processor_timeout at 300. Increasing the dagbag_import_timeout to 180 has solved the problem, but we need to und erstand when this delay is coming from in order to find the pattern that will help to avoid this issues in the future. Especially that it happens only in the newer versiosn mentionned above. **Note that scaling up are worker by doubling the amount of memory and cpu, or decrease the worker_concurrency, didn't have any significant impact on the process timeout issue**. ### What you think should happen instead? If we consider that python 3.11 is faster ([check_this_article](https://www.phoronix.com/review/python-311-performance)) than 3.8, we are supposed to have more efficient and faster import and file processing. ### How to reproduce Run a dag with a relative large amount of libs dependencies, set the dagbag_impot_timeout just above the actual import delay in an composer(2.5.4) airflow(2.5.3) python(3.8), and then do run the same dag with the same timeout configs in composer (2.9.5) and airflow (2.9.3) on python(3.11). use small sized composer cluster. ### Operating System Embedded in cloud composer 2.9.5 ### Versions of Apache Airflow Providers _No response_ ### Deployment Google Cloud Composer ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
