ksidata opened a new issue, #44372:
URL: https://github.com/apache/airflow/issues/44372

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.9.3
   
   ### What happened?
   
   We're using cloud composer (2.9.5) and airflow (2.9.3) on python(3.11) and, 
after making an upgrade from composer(2.5.4) airflow(2.5.3) python(3.8), we 
started  to face an issue concerning our Dagbag import process. We use python 
scripts to generate dags. one of those script, generate a specific type of dags 
called "global", this dags are made and scheduled to control/orchestrate all 
the other subdags using TriggerDagRunOperators. When executing the triggering 
tasks in the workers, after the worker process fork, it started to 
self._process_file() and that starts by making the import of the corresponding 
python file. Well when the worker forked process starts "Filling up the dagbag 
[dag_file_folder]" as the logs say, it lasts more than 60 secs and it gets 
terminated with an error Process timeout (PID:xxxxx). In our configs we have 
dagbag_import_timeout at 60 and dag_file_processor_timeout at 300. Increasing 
the dagbag_import_timeout to 180 has solved the problem, but we need to und
 erstand when this delay is coming from in order to find the pattern that will 
help to avoid this issues in the future. Especially that it happens only in the 
newer versiosn mentionned above. **Note that scaling up are worker by doubling 
the amount of memory and cpu, or decrease the worker_concurrency, didn't have 
any significant impact on the process timeout issue**.
   
   ### What you think should happen instead?
   
   If we consider that python 3.11 is faster 
([check_this_article](https://www.phoronix.com/review/python-311-performance)) 
than 3.8, we are supposed to have more efficient and faster import and file 
processing.
   
   ### How to reproduce
   
   Run a dag with a relative large amount of libs dependencies, set the 
dagbag_impot_timeout just above the actual import delay in an composer(2.5.4) 
airflow(2.5.3) python(3.8), and then do run the same dag with the same timeout 
configs in composer (2.9.5) and airflow (2.9.3) on python(3.11). use small 
sized composer cluster.
   
   ### Operating System
   
   Embedded in cloud composer 2.9.5
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Google Cloud Composer
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to