potiuk commented on PR #49839: URL: https://github.com/apache/airflow/pull/49839#issuecomment-2833437089
> I wonder if we should respect it instead of? I have not looked at how we we are parsing the dags now but if we are still forking Dag File processor processes, it might make sense to pre-import the modules before we actually fork. Though probably what would make sense we could import ALL "airflow" modules this way, not only those that are present in DAGs when we start DAG file processor - this way we could just delay the dag file processor startup a "little". but all airflow modules would be imported before we fork. That could be IMHO a bit better optimisation. As I've learned forking does not give **that** much of memory sharing in Python as I thought (due to reference counting) , the elapsed time neeed to run the import for each DAG will be still smaller. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
