ashb commented on PR #49839: URL: https://github.com/apache/airflow/pull/49839#issuecomment-2854055327
> As I've learned forking does not give that much of memory sharing in Python as I thought (due to reference counting) , the elapsed time neeed to run the import for each DAG will be still smaller. The main benefit to forking here is (or at least was, when I mage this change around v1.10.7 period) due to two things - 1. not having to spawn an entirely fresh python process, and 2. Not having to import `airflow`, all of its deps, parse all it's config and connect to DB etc. Some of that is lessened these days, but imports are still surprisingly slow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
