Hello, everyone! I hope this is the right place to ask about this, please redirect me otherwise :)
I was looking into how dag files are imported, and noticed that airflow creates a whole new process for each file that can potentially contain DAGs, and then closes the process after only processing a single file. It would seem to me that keeping the process around to parse multiple files would be much more efficient (keeps sqlalchemy connections around, for example). Is there a specific reason this design was selected, and if no - is there any interest in changing this? The initial reason for me to look into this is that DagBag filling time seems to be rather slow when we have a significant amount of dag files (more than a thousand files) Regards, Oleksandr -- This email and any files transmitted with it contain confidential information and/or privileged or personal advice. This email is intended for the addressee(s) stated above only. If you are not the addressee of the email please do not copy or forward it or otherwise use it or any part of it in any form whatsoever. If you have received this email in error please notify the sender and remove the e-mail from your system. Thank you. This is an email from the company Just Eat Takeaway.com N.V., a public limited liability company with corporate seat in Amsterdam, the Netherlands, and address at Oosterdoksstraat 80, 1011 DK Amsterdam, registered with the Dutch Chamber of Commerce with number 08142836 and where the context requires, includes its subsidiaries and associated undertakings.
