potiuk commented on issue #32966: URL: https://github.com/apache/airflow/issues/32966#issuecomment-1660896619
No. Dag FileProcessor generally does not manipulate Dag Runs and Task instance - and those are the ones that locks are mainly used for. Creating DagRuns and queuing tasks is all happening in schduler. We are also heavily using multiprocessing (Especially for Dag FileProcessor) - basically each file is processed in a new forked python process, so gil and singletons have nothing to do with it. BTW. Horizontal scaling of DAG file processor would really be needed if your Python Parsing gets a lot of CPU cycles. And mostly when you have big number of smal Python files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
