[ https://issues.apache.org/jira/browse/AIRFLOW-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992434#comment-16992434 ]
Felix Uellendall commented on AIRFLOW-5931: ------------------------------------------- [~ash] don't forget to resolve the issue :) 1.10.7? > Spawning new python interpreter for every task slow > --------------------------------------------------- > > Key: AIRFLOW-5931 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5931 > Project: Apache Airflow > Issue Type: Improvement > Components: executors, worker > Affects Versions: 2.0.0 > Reporter: Ash Berlin-Taylor > Assignee: Ash Berlin-Taylor > Priority: Major > > There are a number of places in the Executors and Task Runners where we spawn > a whole new python interpreter. > My profiling has shown that this is slow. Rather than running a fresh python > interpreter which then has to re-load all of Airflow and its dependencies we > should use {{os.fork}} when it is available/suitable which should speed up > task running, espeically for short lived tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)