[jira] [Commented] (AIRFLOW-2615) Webserver not using cached app
[ https://issues.apache.org/jira/browse/AIRFLOW-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512057#comment-16512057 ] Kevin Yang commented on AIRFLOW-2615: - [~joygao] Not very confident in the webserver area, would you kindly provide you opinion here please? Thank you! > Webserver not using cached app > -- > > Key: AIRFLOW-2615 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2615 > Project: Apache Airflow > Issue Type: Bug >Reporter: Kevin Yang >Priority: Major > > From what I can tell, the app cached here attempt to cache the app for later > use-likely to be for the expensive DagBag() creation. Before I dive into the > webserver parsing everything in one process problem, I was hoping this cached > app would save me sometime. However it seems to me that every subprocess spun > up by gunicorn is trying to create the DagBag() right after they've been > created--make sense to me since we didn't share the cached app to the > subprocess( doubt we can). If what I observed is true, why do we cache the > app at all? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2615) Webserver not using cached app
[ https://issues.apache.org/jira/browse/AIRFLOW-2615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512059#comment-16512059 ] Kevin Yang commented on AIRFLOW-2615: - Adding a little bit context here: Airbnb has ~2000 DAG file in our centralized DAG repo and it takes a long time to parse the entire repo, this extra app creation is basically doubling the time we need to refresh webserver worker. > Webserver not using cached app > -- > > Key: AIRFLOW-2615 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2615 > Project: Apache Airflow > Issue Type: Bug >Reporter: Kevin Yang >Priority: Major > > From what I can tell, the app cached > [here|https://github.com/apache/incubator-airflow/blob/master/airflow/bin/cli.py#L790] > attempt to cache the app for later use-likely to be for the expensive > DagBag() creation. Before I dive into the webserver parsing everything in one > process problem, I was hoping this cached app would save me sometime. However > it seems to me that every subprocess spun up by gunicorn is trying to create > the DagBag() right after they've been created--make sense to me since we > didn't share the cached app to the subprocess( doubt we can). If what I > observed is true, why do we cache the app at all? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
