[jira] [Updated] (AIRFLOW-2372) SubDAGs should share dag_concurrency of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share dag_concurrency of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share dag_concurrency of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share dag_concurrency of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Summary: SubDAGs should share dag_concurrency of parent DAG (was: SubDAGs should share parallelism

[jira] [Issue Comment Deleted] (AIRFLOW-2340) SQLalchemy pessimistic connection handling not working

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2340: -- Comment: was deleted (was: We see that with our mysql db too) > SQLalchemy pessimistic connection

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share parallelism of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Updated] (AIRFLOW-2372) SubDAGs should share parallelism of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2372: -- Description: It seems like right now subDAGs are scheduled just like normal DAGs, so if a DAG has a

[jira] [Created] (AIRFLOW-2372) SubDAGs should share parallelism of parent DAG

2018-04-24 Thread Xiao Zhu (JIRA)
Xiao Zhu created AIRFLOW-2372: - Summary: SubDAGs should share parallelism of parent DAG Key: AIRFLOW-2372 URL: https://issues.apache.org/jira/browse/AIRFLOW-2372 Project: Apache Airflow Issue

[jira] [Updated] (AIRFLOW-2371) SubdagOperator ignores pooling

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2371: -- Summary: SubdagOperator ignores pooling (was: SubdagOperator ignores pooling when not using

[jira] [Updated] (AIRFLOW-2371) SubdagOperator ignores pooling when not using SequentialExecutor

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2371: -- Description: Basically we need a long term solution for

[jira] [Created] (AIRFLOW-2371) SubdagOperator ignores pooling when not using SequentialExecutor

2018-04-24 Thread Xiao Zhu (JIRA)
Xiao Zhu created AIRFLOW-2371: - Summary: SubdagOperator ignores pooling when not using SequentialExecutor Key: AIRFLOW-2371 URL: https://issues.apache.org/jira/browse/AIRFLOW-2371 Project: Apache Airflow

[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450416#comment-16450416 ] Xiao Zhu commented on AIRFLOW-74: - [~TaoFeng] fair. Can we have a JIRA ticket for the long term solution?

[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes

2018-04-24 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450401#comment-16450401 ] Xiao Zhu commented on AIRFLOW-74: - So we have to use SequentialExecutor for subdags? isn't it possibly

[jira] [Updated] (AIRFLOW-2368) Relative path should be used when enqueuing task instace

2018-04-23 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Zhu updated AIRFLOW-2368: -- Description: In our Airflow setup, scheduler and workers (connected via Celery) runs on different

[jira] [Commented] (AIRFLOW-2340) SQLalchemy pessimistic connection handling not working

2018-04-23 Thread Xiao Zhu (JIRA)
[ https://issues.apache.org/jira/browse/AIRFLOW-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449053#comment-16449053 ] Xiao Zhu commented on AIRFLOW-2340: --- We see that with our mysql db too > SQLalchemy pessimistic

[jira] [Created] (AIRFLOW-2368) Relative path should be used when enqueuing task instace

2018-04-23 Thread Xiao Zhu (JIRA)
Xiao Zhu created AIRFLOW-2368: - Summary: Relative path should be used when enqueuing task instace Key: AIRFLOW-2368 URL: https://issues.apache.org/jira/browse/AIRFLOW-2368 Project: Apache Airflow