James Davidheiser created AIRFLOW-2317: ------------------------------------------
Summary: Support for multiple resource pools Key: AIRFLOW-2317 URL: https://issues.apache.org/jira/browse/AIRFLOW-2317 Project: Apache Airflow Issue Type: Wish Components: pools Reporter: James Davidheiser We are migrating to Airflow from Luigi, where we have the capability to require multiple pools, and multiple "units" of a pool, for a given task. This is very useful for a variety of use cases, but the two core examples are: * If a task accessing a data store is extremely resource-intensive, we might define a pool size of 10 resources, but use 5 of those in a single task to denote the extra load. When smaller tasks are using the same data store, it's completely fine to allow more of them to run at once. If a task is connecting to two different data stores, to transfer data between them, we might want to require a resource for each data store, so we can limit concurrency on both simultaneously. I know there are a lot of other issues related to capacity scheduling, which are tracked more broadly in https://issues.apache.org/jira/browse/AIRFLOW-72, and using more than one pool slot was suggested in https://issues.apache.org/jira/browse/AIRFLOW-1467, so this seems like it could be a useful thing to consider as part of larger efforts around managing capacity. -- This message was sent by Atlassian JIRA (v7.6.3#76005)