hterik opened a new issue, #35174: URL: https://github.com/apache/airflow/issues/35174
### Description Schedule tasks based on labels/capabilities with and+or matching, instead of just queue names. ### Use case/motivation Imagine you have tasks that can only run on specific nodes, for example GPUs, different operating systems or other external HW-peripherals connected. Some tasks have partial constraints, Task A: requires `gpu` Task B: requires `linux` Task C: requires `gpu && linux` Task D: requires `gpu && windows` Task E: requires `gpu || mac-m1-cpu` To serve the above you might have 5 different nodes Node 1: Linux (can serve task B only) Node 2: Linux+Gpu (can serve A+B+C+D+E) Node 3: Windows+Gpu (can serve A+D) Optimizing this type of planning is, from what i understand, not possible with CeleryExecutor today. Airflow celery workers can only listen to a list of queues. In the scenario above, task D should not be assigned to a single queue, because there are two workers that could potentially execute it. Node2+3 can't listen to a common queue though because they support different features. Instead you would have to compromise and choose one of the nodes for task D. Is this even possible to solve with Celery? I've scrutinized the underlying Celery documents deeply and it has lots of advanced features for _inserting_ items to the queues using exchanges and routers, but it seems like _consumers_ only read from fixed queue-names. Am i missing something or is this correct? It feels like this should be a very common scenario. Compared with other systems, Kubernetes can do it with [nodeSelectors, affinity and anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node). Using Airflow KubernetesExecutor these can be injected in `pod_override`. Jenkins has [label conditions](https://www.jenkins.io/doc/book/pipeline/syntax/#agent) ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
