utkarsharma2 commented on issue #41055: URL: https://github.com/apache/airflow/issues/41055#issuecomment-2254177871
my 2c: I think there is a valid case for the infinite/unknown parallelism for KE. For example, if we have a cluster with some capacity X Gi of memory, now how do we decide the optimal parallelism in such scenarios? We can defer it to the Kubernetes scheduler to do it. As it totally depends on the resource requirement of the task, the X could be distributed say, X / 10 or X / 20 or even non-uniformly across pods. However, IMO we can default it to -1 in such cases instead of 0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
