Also it's worth noting that in 1.10.3 we added a fix for the pod creation rate https://issues.apache.org/jira/browse/AIRFLOW-3516. Have you upgraded to 1.10.3?
On Tue, Apr 16, 2019 at 9:40 AM Daniel Imberman < [email protected]> wrote: > Hi Kamil, > > Could you explain your use-case a little further? Is it that your k8s > cluster runs into issues launching 250 tasks at the same time or that > airflow runs into issues launching 250 tasks at the same time? I'd love to > know more so I could try to address it in a future airflow release. > > Thanks! > > Daniel > > On Tue, Apr 16, 2019 at 3:32 AM Kamil Gałuszka <[email protected]> wrote: > >> Hey, >> >> We are quite interested in that Executor too but my main concern isn't it >> a >> > waste of resource to start a whole pod to run thing like DummyOperator >> for >> > example ? We have a cap of 200 tasks at any given time and we regularly >> hit >> > this cap, we cope with that with 20 celery workers but with the >> > KubernetesExecutor that would mean 200 pods, does it really scale that >> > easily ? >> > >> >> Unfortunately no. >> >> We are now having problem of having a DAG with 300 tasks in DAG that >> should >> start parallel at once, and there is only about 140 task instances >> started. >> Setting parallelism to 256 didn't help and system struggles to get the >> numbers up that high for running tasks. >> >> The biggest problem that we have now, is to find bottleneck in scheduler, >> but it's taking time to debug it. >> >> We will definitely be investigating that further and share findings but as >> for now, I wouldn't say it's "non-problematic" as some other people >> stated. >> >> Thanks >> Kamil >> >
