On Wed, Jun 26, 2019 at 2:35 AM Ry Walker <r...@rywalker.com> wrote: > The nice thing about k8s executor is that you can redeploy airflow without > the need to drain the system to ensure work won’t be affected. > > Also, going all k8s gives you autoscaling. > > If you’d like to try our both, you can install trial of Astronomer platform > in your k8s - it lets you easily switch between executors.
Being able to redeploy without draining seems like a big advantage to me. I will check out astronomer platform as well. On Wed, Jun 26, 2019 at 9:14 AM Austin Weaver <aus...@flyrlabs.com> wrote: > > We attempted to migrate our existing production airflow (celery executor) > to k8 executor, and saw many issues scaling tasks ie. Couldn't run 100+ > tasks concurrently, as described here > https://issues.apache.org/jira/browse/AIRFLOW-4346 . > Thanks for the info. Looking at the symptom, it does look like you are hitting the k8s maximum pod per node restriction. I will keep an eye on the ticket and try to see if I can reproduce it from my side.
