We attempted to migrate our existing production airflow (celery executor) to k8 executor, and saw many issues scaling tasks ie. Couldn't run 100+ tasks concurrently, as described here https://issues.apache.org/jira/browse/AIRFLOW-4346 .
Although there has not been much movement on the ticket, I am curious if people haven't seen these issues running an airflow k8 executor deployment at scale? Thanks, Austin On Wed, Jun 26, 2019 at 2:35 AM Ry Walker <r...@rywalker.com> wrote: > If tasks are short, the k8s startup/shutdown time can be a negative factor. > > The nice thing about k8s executor is that you can redeploy airflow without > the need to drain the system to ensure work won’t be affected. > > Also, going all k8s gives you autoscaling. > > If you’d like to try our both, you can install trial of Astronomer > platform in your k8s - it lets you easily switch between executors. > > Sent via Superhuman iOS ( https://sprh.mn/?vip=r...@rywalker.com ) > > On Wed, Jun 26 2019 at 7:29 AM, < q...@scribd.com > wrote: > > > > > > > > > Hi all, > > > > > > > > We have made the decision to migrate all our existing ETL jobs to > Airflow. > > Right now, we are deciding whether celery or kubernetes executor would be > > a better fit. > > > > > > > > Is anyone currently using kubernetes executor in production? If so, how > > well does it scale and how stable is it compared to celery executor? We > > are planning to run about 2000 tasks per day to start with. > > > > > > > > From what I have seen so far, with kubernetes executor, we will be able > to > > restrict resource usage for each task run. Can anyone share some insights > > on what other advantages does celery executor provide over kubernetes > > executor except maturity? > > > > > > > > Thanks, > > QP > > > > > > -- Austin Weaver Software Engineer FLYR, Inc. www.flyrlabs.com
