If tasks are short, the k8s startup/shutdown time can be a negative factor.
The nice thing about k8s executor is that you can redeploy airflow without the need to drain the system to ensure work won’t be affected. Also, going all k8s gives you autoscaling. If you’d like to try our both, you can install trial of Astronomer platform in your k8s - it lets you easily switch between executors. Sent via Superhuman iOS ( https://sprh.mn/[email protected] ) On Wed, Jun 26 2019 at 7:29 AM, < [email protected] > wrote: > > > > Hi all, > > > > We have made the decision to migrate all our existing ETL jobs to Airflow. > Right now, we are deciding whether celery or kubernetes executor would be > a better fit. > > > > Is anyone currently using kubernetes executor in production? If so, how > well does it scale and how stable is it compared to celery executor? We > are planning to run about 2000 tasks per day to start with. > > > > From what I have seen so far, with kubernetes executor, we will be able to > restrict resource usage for each task run. Can anyone share some insights > on what other advantages does celery executor provide over kubernetes > executor except maturity? > > > > Thanks, > QP > > >
