Once Daniel Imberman is back from his vacation, I do know he’s planning to jump into that JIRA issue to see what is going on. -Ry
On Wednesday, June 26, 2019, Austin Weaver <aus...@flyrlabs.com> wrote: > We attempted to migrate our existing production airflow (celery executor) > to k8 executor, and saw many issues scaling tasks ie. Couldn't run 100+ > tasks concurrently, as described here > https://issues.apache.org/jira/browse/AIRFLOW-4346 . > > Although there has not been much movement on the ticket, I am curious if > people haven't seen these issues running an airflow k8 executor deployment > at scale? > > Thanks, > Austin > > On Wed, Jun 26, 2019 at 2:35 AM Ry Walker <r...@rywalker.com> wrote: > > > If tasks are short, the k8s startup/shutdown time can be a negative > factor. > > > > The nice thing about k8s executor is that you can redeploy airflow > without > > the need to drain the system to ensure work won’t be affected. > > > > Also, going all k8s gives you autoscaling. > > > > If you’d like to try our both, you can install trial of Astronomer > > platform in your k8s - it lets you easily switch between executors. > > > > Sent via Superhuman iOS ( https://sprh.mn/?vip=r...@rywalker.com ) > > > > On Wed, Jun 26 2019 at 7:29 AM, < q...@scribd.com > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > We have made the decision to migrate all our existing ETL jobs to > > Airflow. > > > Right now, we are deciding whether celery or kubernetes executor would > be > > > a better fit. > > > > > > > > > > > > Is anyone currently using kubernetes executor in production? If so, how > > > well does it scale and how stable is it compared to celery executor? We > > > are planning to run about 2000 tasks per day to start with. > > > > > > > > > > > > From what I have seen so far, with kubernetes executor, we will be able > > to > > > restrict resource usage for each task run. Can anyone share some > insights > > > on what other advantages does celery executor provide over kubernetes > > > executor except maturity? > > > > > > > > > > > > Thanks, > > > QP > > > > > > > > > > > > > -- > Austin Weaver > Software Engineer > FLYR, Inc. www.flyrlabs.com > -- Sent from Gmail Mobile
