Hey, We are quite interested in that Executor too but my main concern isn't it a waste of resource to start a whole pod to run thing like DummyOperator for example ? We have a cap of 200 tasks at any given time and we regularly hit this cap, we cope with that with 20 celery workers but with the KubernetesExecutor that would mean 200 pods, does it really scale that easily ?
Also I am very interest to know if ppl get this to work with the cluster autoscaler that can kill pod at any given moment. Thx, E On Thu, Apr 11, 2019 at 12:04 AM Daniel Imberman <[email protected]> wrote: > Hi all, > > Will try to answer questions one-by-one: > > > *Is the k8s executor stable?* > I've spoken to multiple companies that run the k8s executor in production > and have not heard any stories of major issues. It's a feature that's still > actively in development, but it has a pretty good track record for > stability. Astronomer.io uses a lightly modified version of the k8s > executor for their on-prem offering so they could speak to their stories > pretty well (cc: @Greg Neiheisel <[email protected]> ) > > > *What are major benefits of the k8s executor?* > There are a few major advantages to the k8s executor: > > 1. No need for capacity planning > > Unlike other executors where you need to determine and resize the number of > workers requires, the k8s executor dynamically resized. This also prevents > two major scheduling issues: under-utilization and backlog. Preventing > backlog is really valuable if you want all jobs completed in a timely > manner, and preventing under-utilization will reduce your cloud > bill/compute usage. > > 2. Better fault tolerance: > > The k8s executor takes advantage of the kubernetes event stream to ensure > that state is not held locally within the airflow pod. This means if a > scheduler goes down, you can recreate the state of the scheduler by > re-reading this event log. This means your system can go down for > hours/days and pick up right where it left off with no duplication/skipped > steps > > 3. Custom airflow images/resources per-task > > You can launch docker images on a per-task basis to handle a lot of > lower-level dependencies. Say you have a task requiring certain DPKGs or > system libraries, but don't want to lug those libraries everywhere, now > they only exist where needed (i.e. you could have an image with java) > > 4. Per-task security > > You can use k8s secrets to inject credentials on a per-task basis, so you > don't need to have all of your accounts available on the same pods. > > To Magnus' point the benefits of the k8s operator vs. executor are pretty > different (though they are complimentary). The point of the k8s executor > isn't that you can launch pods in k8s (the operator can do that just fine), > but it scales really well (you can have hundreds/potentially thousands of > tasks running in parallel and then have the cluster shrink back to a single > pod), offers valuable stability/security features, and is significantly > easier WRT maintenance (no need to monitor celery/redis/, launches from a > helm script, restarts itself on any failures). > > Hope this helps :) > > On Wed, Apr 10, 2019 at 12:32 PM Kyle Hamlin <[email protected]> wrote: > > > I would really love to know what are the advantages of the > > KubernetesExecutor are over the CeleryExecutor. Also, what are the types > of > > use cases that KubernetesExecutor is well suited for vs CeleryExecutor > and > > vise versa? > > > > On Wed, Apr 10, 2019 at 1:09 PM Magnus Runesson <[email protected]> > > wrote: > > > > > Hi! > > > > > > I were in similar position early this year at my work at Tink and went > > > with Kubernetes. We do run all our jobs via the Kubernetes pod operator > > > and rely on K8s to schedule resources etc. A big advantage for us is > > > that we reuse same K8s environment as our services uses for our data > > > processing. Airflow has it own namespace and our infrastructure team > > > takes care about K8s. We have not had any stability issues and > basically > > > not have to think that much about maintenance. So in my perspective > much > > > less than using Celery. > > > > > > Each task running its own pod means we can ha different software for > > > different tasks and we run them as different service accounts with > > > different permissions regarding access to databases, network etc. We > > > have not found any major drawbacks, rather the opposite with the > > > advantages mentioned above. > > > > > > We needed to add some capabilities to control the security context of > > > our pods since our environment is quite locked down. See > > > > > > > > > https://github.com/mrunesson/airflow/commit/e2bc358ab094bb519932d870cb3623c127927e37 > > > and https://issues.apache.org/jira/browse/AIRFLOW-4156 > > > > > > We evaluated the Kube executor but decided not to use it. Since our > > > Airflow instance only have to mange tasks that spawns K8s pods we found > > > the Kube executor did not add any benefits for us. The load on the > > > airflow instance to manage our pods is so small. Some drawbacks not > > > using the Kube executor is that we "cannot" use other operators than > the > > > Kubernetes operator. But we do not want that since that would remove > the > > > isolation we have between tasks. I should say, the tests we did with > the > > > Kube executor showed no stability issues. > > > > > > Hope this gives you some input. > > > > > > /Magnus > > > > > > > > > On 2019-04-10 08:53, Ashwin Sai Shankar wrote: > > > > Hi Airflow devs, > > > > I am to new airflow and trying to figure out some details which will > > help > > > > me choose the right deployment for my company. I want to create a > > > > distributed airflow deployment either using Kube or Celery executor, > > and > > > > had a few questions. > > > > > > > > 1. I see that airflow on kube is a recent feature. How stable is this > > > > feature for production deployments in the most recent oss version? > Are > > > > there any important known problems with airflow on kube? > > > > > > > > 2. I notice that some jiras are still pending in the umbrella jira( > > > > https://issues.apache.org/jira/browse/AIRFLOW-1314). Is it better to > > > wait > > > > for these tickets to be closed? > > > > > > > > 3. How does kube deployment compare to Celery in terms of stability, > > > > performance, feature set, ease of deployment, maintenance? > > > > > > > > Thanks, > > > > Ash > > > > > > > > > > > > > -- > > Kyle Hamlin > > > -- GetYourGuide AG Stampfenbachstrasse 48 8006 Zürich Switzerland <https://www.facebook.com/GetYourGuide> <https://twitter.com/GetYourGuide> <https://www.instagram.com/getyourguide/> <https://www.linkedin.com/company/getyourguide-ag> <http://www.getyourguide.com>
