+1 for pod mutation in config. It's not a "yamly" thing - it's a python code to run, so it's quite appropriate to have it in config.
On Wed, Aug 12, 2020 at 9:43 PM Daniel Imberman <daniel.imber...@gmail.com> wrote: > I think pod_mutation is important as an “admin override”. E.g. there might > be a label that is required by security policy for all tasks launched by > airflow. I do agree that the executor_config should take in a V1Pod object > > One option is that the executor_config could take in a pod_mutation_hook > function as well? So users can manually override the pod functionally > rather than just giving an object and hoping it merges the way they expect > it to? > via Newton Mail [ > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2 > ] > On Wed, Aug 12, 2020 at 12:19 PM, David Lum <davidlu...@gmail.com> wrote: > I'll just leave some passing thoughts here. How the pod template file is > modified by the executor config might not be very well defined. Currently > when a user passes in a k8s.V1Pod object through executor_config we kind of > arbitrarily merge the two pod definitions to what we imagine the client > would want, but it could be often misinterpreted. Previously the > Airflow-defined Pod class was fairly flat. This made merging the two > objects fairly straight forward. If the user wrote the field in > executor_config, it overwrote that field in the Pod definition, but the > rest stayed the same. When attributes are arbitrarily nested like in the > k8s.V1Pod, some aspects can be mistakenly overwritten. For example, > specifying the volumeMounts of a container requires also specifying the > rest of the V1Pod. How do we handle this? Do we just overwrite > volumeMounts? The client must have specified a container, so do we > overwrite that too? The fact that there is a choice means the API could be > unintuitive to the user. > > What I suggest might be easier is that executor_config completely replaces > the pod_template_file if it is specified, but then also offer something > like pod_mutation_hook at the operator level. This makes it quite clear to > the user, you are either replacing the pod_template, or you are modifying > the pod_template with pod_mutation_hook and you are doing it by your own > rules. the argument to executor_config can be something like > ``` > executor_config: { 'KubernetesExector': {'pod_mutation': some_func, > 'pod_template': k8s.V1Pod(...) }} # Pod template could also be a path to a > file > ``` > > If both are specified then pod_mutation is a function applied to > pod_template. Thoughts? > > On Wed, Aug 12, 2020 at 11:24 AM Tomasz Urbaszek <turbas...@apache.org> > wrote: > > > +1 for the idea, thanks Daniel! I agree that multi-executor and pod > > templates should be 2.1. > > > > T. > > > > On Wed, Aug 12, 2020 at 5:13 PM Daniel Imberman < > daniel.imber...@gmail.com> > > wrote: > > > >> (Also funny enough we could use a lot of the existing infrastructure int > >> 1.10 to create that migration script. Would just need to take the python > >> dictionary and use the python yaml library) > >> > >> via Newton Mail [ > >> > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2 > >> ] > >> On Wed, Aug 12, 2020 at 8:09 AM, Daniel Imberman < > >> daniel.imber...@gmail.com> wrote: > >> 100% agreed on timing. I think 2.0 should be for the breaking aspect > >> (losing the configurations) and then 2.1/2.2 we can start adding on the > new > >> features. I also like the idea of a migration tool. We can make a script > >> that takes your airflow.cfg and converts that into a yaml file. > >> > >> via Newton Mail [ > >> > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2 > >> ] > >> On Wed, Aug 12, 2020 at 7:48 AM, Jarek Potiuk <jarek.pot...@polidea.com > > > >> wrote: > >> Big +1. All the arguments are very appealing to me and simplifying the > >> Kubernetes Executor down to YAML-configurable one seems like a > no-brainer > >> especially if we provide some migration tools. I've lost countless hours > >> on > >> debugging some configuration problems, simply because the relevant > >> Kubernetes-related configuration was in the least expected place - i.e. > >> airflow.cfg. YAML configuration. > >> > >> I am also a big fan of both 1. and 2. I've implemented a POC of > >> queue-based multi-scheduler once but having it embedded as part of core > >> Airflow rather than based it on queues (which are basically a Celery > >> Executor concept) is I think much better approach. Both 1. and 2. are > >> cool. > >> > >> Now - question about timing. If we decide to go that route - my view is > >> that simplifying Kubernetes should be an Airflow 2.0 task - alongside > more > >> comprehensive tests (which will be much easier to write in this case). > The > >> new features/ideas 1. 2. for KE I think should come after that - when we > >> release and stabilize 2.0. Sounds like great candidates for 2.1 to me. > >> > >> J. > >> > >> > >> On Wed, Aug 12, 2020 at 4:24 PM Daniel Imberman < > >> daniel.imber...@gmail.com> > >> wrote: > >> > >> > Hello, fellow Airflowers! I hope you are all well in these trying > times. > >> > > >> > > >> > With the recent launch of Airflow 2.0 preparation, it now seems like a > >> > good time to review the project's state and where we can fit in some > >> > breaking changes that will improve the project for the future. > >> > > >> > > >> > When we first created the KubernetesExecutor, we had two goals in > mind. > >> > The first goal was to improve the airflow Auto scaling story. > >> Previously, > >> > airflow users would have to manually provision celery workers, which > >> could > >> > lead to wasted resources or missed SLAs. The other goal was to > >> introduce a > >> > community that was not yet well versed in the Kubernetes API to the > >> > Kubernetes system. > >> > > >> > > >> > To ease the community's transition, we abstracted many of the > >> complexities > >> > of creating a Kubernetes object. We chose to offer a limited number of > >> > configurations and keep much of the pod creation process internal to > >> > airflow. In the short-term, this system lowered the barrier to entry. > >> Over > >> > time, however, this abstraction has become a nightmare of tech debt as > >> the > >> > Kubernetes API is expensive and constantly changing. > >> > > >> > > >> > With this in mind, I think it's time for us to consider a more > >> > straightforward approach that takes the complexity out of Airflow and > >> > offers the full Kubernetes API to the airflow user. > >> > > >> > > >> > What I'm proposing here is pretty straightforward. We remove all > >> > Kubernetes pod creation configurations from the airflow.cfg and > instead > >> > offer only one way to use the KubernetesExecutor: with a YAML file. > >> > > >> > > >> > We can easily supply all of the configurations to the > KubernetesExecutor > >> > by offering example YAMLs (git sync mode is just a sidecar and an init > >> > container, DAG volumes are just an example volume and volume mount, > >> etc.). > >> > > >> > > >> > This system would simplify a user's ability to predict what a pod will > >> > look like once it is launched by airflow. They will know it's a base > pod > >> > and will be able to simply modify the pod object using the executor > >> config > >> > and the pod mutation hook. > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > This simplification could also lead to some pretty great new features > in > >> > the KubernetesExecutor > >> > > >> > > >> > Idea 1: Picking a pod_template_file per-task > >> > Along with the existing customization with the executor config, solely > >> > relying on pod files can allow users to pick the pod template file > that > >> > they want to use as their base pod on a per-task basis. An Airflow > >> engineer > >> > could supply several pre-made templates for their data scientists to > >> reduce > >> > the amount of customization an airflow user would need to use. > >> > > >> > > >> > Idea 2: Merging the KubernetesExecutor into the CeleryExecutor > >> > One idea that we've been excited about recently has been the idea of > >> > creating merged Celery and Kubernetes executor. This hybrid executor > >> would > >> > default to launching celery workers with KEDA and would have the > option > >> to > >> > launch individual tasks using the Kubernetes executor when a user > wants > >> > isolation or customization. Simplifying the Kubernetes executor > reduces > >> the > >> > number of fail-points that this merged executor would need to account > >> for. > >> > > >> > > >> > > >> > What would we need to do to implement this? > >> > The good news here is that the hard work has already been done! As of > >> > AIRFLOW-5413 [https://issues.apache.org/jira/browse/AIRFLOW-5413] by > >> > David Lum, airflow already has the ability to use base worker pods on > a > >> > template file. This would involve primarily code deletion and very > >> little > >> > new code. > >> > > >> > > >> > Thank you for your time and I look forward to the community’s > >> discussion. > >> > > >> > Daniel > >> > >> > >> > >> -- > >> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> M: +48 660 796 129 <+48660796129> > >> [image: Polidea] <https://www.polidea.com/> > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
