Would be great if this also worked for KubernetesExecutor config. I made a PR: https://github.com/apache/airflow/pull/4456 to add a default_executor_config because it doesn't make much sense configuring every operator with the same config. I think it would be much more preferable to use YAML, still use the merge functionality, and be able to have much more control over worker pod configuration maybe add additional labels stuff like that.
On Wed, Mar 6, 2019 at 1:14 PM Marwan Nabil <mrwanbaghda...@gmail.com> wrote: > Thanks for starting the discussion. > > I think it would be great. And a great first step would be to supply the > filepath. > > A decorator approach would be suitable I think and would allow for > extendability and would allow for much than yaml. I'm thinking helm charts > :D > > On 2019/03/06 14:58:03, da...@ssense.com <d...@ssense.com> wrote: > > Hi,> > > > > I would like to discuss parsing YAML for the Kubernetes worker > configuration instead of the current process of programmatically generating > the YAML from the Pod and PodRequest Factory as is done currently.> > > > > *Motivation:*> > > > > Kubernetes configuration is quite complex. Instead of using the > configuration system that is offered natively by Kubernetes (YAML),the > current method involves programmatically recreating this YAML file. Fully > re-implementing the configuration in Airflow is taking a lot of time, and > at the moment many features available through YAML configuration are not > available in Airflow. Furthermore, as the Kubernetes API evolves, the > Airflow codebase will have to change with it, and Airflow will be in a > constant state of catching up with missing features available. This can all > be solved by simply parsing the YAML file.> > > > > *Idea:*> > > > > Either pass in the YAML as string or have a path to the YAML file.> > > > -- Kyle Hamlin
