I agree there should be an option with more flexibility than the current
k8s pod operator. The design was a mixture of trying to introduce k8s to a
larger community by abstracting details and frankly my own freshness with
the technology (I had only been using it for 3 months when this effort
starte
The challenge with using yaml to define the pod spec is we need to inject
values into the yaml in order for the pod to work properly.
For example, if you try setting the command property, then the pod will not
actually run the airflow command to start the task. Same idea with needed
environmental
Would be great if this also worked for KubernetesExecutor config. I made a
PR: https://github.com/apache/airflow/pull/4456 to add a
default_executor_config because it doesn't make much sense configuring
every operator with the same config. I think it would be much more
preferable to use YAML, still
Thanks for starting the discussion.
I think it would be great. And a great first step would be to supply the
filepath.
A decorator approach would be suitable I think and would allow for
extendability and would allow for much than yaml. I'm thinking helm charts
:D
On 2019/03/06 14:58:03, da...@ss
Thanks for starting the discussion David.
Any templating should apply for both kubernetes airflow workers and
kubernetes pod operators. I estimate there are currently around 20 objects
in the pod spec missing (kubectl explain pod.spec --recursive).
The main challenge would probably be getting air
I'm in favor of having a YAML-based option for Kubernetes. We've had to
internally subclass the Kubernetes operator because it really isn't doing
what we need out of the box; such as intercepting the object it creates
right before it sends it so that we can patch in missing features. I think
it wou
Hi,
I would like to discuss parsing YAML for the Kubernetes worker configuration
instead of the current process of programmatically generating the YAML from the
Pod and PodRequest Factory as is done currently.
*Motivation:*
Kubernetes configuration is quite complex. Instead of using the config
