davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S
worker pod to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#discussion_r357424985
##########
File path: airflow/kubernetes/worker_configuration.py
##########
@@ -362,24 +366,17 @@ def generate_dag_volume_mount_path(self) -> str:
return self.kube_config.git_dags_folder_mount_point
- def make_pod(self, namespace, worker_uuid, pod_id, dag_id, task_id,
execution_date,
Review comment:
> This comment doesn't make sense - airflow.cfg is already only parsed once
at start-up
The functionality of `make_pod` was moved to the `construct_pod` function in
the `PodGenerator`. Currently `WorkerConfiguration` takes the parsed
`airflow.cfg` and converts them into Kubernetes client objects. On top of that
the `make_pod` function in `WorkerConfiguration` takes a series of dynamic
arguments from `AirflowKubernetesScheduler` and adds that to the static
arguments from the `airflow.cfg` and outputs a pod. Following that the output
pod is possibly further modified by the `executor_config` argument and possibly
the `pod_mutation_hook`. The heuristics on what overrides what here was
extremely difficult to grasp and I found that the `make_pod` function, which
already triggers the huge conversion of `airflow.cfg` -> Kubernetes models was
already sufficiently complex. Most of the functionality really only needed to
be triggered once, as most of the arguments were static coming from
`airflow.cfg`. With this change `make_pod` gets triggered in the constructor of
`AirflowKubernetesScheduler` only once.
The idea here is to have all pod modification done in one spot, being
`construct_pod`.
`construct_pod` takes the dynamic arguments from the scheduler, the
worker_configuration and the executor_config. It then creates a pod from the
dynamic arguments of the scheduler and reconciles these 3 pods using the same
method `reconcile_pods`, outputting one pod. Of course this pod may get further
modified by the mutation_hook, but that's not in our hands.
I'm sorry if this seems overly complex, I couldn't think of a more
straightforward way with the current setup in place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services