davlum commented on a change in pull request #6230: [AIRFLOW-5413] Allow K8S 
worker pod to be configured from JSON/YAML file
URL: https://github.com/apache/airflow/pull/6230#discussion_r357424985
 
 

 ##########
 File path: airflow/kubernetes/worker_configuration.py
 ##########
 @@ -362,24 +366,17 @@ def generate_dag_volume_mount_path(self) -> str:
 
         return self.kube_config.git_dags_folder_mount_point
 
-    def make_pod(self, namespace, worker_uuid, pod_id, dag_id, task_id, 
execution_date,
 
 Review comment:
   > This comment doesn't make sense - airflow.cfg is already only parsed once 
at start-up
   
   The functionality of `make_pod` was moved to the `construct_pod` function in 
the `PodGenerator`. Currently `WorkerConfiguration` takes the parsed 
`airflow.cfg` and converts them into Kubernetes client objects. On top of that 
the `make_pod` function in `WorkerConfiguration` takes a series of dynamic 
arguments from `AirflowKubernetesScheduler` and adds that to the static 
arguments from the `airflow.cfg` and outputs a pod. Following that the output 
pod is possibly further modified by the `executor_config` argument and possibly 
the `pod_mutation_hook`. The heuristics on what overrides what here was 
extremely difficult to grasp and I found that the `make_pod` function, which 
already triggers the huge conversion of `airflow.cfg` -> Kubernetes models was 
already sufficiently complex. Most of the functionality really only needed to 
be triggered once, as most of the arguments were static coming from 
`airflow.cfg`. With this change `make_pod` gets triggered in the constructor of 
`AirflowKubernetesScheduler` only once. 
   
   The idea here is to have all pod modification done in one spot, being 
`construct_pod`.
   `construct_pod` takes the dynamic arguments from the scheduler, the 
worker_configuration and the executor_config. It then creates a pod from the 
dynamic arguments of the scheduler and reconciles these 3 pods using the same 
method `reconcile_pods`, outputting one pod. Of course this pod may get further 
modified by the mutation_hook, but that's not in our hands.
   
   I'm sorry if this seems overly complex, I couldn't think of a more 
straightforward way with the current setup in place.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to