johngtam opened a new issue #22298: URL: https://github.com/apache/airflow/issues/22298
### Apache Airflow version 2.0.2 ### What happened I've attempted to use the pod_override as per the documentation [here](https://airflow.apache.org/docs/apache-airflow/2.0.2/executor/kubernetes.html#pod-override). Labels, annotations, and service accounts work. However, attempting to override the namespace does not: I would expect the pod to be created in the namespace that I've set `pod_override` to, but it does not. My speculation is that the `pod_generator.py` code is incorrect [here](https://github.com/apache/airflow/blob/2.0.2/airflow/kubernetes/pod_generator.py#L405). The order of reconciliation goes: ``` # Reconcile the pods starting with the first chronologically, # Pod from the pod_template_File -> Pod from executor_config arg -> Pod from the K8s executor pod_list = [base_worker_pod, pod_override_object, dynamic_pod] ``` Note that dynamic pod takes precedence. Note that `dynamic_pod` has the namespace from whatever namespace is [passed](https://github.com/apache/airflow/blob/2.0.2/airflow/kubernetes/pod_generator.py#L373) in. It is initialized from `self.kube_config.kube_namespace` [here](https://github.com/apache/airflow/blob/ac77c89018604a96ea4f5fba938f2fbd7c582793/airflow/executors/kubernetes_executor.py#L245). Therefore, the kube config's namespace takes precedence and will override the namespace of the `pod_override`, because only one namespace can exist (unlike labels and annotations, which can be additive). This unfortunately means pods created by the KubernetesExecutor will run in the config's namespace. This code flow remains the same on the latest version of Airflow -- I speculate that the bug still might be in. ### What you think should happen instead If we use pod_override with a namespace, the namespace where the pod is run should take the namespace desired in pod_override. ### How to reproduce Create a DAG using a PythonOperator or any other operator. Use the pod_override: ``` "pod_override": k8s.V1Pod( metadata=k8s.V1ObjectMeta(namespace="fakenamespace", annotations={"test": "annotation"}, labels={'foo': 'barBAZ'}), spec=k8s.V1PodSpec( containers=[ k8s.V1Container( name="base" ) ], service_account_name="foobar" ) ) ``` Look at the k8s spec file or use `kubectl` to verify where this pod is attempted to start running. It should be `fakenamespace`, but instead will likely be in the same namespace as whatever is set in the config file. ### Operating System EKS ### Versions of Apache Airflow Providers _No response_ ### Deployment Other 3rd-party Helm chart ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
