victor-perchwell opened a new issue, #29365:
URL: https://github.com/apache/airflow/issues/29365
### Official Helm Chart version
1.7.0 (latest released)
### Apache Airflow version
2.4.1
### Kubernetes Version
v1.26.1
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### What happened
After installing the official airflow helm chart and trying to run a DAG,
the airflow workers attempt to create pods in the default namespace instead of
the airflow namespace, and they don't have permission for that.
Here's the output from the worker logs:
```
[2023-02-03, 18:31:30 UTC] {taskinstance.py:1383} INFO - Executing
<Task(KubernetesPodOperator): passing-task> on 2023-02-03 18:31:24.873405+00:00
[2023-02-03, 18:31:30 UTC] {standard_task_runner.py:54} INFO - Started
process 78 to run task
[2023-02-03, 18:31:30 UTC] {standard_task_runner.py:82} INFO - Running:
['airflow', 'tasks', 'run', 'kube_sample', 'passing-task',
'manual__2023-02-03T18:31:24.873405+00:00', '--job-id', '4', '--raw',
'--subdir', 'DAGS_FOLDER/kube_sample_dag.py', '--cfg-path', '/tmp/tmp_yzq00jr']
[2023-02-03, 18:31:30 UTC] {standard_task_runner.py:83} INFO - Job 4:
Subtask passing-task
[2023-02-03, 18:31:30 UTC] {dagbag.py:525} INFO - Filling up the DagBag from
/opt/airflow/dags/kube_sample_dag.py
[2023-02-03, 18:31:31 UTC] {task_command.py:384} INFO - Running
<TaskInstance: kube_sample.passing-task
manual__2023-02-03T18:31:24.873405+00:00 [running]> on host
airflow-worker-0.airflow-worker.airflow.svc.cluster.local
[2023-02-03, 18:31:32 UTC] {taskinstance.py:1592} INFO - Exporting the
following env vars:
[email protected]
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=kube_sample
AIRFLOW_CTX_TASK_ID=passing-task
AIRFLOW_CTX_EXECUTION_DATE=2023-02-03T18:31:24.873405+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-02-03T18:31:24.873405+00:00
[2023-02-03, 18:31:32 UTC] {kubernetes_pod.py:587} INFO - Creating pod
passing-test-6055b8067ab345bdb6d46939603268fd with labels: {'dag_id':
'kube_sample', 'task_id': 'passing-task', 'run_id':
'manual__2023-02-03T183124.8734050000-88eab1eb0', 'kubernetes_pod_operator':
'True', 'try_number': '1'}
[2023-02-03, 18:31:32 UTC] {taskinstance.py:1851} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 419, in execute
context=context,
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 387, in get_or_create_pod
pod = self.find_pod(self.namespace or
pod_request_obj.metadata.namespace, context=context)
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 371, in find_pod
label_selector=label_selector,
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py",
line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) #
noqa: E501
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py",
line 15826, in list_namespaced_pod_with_http_info
collection_formats=collection_formats)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py",
line 353, in call_api
_preload_content, _request_timeout, _host)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py",
line 184, in __call_api
_request_timeout=_request_timeout)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py",
line 377, in request
headers=headers)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py",
line 244, in GET
query_params=query_params)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py",
line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'6c8ef6d6-b713-4080-8872-22377eeb9143', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff',
'X-Kubernetes-Pf-Flowschema-Uid': 'ec584bbc-619e-4370-ae0c-98b9e6e4b4d1',
'X-Kubernetes-Pf-Prioritylevel-Uid': '9cc2b786-09db-4bfc-85ec-4155174d6446',
'Date': 'Fri, 03 Feb 2023 18:31:32 GMT', 'Content-Length': '289'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"system:serviceaccount:airflow:airflow-worker\" cannot
list resource \"pods\" in API group \"\" in the namespace
\"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```
### What you think should happen instead
The airflow worker should instead attempt to create pods in the airflow
namespace.
### How to reproduce
Install the official helm chart listed here:
https://airflow.apache.org/docs/helm-chart/stable/index.html
Attempt to execute a DAG via the UI.
### Anything else
I checked role bindings and they appear to exists:
```
~ ❯ kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide |
grep airflow
14:04:07
airflow
rolebinding.rbac.authorization.k8s.io/airflow-pod-launcher-rolebinding
Role/airflow-pod-launcher-role 36m
airflow/airflow-worker
airflow
rolebinding.rbac.authorization.k8s.io/airflow-pod-log-reader-rolebinding
Role/airflow-pod-log-reader-role 36m
airflow/airflow-webserver,
airflow/airflow-triggerer
```
Checking via kubectl:
```
~ ❯ kubectl auth can-i get pods --as
system:serviceaccount:airflow:airflow-webserver --namespace airflow
Yes
~ ❯ kubectl auth can-i get pods --as
system:serviceaccount:airflow:airflow-webserver --namespace airflow
No
```
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
