arjunanan6 commented on PR #29012:
URL: https://github.com/apache/airflow/pull/29012#issuecomment-1398288734
Alright, so I attempted this locally where I have no restrictions. As
discussed on #28394, scheduled tasks run just fine, but there is an exception
when running a task manually:
```
[2023-01-20T11:57:44.263+0000] {kubernetes_executor.py:527} INFO - Start
Kubernetes executor
[2023-01-20T11:57:44.306+0000] {kubernetes_executor.py:476} INFO - Found 0
queued task instances
[2023-01-20T11:57:44.309+0000] {base_executor.py:95} INFO - Adding to queue:
['airflow', 'tasks', 'run', 'HELLO_WORLD', 'hello',
'scheduled__2023-01-20T11:40:00+00:00', '--ignore-all-dependencies',
'--ignore-dependencies', '--force', '--local', '--pool', 'default_pool',
'--subdir', 'DAGS_FOLDER/hello.py']
[2023-01-20T11:57:44.310+0000] {kubernetes_executor.py:559} INFO - Add task
TaskInstanceKey(dag_id='HELLO_WORLD', task_id='hello',
run_id='scheduled__2023-01-20T11:40:00+00:00', try_number=4, map_index=-1) with
command ['airflow', 'tasks', 'run', 'HELLO_WORLD', 'hello',
'scheduled__2023-01-20T11:40:00+00:00', '--ignore-all-dependencies',
'--ignore-dependencies', '--force', '--local', '--pool', 'default_pool',
'--subdir', 'DAGS_FOLDER/hello.py']
[2023-01-20T11:57:44.310+0000] {kubernetes_executor.py:130} INFO - Event:
and now my watch begins starting at resource_version: 0
[2023-01-20T11:57:44.383+0000] {kubernetes_executor.py:339} INFO - Creating
kubernetes pod for job is TaskInstanceKey(dag_id='HELLO_WORLD',
task_id='hello', run_id='scheduled__2023-01-20T11:40:00+00:00', try_number=2,
map_index=-1), with pod name hello-world-hello-be2bad2bd8dc4568bd1ba73082ecef4a
[2023-01-20T11:57:44.392+0000] {kubernetes_executor.py:274} ERROR -
Exception when attempting to create Namespaced Pod: {
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"annotations": {
"dag_id": "HELLO_WORLD",
"task_id": "hello",
"try_number": "2",
"run_id": "scheduled__2023-01-20T11:40:00+00:00"
},
"labels": {
"tier": "airflow",
"component": "worker",
"release": "airflowlocal",
"airflow-worker": "None",
"dag_id": "HELLO_WORLD",
"task_id": "hello",
"try_number": "2",
"airflow_version": "2.5.0",
"kubernetes_executor": "True",
"run_id": "scheduled__2023-01-20T1140000000-c15690dab"
},
"name": "hello-world-hello-be2bad2bd8dc4568bd1ba73082ecef4a",
"namespace": "airflow"
},
"spec": {
"affinity": {},
"containers": [
{
"args": [
"airflow",
"tasks",
"run",
"HELLO_WORLD",
"hello",
"scheduled__2023-01-20T11:40:00+00:00",
"--ignore-all-dependencies",
"--ignore-dependencies",
"--force",
"--local",
"--pool",
"default_pool",
"--subdir",
"DAGS_FOLDER/hello.py"
],
"env": [
{
"name": "AIRFLOW__CORE__EXECUTOR",
"value": "LocalExecutor"
},
{
"name": "AIRFLOW__CORE__FERNET_KEY",
"valueFrom": {
"secretKeyRef": {
"key": "fernet-key",
"name": "airflowlocal-fernet-key"
}
}
},
{
"name": "AIRFLOW__CORE__SQL_ALCHEMY_CONN",
"valueFrom": {
"secretKeyRef": {
"key": "connection",
"name": "airflowlocal-airflow-metadata"
}
}
},
{
"name": "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN",
"valueFrom": {
"secretKeyRef": {
"key": "connection",
"name": "airflowlocal-airflow-metadata"
}
}
},
{
"name": "AIRFLOW_CONN_AIRFLOW_DB",
"valueFrom": {
"secretKeyRef": {
"key": "connection",
"name": "airflowlocal-airflow-metadata"
}
}
},
{
"name": "AIRFLOW__WEBSERVER__SECRET_KEY",
"valueFrom": {
"secretKeyRef": {
"key": "webserver-secret-key",
"name": "airflowlocal-webserver-secret-key"
}
}
},
{
"name": "AIRFLOW_IS_K8S_EXECUTOR_POD",
"value": "True"
}
],
"image": "my-dags:0.0.1",
"imagePullPolicy": "IfNotPresent",
"name": "base",
"resources": {},
"volumeMounts": [
{
"mountPath": "/opt/airflow/logs",
"name": "logs"
},
{
"mountPath": "/opt/airflow/airflow.cfg",
"name": "config",
"readOnly": true,
"subPath": "airflow.cfg"
},
{
"mountPath": "/opt/airflow/config/airflow_local_settings.py",
"name": "config",
"readOnly": true,
"subPath": "airflow_local_settings.py"
}
]
}
],
"restartPolicy": "Never",
"securityContext": {
"fsGroup": 0,
"runAsUser": 50000
},
"serviceAccountName": "airflowlocal-worker",
"volumes": [
{
"emptyDir": {},
"name": "logs"
},
{
"configMap": {
"name": "airflowlocal-airflow-config"
},
"name": "config"
}
]
}
}
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py",
line 269, in run_pod_async
resp = self.kube_client.create_namespaced_pod(
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py",
line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs) # noqa: E501
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py",
line 7455, in create_namespaced_pod_with_http_info
return self.api_client.call_api(
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py",
line 348, in call_api
return self.__call_api(resource_path, method,
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py",
line 180, in __call_api
response_data = self.request(
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py",
line 391, in request
return self.rest_client.POST(url,
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py",
line 275, in POST
return self.request("POST", url,
File
"/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py",
line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'c2225504-16da-4966-ae42-36241a0d49cb', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff',
'X-Kubernetes-Pf-Flowschema-Uid': '3dd6f880-05b3-46ee-a0f2-a45a4c663a50',
'X-Kubernetes-Pf-Prioritylevel-Uid': 'b8f0d7a8-7a33-4a3e-8e49-112cfe10ad1d',
'Date': 'Fri, 20 Jan 2023 11:57:44 GMT', 'Content-Length': '299'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User \"system:serviceaccount:airflow:airflowlocal-webserver\"
cannot create resource \"pods\" in API group \"\" in the namespace
\"airflow\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```
Which is strange, since this is the same service account that is spinning up
pods to execute a scheduled task, here is a yaml of that pod definition:
```
apiVersion: v1
kind: Pod
metadata:
annotations:
dag_id: HELLO_WORLD
run_id: scheduled__2023-01-20T11:40:00+00:00
task_id: hello
try_number: "3"
creationTimestamp: "2023-01-20T11:57:11Z"
labels:
airflow-worker: "80"
airflow_version: 2.5.0
component: worker
dag_id: HELLO_WORLD
kubernetes_executor: "True"
release: airflowlocal
run_id: scheduled__2023-01-20T1140000000-c15690dab
task_id: hello
tier: airflow
try_number: "3"
name: hello-world-hello-3c579ea84688467bab7036d3bc940c64
namespace: airflow
resourceVersion: "40588"
uid: 53017417-d794-4532-bdb8-fa92db4f97fd
spec:
affinity: {}
containers:
- args:
- airflow
- tasks
- run
- HELLO_WORLD
- hello
- scheduled__2023-01-20T11:40:00+00:00
- --local
- --subdir
- DAGS_FOLDER/hello.py
env:
- name: AIRFLOW__CORE__EXECUTOR
value: LocalExecutor
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
key: fernet-key
name: airflowlocal-fernet-key
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
key: connection
name: airflowlocal-airflow-metadata
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
key: connection
name: airflowlocal-airflow-metadata
- name: AIRFLOW_CONN_AIRFLOW_DB
valueFrom:
secretKeyRef:
key: connection
name: airflowlocal-airflow-metadata
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
key: webserver-secret-key
name: airflowlocal-webserver-secret-key
- name: AIRFLOW_IS_K8S_EXECUTOR_POD
value: "True"
image: my-dags:0.0.1
imagePullPolicy: IfNotPresent
name: base
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/airflow/logs
name: logs
- mountPath: /opt/airflow/airflow.cfg
name: config
readOnly: true
subPath: airflow.cfg
- mountPath: /opt/airflow/config/airflow_local_settings.py
name: config
readOnly: true
subPath: airflow_local_settings.py
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-w5gcp
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: kind-control-plane
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext:
fsGroup: 0
runAsUser: 50000
serviceAccount: airflowlocal-worker
serviceAccountName: airflowlocal-worker
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: logs
- configMap:
defaultMode: 420
name: airflowlocal-airflow-config
name: config
- name: kube-api-access-w5gcp
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2023-01-20T11:57:11Z"
reason: PodCompleted
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2023-01-20T11:57:20Z"
reason: PodCompleted
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2023-01-20T11:57:20Z"
reason: PodCompleted
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2023-01-20T11:57:11Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID:
containerd://a81155f6103c7fee21ab8b06298d8ba04a112f91356685f2d6414dd68136eb3b
image: docker.io/library/my-dags:0.0.1
imageID:
docker.io/library/import-2023-01-20@sha256:9298bf3504bc8279b2270f7d41e9d2c1244a39e22e0ac0c534c5e881d5621ca9
lastState: {}
name: base
ready: false
restartCount: 0
started: false
state:
terminated:
containerID:
containerd://a81155f6103c7fee21ab8b06298d8ba04a112f91356685f2d6414dd68136eb3b
exitCode: 0
finishedAt: "2023-01-20T11:57:19Z"
reason: Completed
startedAt: "2023-01-20T11:57:11Z"
hostIP: 172.18.0.2
phase: Running
podIP: 10.244.0.42
podIPs:
- ip: 10.244.0.42
qosClass: BestEffort
startTime: "2023-01-20T11:57:11Z"
```
I verified whether this SA is able to spin up a pod, and that checks out too:
```
kubectl auth can-i get pods --as
system:serviceaccount:airflow:airflowlocal-webserver
yes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
