yogyang opened a new issue #15960:
URL: https://github.com/apache/airflow/issues/15960
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the
context.
-->
**Apache Airflow version**: 2.0.2
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`):
```
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7",
GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean",
BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc",
Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.4",
GitCommit:"8d8aa39598534325ad77120c120a22b3a990b5ea", GitTreeState:"clean",
BuildDate:"2020-03-12T20:55:23Z", GoVersion:"go1.13.8", Compiler:"gc",
Platform:"linux/amd64"}
```
**Environment**:
- **Cloud provider or hardware configuration**: rancher ec2
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
- **Kernel** (e.g. `uname -a`): Linux airflow-webserver-57ccd474bc-8rfg5
5.4.0-1045-aws #47-Ubuntu SMP Tue Apr 13 07:02:25 UTC 2021 x86_64 GNU/Linux
- **Install tools**:
- **Others**: K8s executor mode
**What happened**:
click task -> run ignore state -> Task would not be executed
webserver gives error:
```
WARNING - ApiException when attempting to run task, re-queueing. Message:
pods is forbidden: User
"system:serviceaccount:production-airflow:airflow-webserver" cannot create
resource "pods" in API group "" in the namespace "production-airflow"
[2021-05-20 04:28:07,532] {kubernetes_executor.py:275} INFO - Kubernetes job
is (TaskInstanceKey xxxxx)
[2021-05-20 04:28:07,534] {pod_launcher.py:86} ERROR - Exception when
attempting to create Namespaced Pod:
"metadata": {
"annotations": {
"dag_id": "xxxx",
"task_id": "xxxx",
"execution_date": "2021-05-20T03:00:00+00:00",
"try_number": "3",
"ad.datadoghq.com/tags": "{ \"type\": \"job\",\"task\": \"xx.xx\" }"
},
"labels": {
"airflow-worker": "manual",
"dag_id": "xxx",
"task_id": "xxx",
"execution_date": "2021-05-20T03_00_00_plus_00_00",
"try_number": "3",
"airflow_version": "2.0.2",
"kubernetes_executor": "True"
},
"name": "xxxx.b2bd66df4f3b44198d7f23cdcaae07d1",
"namespace": "production-airflow"
},
"spec": {...}
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.7/site-packages/airflow/kubernetes/pod_launcher.py",
line 82, in run_pod_async
body=sanitized_pod, namespace=pod.metadata.namespace, **kwargs
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py",
line 6174, in create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs) # noqa: E501
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py",
line 6265, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py",
line 345, in call_api
_preload_content, _request_timeout)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py",
line 176, in __call_api
_request_timeout=_request_timeout)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py",
line 388, in request
body=body)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py",
line 278, in POST
body=body)
File
"/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py",
line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json',
'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 20 May 2021 04:28:07 GMT',
'Content-Length': '316'})
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
is forbidden: User
\"system:serviceaccount:production-airflow:airflow-webserver\" cannot create
resource \"pods\" in API group \"\" in the namespace
\"production-airflow\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
```
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
Task can be ran
<!-- What do you think went wrong? -->
**How to reproduce it**:
set up a K8s airflow, manually trigger task run by task -> run (Ignore
state)
<!---
As minimally and precisely as possible. Keep in mind we do not have access
to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using
minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a
youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an
unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
