johnhoran commented on issue #51789:
URL: https://github.com/apache/airflow/issues/51789#issuecomment-2976823177
```
[2025-06-16T00:14:07.434+0000] {local_task_job_runner.py:123} INFO -
::group::Pre task execution logs
[2025-06-16T00:14:07.450+0000] {taskinstance.py:2614} INFO - Dependencies
all met for dep_context=non-requeueable deps ti=<TaskInstance:
zdp_dbt_regional_support_fdn_models_amer_staging.regional_product_support.base_rules_run
scheduled__2025-06-15T00:00:00+00:00 [queued]>
[2025-06-16T00:14:07.458+0000] {taskinstance.py:2614} INFO - Dependencies
all met for dep_context=requeueable deps ti=<TaskInstance:
zdp_dbt_regional_support_fdn_models_amer_staging.regional_product_support.base_rules_run
scheduled__2025-06-15T00:00:00+00:00 [queued]>
[2025-06-16T00:14:07.458+0000] {taskinstance.py:2867} INFO - Starting
attempt 3 of 3
[2025-06-16T00:14:07.469+0000] {taskinstance.py:2890} INFO - Executing
<Task(DbtRunKubernetesOperator): regional_product_support.base_rules_run> on
2025-06-15 00:00:00+00:00
[2025-06-16T00:14:07.475+0000] {warnings.py:112} WARNING -
/usr/local/lib/python3.12/site-packages/airflow/task/task_runner/standard_task_runner.py:70:
DeprecationWarning: This process (pid=170) is multi-threaded, use of fork()
may lead to deadlocks in the child.
pid = os.fork()
[2025-06-16T00:14:07.476+0000] {standard_task_runner.py:72} INFO - Started
process 172 to run task
[2025-06-16T00:14:07.477+0000] {standard_task_runner.py:104} INFO - Running:
['airflow', 'tasks', 'run', 'zdp_dbt_regional_support_fdn_models_amer_staging',
'regional_product_support.base_rules_run',
'scheduled__2025-06-15T00:00:00+00:00', '--job-id', '90029', '--raw',
'--subdir',
'DAGS_FOLDER/external/zdp_dbt_regional_support/dbt_regional_support_fdn_models.py',
'--cfg-path', '/tmp/tmp7igxne9x']
[2025-06-16T00:14:07.478+0000] {standard_task_runner.py:105} INFO - Job
90029: Subtask regional_product_support.base_rules_run
[2025-06-16T00:14:07.517+0000] {task_command.py:467} INFO - Running
<TaskInstance:
zdp_dbt_regional_support_fdn_models_amer_staging.regional_product_support.base_rules_run
scheduled__2025-06-15T00:00:00+00:00 [running]> on host 172.20.6.76
[2025-06-16T00:14:07.597+0000] {taskinstance.py:3134} INFO - Exporting env
vars: AIRFLOW_CTX_DAG_OWNER='Foundation-Data'
AIRFLOW_CTX_DAG_ID='zdp_dbt_regional_support_fdn_models_amer_staging'
AIRFLOW_CTX_TASK_ID='regional_product_support.base_rules_run'
AIRFLOW_CTX_EXECUTION_DATE='2025-06-15T00:00:00+00:00'
AIRFLOW_CTX_TRY_NUMBER='3'
AIRFLOW_CTX_DAG_RUN_ID='scheduled__2025-06-15T00:00:00+00:00'
[2025-06-16T00:14:07.605+0000] {warnings.py:112} WARNING -
/usr/local/lib/python3.12/site-packages/airflow/providers/openlineage/plugins/listener.py:374:
DeprecationWarning: This process (pid=172) is multi-threaded, use of fork()
may lead to deadlocks in the child.
pid = os.fork()
[2025-06-16T00:14:07.714+0000] {client.py:110} INFO - OpenLineageClient will
use `http` transport
[2025-06-16T00:14:08.019+0000] {taskinstance.py:732} INFO - ::endgroup::
[2025-06-16T00:14:08.022+0000] {baseoperator.py:423} WARNING -
DbtRunKubernetesOperator.execute cannot be called outside TaskInstance!
[2025-06-16T00:14:08.027+0000] {kubernetes.py:73} INFO - Running command: ...
[2025-06-16T00:14:08.027+0000] {baseoperator.py:423} WARNING -
DbtRunKubernetesOperator.execute cannot be called outside TaskInstance!
[2025-06-16T00:14:08.038+0000] {pod.py:1134} INFO - Building pod
regional-product-support-base-rules-run-dbbsoj9h with labels: ...
[2025-06-16T00:14:08.373+0000] {pod_manager.py:334} ERROR - Exception when
attempting to create Namespaced Pod: ...
Traceback (most recent call last):
File
"/usr/local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 329, in run_pod_async
resp = self._client.create_namespaced_pod(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py",
line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py",
line 7455, in create_namespaced_pod_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py", line
348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py", line
180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py", line
391, in request
return self.rest_client.POST(url,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kubernetes/client/rest.py",
line 279, in POST
return self.request("POST", url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kubernetes/client/rest.py",
line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
\"regional-product-support-base-rules-run-dbbsoj9h\" is forbidden: exceeded
quota: ancient-meteoroid-9548-default, requested: cpu=501m,memory=1034Mi, used:
cpu=49599m,memory=102366Mi, limited:
cpu=50,memory=100Gi","reason":"Forbidden","details":{"name":"regional-product-support-base-rules-run-dbbsoj9h","kind":"pods"},"code":403}
[2025-06-16T00:14:08.378+0000] {taskinstance.py:3313} ERROR - Task failed
with exception
Traceback (most recent call last):
File
"/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line
768, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line
734, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py",
line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line
424, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/airflow/packages/custom/src/zdp/callbacks/multi_callback.py", line
32, in execute
super().execute(context)
File
"/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line
424, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/cosmos/operators/base.py",
line 268, in execute
self.build_and_run_cmd(context=context, cmd_flags=self.add_cmd_flags())
File
"/usr/local/lib/python3.12/site-packages/cosmos/operators/kubernetes.py", line
74, in build_and_run_cmd
result = KubernetesPodOperator.execute(self, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line
424, in wrapper
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 594, in execute
return self.execute_sync(context)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 604, in execute_sync
self.pod = self.get_or_create_pod( # must set `self.pod` for `on_kill`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py",
line 566, in get_or_create_pod
self.pod_manager.create_pod(pod=pod_request_obj)
File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line
336, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line
475, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line
376, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line
398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in
result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in
__get_result
raise self._exception
File "/usr/local/lib/python3.12/site-packages/tenacity/__init__.py", line
478, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 359, in create_pod
return self.run_pod_async(pod)
^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 337, in run_pod_async
raise e
File
"/usr/local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py",
line 329, in run_pod_async
resp = self._client.create_namespaced_pod(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py",
line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs) # noqa: E501
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api/core_v1_api.py",
line 7455, in create_namespaced_pod_with_http_info
return self.api_client.call_api(
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py", line
348, in call_api
return self.__call_api(resource_path, method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py", line
180, in __call_api
response_data = self.request(
^^^^^^^^^^^^^
File
"/usr/local/lib/python3.12/site-packages/kubernetes/client/api_client.py", line
391, in request
return self.rest_client.POST(url,
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kubernetes/client/rest.py",
line 279, in POST
return self.request("POST", url,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kubernetes/client/rest.py",
line 238, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
