jnstockley opened a new issue, #53713:
URL: https://github.com/apache/airflow/issues/53713
### Apache Airflow version
3.0.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Longer running tasks will occasionally fail because of an invalid auth token
that has expired.
```
2025-07-23 01:27:58,629 {_client.py:1025} INFO - HTTP Request: PATCH
http://127.0.0.1:8085/airflow/execution/task-instances/019835de-0967-7bc4-b053-e6663b1a45ec/state
"HTTP/1.1 204 No Content"
2025-07-23 01:27:58 [info ] Task finished [supervisor]
duration=1076.4064389939886 exit_code=0 final_state=success
2025-07-23 01:27:58 [info ] Secrets backends loaded for worker
[supervisor] backend_classes=['SecretsManagerBackend',
'EnvironmentVariablesBackend'] count=2
2025-07-23 01:27:58,919 {_client.py:1025} INFO - HTTP Request: PATCH
http://127.0.0.1:8085/airflow/execution/task-instances/019835de-0955-74d4-8db6-2aea4332ab1b/run
"HTTP/1.1 403 Forbidden"
2025-07-23 01:27:58 [warning ] Server error
[airflow.sdk.api.client] detail=None
2025-07-23 01:27:58 [info ] Process exited [supervisor]
exit_code=<Negsignal.SIGKILL: -9> pid=1258981 signal_sent=SIGKILL
2025-07-23 01:27:58,929 {local_executor.py:96} ERROR - uhoh
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py",
line 92, in _run_worker
_execute_work(log, workload)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/local_executor.py",
line 120, in _execute_work
supervise(
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 1683, in supervise
process = ActivitySubprocess.start(
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 851, in start
proc._on_child_started(ti=what, dag_rel_path=dag_rel_path,
bundle_info=bundle_info)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 862, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid,
start_date)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 149, in start
resp = self.client.patch(f"task-instances/{id}/run",
content=body.model_dump_json())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 1218, in patch
return self.request(
^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
336, in wrapped_f
return copy(f, *args, **kw)
^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
475, in __call__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
376, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in
result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
478, in __call__
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 707, in request
return super().request(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 825, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 914, in send
response = self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 942, in _send_handling_auth
response = self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 999, in _send_handling_redirects
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py",
line 982, in _send_handling_redirects
hook(response)
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 120, in raise_on_4xx_5xx_with_note
return get_json_error(response) or response.raise_for_status()
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py",
line 110, in get_json_error
raise err
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature
has expired
Correlation-id=019835f7-a470-78d0-b268-b388c323926f
```
### What you think should happen instead?
The task instance should get a new token and continue
### How to reproduce
Create with multiple tasks, and different connections in the tasks where the
first task runs for 30+ minutes. Store connections in AWS Secrets Manager.
### Operating System
Debain
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.10.0
apache-airflow-providers-common-compat==1.7.2
apache-airflow-providers-common-io==1.6.1
apache-airflow-providers-common-sql==1.27.3
apache-airflow-providers-fab==2.3.0
apache-airflow-providers-google==16.1.0
apache-airflow-providers-http==5.3.2
apache-airflow-providers-microsoft-mssql==4.3.1
apache-airflow-providers-openai==1.6.1
apache-airflow-providers-oracle==4.1.2
apache-airflow-providers-postgres==6.2.1
apache-airflow-providers-sftp==5.3.2
apache-airflow-providers-smtp==2.1.1
apache-airflow-providers-ssh==4.1.1
apache-airflow-providers-standard==1.4.1
### Deployment
Other Docker-based deployment
### Deployment details
When running in production, we use AWS ECS tasks, I have only see the issue
happen here, but also haven't run the longer running tasks locally.
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
