seanghaeli opened a new pull request, #76571:
URL: https://github.com/apache/airflow/pull/76571
Since #71192, a worker swaps a deadline callback's single-use token for an
execution token before running it (`PATCH /execution/callbacks/{id}/run`). If
that request took longer than the 5 s client timeout, the client retried, the
retry got a 409 because the token was already used, and the callback was marked
failed without running. The same happened if the API was unreachable for more
than about 15 s, for example during a restart.
This changes how that one call retries: it waits up to 60 s for an answer
instead of retrying, and only retries when the request never reached the server
(connection errors), for up to 2 minutes. The token stays single-use.
It still fails closed if the server takes over 60 s, or if the connection
drops after the request was sent, since the server may already have used the
token.
```text
# Breeze, Postgres, LocalExecutor, a SyncCallback that reads a Variable. A:
the first exchange response is held 6 s
# after the server handled it. B: API server stopped from before the run
until 30 s after it. C: connection dropped
# after the server handled the exchange.
# before (main)
[main3 21:34:45] A1 slow exchange (6s): callback failed output='Execution
failed'; user code ran: 0 (cumulative)
[main3 21:35:43] A2 slow exchange (6s): callback failed output='Execution
failed'; user code ran: 0 (cumulative)
[main3 21:37:40] B1 API down 30s at callback start: callback failed
output='Execution failed'; user code ran: 0 (cumulative)
[main3 21:39:37] B2 API down 30s at callback start: callback failed
output='Execution failed'; user code ran: 0 (cumulative)
[main3 21:40:49] C1 response lost after the exchange: callback failed
output='Execution failed'; user code ran: 0 (cumulative)
Server error detail: {'reason': 'invalid_state', 'message': 'Callback
01a1ada1-f894-7ba4-941d-fd266a3efc60 is in state running; its token can only be
exchanged once while QUEUED.', 'previous_state': 'running'}
# after
[fix3 21:25:04] A1 slow exchange (6s): callback success output=''; user code
ran: 1 (cumulative)
[fix3 21:26:02] A2 slow exchange (6s): callback success output=''; user code
ran: 2 (cumulative)
[fix3 21:27:59] B1 API down 30s at callback start: callback success
output=''; user code ran: 3 (cumulative)
[fix3 21:29:56] B2 API down 30s at callback start: callback success
output=''; user code ran: 4 (cumulative)
[fix3 21:31:09] C1 response lost after the exchange: callback failed
output='Execution failed'; user code ran: 4 (cumulative)
httpx2.RemoteProtocolError: peer closed connection without sending complete
message body (incomplete chunked read)
```
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
