seanghaeli opened a new issue, #76735:
URL: https://github.com/apache/airflow/issues/76735
Since #71192, a worker swaps a deadline callback's single-use token for an
execution token with `PATCH /execution/callbacks/{id}/run` before it runs the
callback. The server uses up the token on the first request it handles, so if
the response is lost (connection dropped, a proxy 502 or 504 after the server
handled it, or a server slower than the client's read timeout), the client's
retry gets a 409 and the callback fails without running. #76571 makes this
rarer for 3.4.1 by giving that call a longer timeout and retry budget, but a
retry after a lost response still fails.
Proposal, from the review of #76571: the worker generates a random nonce
once per run of the callback and sends it with every attempt of the exchange.
On the first exchange the server stores a hash of it on the callback with the
QUEUED to RUNNING change. A later exchange while RUNNING gets a new execution
token only if its nonce matches; anything else still gets a 409. A copy of the
callback token from the broker message, or a broker redelivery, doesn't have
the nonce, so the token stays single-use for everyone except the worker that
already used it.
Notes:
- Send the nonce in the request body, not a header, so it doesn't end up in
proxy logs.
- This needs a new Execution API version. Older workers don't send a nonce
and keep today's behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
