kaxil opened a new pull request, #73893:
URL: https://github.com/apache/airflow/pull/73893
Stacked on #73873, which this branch includes; only the last commit is new
here.
With `durable=True`, a retry replays every cached model response and tool
call, and replayed steps count toward `usage_limits` like live ones. A run that
reached its limit therefore usually stops at the same step on every retry.
Until now the task log only showed the same `UsageLimitExceeded` on each try,
with nothing saying why the retries were not helping.
`AgentOperator` now logs a warning when a durable run reaches a usage limit.
It includes the error, says a retry will likely stop at the same step if that
is the task's own limit, and says that raising `usage_limits` is what lets the
run go further. The durable execution guide says the same, including that an
unset `usage_limits` still caps a run at pydantic-ai's default of 50 requests.
Retry behaviour does not change.
**Why not stop retrying instead.** Failing the task once a retry reaches the
limit purely from the cache looks like the obvious fix, but that only holds
when everything in the run replays exactly. Several things run live on every
attempt and can change the outcome: output functions and validators, tools
asking for a retry, tool results the cache could not store, tools on routes the
cache does not wrap, and agents run inside a tool, which can add their own
usage or hit their own limits. Each of these can make an attempt look fully
replayed and still leave a later attempt able to finish, so an operator-side
fail-fast would sometimes drop a retry that would have succeeded. A redundant
retry that only replays costs no model tokens, so the warning is the trade
worth making.
Run end to end on a local Airflow 3.4.0 with a durable agent,
`usage_limits={"request_limit": 1}` and `retries=1`. On try 1 the run makes its
model call and tool call live, reaches the limit, and logs the warning:
On try 2 both steps replay from the cache (`replayed 2 cached steps ...
cached 0 new steps`) and the run stops at the same limit, as the warning says:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
