kaxil opened a new pull request, #76882:
URL: https://github.com/apache/airflow/pull/76882
`AgentOperator(durable=True)` and `@task.agent(durable=True)` could not use
tool approval. A tool marked `requires_approval=True` or wrapped with
`.approval_required()` failed the task:
```
UnsupportedToolDeferralError: The agent called tools that need approval
(issue_refund), but tool approval needs Airflow 3.3+ and is not available with
durable, enable_hitl_review, a CodeMode capability or a SandboxToolset.
```
Removing that check would not make it work. The non-durable resume runs a
second agent run from a saved transcript, outside the task's journal, so a
failure after the resume would run the approved tool again on retry. Nothing
recorded the answer either: the retry would replay up to the same pause and ask
the reviewer again. On Airflow 3.3 that second request fails the task with
`ToolApprovalAlreadyRequestedError`. On 3.4, a reviewer who rejected this time
would have the agent told "denied" about a refund that had already gone out.
With this PR, on Airflow 3.3 and later, a durable agent task pauses for
approval like a non-durable one, and the reviewer's answer becomes a step of
the journal. When the reviewer answers, the task runs the agent again from the
prompt. Every model request and tool call before the pause replays from the
journal, and the approval step gets the answer. A retry after that replays the
decision too: the reviewer is not asked again, and an approved call that
already ran is not run again.
## Design rationale
**Approval step.** The approval is a pydantic-ai capability whose decision
is a `@durable_operation`; `handle_deferred_tool_calls` is not one of the hooks
pydantic-ai forbids as durable operations. While the reviewer has not answered,
the step writes the approval request and parks the task (`TaskAwaitingInput`).
A parked step is not recorded, so the resumed run reaches it again. Once it
returns `DeferredToolResults`, the journal records it. The step is
fingerprinted by the calls it settles, through
`fingerprint_deferred_tool_requests`, so a recorded answer replays only for
those calls. This replaces the saved transcript, its hash check and the
`{ti.id}-resumed-{n}` run id on the durable path. The non-durable path keeps
them, because without a journal, running again from the prompt would call the
model again for every step before the pause.
**Stored answers.** Answers live in the task state store, bound to the
approval step that asked. Each answer is stored with:
- the approval step's number in the run, counted the same way on replay and
on a live run;
- the tool name and an argument fingerprint of each call the reviewer saw;
- the agent's rendered toolset ids.
They are deleted with the journal when the task succeeds. Keeping them only
in the resume kwargs broke in these cases:
- A handler that changes its mind between resumes. Each resume's answer
stays available, so the run does not re-ask in a loop.
- A handler that fails after an answer was applied. The retry still has the
answer.
- A model request asked again on resume, after a tool opted out of the
journal returned something new. A real provider names the same call with a new
id, so matching on ids would never apply the answer.
Matched by content alone, an earlier answer reached a later call with the
same tool and arguments, and a rejected refund ran. Answers given for other
toolset ids are not applied, so a retry never runs an approved call against a
connection the reviewer did not see.
**Other handlers.** The approval capability declares `outermost` ordering,
and its one durable step does three things in order:
1. Applies the stored answers.
2. Lets the run's other `handle_deferred_tool_calls` handlers (a
decision-model judge, say) settle what is left. It reaches them through
`ctx.root_capability`, so pydantic-ai's own chain decides which capabilities
are active and in what order.
3. Asks the reviewer about anything still left.
Left where pydantic-ai's list order puts it, ahead of the approval step, a
judge that changed its verdict between pause and resume overruled a Reject;
placed after it, the judge would only see calls a person had already been asked
about. A retry replays the whole decision, so the judge is not consulted again.
**Journal.** On a pause, `DurableRun.pause()` trusts every step the paused
attempt recorded and deletes what an earlier attempt left at positions this one
did not record. `fail()` would hand the resume back to an earlier attempt's
failure point, so the model would be asked again on every resume and the
reviewer with it.
**Nested approvals.** An approval reached inside another tool's work fails
the task, such as one in an agent that a tool runs (`ctx.agent.run`) or in a
call a tool makes inline. Once that tool is recorded, a replay skips its
insides, so every later approval step would be numbered one lower and get
another step's answer. The refusal uses the journal's executing-step context
and raises `UnsupportedToolDeferralError`, as approval under CodeMode already
does.
The `@task.agent` resume goes through `execute`, so the prompt function runs
again on each resume, as it does on a retry. pydantic-ai-slim 2.46.0, the
current floor, has everything this needs; the durable approval, agent and
journal tests pass at 2.46.0 and at the locked 2.63.0.
## End to end
`AgentOperator` and `@task.agent` with `durable=True` on a live stack
(Airflow `main`, 3.5.0.dev), with a real model (`gpt-5.4-mini`), from the PR's
commit. Each tool logs `TOOL RAN LIVE` when its code runs.
In `refund_approved` the reviewer approved, `issue_refund` ran once, and
`notify_customer` failed on purpose; try 2 replayed 6 steps (3 model, 2 tool, 1
approval) with no new request, ran only `notify_customer`, and succeeded.
After a reject with a reason, the refund never ran and the model replied
"The refund was not issued because refunds over $15 require a support ticket
number." `two_refunds` paused twice in one try; the second request was answered
on the Required Actions page, and each refund ran once:
Through `@task.agent`, the prompt function ran again on the resume and the
refund ran once.
The issue also asks for these cases on Airflow 3.3 and 3.4 releases. Those
legs were not run against released Airflow. The unit tests cover both version
branches (one request per task instance on 3.3, repeat requests on 3.4) by
patching the version check.
## Known issues
- A tool that runs the agent in a bare thread (`run_in_executor` with
`asyncio.run`) does not carry the journal's context, so an approval there is
not refused and is numbered as a top-level step.
- A tool opted out of the journal that returns something new on resume makes
the model requests and calls after it run again, approved calls included. The
answer still applies when the same call comes back at the same approval step.
- Under durable, `ToolLoggingCapability` logs "waiting to be approved" each
time a gated call replays, although nothing waits then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
