kaxil commented on code in PR #76801:
URL: https://github.com/apache/airflow/pull/76801#discussion_r4606327065
##########
providers/common/ai/docs/recovery.rst:
##########
@@ -131,11 +131,26 @@ then sends its first live request through the chain from
the top. The run contin
the first connection that answers, whether or not that is the one that
answered before.
A recorded model step replays only if the retry's request has the same
fingerprint, and
-for a chain the model in the fingerprint is the whole chain, whichever
connection
-answered. Adding, removing or reordering a connection in
``fallback_conn_ids``, or
-changing the ``model`` of a connection in it, between attempts therefore makes
the retry
-run every step live, tools included. Changing a connection's credentials or
host does
-not. "How a retry replays" in :doc:`durable_execution` gives the full matching
rules.
+for a chain the fingerprint names the chain as a whole, not its connections.
Each
+recorded response also notes which model of the chain answered it, and the
step replays
+only if that model is still in the chain. Adding, removing or reordering other
Review Comment:
Not quite for the last fallback. Drop `llm_standby` from the example above
and the hook hands back the bare model, so the identity goes from `fallback` to
`system:model_name` and every tool runs again. Same going from one connection
to two. Worth saying here that the chain needs to keep at least one fallback?
##########
providers/common/ai/src/airflow/providers/common/ai/durable/capability.py:
##########
@@ -686,9 +693,12 @@ async def _fingerprint_model_request(
# here as well as in the request itself is safe.
model = await self._durability._model_for_fingerprint(model_id,
run_context)
prepared_settings, prepared_parameters =
model.prepare_request(model_settings, parameters)
- return fingerprint_model_request(
- f"{model.system}:{model.model_name}", messages, prepared_settings,
prepared_parameters
- )
+ # A chain whose members all stamp their answers is fingerprinted by a
fixed name, not its
+ # members: editing the chain must not invalidate steps a surviving
member answered.
+ # ``_find_unusable_answerer`` rejects the replay of a step whose
answering member left
+ # the chain. Any other chain cannot say who answered, so it keeps the
member-listing name.
+ identity = "fallback" if _is_stamped_chain(model) else
f"{model.system}:{model.model_name}"
Review Comment:
I think this still breaks from the second model step on. On pydantic-ai
2.64+ the agent graph sets `model_id` on each response after the step, and for
a chain that's `fallback:a,b`. A replayed step 1 comes back with the edited
chain's id, so request 2 hashes differently. Probed it with two tools on 2.66:
`a,b` -> `a,c` replays one model step and `send_email` runs twice. Leaving
`model_id` out of the hashed history fixes it. Can the stamped-chain path drop
it? The tests in `TestFallbackChainEdits` all stop after one model step, so a
two-tool case would pin this.
##########
providers/common/ai/src/airflow/providers/common/ai/durable/member_stamp.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Record which member of a ``FallbackModel`` chain answered a model
request."""
+
+from __future__ import annotations
+
+from collections.abc import AsyncIterator
+from contextlib import asynccontextmanager
+from typing import TYPE_CHECKING, Any
+
+from pydantic_ai.models.wrapper import WrapperModel
+
+if TYPE_CHECKING:
+ from pydantic_ai.messages import ModelMessage, ModelResponse
+ from pydantic_ai.models import ModelRequestParameters, StreamedResponse
+ from pydantic_ai.settings import ModelSettings
+ from pydantic_ai.tools import RunContext
+
+MEMBER_STAMP_KEY = "__airflow_common_ai__"
+MEMBER_MODEL_ID_KEY = "member_model_id"
+
+
+def get_member_model_id(response: ModelResponse) -> str | None:
+ """Return the ``model_id`` of the chain member that produced ``response``,
if it was stamped."""
+ stamp = (response.metadata or {}).get(MEMBER_STAMP_KEY)
+ if not isinstance(stamp, dict):
+ return None
+ member_model_id = stamp.get(MEMBER_MODEL_ID_KEY)
+ return member_model_id if isinstance(member_model_id, str) else None
+
+
+class MemberStampingModel(WrapperModel):
+ """
+ Stamp each response with the ``model_id`` of the wrapped model.
+
+ pydantic-ai's ``FallbackModel`` only records which member answered on
suspended responses,
+ so a durable replay cannot otherwise tell whether the recorded answer came
from a model
+ that is still in the chain. The stamp lives under its own key rather than
the
+ ``__pydantic_ai__`` namespace that pydantic-ai owns.
+ """
+
+ def _stamp(self, response: ModelResponse | StreamedResponse) -> None:
+ if response.metadata is None:
+ response.metadata = {}
+ response.metadata[MEMBER_STAMP_KEY] = {MEMBER_MODEL_ID_KEY:
self.wrapped.model_id}
Review Comment:
The hook knows each member's conn_id when it builds the chain. Would
stamping that next to `model_id` let replay tell two connections that serve the
same model apart? Then answers from a removed gateway connection serving
`openai:gpt-4o` would stop replaying, and the docs wouldn't need the caveat.
##########
providers/common/ai/docs/recovery.rst:
##########
@@ -131,11 +131,26 @@ then sends its first live request through the chain from
the top. The run contin
the first connection that answers, whether or not that is the one that
answered before.
A recorded model step replays only if the retry's request has the same
fingerprint, and
-for a chain the model in the fingerprint is the whole chain, whichever
connection
-answered. Adding, removing or reordering a connection in
``fallback_conn_ids``, or
-changing the ``model`` of a connection in it, between attempts therefore makes
the retry
-run every step live, tools included. Changing a connection's credentials or
host does
-not. "How a retry replays" in :doc:`durable_execution` gives the full matching
rules.
+for a chain the fingerprint names the chain as a whole, not its connections.
Each
+recorded response also notes which model of the chain answered it, and the
step replays
+only if that model is still in the chain. Adding, removing or reordering other
+connections in ``fallback_conn_ids`` between attempts therefore keeps the
steps that a
+remaining model answered. A step answered by a model that was removed from the
chain, or
+whose ``model`` changed, runs live along with every step after it, tools
included, and
+the task log names the reason. The check compares models, not connections: two
chain
+members that resolve to the same model cannot be told apart.
+A ``FallbackModel`` built outside the hook, whose members do not record which
one
+answered, keeps the earlier rule: its steps replay while the chain is
unchanged and run
+live once it is edited.
+
+Changing a connection's credentials or host does not invalidate a recorded
step. Nor
+does changing settings on a chain connection, such as ``temperature``: a chain
never
Review Comment:
Which setting is this? The hook builds members with `infer_model(...)`, and
no connection extra carries `temperature`. A `temperature` in the task's
`model_settings` is part of the fingerprint and does re-run the step, so a
reader could take this the wrong way. I'd drop the sentence here and in the
changelog.
##########
providers/common/ai/tests/unit/common/ai/durable/test_capability.py:
##########
@@ -1928,3 +1938,230 @@ async def
test_a_retry_whose_budget_is_spent_can_still_replay(self, memory_stora
assert result.output == "done"
assert usage.requests == 3
+
+
+class TestFallbackChainEdits:
+ """Editing a fallback chain between attempts keeps the steps a surviving
member answered."""
+
+ @staticmethod
+ def build_chain(
+ calls: Calls,
+ members: list[tuple[str, float | None]],
+ *,
+ up: set[str],
+ stamped: bool = True,
+ ) -> FallbackModel:
+ def build_member(name: str, temperature: float | None) -> Model:
+ def respond(messages: list[ModelMessage], info: AgentInfo) ->
ModelResponse:
+ calls.bump(name)
+ if name not in up:
+ raise ModelHTTPError(500, name)
+ return ModelResponse(parts=[TextPart(f"{name} answered")])
+
+ model = FunctionModel(
+ respond,
+ model_name=name,
+ settings=ModelSettings(temperature=temperature) if temperature
is not None else None,
+ )
+ return MemberStampingModel(model) if stamped else model
+
+ return FallbackModel(
+ *(build_member(name, temperature) for name, temperature in
members),
+ fallback_on=(ModelAPIError,),
+ )
+
+ @staticmethod
+ def build_agent(chain: FallbackModel) -> Agent[None, str]:
+ return Agent(chain, name="chained", capabilities=[AirflowDurability()])
+
+ @pytest.mark.parametrize(
+ ("members", "up", "output", "replayed", "calls_by_member"),
+ [
+ pytest.param(
+ [("c", None), ("b", None)], {"c"}, "b answered", 1, {"b": 1,
"c": 0}, id="member added before"
+ ),
+ pytest.param(
+ [("b", None), ("a", None)], {"a"}, "b answered", 1, {"b": 1,
"a": 1}, id="members reordered"
+ ),
+ pytest.param(
+ [("a", None), ("b", 0.9)], set(), "b answered", 1, {"b": 1},
id="member settings changed"
+ ),
+ pytest.param(
+ [("a", None)], {"a"}, "a answered", 0, {"a": 2, "b": 1},
id="answering member removed"
+ ),
+ pytest.param(
+ [("a", None), ("b2", None)],
+ {"b2"},
+ "b2 answered",
+ 0,
+ {"a": 2, "b": 1, "b2": 1},
+ id="answering member's model changed",
+ ),
+ ],
+ )
+ @pytest.mark.asyncio
+ async def
test_a_step_replays_only_while_its_answering_member_is_in_the_chain(
+ self, memory_storage, members, up, output, replayed, calls_by_member
+ ):
+ calls = Calls()
+ first = self.build_chain(calls, [("a", None), ("b", None)], up={"b"})
+ await attempt(memory_storage, self.build_agent(first))
+
+ journal = DurableJournal(memory_storage)
+ with (
+ journal_scope(journal),
+ mock.patch.object(DurableRun, "reject", autospec=True,
side_effect=DurableRun.reject) as reject,
+ ):
+ result = await self.build_agent(self.build_chain(calls, members,
up=up)).run("go")
+
+ assert result.output == output
+ assert journal.stats.replayed["model"] == replayed
+ assert {name: calls[name] for name in calls_by_member} ==
calls_by_member
+ reason = "the model that answered this step, function:b, is no longer
in the fallback chain"
+ assert [call.kwargs["reason"] for call in reject.call_args_list] == (
+ [reason] if replayed == 0 else []
+ )
+
+ @pytest.mark.parametrize(
+ ("second_attempt", "replayed", "calls_by_member"),
+ [
+ pytest.param([("a", None), ("b", None)], 1, {"a": 1, "b": 1},
id="chain unchanged"),
+ pytest.param([("b", None)], 0, {"a": 1, "b": 2}, id="chain
edited"),
Review Comment:
With `stamped_members=1`, the only member of this chain gets wrapped, so the
edited chain is fully stamped, not mixed. The "mixed chain" + "chain edited"
case passes on the `fallback` vs member-list identity split and never builds an
edited mixed chain. Two members here (`[("b", None), ("c", None)]`) would keep
it mixed.
##########
providers/common/ai/tests/unit/common/ai/hooks/test_pydantic_ai.py:
##########
@@ -867,12 +877,21 @@ def test_param_builds_chain_in_order(self, registry,
infer_model_stub):
model = hook.get_conn()
assert isinstance(model, FallbackModel)
- assert model.models == [
+ assert get_chain_members(model) == [
infer_model_stub.models["openai:gpt-5.6-sol"],
infer_model_stub.models["anthropic:claude-opus-4-6"],
infer_model_stub.models["groq:llama-4"],
]
+ def test_every_chain_member_is_stamped_with_its_model_id(self, registry,
infer_model_stub):
+ registry.add("primary", extra={"model": "openai:gpt-5.6-sol"})
+ registry.add("second", extra={"model": "anthropic:claude-opus-4-6"})
+
+ model = PydanticAIHook(llm_conn_id="primary",
fallback_conn_ids=["second"]).get_conn()
+
+ assert isinstance(model, FallbackModel)
+ assert all(isinstance(member, MemberStampingModel) for member in
model.models)
Review Comment:
`get_chain_members` already asserts this isinstance in every chain test, and
nothing here reads a stamp. Could this send a request through the hook-built
chain and check `get_member_model_id` on the response? If not, it can go.
##########
providers/common/ai/src/airflow/providers/common/ai/durable/capability.py:
##########
@@ -207,6 +209,7 @@ def _lacks_id(leaf: AbstractToolset[Any]) ->
TypeGuard[FunctionToolset[Any] | Dy
# Why a recorded step that matched is run live anyway: it was written by a
version whose
# types have changed since, so replaying it would fail on every retry.
_UNLOADABLE = "the recorded result no longer loads, so it was written by
another version"
+_UNSTAMPED_ANSWERER = "the recorded response does not say which fallback chain
member answered it"
Review Comment:
The comment above only describes `_UNLOADABLE` ("written by a version whose
types have changed"), but now it reads as if it covers this one too. Could this
get its own comment line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]