GitHub user weiqingy added a comment to the discussion: [Discussion] Pluggable
model routing — API & design (PR #852 follow-up)
+1 to @wenjin272's questions, especially around the runtime/event contract and
whether caching is still needed in the framework-owned design.
Thanks for writing this up. I like the direction of treating routing as a
framework capability rather than a specialized `ChatModelSetup`. That seems
like a promising way to address the observability and metrics concerns raised
on #852, while keeping the agent-side usage simple.
A few additional questions came to mind while reading:
1. **Could we make the event/control-flow contract more explicit?**
If a user sends `ChatRequestEvent("router", ...)`, would `ChatModelAction`
resolve the selected model and call it directly, or would the framework
create/record a selected-model chat request as a separate step? If it is a
direct call, I wonder whether the selected backend invocation would still be
visible as an ordinary model invocation in the EventLog. If it is represented
as a second/internal chat request, it may be helpful to define request IDs,
parent/original request IDs, and whether user-defined actions listening to chat
requests can observe it.
2. **What role should `ModelRoutingEvent` play?**
Should it be purely an observability event for logging/tracing/evaluation,
or is it expected to drive another framework step that dispatches the request
to the selected model? I think either shape can work, but making this explicit
would help clarify the runtime call path and avoid ambiguity for users who
consume events.
3. **Would a structured `RoutingDecision` be more useful than returning only a
`String`?**
A selected model name may be enough for execution, but the routing event may
want to record why the model was selected, whether the strategy abstained,
whether default/fallback was used, score/confidence, strategy metadata, or an
LLM judge response. Maybe the common custom case can still stay lightweight
with helpers, but the SPI could return a structured decision so this
information is not lost.
4. **Is router-level caching still needed once routing is owned by the
framework?**
In #852, caching helped because the router was itself a chat model and could
be invoked again on each tool-call round. If routing moves into
`ChatModelAction`, could it route once for the initial request, associate the
selected concrete model with `initialRequestId`, and reuse that model for
subsequent tool-call rounds? That might avoid a separate LRU keyed by the first
user message, which could be tricky when different conversations share the same
first message but differ in prompt args, tools, tenant, or metadata.
5. **How should LLM-as-router be represented?**
A rule-based strategy feels like a pure selection function, while an LLM
judge is itself a model invocation with latency, retries, metrics, durability,
and EventLog implications. In the current shape, a resource/strategy does not
have `RunnerContext.sendEvent(...)`, and a direct `judge.chat(...)` call would
bypass the current action-level EventLog and token-recording path. Would it
make sense for the judge call to go through a framework-managed observable path
from the beginning, or should the first version focus on the basic routing
contract and leave fully observable LLM routing as a follow-up?
6. **How should routing decisions behave during recovery/replay?**
Routing makes the selected backend a runtime decision. If a job fails after
the original decision but before the run finishes, should recovery replay the
original selected model, or is it acceptable to re-run the strategy and
possibly choose a different backend? This seems especially important because
the current chat durable call identity is generic to the chat operation rather
than to the selected model/decision. My intuition is that the decision should
become replayable data, not re-derived data, so a recovered run reaches the
same backend or can detect that it would not. Would `RoutingDecision`,
`ModelRoutingEvent`, or the existing action/durable state be the right place to
persist that selected model and related metadata?
7. **Could fallback semantics be clarified relative to existing retry/error
handling?**
`ChatModelAction` already has retry/ignore/fail behavior. It would be useful
to know whether fallback happens before retries, after retries are exhausted,
or only for certain exception categories. For observability, it may also help
if the routing/response data records both the initially selected model and the
model that ultimately produced the response.
8. **What is the deployable shape for custom strategies?**
The lambda example is very ergonomic for local Java code, but I wonder how
that function reaches TaskManagers once the agent plan is serialized. Existing
function/resource paths tend to move named classes or descriptors rather than
closures. Is the lambda form intended mainly as an embedded/local convenience,
with a named `RoutingStrategy` class as the deployable shape? Or should custom
routing follow a named-method hook similar to actions/tools?
9. **What `ResourceType` should a framework router use?**
If `ChatModelAction` needs to detect the router, should the router still be
registered as `CHAT_MODEL`, or should there be a distinct router resource type?
Keeping it as `CHAT_MODEL` preserves the agent-side experience but may keep
some of the type-level conflation that the framework design is trying to
remove, and it also raises questions like whether routers can point to other
routers. A distinct type would make the contract clearer, but it would need the
usual Java/Python/YAML parity work.
10. **Should abstain and invalid candidate be treated differently?**
Using `defaultModel` when a strategy returns `null` seems intuitive. If a
strategy returns a non-candidate model name, though, could that indicate a typo
or buggy strategy? I wonder whether failing clearly, or at least recording it
as an invalid decision, would be safer than treating it the same as abstain.
11. **Could we record the Python/YAML scope explicitly?**
On #852, the Python story was intentionally narrow: Python users could
reference the Java router/strategies by FQCN, while a native Python router/SPI
was deferred. Does that remain the intended v1 scope for the framework-level
design? If so, it may be worth recording as a deliberate non-goal, together
with the trigger for when native Python routing should be added.
Overall, I'm supportive of the framework-level router direction and the single
`RoutingStrategy` extension point with built-in factories. The main thing I'd
love to understand better is the runtime/event contract: when routing runs, how
the selected model is stored for tool-call rounds and recovery, what exactly
`ModelRoutingEvent` represents, and how selected backend calls remain
observable with clear IDs and metrics attribution.
GitHub link:
https://github.com/apache/flink-agents/discussions/897#discussioncomment-17695346
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
