purushah opened a new pull request, #1363:
URL: https://github.com/apache/flink-agents/pull/1363
## What is the purpose of the change
Implements the quality-triggered escalation agreed in #1328: a
`MODEL_ROUTER` can declare cheapest-first **tiers** and **quality signals**.
After each model round the router checks the answer; when a signal rejects it,
the rejected answer is discarded before any of its tool calls runs and the same
conversation is sent to the next tier, where the request then stays. Escalation
sits beside fallback and does not change existing routers: a router without
tiers behaves exactly as before.
## Brief change log
- `api`: `Escalation` (tiers, signals, `maxEscalations`, descriptor round
trip and diagnostics), `QualitySignal` / `QualityContext` / `QualityVerdict`,
`QualitySignals.decision(...)` and `QualitySignals.custom(...)`,
`ModelRouter.Builder.escalation(...)` (strategy becomes optional when tiers are
set; the request starts on the first tier or `defaultModel`),
`ModelRoutingEvent.SOURCE_ESCALATION`.
- `plan`: `EscalationGate` runs the built-in structural checks (empty
answer, call to a tool the model does not have, arguments failing the tool's
JSON schema, identical tool call repeated in consecutive rounds) and then the
declared signals in order. A decision-model signal asks a predicate question
through `DecisionModelInvoker.decideWithRetries` under the durable id
`escalate:<router>:<candidate>`; a custom signal's verdict is persisted under
`escalate:<router>:<candidate>:<n>`, so recovery reuses verdicts.
`ChatModelAction` applies the gate after each round, keeps the policy across
tool rounds (`ResolvedModelRoute.continued`), skips a rejected tier in
fallback, fails the request as a failed `ChatResponseEvent` when a signal
cannot judge (verifier out of retries, custom signal threw), and records hops
in the `model_routing` block (`escalation_attempted`, `escalations`,
`escalated_tokens`), as `ModelRoutingEvent`s with `decision_source:
escalation`, and in the `routingEscalation
Count` / `routingEscalatedTokens` counters. `AgentPlan` validates tiers, the
signal's decision model and the custom class at plan construction.
- `docs`: "Quality-Triggered Escalation" section in Model Routing
(declaration, signals, interplay table), observability and recovery updates,
two metric rows in Monitoring.
- `examples`: `ModelRoutingEscalationExample` (two Ollama tiers, no
strategy, a custom SQL-shape signal, an agent that prints the tier that
answered and the hops).
- `e2e-test`: `QualityEscalationE2ETest` runs five requests through a real
job with scripted Java chat models, a scripted verifier decision model and a
tool: accepted on the first tier, verifier rejection, structural rejection,
custom rejection on a tool round (the stronger tier continues with the tool
result), and an escalation target that is down (fallback skips the rejected
tier and the request fails).
## Verifying this change
Unit and integration tests:
```
mvn -o -pl api,plan test # api 845 pass, plan 731 pass (incl.
EscalationDeclarationTest 9,
# EscalationGateTest 8,
ChatModelActionEscalationTest 14,
# AgentPlanEscalationValidationTest 6)
mvn -o -pl e2e-test/flink-agents-end-to-end-tests-integration test
-Dtest=QualityEscalationE2ETest # pass
```
Running job: `ModelRoutingEscalationExample` on a standalone Flink 2.3.0
cluster with a local Ollama (`qwen3:0.6b` -> `qwen3:8b`), four requests: the
greeting and the plain SQL request were accepted on `small`; the prose-only SQL
request was escalated to `big` by the SQL signal; the unanswerable fact was
escalated by the hedge check and, since `big` hedged too, the last tier's
answer was kept. The output carries the hops with their reasons and
`escalated_tokens`, and the TaskManager log has the `decision_source:
escalation` routing events.
Also run: `mvn -o -pl runtime test` (1561 tests; the only errors are the six
Kafka Testcontainers tests, which cannot start the ryuk container on this
machine) and the end-to-end modules: agent-plan compatibility passes; the
integration module passes except for one flaky run of the live-Ollama
`ReActAgentTest` (a model-output-dependent tool-parameter cast, green on
rerun); the cross-language module passes 25 of 27 with the project venv, the
two failures being the skills-wheel fixture calling `pip` in a venv without it.
Docs built with the pinned Hugo 0.110.0.
## Does this pull request potentially affect one of the following parts
- Dependencies (does it add or upgrade a dependency): no
- The public API: yes, additive (`Escalation`, `QualitySignal*`,
`ModelRouter.Builder.escalation`, `ModelRoutingEvent.SOURCE_ESCALATION`).
Routers remain Java-only, as documented; the Python side follows the routing
parity work in #1333.
- The runtime per-record code paths: yes, only for routed requests whose
router declares tiers.
- Anything that affects deployment or dependency management: no
## Documentation
- Does this pull request introduce a new feature: yes
- If yes, how is the feature documented: docs (Model Routing:
Quality-Triggered Escalation; Monitoring: two counters) and JavaDocs
Generated-by: Claude Code 2.1.309 (Claude Fable 5.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
