wenjin272 opened a new issue, #1324: URL: https://github.com/apache/flink-agents/issues/1324
### Search before asking - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description Add an optional integration with the **LiteLLM Python Router SDK**, allowing Flink Agents to use multiple model deployments, retries, fallbacks, and routing strategies without running a separate LiteLLM Proxy service. **Target:** the next release, subject to community agreement and implementation readiness. #### Motivation Model rate limits and transient failures can interrupt agent execution. Applications may need to retry a request, switch to another model or deployment, or select a backend based on latency or available capacity. LiteLLM Router provides these capabilities, but Flink Agents currently requires a custom adapter to use it in process. #### Proposed integration Reuse the existing Connection / Setup resource model; the names below are illustrative: - **LiteLLM Router Connection:** declare `model_list`, routing strategy and arguments, retry/timeout settings, fallback chains, and optional Redis-backed shared state. - **Chat Model Setup:** reference the connection and a logical model group, together with the existing prompt, tools, skills, and generation settings. - **Invocation:** keep the existing chat APIs and tool execution flow. The connection delegates model requests to `litellm.Router` and converts the result to `ChatResult`. Carry serializable configuration in the resource descriptor and create/reuse/close the live Router at runtime. Custom routing strategies should be declared through an importable class path and arguments rather than a live Router or closure in the job plan. This complements Flink Agents' existing `MODEL_ROUTER`: framework routing can select a business-level model/setup, while LiteLLM routes requests among backend deployments. Using LiteLLM should not require a `MODEL_ROUTER` resource. #### Acceptance criteria - [ ] Provide an optional Python integration and a documented resource declaration API, including built-in routing strategies and a custom strategy example. - [ ] Support multiple deployments, rate-limit retries, timeout handling, and cross-model fallback through LiteLLM Router. - [ ] Preserve messages, tool calls/results, finish reasons, usage, and actual backend model information where available; define and test structured-output behavior across fallback models. - [ ] Define Router lifecycle, concurrent use, and recovery behavior within the framework's existing durable model-call path. Document retry ownership and how parallel instances share quota/cooldown state. - [ ] Add deterministic tests for 429 retry, timeout/fallback, exhausted candidates, tool calls, and runtime reconstruction, without requiring live provider credentials. - [ ] Document dependencies and Python/Java cross-language support boundaries, and provide a Bailian example using a primary and backup model. #### References - [LiteLLM Router](https://docs.litellm.ai/docs/routing) - [LangChain ChatLiteLLMRouter](https://docs.langchain.com/oss/python/integrations/chat/litellm) - [smolagents LiteLLMRouterModel](https://huggingface.co/docs/smolagents/reference/models#smolagents.LiteLLMRouterModel) ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
