Zhuoxi2000 commented on code in PR #945:
URL: https://github.com/apache/flink-agents/pull/945#discussion_r3744899910
##########
docs/content/docs/development/chat_models.md:
##########
@@ -1207,6 +1207,113 @@ Some popular options include:
Model availability and specifications may change. Always check the official
DashScope documentation for the latest information before implementing in
production.
{{< /hint >}}
+### vLLM
+
+[vLLM](https://docs.vllm.ai) serves open-weight models behind an
OpenAI-compatible API and is a popular choice for self-hosted production
deployments. Flink Agents provides a dedicated connection that reuses the
OpenAI integration with vLLM-friendly defaults, in both Java and Python.
+
+#### Prerequisites
+
+1. Install vLLM and start a server: `vllm serve Qwen/Qwen2.5-7B-Instruct`
Review Comment:
Done the docs now show the tool-enabled command for Qwen2.5:
`--enable-auto-tool-choice --tool-call-parser hermes`
I also noted that the parser is model-specific and linked to the vLLM
tool-calling docs. Without those flags, normal chat still works, but tool calls
won’t show up in `tool_calls`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
