wenjin272 commented on code in PR #945: URL: https://github.com/apache/flink-agents/pull/945#discussion_r3742809992
########## integrations/chat-models/openai/src/main/java/org/apache/flink/agents/integrations/chatmodels/openai/VLLMChatModelConnection.java: ########## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.agents.integrations.chatmodels.openai; + +import org.apache.flink.agents.api.resource.ResourceContext; +import org.apache.flink.agents.api.resource.ResourceDescriptor; + +import java.util.HashMap; +import java.util.Map; + +/** + * Chat model connection for a <a href="https://docs.vllm.ai";>vLLM</a> server. + * + * <p>vLLM exposes an OpenAI-compatible API, so this connection reuses {@link + * OpenAICompletionsConnection} with vLLM-friendly defaults: + * + * <ul> + * <li><b>api_base_url</b> (optional): defaults to {@code http://localhost:8000/v1}, the default + * address of {@code vllm serve} + * <li><b>api_key</b> (optional): defaults to a placeholder value, since vLLM servers started + * without {@code --api-key} do not require a credential (the underlying OpenAI SDK requires a + * non-empty key, but the server ignores it). Set it explicitly when the server is started + * with {@code --api-key}. + * </ul> + * + * <p>All other connection arguments ({@code timeout}, {@code max_retries}, {@code default_headers}, + * {@code model}) behave exactly as in {@link OpenAICompletionsConnection}. + * + * <p>Example usage: + * + * <pre>{@code + * public class MyAgent extends Agent { + * @ChatModelConnection + * public static ResourceDescriptor vllm() { + * return ResourceDescriptor.Builder.newBuilder(VLLMChatModelConnection.class.getName()) + * .addInitialArgument("api_base_url", "http://my-vllm-host:8000/v1";) + * .build(); + * } + * } + * }</pre> + */ +public class VLLMChatModelConnection extends OpenAICompletionsConnection { Review Comment: [P1] Please use a vLLM-specific structured-output capability check. Because this class inherits `supportsNativeStructuredOutput()` unchanged, capability is still decided by the OpenAI model-name allowlist. Consequently, the documented `Qwen/Qwen2.5-7B-Instruct` model (and ordinary Llama names) returns `false`, so `chat(..., outputSchema)` silently omits `response_format`, while a model named `gpt-4o` returns `true`. vLLM supports the OpenAI `json_schema` response format for served models independently of OpenAI model names: https://docs.vllm.ai/en/stable/examples/features/structured_outputs/. Could we override this in both Java and Python (or introduce a vLLM-specific capability setting) and add request-building tests with a Qwen model? ########## docs/content/docs/development/chat_models.md: ########## @@ -1207,6 +1207,113 @@ Some popular options include: Model availability and specifications may change. Always check the official DashScope documentation for the latest information before implementing in production. {{< /hint >}} +### vLLM + +[vLLM](https://docs.vllm.ai) serves open-weight models behind an OpenAI-compatible API and is a popular choice for self-hosted production deployments. Flink Agents provides a dedicated connection that reuses the OpenAI integration with vLLM-friendly defaults, in both Java and Python. + +#### Prerequisites + +1. Install vLLM and start a server: `vllm serve Qwen/Qwen2.5-7B-Instruct` Review Comment: [P2] Please document the server flags required for tool calling. The command shown starts basic chat serving, but Flink Agents sends tools without a named `tool_choice` and relies on vLLM automatic tool calling. vLLM requires `--enable-auto-tool-choice` and a model-specific `--tool-call-parser` for that path; for Qwen2.5 it recommends the `hermes` parser: https://docs.vllm.ai/en/stable/features/tool_calling/. Without those flags, tool calls are not parsed into the OpenAI `tool_calls` field. Could the prerequisite show a tool-enabled command and note that the parser is model-specific, so users do not get a server that can chat but cannot drive an agent’s tools? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
