wenjin272 commented on code in PR #1374:
URL: https://github.com/apache/flink-agents/pull/1374#discussion_r4536326558
##########
docs/content/docs/development/calling-llms/chat-models/providers.md:
##########
@@ -49,7 +49,7 @@ model, deployment, and server must also support the requested
input and tool cal
| Integration | Implementation language | Media input accepted by the
integration |
|---|---|---|
-| [Alibaba Cloud Model Studio
(DashScope)](#alibaba-cloud-model-studio-dashscope) | Python | No media input;
media blocks are rejected. |
+| [Alibaba Cloud Model Studio
(DashScope)](#alibaba-cloud-model-studio-dashscope) | Python | User images and
video files with `api_type="multimodal"` or an `auto` route to that interface.
Speech recognition and image editing select it through `task`. Media sources
can be HTTP(S) URLs or Base64. |
Review Comment:
Could we simplify this row to describe only the supported media inputs?
```markdown
| [Alibaba Cloud Model Studio
(DashScope)](#alibaba-cloud-model-studio-dashscope) | Python | User images and
video files; audio for speech recognition. |
```
The `api_type`, `auto`, `task`, and URL/Base64 details would fit better in
the DashScope section below. This table is a capability overview, so readers
should be able to quickly see which media types each integration accepts.
##########
api/src/main/java/org/apache/flink/agents/api/embedding/model/BaseEmbeddingModelConnection.java:
##########
@@ -89,4 +94,23 @@ public EmbeddingResult<List<float[]>> embedWithUsage(
List<String> texts, Map<String, Object> parameters) {
return new EmbeddingResult<>(embed(texts, parameters), null);
}
+
+ /**
+ * Generate one vector per supported data content block in input order,
with token usage when
+ * available.
+ */
+ public EmbeddingResult<List<float[]>> embedBlocksWithUsage(
Review Comment:
The embedding API currently has separate `embed` and `embedWithUsage`
methods, while `chat()` consistently returns a `ChatResult` containing optional
usage. Longer term, I think we should align these designs: embedding calls
should return `EmbeddingResult` with optional usage, with consistent
single-input and batch semantics across Java and Python.
That cleanup would affect existing providers, callers, and cross-language
bridges, so it belongs in a separate issue and PR. I don’t think we need to
reshape the APIs in this PR just to introduce an intermediate design. I’m fine
with keeping the current approach here and tracking the API unification as a
follow-up.
##########
docs/content/docs/development/calling-llms/chat-models/providers.md:
##########
@@ -143,9 +145,136 @@ class MyAgent(Agent):
...
```
+#### Interface Selection
+
+Use the same setup options in Python, Java, and YAML.
+
+| Effective `api_type` | Operation | Setup options | Result content |
+|---|---|---|---|
+| `text` (default) | Text chat | `model="qwen-plus"` (default) | Text and,
when configured, tool calls |
+| `multimodal` | [Image](#image-input) or [video](#video-input) understanding
| `api_type="multimodal"` with a vision model such as `qwen3-vl-plus` | Text
and, when supported, tool calls |
+| `multimodal` (automatic) | [Speech
recognition](#audio-input-speech-recognition) |
`model="qwen-audio-3.1-asr-flash"`, `task="asr"` | Transcript text |
Review Comment:
Could we organize this table around the operation and the configuration
users need? `multimodal (automatic)` is easy to confuse with `api_type="auto"`,
but here it means that setting `task` selects the multimodal interface.
For example:
| Operation | Configuration |
|---|---|
| Text chat | Default configuration |
| Image/video understanding | Set `api_type="multimodal"` and a compatible
model |
| Speech recognition | Set `task="asr"` and a compatible model; `api_type`
can be omitted |
| Image generation/editing | Set `task="image_generation"` and a compatible
model; `api_type` can be omitted |
We could explain the model-name routing behavior of `api_type="auto"`
separately. The paragraph below could also name `asr` and `image_generation`
explicitly instead of referring to “either task,” and avoid introducing the MMC
abbreviation.
##########
docs/content/docs/development/calling-llms/chat-models/providers.md:
##########
@@ -110,10 +110,12 @@ The DashScope integration currently has a Python
implementation only. To use it
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
-| `model` | str | `"qwen-plus"` | Name of the chat model to use (an
`output_schema` is enforced natively only on `qwen3.7-max`,
`qwen3.7-max-preview`, `qwen3.7-max-2026-05-17` and `qwen3.7-max-2026-05-20`,
the schema-capable models DashScope serves on the text-generation interface
this connection calls; every other model, the `qwen-plus` default included,
falls back to prompting) |
+| `model` | str | `"qwen-plus"` | Name of the chat model to use. See
[Interface Selection](#interface-selection) for native `output_schema` support.
|
+| `api_type` | str | `"text"` | `text` uses Generation; `multimodal` uses
MultiModalConversation; `auto` selects the interface from the model name as
described below |
+| `task` | str | `None` | Omit for chat and media understanding. Set `asr` for
speech recognition or `image_generation` for image generation and editing. |
| `temperature` | float | `0.7` | Sampling temperature (0.0 to 2.0) |
| `extract_reasoning` | bool | `False` | Extract reasoning content from
response |
-| `additional_kwargs` | dict | `{}` | Additional DashScope API parameters. A
`response_format` supplied here is rejected only when it would be overwritten,
that is when an `output_schema` carrying a Pydantic model is set on a model
that applies it natively; otherwise it is passed through unchanged |
+| `additional_kwargs` | dict | `{}` | Additional DashScope API parameters. For
chat and media understanding, a supplied `response_format` is rejected when a
native `output_schema` would overwrite it. Otherwise it is passed through
unchanged. |
Review Comment:
Could we keep the `additional_kwargs` description brief in this table?
> Additional parameters passed to the DashScope API.
The `response_format` conflict rule would be clearer in the structured
output section, where both concepts can be explained together. For example:
> When using native structured output, do not also set `response_format` in
`additional_kwargs`. Flink Agents derives it from `output_schema` and raises an
error if both are supplied.
##########
docs/content/docs/development/calling-llms/chat-models/providers.md:
##########
@@ -143,9 +145,136 @@ class MyAgent(Agent):
...
```
+#### Interface Selection
+
+Use the same setup options in Python, Java, and YAML.
+
+| Effective `api_type` | Operation | Setup options | Result content |
+|---|---|---|---|
+| `text` (default) | Text chat | `model="qwen-plus"` (default) | Text and,
when configured, tool calls |
+| `multimodal` | [Image](#image-input) or [video](#video-input) understanding
| `api_type="multimodal"` with a vision model such as `qwen3-vl-plus` | Text
and, when supported, tool calls |
+| `multimodal` (automatic) | [Speech
recognition](#audio-input-speech-recognition) |
`model="qwen-audio-3.1-asr-flash"`, `task="asr"` | Transcript text |
+| `multimodal` (automatic) | [Image generation and
editing](#synchronous-image-generation-and-editing) | `model="qwen-image-3.0"`,
`task="image_generation"` | Image blocks |
+
+The multimodal interface uses MultiModalConversation (MMC) in non-streaming
+mode. Setting either task selects MMC automatically, so you can omit
+`api_type`. For these tasks, leave `prompt`, `tools`, `skills`, and `subagents`
+unset on the setup and omit `output_schema` and `response_format`.
+
+Add media to user messages. Model responses use `ChatResult`, including token
+usage and the finish reason when available. If the service reports only audio
+duration or image count for a task, token usage is absent.
+
+`api_type="auto"` recognizes the following MMC model families and exceptions:
+
+| Models | Matching rule |
+|--------|---------------|
+| `qwen-vl-plus`, `qwen-vl-max`, `qwen3-vl-plus`, `qwen3-vl-flash` | Exact
names and variants beginning with the name followed by `-` |
+| `qwen3.5-plus`, `qwen3.5-flash`, `qwen3.6-plus`, `qwen3.6-flash`,
`qwen3.7-plus`, `qwen3.7-flash`, `qwen3.8-max`, `qwen3.8-flash` | Exact names
and variants beginning with the name followed by `-` |
+| Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5, Qwen3.6 and Qwen3.8 open-weight
models | A parameter size beginning with a digit after the family prefix, such
as `qwen3-vl-8b-instruct` or `qwen3.8-2.4t-a95b` |
+| QVQ | `qvq` or names beginning with `qvq-` |
+| `qwen3.7-max-2026-06-08`, `kimi-k2.5` | Exact names |
+
+Names not matching these rules use the text interface. In particular,
`qwen3.7-max`, its preview,
+and its `2026-05-17` and `2026-05-20` snapshots remain on text generation.
+Unlisted models and deployment aliases can select `multimodal` explicitly.
+VL embedding, rerank and OCR models are outside this chat model list.
+
+Routing identifies the API, not the model's availability or supported
invocation modes.
Review Comment:
Could we shorten this paragraph to the restrictions users need when
configuring the integration?
> The multimodal interface supports only non-streaming calls. Choose a model
that supports this mode. Including media in a message does not automatically
select the multimodal interface.
The model-specific streaming and thinking restrictions could be covered by a
link to the provider documentation. Listing models in the routing table and
then explaining why some cannot be invoked through this integration makes the
section harder to follow.
##########
docs/content/docs/development/calling-llms/chat-models/providers.md:
##########
@@ -143,9 +145,136 @@ class MyAgent(Agent):
...
```
+#### Interface Selection
+
+Use the same setup options in Python, Java, and YAML.
+
+| Effective `api_type` | Operation | Setup options | Result content |
+|---|---|---|---|
+| `text` (default) | Text chat | `model="qwen-plus"` (default) | Text and,
when configured, tool calls |
+| `multimodal` | [Image](#image-input) or [video](#video-input) understanding
| `api_type="multimodal"` with a vision model such as `qwen3-vl-plus` | Text
and, when supported, tool calls |
+| `multimodal` (automatic) | [Speech
recognition](#audio-input-speech-recognition) |
`model="qwen-audio-3.1-asr-flash"`, `task="asr"` | Transcript text |
+| `multimodal` (automatic) | [Image generation and
editing](#synchronous-image-generation-and-editing) | `model="qwen-image-3.0"`,
`task="image_generation"` | Image blocks |
+
+The multimodal interface uses MultiModalConversation (MMC) in non-streaming
+mode. Setting either task selects MMC automatically, so you can omit
+`api_type`. For these tasks, leave `prompt`, `tools`, `skills`, and `subagents`
+unset on the setup and omit `output_schema` and `response_format`.
+
+Add media to user messages. Model responses use `ChatResult`, including token
+usage and the finish reason when available. If the service reports only audio
+duration or image count for a task, token usage is absent.
+
+`api_type="auto"` recognizes the following MMC model families and exceptions:
Review Comment:
Could we describe this as model-name routing rules? “Recognizes the
following MMC model families and exceptions” can read like a list of supported
models, while the implementation selects an API through fixed name matching.
For example:
> For ordinary chat, `api_type="auto"` selects the API based on the model
name. Models matching the rules below use the multimodal API; all other models
use the text API.
##########
docs/content/docs/development/calling-llms/chat-models/providers.md:
##########
@@ -143,9 +145,136 @@ class MyAgent(Agent):
...
```
+#### Interface Selection
+
+Use the same setup options in Python, Java, and YAML.
+
+| Effective `api_type` | Operation | Setup options | Result content |
+|---|---|---|---|
+| `text` (default) | Text chat | `model="qwen-plus"` (default) | Text and,
when configured, tool calls |
+| `multimodal` | [Image](#image-input) or [video](#video-input) understanding
| `api_type="multimodal"` with a vision model such as `qwen3-vl-plus` | Text
and, when supported, tool calls |
+| `multimodal` (automatic) | [Speech
recognition](#audio-input-speech-recognition) |
`model="qwen-audio-3.1-asr-flash"`, `task="asr"` | Transcript text |
+| `multimodal` (automatic) | [Image generation and
editing](#synchronous-image-generation-and-editing) | `model="qwen-image-3.0"`,
`task="image_generation"` | Image blocks |
+
+The multimodal interface uses MultiModalConversation (MMC) in non-streaming
+mode. Setting either task selects MMC automatically, so you can omit
+`api_type`. For these tasks, leave `prompt`, `tools`, `skills`, and `subagents`
+unset on the setup and omit `output_schema` and `response_format`.
+
+Add media to user messages. Model responses use `ChatResult`, including token
Review Comment:
Could we remove this paragraph and keep the relevant input restrictions in
the individual task sections? `ChatResult`, optional usage, and finish reasons
are already part of the general Chat API contract. The
audio-duration/image-count detail seems unnecessary for explaining how to use
this provider.
If the user-message restriction is not stated elsewhere, we could retain
just:
> Media inputs are supported only in user messages.
##########
docs/content/docs/development/calling-llms/chat-models/providers.md:
##########
@@ -143,9 +145,136 @@ class MyAgent(Agent):
...
```
+#### Interface Selection
+
+Use the same setup options in Python, Java, and YAML.
+
+| Effective `api_type` | Operation | Setup options | Result content |
+|---|---|---|---|
+| `text` (default) | Text chat | `model="qwen-plus"` (default) | Text and,
when configured, tool calls |
+| `multimodal` | [Image](#image-input) or [video](#video-input) understanding
| `api_type="multimodal"` with a vision model such as `qwen3-vl-plus` | Text
and, when supported, tool calls |
+| `multimodal` (automatic) | [Speech
recognition](#audio-input-speech-recognition) |
`model="qwen-audio-3.1-asr-flash"`, `task="asr"` | Transcript text |
+| `multimodal` (automatic) | [Image generation and
editing](#synchronous-image-generation-and-editing) | `model="qwen-image-3.0"`,
`task="image_generation"` | Image blocks |
+
+The multimodal interface uses MultiModalConversation (MMC) in non-streaming
+mode. Setting either task selects MMC automatically, so you can omit
+`api_type`. For these tasks, leave `prompt`, `tools`, `skills`, and `subagents`
+unset on the setup and omit `output_schema` and `response_format`.
+
+Add media to user messages. Model responses use `ChatResult`, including token
+usage and the finish reason when available. If the service reports only audio
+duration or image count for a task, token usage is absent.
+
+`api_type="auto"` recognizes the following MMC model families and exceptions:
+
+| Models | Matching rule |
+|--------|---------------|
+| `qwen-vl-plus`, `qwen-vl-max`, `qwen3-vl-plus`, `qwen3-vl-flash` | Exact
names and variants beginning with the name followed by `-` |
+| `qwen3.5-plus`, `qwen3.5-flash`, `qwen3.6-plus`, `qwen3.6-flash`,
`qwen3.7-plus`, `qwen3.7-flash`, `qwen3.8-max`, `qwen3.8-flash` | Exact names
and variants beginning with the name followed by `-` |
+| Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3.5, Qwen3.6 and Qwen3.8 open-weight
models | A parameter size beginning with a digit after the family prefix, such
as `qwen3-vl-8b-instruct` or `qwen3.8-2.4t-a95b` |
+| QVQ | `qvq` or names beginning with `qvq-` |
+| `qwen3.7-max-2026-06-08`, `kimi-k2.5` | Exact names |
+
+Names not matching these rules use the text interface. In particular,
`qwen3.7-max`, its preview,
+and its `2026-05-17` and `2026-05-20` snapshots remain on text generation.
+Unlisted models and deployment aliases can select `multimodal` explicitly.
+VL embedding, rerank and OCR models are outside this chat model list.
+
+Routing identifies the API, not the model's availability or supported
invocation modes.
+QVQ requires streaming, which this connection does not support. Some
open-weight
+models also require streaming when thinking is enabled. See the
+[streaming
restrictions](https://help.aliyun.com/en/model-studio/deep-thinking).
+The connection does not change thinking parameters automatically.
+See the [text
generation](https://help.aliyun.com/en/model-studio/text-generation),
+[vision](https://help.aliyun.com/en/model-studio/vision), and
+[Kimi](https://help.aliyun.com/en/model-studio/kimi-api) documentation for
model
+and interface details.
+The presence of media in a message does not change the selected interface.
+
+For chat and media understanding in Python, `structured_output_strategy=AUTO`
Review Comment:
Could we move the native JSON Schema model/interface table into a dedicated
Structured Output subsection? Interface Selection would be easier to follow if
it focused on choosing the API.
The general strategy behavior, schema types, and extra text-only conversion
call are already explained in the Structured Output chapter, so a link should
be sufficient here.
Also, “in Python” needs a brief explanation. Java callers can still use
structured output through the prompt-based path, but the current cross-language
setup cannot carry a schema for native enforcement. We could state this
explicitly:
> Native structured output is available when using DashScope directly from
Python. For requests with an output schema, Java callers using the
cross-language setup use the prompt-based path under `AUTO` or `PROMPT`;
`NATIVE` raises an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
