joeyutong opened a new pull request, #1374: URL: https://github.com/apache/flink-agents/pull/1374
Linked issue: #1059 ### Purpose of change DashScope setups can now handle images and videos in chat, transcribe recordings, generate or edit images, and embed text and media. This reuses the existing content blocks and works from Python and from Java through the Python resource bridge. #### Runtime flow Chat merges setup and call options, selects Generation or MultiModalConversation (MMC), and maps ordered blocks and results. The agent checks task schema acceptance before dispatch. Embedding selects the text or multimodal SDK call and orders vectors by input index. The Java resource bridge carries block lists, vectors, and usage. #### Key decisions Reuse shared block types and preserve text defaults. Independent vectors let callers correlate results by position. The SDK calls preserve path-like text as literal input. ### Behavioral Semantics #### Interaction decisions | Setup and request | Behavior | | --- | --- | | Chat without a task and with `api_type` omitted | Uses `text` and the existing Generation path. | | Chat without a task and with `api_type="auto"` | Selects MMC for recognized model names. Other models use Generation. Media does not change the route. | | Ordinary chat with a schema | The existing structured-output flow applies. Native finalization uses a separate text-only call on supported model/interface pairs. | | Chat with `task="asr"` or `task="image_generation"` | The task selects non-streaming MMC. Requests with tools, `output_schema`, or `response_format` fail before the provider call. | | Embedding with `api_type` omitted | Uses `text` with `text-embedding-v4` by default. Text-only providers also accept `TextBlock`. | #### Behavioral contracts - **Image understanding.** Set `api_type="multimodal"` on a vision-capable Chat setup such as `qwen3-vl-plus`. Send ordered user image and text blocks. Images accept HTTP(S) URLs and Base64. [DashScope image understanding](https://help.aliyun.com/en/model-studio/vision#multiple-image-inputs) - **Video understanding.** Use `api_type="multimodal"` and send a `VideoBlock` with a video file URL or Base64. Images, video and text can share the request. Per-block `provider_options` for settings such as video `fps`, and ordered frame-list inputs, are deferred. They need extensions to the shared block and media-source contracts. [DashScope video files, frames and sampling](https://help.aliyun.com/en/model-studio/vision#video-understanding-2) - **Speech recognition.** Use `task="asr"` with a Qwen-Audio-ASR model such as `qwen-audio-3.1-asr-flash`. One user `AudioBlock` by HTTP(S) URL or Base64 yields a transcript `TextBlock`. The media type supplies `format` unless the caller supplies it. [DashScope non-streaming ASR](https://help.aliyun.com/en/model-studio/qwen-audio-asr-http-api#non-streaming-h4) - **Image generation and editing.** Use `task="image_generation"` with a compatible Qwen-Image model such as `qwen-image-3.0`. Send one text prompt and add reference `ImageBlock` values by URL or Base64 for editing. The response contains ordered `ImageBlock` URLs that can be reused as references. [DashScope synchronous image generation and editing](https://help.aliyun.com/en/model-studio/qwen-image-generation-and-editing-api-reference#dashscope-synchronous-api-recommended) - **Multimodal Embedding.** Set `api_type="multimodal"` on the Embedding setup. The default model is `qwen3-vl-embedding`. Set `dimension` to choose the vector size. Each text, image URL or Base64 image, and video URL yields one vector. Single inputs return one vector in Python and batches preserve input order. Fused embeddings across items are deferred. [DashScope multimodal Embedding API](https://help.aliyun.com/en/model-studio/multimodal-embedding-api-reference) - ASR and image tasks leave token usage absent when the response supplies only duration or image count. Java setup wrappers expose the Python-resolved model name when the declaration omits it. #### Failure behavior Unsupported roles, media sources, and task inputs fail before making a request. Non-OK SDK responses raise with the provider's message. Embedding responses with an unexpected vector count or invalid indices fail the call. ### Tests | Contract | Coverage | | --- | --- | | Chat interface selection | `test_dashscope_chat_selects_interface` | | Image and video order and role validation | DashScope media request and rejection tests | | Ordinary chat schema path | DashScope native schema and Chat action tests | | ASR transcript and usage | `test_dashscope_asr_transcribes_one_recording` | | Image output and reuse | `test_dashscope_image_generation_can_edit_returned_image` | | Task shape and schema rejection | DashScope task rejection tests | | Embedding defaults, vector shape, order and usage | DashScope single and mixed input tests | | Unsupported media and invalid vector indices | Inline video and duplicate index tests | | Java/Python model and result fidelity | Java Setup and Python Java Embedding wrapper tests | On the updated main, 190 Java and 330 Python tests passed (20 and 18 skipped). All four PyFlink example modes and the Java YAML job passed with real DashScope requests. Hugo, Ruff, and Spotless passed. The run covers the example models on local Flink. The full model and region matrix, distributed recovery of the new tasks, and repository-wide suites remain unverified. <details> <summary>Request and bridge verification details</summary> The Embedding request uses a uniform object form for its contents, including text. It sets `enable_fusion=false` and validates that indices cover the batch exactly. The SDK-boundary test sends literal text matching a local path and guards against file upload. The Java Embedding adapter transfers discriminated block maps and primitive result values. The Chat Setup preflight asks the Python setup about schema acceptance without transferring a Java schema object. </details> ### API Python Embedding APIs accept `content` as a string, data block, or batch. Positional text calls keep their return shape. Keyword callers use `content=` in place of `text=`. Java retains the string methods and adds `embedBlocksWithUsage` for ordered blocks. Python, Java, and YAML resource declarations use the same DashScope options. ### Documentation - [x] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes Generated-by: Codex CLI 0.167.0-alpha.3.2 (GPT-6.1 Sol) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
