joeyutong opened a new pull request, #1382: URL: https://github.com/apache/flink-agents/pull/1382
Linked issue: #1381. Related: #1374 (Chat/Embedding). ### Purpose of change Add MediaModel and RerankModel resources for generation, transcription, and reranking. Async job handles support resumed queries. DashScope providers run in Python, with Java wrappers. #### Capability map | Resource / task | Input | Execution and output | DashScope documentation | | --- | --- | --- | --- | | Media / `speech_synthesis` | Text and voice | Synchronous request → audio URL | [TTS](https://help.aliyun.com/en/model-studio/qwen-audio-tts-http-api) | | Media / `transcription` | One HTTP(S) audio/video URL | Async job → transcript and metadata | [File transcription](https://help.aliyun.com/en/model-studio/qwen-audio-asr-filetrans-http-api) | | Media / `image_generation` | Prompt, optional reference images | Async job → image URLs | [Generation/editing](https://help.aliyun.com/en/model-studio/qwen-image-generation-and-editing-api-reference) | | Media / `video_generation` | Prompt and/or media with roles | Async job → video URL | [Wan](https://help.aliyun.com/en/model-studio/wan3-video-generation-api-reference), [HappyHorse](https://help.aliyun.com/en/model-studio/happyhorse-image-to-video-api-reference) | | Rerank | Text/image query; text/image/video candidates | Synchronous request → ranked indices and scores | [Reranking](https://help.aliyun.com/en/model-studio/text-rerank-api) | #### Runtime flow and key decisions Decorators, annotations, descriptors, or YAML register a resource. Python invokes the provider; Java wrappers transfer typed blocks, results, and jobs. Each resource owns and closes its SDK client. Media binds `model` and `task`; task selects the protocol without model-name routing. `generate()` sends one TTS request or submits and waits for an async task. Only async tasks support job operations. Polling shares one state/deadline loop; request/result conversion remains task-specific. Chat's `api_type` is not exposed here. Rerank uses one SDK interface for text and multimodal inputs. Two Python examples cover generation chaining and ranking. ### Behavioral Semantics #### Execution contracts - Call parameters override resource defaults. Unset rerank options use configuration, then service defaults. `video_media_roles` must align with media blocks; the service validates model-specific combinations. - Jobs bind provider, model, task, and ID; mismatched handles are rejected. `fetch()` returns status and successful results. - `wait()` raises for failed, canceled, unknown, or result-less completion. A failed transcription subtask overrides parent success. | Operation | Timeout meaning | Task after timeout | | --- | --- | --- | | TTS `generate`, `rerank` | One request; unset uses `request_timeout` | No queryable job | | Async `generate` | Polling budget after submission | Continues; exception retains handle | | `wait(job)` | Budget from this wait; unset uses `wait_timeout` | Continues; handle can be reused | Polling limits query start times; in-flight queries may finish later. Only pending tasks can be canceled. Callers can durably record `submit()` and resume `wait()`. Failure before recording the handle can repeat submission; there is no exactly-once task creation guarantee. #### Results and failures Outputs reuse typed blocks; rerank preserves provider order and optional original documents by request index. Invalid indices raise. Result URLs may expire. Transcription files are downloaded without the SDK client's credentials. Invalid inputs/job operations, missing image/video/audio URLs, and invalid rerank indices raise. Query/transport errors propagate. Non-pending cancellation raises a job error; failed diagnostic SDK/HTTP queries report UNKNOWN. New providers disable SDK retries and local config-file loading. Java preserves interruption/cancellation separately from job failures. ### Tests | Contract | Coverage | | --- | --- | | Task support and TTS request | `test_async_support_follows_the_task`, `test_generate_sends_text_voice_and_format` | | Submit once, then wait | `test_generate_submits_once_and_waits` | | Serialized job identity | `test_media_job_json_roundtrip`, `test_wait_rejects_job_from_another_resource` | | Wait success/failure/deadline | `test_media_model.py` polling cases | | Recorded job reused after interruption | `test_interrupted_wait_replays_the_submitted_job` | | Transcription download isolation / subtask failure | `test_dashscope_transcription.py` | | Image/video conversion and roles | Image, Wan, and HappyHorse protocol tests | | Rerank defaults, indices and original content | `test_dashscope_rerank_model.py` | | Existing Chat/Embedding SDK calls | 37 offline schema and mock cases | | Java/Python declarations and YAML aliases | `AgentPlanTest`, `AliasesTest`, Python YAML/Plan tests | | Bridge metadata, job errors and control signals | `PythonMediaModelTest`, `PythonRerankModelTest`, `test_media_rerank_bridge.py` | On the rebased tree: 828 Python tests passed (1 skipped, 8 live cases deselected); Java API/Plan plus MCP dependency tests: 1,628 passed, 20 skipped. Ruff, Spotless, Apache RAT, generated-schema and resource-name consistency checks passed. Unverified here: live providers, full Flink restart jobs, runtime/E2E suites, and other models/regions. Recovery uses a mock provider; examples do not cover recovery. <details> <summary>SDK compatibility and protocol details</summary> DashScope's SDK transport is used directly for Qwen's documented asynchronous image endpoint and shared task query/cancel endpoints. TTS, transcription submission, video submission, and rerank use their SDK methods. The SDK upgrade was previously checked with live Chat/Embedding requests and clean Python 3.10–3.12 installations. Python 3.10 with PyFlink has an existing grpcio wheel-platform metadata warning, reproduced with the old SDK too; it is not claimed as a clean runtime dependency check. Those earlier checks are not new full-job validation of this rebased commit. </details> ### API Adds Java/Python contracts, registration, YAML `media_models`/`rerank_models`, and Java-to-Python wrappers. Native Java providers, reverse wrappers, and built-in Events/Actions are out of scope. Chat/Embedding source APIs are unchanged; upgrading DashScope `~=1.24.2` to `~=2.0,>=2.0.4` changes SDK defaults/dependencies for existing callers. ### Documentation - [x] `doc-needed` User docs follow after #1374 merges. YAML schemas are updated here. ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes Generated-by: Cursor (Grok 4.7) Generated-by: Codex CLI 0.167.0-alpha.3.2 (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
