joeyutong opened a new issue, #1381: URL: https://github.com/apache/flink-agents/issues/1381
### Search before asking - [x] I searched the existing issues. #1059 and #1374 cover multimodal ChatMessage/provider integration; this proposal adds separate media-generation and reranking resources. ### Description Applications need to synthesize speech, transcribe recordings, generate images and videos, and rerank retrieval candidates without treating every operation as a chat completion. Long-running provider jobs also need a serializable handle so callers can query their state and resume waiting without submitting a new job. Introduce provider-independent MediaModel and RerankModel resource contracts, with DashScope as the first Python provider and Java access through the existing Python resource bridge. | Resource | Capability | Execution and result | | --- | --- | --- | | MediaModel | Speech synthesis | Synchronous request returning an audio URL | | MediaModel | File transcription | Asynchronous job returning transcript blocks and metadata | | MediaModel | Image generation/editing | Asynchronous job returning image URLs | | MediaModel | Video generation | Asynchronous job returning a video URL | | RerankModel | Text and multimodal reranking | Ranked candidate indices, relevance scores, and optional original content | ### Proposed contract - Reuse the existing typed content blocks. Each media resource binds a model and an explicit task; model names do not select the invocation protocol. - Expose `generate()` for a completed result and `submit/fetch/wait/cancel` for asynchronous jobs. Synchronous speech synthesis has no job operations. - Keep asynchronous polling, terminal states, and timeout behavior common across task types. A polling timeout leaves the provider task running and retains its handle. - Let callers persist a successful `submit()` outcome and resume `wait()` separately. This does not guarantee exactly-once task creation if the handle was not durably recorded. - Keep request indices stable in rerank results and preserve the provider's ranking order. - Support Python decorators, Java annotations, programmatic descriptors, and YAML declarations, with aligned Java/Python results and control-flow exceptions. ### Delivery scope and verification The initial implementation includes Python providers, Java-to-Python wrappers, YAML schemas/aliases, focused contract and protocol tests, and two runnable Python examples. Native Java providers, reverse Python-to-Java wrappers, and built-in request Events/Actions are outside this delivery. DashScope SDK 2.0.4 or later in the 2.x series is required. Existing Chat and Embedding source APIs remain unchanged; the SDK upgrade needs compatibility and installation checks. User documentation will follow after the first-phase DashScope multimodal PR #1374 merges, so the combined capability guide describes the final public contracts. The implementation PR will be marked `doc-needed`. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
