ShawnGarello opened a new pull request, #1364: URL: https://github.com/apache/flink-agents/pull/1364
Linked issue: #1216 ### Purpose of change The OpenAI Responses integration now sends images and documents returned by a tool to the model. A tool result holding media becomes one `function_call_output` item whose `output` is a list of the result's text, image and document blocks in order. Before, any media in a tool result failed the request. Tools can return screenshots, charts or documents the model must read before it continues, and the Responses API accepts them in a function output. This is part of the plan in the issue, after the Anthropic change in #1331; the other integrations still reject media in tool results. The fields below cover the adapter change; the docs part gives the tool-result media table the same rows as the Provider Capabilities table, and updates the Responses section and the Tool Use warning. #### Runtime flow `OpenAIResponsesUtils.convertSingleMessage` passes a tool message to `convertToolResult`. A result of only `TextBlock`s is sent as `output(String)`, the concatenated text, as before. Otherwise `toOutputItem` turns each block into a `ResponseFunctionCallOutputItem` in order: `ofInputText`, `ofInputImage` with `urlOrDataUri`, or `ofInputFile` with the fields from `toInputFile`. Those two helpers are shared with user messages. Other media throws. Assistant and system messages still go through `requireTextOnly`. #### Key decisions - Tool results reuse the user-message helpers, so one place decides each block's values. The SDK has separate classes for user content and function output, so `toInputFileContent` copies the fields `toInputFile` set, and unset fields stay off the wire. - Text-only results keep their wire form, so existing requests don't change. - `is_error` is not sent. A function output has no error field, and `status` is the item's progress, not an error. Text-only error results already work this way: the text says what failed. - No dependency change: openai-java 4.8.0 has `outputOfResponseFunctionCallOutputItemList`. ### Behavioral Semantics #### Interaction decisions | Message | Text only | Image or document | Audio or video | |---|---|---|---| | User | unchanged | unchanged | raises, unchanged | | Tool result | unchanged: one string | list `output` in block order | raises, same message as a user message | | Tool result, `is_error=true` | unchanged: one string, flag not sent | list `output`, flag not sent | raises, same message | | Assistant | unchanged | raises, role reason (new wording) | raises, role reason | #### Behavioral contracts 1. A tool result with media is sent as one `function_call_output` item: `call_id` is the call ID, and `output` a list of its blocks in order. 2. `TextBlock` becomes `input_text`. `ImageBlock` becomes `input_image` with `image_url` (the URL, or a `data:` URI for Base64 data) and `detail: auto`. `DocumentBlock` becomes `input_file`: Base64 data as `file_data` (a `data:` URI) with `filename` the block's name, or `document` when unset; a URL as `file_url`, with `filename` only when the block is named. 3. Each block carries the same values as in a user message. 4. A text-only tool result is sent as before: `output` is one string, the concatenated text, and `""` when empty. 5. Audio or video in a tool result raises `UnsupportedContentBlockException` with the same message as in a user message. The message names no URL or data. 6. Media in an assistant message raises with the reason `only user messages and tool results can carry media, not assistant messages`. 7. A tool result's error flag is not sent, with or without media, and no other field is added. 8. Each tool message becomes its own item with its own call ID and output, text-only or not. #### Failure behavior Every failure above is raised while the request is built, so nothing is sent; media is never dropped or stringified. `chat()` doesn't catch it, so `UnsupportedContentBlockException` reaches the caller unchanged. Media the model or API rejects comes back as the service's error. ### Tests | Contract | `OpenAIResponsesMultimodalTest` | |---|---| | 1, 2, 3 | `testToolResultMediaIsSentInOrder` (full structure, and equal to the same blocks' user-message content) | | 4 | `testTextOnlyToolResultIsUnchanged` (exact JSON) | | 5 | `testUnsupportedToolResultMediaFailsAsForUserMessages`; `testChatKeepsMediaErrorTypes` (through `chat()`) | | 6 | `testMediaInAssistantMessagesFails` | | 7 | `testErrorToolResultWithMediaSendsNoErrorField`; `testTextOnlyToolResultIsUnchanged` | | 8 | `testParallelToolResultsKeepTheirCorrelation` | | live | `OpenAIResponsesToolMediaLiveTest`: a tool returns a red PNG; the answer must say "red" | Coverage by risk: each accepted media kind and source appears in a tool result, and each unsupported kind is compared with its user-message error. Not verified: - The live test needs `OPENAI_API_KEY`. CI doesn't run it: no workflow sets that variable, and secrets aren't passed to fork PRs. It has not been run against the real API. - No test sends a document, or an error result with media, in a tool result to the real API. - A Python agent using this Java connection through cross-language resources isn't tested with tool media. - No end-to-end agent job returns media from a tool. <details> <summary>Test runs, test-file changes and implementation invariants</summary> - `mvn -B test -pl integrations/chat-models/openai`: 171 passed (`OpenAIResponsesMultimodalTest` 11; the live classes don't run without a key). - `mvn spotless:check -pl integrations/chat-models/openai` and `./tools/check-license.sh` pass. - With `OpenAIResponsesUtils.java` reverted, the new media, assistant-reason, unsupported, error-flag and parallel tests fail, and the text-only test passes. - The former assistant-and-tool media test is split: the assistant half is contract 6's test; the tool half is replaced by the tests for contracts 1 and 5. Audio and video are a shared list used by the user and tool tests. - The live test defaults to `gpt-6-luna`, which OpenAI's model page lists with image input and Responses support; `gpt-4o` and `gpt-4o-mini` are marked deprecated there. `OPENAI_MULTIMODAL_MODEL` overrides it. - The text-only path is picked with `allMatch(TextBlock)`, so an empty result is still sent as `""`. - `DataContentBlock`'s subtypes are text and the four media types, so only audio and video reach the final `throw` in `toOutputItem`. - `ToolResponse.error(...)` builds a text-only result, so media with `is_error` comes only from a `ToolResultBlock` built directly. </details> ### API No signature or dependency change. For a caller who changes nothing: text-only tool results are sent unchanged; images and documents in a tool result are now sent instead of raising; audio and video in a tool result still raise the same type, now with the user-message reason instead of `only user messages can carry media, not tool messages`. The assistant reason adds "and tool results". ### Documentation - [ ] `doc-needed` - [ ] `doc-not-needed` - [x] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: Claude Code 2.1.314 (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
