ShawnGarello opened a new pull request, #1370:
URL: https://github.com/apache/flink-agents/pull/1370
Linked issue: #1216
### Purpose of change
The Gemini integration now sends images, audio, video and documents returned
by a tool to the model. A tool result holding media becomes one
`functionResponse` whose `parts` are its media blocks in order; its text stays
in `response`. Before, media in a tool result failed the request.
Tools can return screenshots, recordings or documents the model must read
before it continues, and Google documents media in function response parts for
Gemini 3 models. This is part of the plan in the issue, after #1331 (Anthropic)
and #1364 (OpenAI Responses), whose tool-result media table this docs change
edits. The fields below cover the adapter change; the docs part updates the
Gemini row of that table and the Gemini section.
#### Runtime flow
`GeminiChatModelConnection.chat` converts each non-system message with
`GeminiChatUtils.convertToContent`. For a tool message, each media block of the
`ToolResultBlock` becomes a `FunctionResponsePart` through `convertMedia`, and
the text blocks are joined into `response.result`, giving one
`Part.fromFunctionResponse(name, {"result": text}, parts)` in a `user` turn.
User messages call `convertMedia` with `Part::fromUri` and `Part::fromBytes`,
tool results with the `FunctionResponsePart` ones. Media in an assistant
message still throws.
#### Key decisions
- Text stays in `response` and media goes in `parts`, without a `$ref` from
`response` to a part. The API makes `$ref` optional, and using it would change
the `response` of every result with media. No `displayName` is set.
- One helper decides URL or Base64 and decodes the data for both paths. It
takes the SDK's factory methods, because message parts and function response
parts are separate types.
- No model check: aliases, `base_url` proxies and new model names make one
brittle. An older model's rejection comes back as the service's error.
- The tool result's error flag and the function response `id` stay unsent,
as for text-only results.
### Behavioral Semantics
#### Interaction decisions
| Message | Text only | Media | Malformed Base64 |
|---|---|---|---|
| User | unchanged | unchanged | raises, unchanged |
| Tool result | unchanged | `functionResponse.parts` in block order; text in
`response` | raises, as in a user message |
| Tool result, `is_error=true` | unchanged, flag not sent | as with
`is_error=false`, flag not sent | as with `is_error=false` |
| Assistant | unchanged | raises, role reason (new wording) | raises the
same, before decoding |
#### Behavioral contracts
1. A tool result with media is one `user` turn with one `functionResponse`:
`name` as for any tool result, `response` = `{"result": <its text blocks
joined>}` (`""` if none), and `parts` = one part per media block, in block
order.
2. A Base64 block becomes `inlineData` (`mimeType` = the block's media type,
`data` = the decoded bytes); a URL block becomes `fileData` (`fileUri`,
`mimeType`). No `displayName` is set.
3. Each media block carries the same values as in a user message.
4. All four media kinds are accepted; media types aren't restricted.
5. A text-only tool result is sent as before, byte for byte.
6. Malformed Base64 data in a tool result raises `IllegalArgumentException`
with the same message as in a user message.
7. Media in an assistant message raises `UnsupportedContentBlockException`
with the reason `only user messages and tool results can carry media, not
assistant messages`. The message names no URL or data.
8. A tool result's error flag is not sent, with or without media, and
nothing is added to `response`.
9. Each tool message becomes its own turn with its own function name and
response, text-only or not.
#### Failure behavior
Malformed Base64 raises `IllegalArgumentException`, and media in an
assistant message `UnsupportedContentBlockException` (a subclass), while the
request is built, so nothing is sent; media is never dropped or stringified.
`chat()` rethrows both unwrapped. A media type, URL or model the service
rejects comes back as the service's error.
### Tests
| Contract | `GeminiMultimodalTest` |
|---|---|
| 1 | `testToolResultMediaIsSentInOrder` (full structure);
`testMediaOnlyToolResultSendsEmptyText` |
| 2 | `testToolResultMediaIsSentInOrder` |
| 3 | `testToolResultMediaIsSentInOrder` (each part equals the same block's
user-message part) |
| 4 | `testToolResultMediaIsSentInOrder` (image, audio, video, document) |
| 5 | `testTextOnlyToolResultIsUnchanged` (exact JSON; empty and error
results too) |
| 6 | `testMalformedBase64InToolResultFails`; `testChatKeepsMediaErrorTypes`
(through `chat()`) |
| 7 | `testMediaInAssistantMessagesFails` |
| 8 | `testErrorToolResultWithMediaSendsNoErrorField`;
`testTextOnlyToolResultIsUnchanged` |
| 9 | `testParallelToolResultsKeepTheirCorrelation` |
| live | `GeminiToolMediaLiveTest`: a tool returns a red PNG; the answer
must say "red" |
Coverage by risk: every media kind and both sources appear in one tool
result, with text between media blocks.
Not verified:
- The live test needs `GEMINI_API_KEY`. CI doesn't run it: no workflow sets
that variable, and secrets aren't passed to fork PRs. It passed against the
Gemini Developer API with `gemini-3.6-flash`.
- URL sources in tool results, models before Gemini 3 and Vertex AI are not
tested against the service.
- A Python agent using this Java connection through cross-language resources
isn't tested with tool media.
- No end-to-end agent job returns media from a tool.
<details>
<summary>Test runs, test-file changes and implementation invariants</summary>
- `mvn -B test -pl integrations/chat-models/gemini`: 84 passed
(`GeminiMultimodalTest` 12; `GeminiChatModelConnectionTest` 65, unchanged; the
live class doesn't run without a key).
- `mvn spotless:check -pl integrations/chat-models/gemini` and
`./tools/check-license.sh` pass.
- With `GeminiChatUtils.java` reverted, the new media, media-only,
assistant-reason, malformed-Base64, error-flag, parallel and `chat()` tests
fail, and the text-only test passes.
- The former assistant-and-tool media test is split: the assistant half is
contract 7's test; the tool half is replaced by the tests for contracts 1 and 6.
- The live test defaults to `gemini-3.6-flash`, a stable Gemini 3 model with
a free tier. `GEMINI_MULTIMODAL_MODEL` and `GEMINI_BASE_URL` override the model
and endpoint. It passes the first answer back unchanged, so the tool call's
thought signature goes with it.
- The SDK serializes every function response with a `parts` list, so a
text-only result keeps its empty `"parts":[]`.
- `DataContentBlock`'s subtypes are text and the four media types, so
`Gemini cannot send tool result block` stays only for a future subtype.
</details>
### API
No signature or dependency change. For a caller who changes nothing:
text-only tool results are sent unchanged; media in a tool result is now sent
instead of raising `UnsupportedContentBlockException`; the reason for media in
an assistant message adds "and tool results".
### Documentation
- [ ] `doc-needed`
- [ ] `doc-not-needed`
- [x] `doc-included`
### Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
Generated-by: Claude Code 2.1.319 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
