Zhuoxi2000 opened a new pull request, #1383:
URL: https://github.com/apache/flink-agents/pull/1383
<!--
* Thank you very much for contributing to Flink Agents.
* Please add the relevant components in the PR title. E.g., [api],
[runtime], [java], [python], [hotfix], etc.
-->
<!-- Please link the PR to the relevant issue(s). Hotfix doesn't need this.
-->
Linked issue: #1059
### Purpose of change
<!-- What is the purpose of this change? -->
A YAML prompt message can now carry media: its `content` is either text, as
before, or a list of content blocks in the `ChatMessage` block format. This is
the #1059 "Multimodal YAML prompts" follow-up. `Prompt.fromMessages` /
`Prompt.from_messages` already format block messages (placeholders in text
blocks, media passed through); YAML had no way to declare them.
```yaml
prompts:
- name: describe_image
messages:
- {role: system, content: "You describe {kind}."}
- role: user
content:
- {type: text, text: "Describe {subject}."}
- type: image
media_type: image/png
source: {type: url, url: "https://example.com/cat.png"}
```
#### Runtime flow
1. `PromptMessage` (Java and Python) reads `content` as text, or as a
non-empty list of `text`, `image`, `audio`, `video` or `document` blocks,
deserialized by the existing block deserializers.
2. The loaders build each message with `new ChatMessage(role, blocks)` /
`ChatMessage.of(role, blocks)`, so `ChatMessage`'s role rules apply at load
time (for example, a `system` message takes text only).
3. Formatting is unchanged: placeholders are substituted in text blocks only.
#### Key decisions
* **No new block format.** The list uses the JSON shape both languages
already share for `ChatMessage` blocks (`type`, `text`, `media_type`, `source:
{type: url|base64, ...}`, optional `name`, `size_bytes`, `sha256`), so a block
reads the same in YAML, in a serialized message and across the language bridge.
* **Data blocks only.** `reasoning`, `tool_call` and `tool_result` blocks
are rejected in prompts: they belong to model output and tool execution, not
templates.
* **No second validation layer.** Role constraints come from `ChatMessage`;
block fields from the block classes.
* `docs/yaml-schema.json` is regenerated from the Pydantic specs (it now has
`$defs` for the blocks and sources), with the development skill's bundled copy
and its `blob_sha`. `SchemaParityTest` checks the block definitions against the
Java block classes, ignoring the `type` discriminator, which Jackson reads as a
type id.
### Behavioral Semantics
<!-- For a non-trivial code change whose implementation is largely
AI-assisted: interaction decisions, behavioral contracts, and failure behavior.
See `contribution-guides/ai-assisted-pr.md`. Remove this heading and this
comment otherwise. -->
#### Interaction decisions
| `content` | Result |
|---|---|
| text (or another scalar, as before) | unchanged: one text message |
| non-empty list of data blocks | a message with those blocks, in order |
| empty list, an object, `null`, a non-data block, a malformed block | load
fails |
| media in a role that takes text only | load fails with `ChatMessage`'s
error |
#### Behavioral contracts
1. Text `content` behaves exactly as before.
2. A block list becomes the message's blocks in order, with the same block
classes `ChatMessage` deserializes.
3. Formatting substitutes placeholders in text blocks and passes media
blocks through unchanged.
4. Invalid content fails when the YAML file is loaded, never at request time.
5. Java and Python accept and reject the same inputs.
#### Failure behavior
* Invalid `content` or a role violation: the YAML load fails with a
validation error (`IllegalArgumentException` in Java, `ValidationError` /
`ValueError` in Python), as other invalid prompt specs do.
* Media that a chat model integration does not accept is handled by that
integration as before (`UnsupportedContentBlockException` /
`UnsupportedContentBlockError`).
### Tests
<!-- How is this change verified? -->
| Contract | Java | Python |
|---|---|---|
| 1 | existing `PromptSpecTest`, `YamlLoaderBuildersTest` | existing
`test_specs.py`, `test_loader.py` |
| 2 | `PromptSpecTest.promptMessageWithContentBlocks` |
`test_prompt_message_with_content_blocks` |
| 3 | `YamlLoaderBuildersTest.buildPromptWithMediaBlocks` |
`test_build_agents_loads_prompt_media_blocks` |
| 4 | `rejectsEmptyContentBlockList`, `rejectsNonDataContentBlocks`,
`rejectsMalformedContentBlocks`, `buildPromptRejectsMediaInSystemMessages` |
`test_prompt_message_rejects_invalid_content`,
`test_build_agents_rejects_media_in_system_prompt_messages` |
| 5 | the same YAML in both suites; `SchemaParityTest` |
`test_development_skill_main_schema_matches_checked_in_schema` |
Verified locally on main: Java `api` module 877 tests and `plan` YAML tests,
with spotless; Python `api/yaml` 172 tests, `plan` YAML tests, ruff.
<details>
<summary>Implementation invariants and supporting evidence</summary>
* `docs/yaml-schema.json` equals `YamlAgentsDocument.model_json_schema()`
serialized with `indent=2, sort_keys=True`, before and after this change.
* `PromptMessage.getContent()` still returns the text for text content; it
is `null` for block content, which `getBlocks()` returns.
</details>
### API
<!-- Does this change touches any public APIs? -->
The YAML format gains block-list `content` for prompt messages; text
`content` is unchanged. Java `PromptMessage` gains `getBlocks()`, and its
creator takes `content` as a `JsonNode`.
### Documentation
<!-- Do not remove this section. Check the proper box only. -->
- [ ] `doc-needed` <!-- Your PR changes impact docs -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [x] `doc-included` <!-- Your PR already contains the necessary
documentation updates -->
The Prompt section of `create-agent/yaml.md`, and the regenerated
`docs/yaml-schema.json`.
### Was this patch authored or co-authored using generative AI tooling?
<!-- Do not remove this section. Check the proper box only. -->
- [x] Yes
- [ ] No
If yes, include a `Generated-by: <tool name and version> (<model name and
version>)` line, for example `Generated-by: Claude Code 2.1.226 (Claude Opus
4.6)`, in the commit message so it reaches Git history. Repeat the same line
here for reviewer visibility. See the [ASF generative tooling
guidance](https://www.apache.org/legal/generative-tooling.html).
Generated-by: Claude Code 2.1.259 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
![https://example.com/cat.png"}](https://example.com/cat.png"}){kind=link}