weiqingy opened a new pull request, #1265:
URL: https://github.com/apache/flink-agents/pull/1265
Linked issue: Fixes #1253
### Purpose of change
A schema with an open-keyed field now keeps the prompt path on providers
whose native schema cannot express one, instead of going native and losing the
field. In Java that is a `Map` or `JsonNode` field, rendered as an empty object
allowing no keys, so the model could only return `{}`. In Python it is a `dict`
field or an `extra="allow"` model, which OpenAI and Azure OpenAI strict mode
reject and Anthropic closes to the same empty object.
#### Runtime flow
Each connection's native predicate, which feeds both
`supportsNativeStructuredOutput` and the request, gains one last check:
- **Java:** `PojoJsonSchemaGenerator.reachesAny(type, targets, options)`
runs the victools generator with the connection's native options and reports
whether it visits an instance of a target. Strict providers pass `Map` and
`JsonNode`; Gemini passes `JsonNode`.
- **Python:** `contains_open_object(model)` walks `model_json_schema()` for
an object whose `additionalProperties` is not `false`, or that has
`patternProperties`.
#### Key decisions
- **Java asks the generator, not the rendered schema**, because a rendered
`Map<String,Object>` has no `additionalProperties` key to read. It uses the
generation options, which change what is reached: on Gemini,
`Map<String,JsonNode>` reaches `JsonNode`.
- **Python reads the rendered schema**, where pydantic states openness,
including `extra="allow"`.
- **Fall back, not raise**: the prompt path can honour a map.
- **Gemini keeps `Map` native**: its schema keeps the value type.
- **vLLM splits by language.** Java vLLM reuses the OpenAI renderer, which
closes a `Map` to `{}`. Python vLLM sends pydantic's typed
`additionalProperties`, which vLLM's guided decoding enforces, so its predicate
drops the check.
### Behavioral Semantics
#### Interaction decisions
| Schema contains (any depth) | OpenAI, Azure, Anthropic | vLLM | Gemini
(Java) | Others |
|---|---|---|---|---|
| Only declared fields | unchanged | unchanged | unchanged | unchanged |
| Java `Map` (values not `JsonNode`) | `INFEASIBLE` | `INFEASIBLE` |
unchanged | unchanged |
| Java `JsonNode`, incl. as a `Map` value | `INFEASIBLE` | `INFEASIBLE` |
`INFEASIBLE` | unchanged |
| Python `dict` field or `extra="allow"` | `INFEASIBLE` | unchanged | n/a |
unchanged |
The check adds to the existing blockers; any one gives `INFEASIBLE`.
#### Behavioral contracts
1. Java OpenAI, Azure OpenAI, vLLM and Anthropic answer `INFEASIBLE` for a
schema class reaching a `Map` or `JsonNode`, and send no `response_format` /
`output_config`.
2. Java Gemini answers `INFEASIBLE` for one reaching a `JsonNode`, and sends
no response schema. A `Map` of other values is unchanged.
3. Python OpenAI, Azure OpenAI and Anthropic answer `INFEASIBLE` for a model
with a `dict` field or `extra="allow"` at any depth, and send no native
parameter.
4. Python vLLM sends such a model natively, keeping its
`additionalProperties`.
5. A blocked schema uses the prompt path under `AUTO` and `PROMPT`, and
raises before the call under `NATIVE`.
6. Anthropic keeps a requested JSON prefill for a blocked schema (Java and
Python).
7. A `@JsonIgnore`d `Map` field does not block (Java).
8. Every other schema is unchanged on every provider.
#### Failure behavior
- `NATIVE` with a blocked schema raises the existing early error naming the
connection and schema.
- Python: a model whose `model_json_schema()` raises is not reported open,
so the render step still raises its `TypeError`.
- Java: the query now runs the generator, uncaught, under every strategy,
`PROMPT` included.
### Tests
| Contract | Tests |
|---|---|
| 1 | `testNativeNotAppliedForOpenObjectSchema` (OpenAI, Azure),
`testOpenObjectSchemaKeepsFallback` (Anthropic),
`testQueryRecommendsNativeForServedModels` (vLLM),
`testQueryAgreesWithTheNativeBranch` |
| 2 | `nativeSchemaSkippedForJsonNode`, `queryAgreesWithTheNativeBranch`
(Gemini); `reachesMapValuesOnlyUnderTheirOption` |
| 3 | `test_output_schema_render.py`;
`test_schema_admitting_undeclared_keys_keeps_the_prompt_path`,
`test_query_agrees_with_the_native_branch` |
| 4 | `test_map_schema_is_sent_natively_with_its_value_type` |
| 5 | Existing strategy tests |
| 6 | `testOpenObjectSchemaKeepsFallback`;
`test_json_prefill_applied_when_open_schema_falls_back` |
| 7 | `doesNotReachTypesTheSchemaLeavesOut` |
| 8 | Unchanged rows of each agreement matrix; the suites below |
Tests pinning a map schema as native on a blocked provider now pin the
prompt path. Each new explicit test fails with its gate or override reverted;
the agreement tests pass either way, as query and request move together.
Run, all passing: `mvn test` on
`integrations/chat-models/{common,openai,anthropic,gemini,ollama,watsonx,bedrock}`
with `-am`; `pytest -m "not integration"` on
`flink_agents/{api,plan,runtime,integrations/chat_models}`; `ruff check`;
Spotless; `./tools/check-license.sh`.
Not verified:
- No live provider call; responses follow documented strict-mode rules.
- vLLM's acceptance of a typed `additionalProperties` comes from its
guided-decoding source, not a running server.
- Whether Gemini enforces a `Map`'s typed values.
- A Python `Any` field is not flagged (it renders with no `type`), though
Java's `JsonNode` is.
<details>
<summary>Implementation invariants and supporting detail</summary>
- The generator skips `@JsonIgnore`d fields and includes subtypes, as the
native schema does. The new check composes with the Azure api-version,
Anthropic caller `output_config` and Gemini tools blockers as one more conjunct.
- Python vLLM does not apply OpenAI's strict rules; its guided-decoding
backend (xgrammar) lists no `additionalProperties` among its unsupported JSON
features.
- `reachesAny` adds a custom definition provider that records an
`isInstanceOf` match and returns `null`, so the generated schema is identical
to `generate`'s. `generate` and `reachesAny` share one private `configure`.
- Each Java connection keeps one `NATIVE_SCHEMA_OPTIONS` constant used by
both generation and the query, so the two cannot drift. The cheap checks run
first in every `canApply…`, and `reachesAny` runs only for a `Class` schema.
- Recursive classes terminate: victools emits a `$ref` for a type it has
already visited.
- `contains_open_object` follows only keywords that hold subschemas
(`properties` and `$defs` values, `items`, `prefixItems`, `anyOf`, `oneOf`,
`allOf`, `not`), so a field named `additionalProperties` does not count.
- Python vLLM's override returns `_native_output_model(output_schema) is not
None`, the OpenAI predicate without the check; its request path reads the same
predicate.
- `JsonNode` on Ollama, watsonx and Bedrock is unchanged and not examined.
- Suite sizes: common 11, OpenAI 152, Anthropic 104, Gemini 71, api 680,
Ollama 35, watsonx 52, Bedrock 90; Python 2157 passed, 15 skipped.
</details>
### API
No public API is removed or changed. Added helpers:
`PojoJsonSchemaGenerator.reachesAny` and
`flink_agents.api.agents.types.contains_open_object`.
Compatibility: under `AUTO`, a schema with an open-keyed field moves from
native to the prompt path on OpenAI, Azure OpenAI, Anthropic and Java vLLM, as
does a `JsonNode` on Gemini. Under `NATIVE` it fails before the call, instead
of returning `{}` (Java, Python Anthropic) or a provider 400 (Python OpenAI,
Azure OpenAI).
### Documentation
- [ ] `doc-needed` <!-- Your PR changes impact docs -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [x] `doc-included` <!-- Your PR already contains the necessary
documentation updates -->
### Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
Generated-by: GitHub Copilot CLI 1.0.98 (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
