weiqingy opened a new pull request, #981: URL: https://github.com/apache/flink-agents/pull/981
Linked issue: #280 ### Purpose of change Ollama has constrained decoding to a JSON schema since server v0.5.0, through the `format` request field. This connection refused it: Java inherited the base 4-arg `chat`, which throws `UnsupportedOperationException` for a non-null schema, and Python called `_reject_unsupported_output_schema`, which raises `NotImplementedError`. Callers were held on the prompt-engineering fallback, where the model is asked in prose for conforming JSON and the reply is scraped for code fences. This adds the native path in both languages, following the OpenAI, Azure and Anthropic connections. #### Runtime flow Java: `chat` delegates to `doChat`, which calls the new package-private `buildRequest` inside its existing try block and then issues the call. `buildRequest` assembles the request as before, and if the schema is a `Class` and the capability predicate agrees, derives a JSON schema with victools and sets it as the request's `format`. The seam is new, so it is extracted in a separate commit with no behavior change. Python: `chat` converts messages and tools, pops `model`, then consults the predicate and `_native_format`. A translated schema becomes a `format=` argument spliced into the SDK call. With none, no `format` key is passed. #### Key decisions The capability predicate reports capable for every model, in both languages. Ollama applies the grammar in the server's sampler rather than the model, and exposes no model-level signal to key on: `/api/show` reports a capability set carrying nothing schema-related, and `/api/version` reports only a version string. A server also runs arbitrary local models, so an allowlist of the kind the hosted providers use would be invented, and would report not-capable for models that work. Probing at `open()` was rejected because it detects only one of the three deployments that break the guarantee. Java derives the schema itself with victools, the first connection here not to hand a class to a provider SDK. `ObjectMapper.generateJsonSchema` needs no dependency and already serves the prompt fallback, but it is deprecated and draft-03 flavored, which would put the two languages on different dialects. victools moves from transitive runtime to direct compile scope. It already ships in the distribution through two provider SDKs, so no artifact is added and no notice changes. Its version comes from importing the victools BOM: a version on the module declaration would win nearest-wins mediation and become the global victools version while pinning only one of its three artifacts, letting the siblings drift on a later SDK upgrade with no build warning. ### Implementation Description #### Behavioral contracts 1. A POJO `Class` in Java, or an `OutputSchema` wrapping a `BaseModel` in Python, is sent as the request's native `format`. 2. Any other schema form, notably a `RowTypeInfo`, leaves `format` unset and keeps the prompt-engineering fallback. 3. Without a schema the request is what it was before this change. `format` is omitted from the body rather than sent as null. 4. The Java schema declares draft 2020-12, keeps properties in declaration order, gives a `Map` its value schema, excludes accessor-derived properties, and marks every field required except an `Optional` one. 5. The Python payload is `model_json_schema()` verbatim, `$defs` included. 6. The schema travels as a request field, never as a sampling option. 7. Capability is reported for every model name, including an absent or empty one. 8. A schema this connection cannot translate natively is answered rather than refused. #### Failure behavior A server below v0.5.0 rejects `format` with HTTP 400. Java rewraps it as `RuntimeException` through the existing catch, Python propagates the SDK error. This is the intended loud failure. Ollama Cloud accepts the request without enforcing the schema, and the MLX runner accepts the field and discards it. Both return an unconstrained response and no error, and neither is distinguishable from a model name, so neither is detected. The escape hatch for all three is `structured_output_strategy: PROMPT`, which is not yet dispatched. A recursive Java class generates a self reference the server rejects with HTTP 400. The victools option that rewrites it is deliberately unset: it makes the server drop the grammar silently for any class carrying a nested type twice, trading a loud failure on a rare shape for a silent one on a common shape. A `BaseModel` pydantic cannot render raises instead of falling back, matching the merged OpenAI and Azure connections. An explicit `NATIVE` strategy is not visible at this layer, so a caller requesting it for a schema form this branch skips receives an unconstrained response rather than an error. A `TODO(#912)` marks the site in both languages. ### Tests | Contract | Tests | |---|---| | 1 | `buildRequestSetsFormatForPojoSchema`, `test_native_applied_for_base_model` | | 2 | `buildRequestLeavesFormatUnsetForRowTypeInfo`, `test_native_not_applied_for_row_type_info` | | 3 | `buildRequestOmitsFormatWithoutSchema`, `test_format_absent_without_schema` | | 4 | `generatedSchemaShapeIsConstraining` | | 5 | `test_schema_is_model_json_schema` | | 6 | `test_schema_not_passed_as_sampling_option`. No Java test: `setFormat` is a typed request field, so there is no equivalent way to misroute it | | 7 | `supportsNativeStructuredOutputIsServerNotModelGated`, `test_supports_native_structured_output`, both parameterized over several names plus null and empty | | 8 | `test_schema_accepted_not_rejected`. Java is covered by contract 2's test, which returns a request rather than throwing | The Java tests are this connection's first, asserting the built request through `buildRequest` with no network and no mocking framework. The Python tests assert the SDK call kwargs against a mocked client. Both run in the unit arm. The Java `RowTypeInfo` test substitutes a non-`Class` object exercising the same gate, because Flink is a `provided` dependency of the api module and so is off this module's test classpath. The Python test uses a real `RowTypeInfo`. ### API `supportsNativeStructuredOutput` and the 4-arg `chat` are existing extension points, overridden here. `buildRequest` is package-private, and `toNativeFormat` and `_native_format` are private. For a caller that does nothing differently, nothing changes. No framework path passes a schema to a connection today, so the native branch is reachable only from a direct call, and a request without a schema is byte-identical to before. For a caller that does pass one, a previously raised error becomes a constrained response for a POJO or `BaseModel`, and a silent prompt fallback for any other form. victools changes scope from runtime to compile in dependent modules. Resolved versions are unchanged and no packaged bytes change. ### Documentation - [ ] `doc-needed` - [x] `doc-not-needed` - [ ] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No `Generated-by: Claude Code 2.1.226` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
