weiqingy opened a new issue, #1009: URL: https://github.com/apache/flink-agents/issues/1009
### Search before asking - [X] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description `AnthropicChatModelSetup` defaults `json_prefill` to `true` (`AnthropicChatModelSetup.java:77`) and publishes the key on every request (`:143`). When it is enabled and no tools are bound, `AnthropicChatModelConnection.buildRequest` appends a synthetic assistant message whose content is `{` (`AnthropicChatModelConnection.java:223-229`), so the request ends on an assistant turn. That is an assistant-message prefill, and it happens on ordinary chat requests where the caller never asked for JSON. **1. The request fails outright on current models.** From the Messages API guide, section "Prefilling Claude's response": > Prefilling is not supported on Claude 4.6 and later models and Claude Mythos Preview. Requests using prefill with these models return a 400 error. Use structured outputs on models that support it, or system prompt instructions, instead. So any chat through this connection to a Claude 4.6 or later model, with `json_prefill` left at its default and no tools bound, fails before the model is reached. The user sees the wrapped `RuntimeException("Failed to call Anthropic messages API.")` from `AnthropicChatModelConnection.chat`, which does not mention prefill, so the cause is not evident from the error. This is masked today by the setup's default model, `claude-sonnet-4-20250514` (`AnthropicChatModelSetup.java:74`), which predates the cutoff and still accepts prefill. A user who never overrides the model never sees it. The exposure grows with every release: the entire 4.6 generation onward rejects prefill, while the model that keeps the default working is a 2025 snapshot. **2. Prefill is incompatible with structured outputs, and nothing enforces that.** The structured outputs guide lists, under "Feature compatibility" then "Incompatible with": > **Message Prefilling:** Incompatible with JSON outputs A caller can put `output_config` into `additional_kwargs`, which is applied to the request verbatim (`AnthropicChatModelConnection.java:214-216`, `applyAdditionalKwargs` at `:464`), while `json_prefill` sits at its default, producing a request carrying both. **3. Python has no equivalent.** `json_prefill` does not appear anywhere under `python/`. The Python `AnthropicChatModelSetup.model_kwargs` publishes only `model`, `max_tokens` and `temperature` (`anthropic_chat_model.py:309-315`), and the connection sends the caller's messages unmodified. The same agent definition therefore behaves differently depending on which language runs it. `json_prefill` is also the only setup-level parameter in either language that coerces output format by default. Every other structured-output path in the repo is schema-conditional, inert unless the caller passes an `output_schema`, and every comparable boolean defaults to `false`, including `DEFAULT_STRICT_TOOLS` one line below it at `AnthropicChatModelSetup.java:78`. **Proposed contract, identical in both languages.** 1. `json_prefill` defaults to `false`, making it an explicit opt-in. 2. Python gains first-class `json_prefill` support in the setup and the connection. 3. Prefill applies only when all of these hold: the caller explicitly enabled it; no tools are in the request; the request carries no `output_config`, whether derived from an `output_schema` or supplied by the caller; and the effective model supports assistant-message prefilling. 4. The leading `{` is reconstructed on the response only when the request actually carried the prefill. Two implementation notes, because both are easy to get wrong: - **The prefill-capability list is not the structured-output capability list.** Structured output is documented from Claude 4.5 onward; prefill rejection starts at 4.6. The 4.5 generation is therefore structured-output capable *and* prefill-accepting, so a prefill guard cannot reuse a structured-output allowlist without silently disabling prefill across all of 4.5. - **There is no programmatic signal for prefill support.** The Models API exposes a `capabilities.structured_outputs` object but nothing for prefill, so the guard has to be a model-name rule maintained against the documentation. The fix lands in #965 rather than in a separate PR, since that PR already touches every file involved and `json_prefill` interacts directly with the native structured-output path it adds. ### How to reproduce 1. Configure an `AnthropicChatModelConnection` with a valid API key. 2. Configure an `AnthropicChatModelSetup` that sets `model` to any Claude 4.6 or later id, for example `claude-sonnet-4-6`, `claude-opus-4-6`, `claude-sonnet-5`, `claude-opus-5`, `claude-fable-5`, `claude-mythos-5`, and that does **not** set `json_prefill` and does **not** bind any tools. 3. Call `chat(...)` with any user message. Result: HTTP 400 from the provider, surfaced as `Failed to call Anthropic messages API.` Expected: a normal completion. Setting `json_prefill` to `false` explicitly, or binding at least one tool, avoids it. The tool path is already excluded by an existing guard (`AnthropicChatModelConnection.java:224-225`). For the parity half: define the same agent against the Python `AnthropicChatModelSetup` and observe that there is no `json_prefill` parameter to set, and that no prefill is ever sent. ### Version and environment - Flink Agents `0.4-SNAPSHOT` (`main`) - Java Anthropic SDK `2.11.1` (`integrations/pom.xml:38` on `main`) - Java 11, Python 3.10 to 3.12 - Not environment specific, the request is built the same way everywhere. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
