wenjin272 commented on PR #965:
URL: https://github.com/apache/flink-agents/pull/965#issuecomment-5265011561
I think `json_prefill` should be an explicit opt-in and have the same
contract in Java and Python.
Currently Java defaults it to `true`, while the Python Anthropic integration
does not implement `json_prefill` at all. This creates both a surprising
default for ordinary Java chat requests and a Java/Python behavior difference.
Could we align both implementations as follows?
1. Default `json_prefill` to `false`.
2. Add first-class `json_prefill` support to the Python setup and connection.
3. Apply prefill only when all of these conditions hold:
- the caller explicitly enabled it;
- no tools are present;
- the final request contains no `output_config`, whether derived from
`output_schema` or supplied by the caller;
- the effective model supports assistant-message prefilling.
4. Reconstruct the leading `{` only when the request actually carried the
prefill.
The model check is important: Anthropic documents that Claude 4.6 and later
models, as well as Mythos Preview, reject assistant-message prefill with a 400
because the request must end with a user message:
https://platform.claude.com/docs/en/build-with-claude/working-with-messages#prefilling-claudes-response
This means the Java implementation also needs an additional model-capability
guard. For example, a schema-free request to Claude 4.6 currently retains the
default prefill and can be rejected even though no native schema is involved.
The caller-supplied `output_config` case should also suppress prefill.
`testCallerOutputConfigWinsOverSchema` currently expects `jsonPrefillApplied`
to remain `true`, but `output_config.format` and message prefilling are
incompatible, so that expectation should be reversed.
Suggested parity tests:
- absent/default value does not apply prefill;
- an explicit `true` applies it on a supported model;
- tools suppress it;
- any caller- or framework-supplied `output_config` suppresses it;
- Claude 4.6+, Claude 5, and Mythos models suppress or reject it
consistently;
- response reconstruction prepends `{` only when prefill was actually
applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
