weiqingy commented on PR #965:
URL: https://github.com/apache/flink-agents/pull/965#issuecomment-5277367341
> I think `json_prefill` should be an explicit opt-in and have the same
contract in Java and Python.
>
> Currently Java defaults it to `true`, while the Python Anthropic
integration does not implement `json_prefill` at all. This creates both a
surprising default for ordinary Java chat requests and a Java/Python behavior
difference.
>
> Could we align both implementations as follows?
>
> 1. Default `json_prefill` to `false`.
> 2. Add first-class `json_prefill` support to the Python setup and
connection.
> 3. Apply prefill only when all of these conditions hold:
>
> * the caller explicitly enabled it;
> * no tools are present;
> * the final request contains no `output_config`, whether derived from
`output_schema` or supplied by the caller;
> * the effective model supports assistant-message prefilling.
> 4. Reconstruct the leading `{` only when the request actually carried the
prefill.
>
> The model check is important: Anthropic documents that Claude 4.6 and
later models, as well as Mythos Preview, reject assistant-message prefill with
a 400 because the request must end with a user message:
https://platform.claude.com/docs/en/build-with-claude/working-with-messages#prefilling-claudes-response
>
> This means the Java implementation also needs an additional
model-capability guard. For example, a schema-free request to Claude 4.6
currently retains the default prefill and can be rejected even though no native
schema is involved.
>
> The caller-supplied `output_config` case should also suppress prefill.
`testCallerOutputConfigWinsOverSchema` currently expects `jsonPrefillApplied`
to remain `true`, but `output_config.format` and message prefilling are
incompatible, so that expectation should be reversed.
>
> Suggested parity tests:
>
> * absent/default value does not apply prefill;
> * an explicit `true` applies it on a supported model;
> * tools suppress it;
> * any caller- or framework-supplied `output_config` suppresses it;
> * Claude 4.6+, Claude 5, and Mythos models suppress or reject it
consistently;
> * response reconstruction prepends `{` only when prefill was actually
applied.
@wenjin272 Agreed on all four, and thanks for catching this. The prefill
check only looked at whether we had added an `output_config` ourselves, so it
missed the one a caller supplies.
There was one more case than you named. That check sat inside the
`outputSchema instanceof Class` branch, so passing `output_config` with no
schema at all still got the prefill. Fixed as well.
All four are in the PR now. The default is `false`, Python has the
parameter, we only prefill on models that accept it, and we only add the `{`
back when we actually sent it. Your six tests are in too, both languages where
they apply.
One call I would like your view on. The prefill gate has its own model list
instead of reusing the structured-output one, because the two do not line up.
4.5 supports structured output but still accepts prefill, and only 4.6 onward
rejects it, so sharing a list would quietly switch prefill off for all of 4.5.
There is also nothing to query for it, the Models API has
`capabilities.structured_outputs` but nothing for prefill, so it comes down to
a hand-maintained list. Does that sound right? wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
