nic-6443 opened a new pull request, #13251:
URL: https://github.com/apache/apisix/pull/13251
### Description
Adds `override.request_body` to `ai-proxy` and `ai-proxy-multi`, letting
operators set arbitrary nested fields on the outgoing request body, keyed by
target protocol.
The existing `options` field can only overwrite top-level fields and is
protocol-agnostic, so it cannot express protocol-specific params such as
`max_tokens` vs `max_output_tokens` vs `generationConfig.maxOutputTokens`. This
new field fills that gap without breaking any existing behaviour.
### Design notes
- **Keyed by target protocol**, not client protocol. Converters only do
structural format conversion (message layout, SSE event shape); they do not
semantically normalize parameter names across protocols. So an override like
\`max_output_tokens\` cannot be matched to the client protocol and expected to
reach the upstream as-is. The override therefore applies after the converter,
matched against the protocol actually sent to the provider.
- **Reuses the existing \`override\` field** instead of introducing a
parallel \`override_request\`. \`override.endpoint\` and
\`override.request_body\` are both \"tweaks to the outgoing HTTP request\".
- **Deep merge for objects, replace for arrays/scalars**, no
\`null\`-deletion. I walked through every array-shaped parameter in the major
AI APIs (OpenAI Chat/Responses/Embeddings \`tools\`/\`stop\`/\`include\`,
Anthropic \`stop_sequences\`/\`system\` blocks, Gemini
\`safetySettings\`/\`contents\`/\`tools.functionDeclarations\`) and for every
case where an operator would actually want to override, the semantics is \"my
list is the list\" — so wholesale array replace is both simplest and the only
predictable semantics.
- **Applied after \`options\`**, so if the same key is set in both,
\`request_body\` wins (more specific beats less specific).
- **Protocol enum is dynamic**: generated from the registered table in
\`ai-protocols/init.lua\`, so new protocols added later are picked up
automatically.
### Changes
| File | Change |
|---|---|
| \`apisix/plugins/ai-proxy/merge.lua\` | New: pure-function deep_merge
helper |
| \`apisix/plugins/ai-proxy/schema.lua\` | New \`override.request_body\`
schema; shared \`override_schema\` DRY-up |
| \`apisix/plugins/ai-proxy/base.lua\` | Read \`override.request_body\` into
\`extra_opts\` |
| \`apisix/plugins/ai-providers/base.lua\` | Deep-merge the matching entry
into outgoing body after model_options |
| \`apisix/plugins/ai-protocols/init.lua\` | Expose \`names()\` for schema
enum generation |
| \`t/plugin/ai-proxy-request-body-override.t\` | 9 scenarios: schema
reject, chat override, deep-merge, array replace, precedence over options,
non-matching protocol key ignored, per-instance in ai-proxy-multi, converter
path (client anthropic → target openai-chat), backward compat |
| \`docs/{en,zh}/latest/plugins/ai-proxy*.md\` | New field documented in
both tables |
### Example
\`\`\`yaml
plugins:
ai-proxy:
provider: openai
auth: { header: { Authorization: \"Bearer ...\" } }
override:
request_body:
openai-chat: { max_tokens: 500, temperature: 0.2 }
openai-responses: { max_output_tokens: 500 }
anthropic-messages: { max_tokens: 500, stop_sequences: [\"Human:\"] }
\`\`\`
### Checklist
- [x] Done in a [PR to main branch](CONTRIBUTING.md)
- [x] Unit tests added (t/plugin/ai-proxy-request-body-override.t)
- [x] Documentation (both en and zh)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
