janiussyafiq opened a new pull request, #13370: URL: https://github.com/apache/apisix/pull/13370
### Description Extract the instance-override application logic — currently inline inside `build_request` in `apisix/plugins/ai-providers/base.lua` — into a new pure helper `apply_instance_overrides(request_body, ai_instance, ai_provider, target_protocol)` in `apisix/plugins/ai-proxy/base.lua`. `build_request` now delegates to the helper at the same call site the inline code lived (after the converter), so the body sent upstream is identical. **Motivation: prerequisite for a planned `ai-cache` plugin.** ai-cache needs to compute its cache key from the *effective* request body — i.e., what would actually be sent to the LLM after instance overrides apply. Without this helper, ai-cache would have to either: 1. Re-implement the override-application formula in its own module (duplication and drift risk), or 2. Use a cache key that ignores operator-configured overrides (correctness gap: two clients sending different `model` fields that get overridden to the same value would not share a cache entry, and changing `override.request_body` would not bust existing entries). By extracting the formula into a pure callable, ai-cache can compute the effective body without invoking `build_request` (which makes the upstream HTTP call, performs signing, and manages keepalive). The signature is slightly widened from the natural `(body, instance)` shape: it also takes `ai_provider` and `target_protocol` because the `override.llm_options` step dispatches to a provider-specific capability hook (`rewrite_request_body`) keyed by target protocol. The helper itself remains pure — no `ctx` access, no I/O beyond a couple of `core.log.info` lines that already existed in the original inline code. #### Which issue(s) this PR fixes: N/A — refactor. ### Behavior change None. This is a strict refactor: - `model_options` flat overwrite, `override.llm_options` capability-hook rewrite, and `override.request_body[target_protocol]` deep merge are applied in the same order at the same call site. - `build_request` retains the converter step before the helper call, so per-target-protocol patches still apply to the post-conversion body. - `extra_opts` is slimmed: the four override-derived fields (`model_options`, `override_llm_options`, `request_body_override_map`, `request_body_force_override`) are removed and replaced with a single `ai_instance` pass-through, since the helper now reads them from the instance directly. ### Tests **No new tests added** — the existing `t/plugin/ai-proxy-request-body-override.t` suite (16 test blocks, 48 assertions) already characterizes every override mechanism this refactor touches: - TESTS 1–2: schema rejection paths. - TESTS 3–7: `llm_options` per-provider field translation (`max_tokens` → `max_completion_tokens` for openai-chat, `max_output_tokens` for openai-responses, etc.). - TESTS 8–15: `request_body` deep-merge semantics, force-override flag, per-target-protocol routing, post-converter application. - TEST 16: `llm_options` + `request_body` interaction. These tests use the established "echo upstream" pattern that asserts the exact body forwarded by ai-proxy — which is precisely what `apply_instance_overrides` is now responsible for producing. They provided concrete coverage for the refactor: an earlier draft of this change called the helper from `before_proxy` (before the converter), which was caught by TEST 14, which asserts that `override.request_body` applies to the post-conversion target-protocol body, not the pre-conversion client-format body. The final placement — calling the helper from `build_request` immediately after the converter — is structurally equivalent to the inline code it replaces. Verification: - `prove -I../test-nginx/lib -I./ t/plugin/ai-proxy-request-body-override.t` — 48/48 pass. - Broader sanity: 17 ai-proxy test files (677 tests) — all pass. - `make lint` — luacheck and lj-releng both clean. ### Checklist - [x] I have explained the need for this PR and the problem it solves - [x] I have explained the changes or the new features added to this PR - [ ] I have added tests corresponding to this change — N/A for a behavior-preserving refactor; existing tests cover the contract end-to-end. - [ ] I have updated the documentation to reflect this change — N/A, no user-visible surface change. - [x] I have verified that this change is backward compatible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
