janiussyafiq commented on PR #13793: URL: https://github.com/apache/apisix/pull/13793#issuecomment-5261211621
Tested the deepcopy overhead on both axes: string length and message count. **Long strings are free to copy.** LuaJIT strings are immutable, so `deepcopy` copies them by reference and never touches the bytes. The copy cost grows only with the number of messages in the body. For scale, here it is next to the `cjson.decode` that ai-proxy already runs on the same body every request: | body | deepcopy (new cost) | cjson.decode (already paid today) | |---|---|---| | 1 msg, 1KB string | 0.25 µs | 1.2 µs | | 1 msg, 1MB string | 0.21 µs | 668 µs | | 1 msg, 10MB string | 0.22 µs | 6835 µs | | 100 msgs x 1KB | 10 µs | 84 µs | | 1000 msgs x 1KB | 110 µs | 823 µs | Even in the worst case the copy is a fraction of the JSON parse we already do for the same body, so it cannot move total request cost by more than that fraction. **E2E check**: wrk at full saturation against a local 1-worker APISIX with an ai-proxy-multi route to a mock LLM upstream. Each number is the mean of 10 x 15s runs, interleaved and spread across two instances to cancel machine-side bias. | body | req/s without the change | req/s with the change | delta | |---|---|---|---| | 1 msg x 1KB | 14802 | 14682 | -0.8% (within noise) | | 1 msg x 128KB | 2807 | 2790 | -0.6% (within noise) | | 1 msg x 1MB | 418 | 409 | -2.3% (within noise) | | 200 msgs x 1KB | 1789 | 1624 | -9.2% | Single-message bodies show no measurable difference regardless of string size. The only visible case is the 200-message body with the worker CPU fully saturated: ~+53 µs per request, mostly GC churn from the 200 copied message tables. At normal load, +53 µs on an LLM request that takes hundreds of ms is negligible. If the many-messages-at-saturation case ever matters, a copy-on-write follow-up could reclaim it, but the numbers do not seem to warrant the complexity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
