shreemaan-abhishek opened a new pull request, #13767: URL: https://github.com/apache/apisix/pull/13767
### Description AWS Comprehend's [`DetectToxicContent`](https://docs.aws.amazon.com/comprehend/latest/APIReference/API_DetectToxicContent.html) takes **at most 10 text segments per call, each at most 1 KB**, with the list capped at 10 KB. Over that it answers `TextSizeLimitExceededException` (400). `ai-aws-content-moderation` sent the whole prompt, or the whole completion, as a single segment. Consequences on content longer than ~1 KB, which is most real traffic: - Request side: Comprehend returns 400, the plugin turns that into a `500`, so long prompts are unusable. - Response side: the error is logged and the content passes through, so `check_response` silently does nothing. - On that error path the verdict is never computed, yet `final_packet` still annotates the SSE stream with `risk_level`, reporting the request-side (or stale) verdict as if the response had been scored. - `realtime` mostly stays under the cap with the default batch size, but an interval-triggered batch, or a single large upstream SSE event, pushes one batch over 1 KB and that batch leaks. Changes: - Content is split on UTF-8 character boundaries into segments bounded by the new `request_check_length_limit` / `response_check_length_limit` (default `1000` bytes, capped at Comprehend's 1 KB), and the segments are batched up to the per-call limits, so a long body costs as few round trips as possible. - On a Comprehend failure the stream is left without a `risk_level` instead of carrying a verdict that was never computed. - The credentials and the Comprehend client were rebuilt on every call — in `realtime` that is a fresh client per batch. They are now built once per request, and `timeout` / `keepalive` / `keepalive_timeout` are exposed so the connection is reused instead of a new TLS handshake per batch. The Comprehend mock in the tests now enforces the same input limits as the real service and answers with one result per segment, so content that is sent unsplit fails the tests. #### Which issue(s) this PR fixes: N/A ### Checklist - [x] I have explained the need for this PR and the problem it solves - [x] I have explained the changes or the new features added to this PR - [x] I have added tests corresponding to this change - [x] I have updated the documentation to reflect this change - [x] I have verified that this change is backward compatible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
