shreemaan-abhishek commented on code in PR #13735:
URL: https://github.com/apache/apisix/pull/13735#discussion_r3673168219
##########
apisix/plugins/ai-aws-content-moderation.lua:
##########
@@ -257,4 +357,62 @@ function _M.access(conf, ctx)
end
end
+
+function _M.lua_body_filter(conf, ctx, headers, body)
+ if not conf.check_response then
+ core.log.info("skip response check for this request")
+ return
+ end
+
+ if ngx.status >= 400 then
+ core.log.info("skip response check because upstream returned error
status: ", ngx.status)
+ return
+ end
+
+ local request_type = ctx.var.request_type
+
+ -- ai-proxy hands us the fully assembled completion, so one check covers
it.
+ if request_type == "ai_chat" then
+ return moderate_response(ctx, conf, ctx.var.llm_response_text)
+ end
+
+ if request_type ~= "ai_stream" then
+ return
+ end
+
+ if conf.stream_check_mode == "final_packet" then
+ -- llm_response_text only appears once the stream is assembled, so
+ -- earlier chunks pass through untouched.
+ if not ctx.var.llm_response_text then
+ return
+ end
+ if not ctx.aws_cm_response_moderated then
+ ctx.aws_cm_response_moderated = true
+ moderate_response(ctx, conf, ctx.var.llm_response_text)
+ end
+ return nil, annotate_stream(ctx, body)
+ end
+
+ -- realtime: moderate batches as they arrive so a hit can cut the stream
off
+ ctx.aws_cm_cache = ctx.aws_cm_cache or ""
+ ctx.aws_cm_cache = ctx.aws_cm_cache
+ .. table.concat(ctx.llm_response_contents_in_chunk or
{}, "")
Review Comment:
Good catch, confirmed. When a converter fans one upstream chunk into N
downstream chunks, `lua_body_filter` runs N times and re-reads the same
`llm_response_contents_in_chunk`, so the realtime cache double-counts and
`stream_check_cache_size` trips early. Moderation still catches everything, but
Comprehend request volume scales with fan-out. `ai-aliyun-content-moderation`
has the same shape. Tracked in api7/api7-ee-3-gateway#2053 and will be fixed in
a follow-up PR across both plugins and both repos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
