nic-6443 opened a new pull request, #13876: URL: https://github.com/apache/apisix/pull/13876
### Description When an LLM upstream returns `200` with a streaming content type and at least one SSE event has already been forwarded downstream, a later `body_reader()` failure (`closed`, `timeout`) was still converted unconditionally into `500`/`504`. The downstream response is already committed as `200` at that point, so the status cannot be changed. Two things go wrong: - `ai-proxy` returns the 5xx to the plugin runner, which calls `core.response.exit()` on the committed `200` and logs `attempt to set ngx.status after sending out response headers`. - `ai-proxy-multi` treats the 5xx as retryable and sends another billable LLM request after the client has already received partial output. `retry_on_error()` documents the assumption that "the failed attempt's body never reaches the client", which does not hold on this path. The broken upstream connection was also handed to `set_keepalive()` instead of being closed. `parse_streaming_response()` already tracks `output_sent` for the EOF branch. This uses it on the error branch as well, mirroring what the `max_stream_duration_ms` / `max_response_bytes` path already does once output has been committed: stop reading, close the upstream connection, run one last `body_filter` pass with `llm_request_done` set so plugins that buffer the whole stream flush what they hold, and return no status. No terminator is synthesized, so the client detects the truncation from the missing `[DONE]` / `message_stop` / `response.completed`. A read error before the first downstream byte is unchanged: still `504` for a timeout and `500` otherwise, and still eligible for fallback. A read error that arrives after the protocol's completion event is treated as a complete response rather than an aborted one. Setting `llm_request_done` on that last pass is also what would make `ai-aws-content-moderation` and `ai-aliyun-content-moderation` append a completion event to a stream that was cut short, reporting a truncated response as complete. That synthesis is now gated on `ctx.ai_stream_aborted`, which already marks every cut-short stream, so it also stops firing on the pre-existing stream-limit path. ### Tests `t/plugin/ai-proxy-stream-truncated.t` drives a raw upstream that commits `200` with `Content-Type: text/event-stream`, flushes one valid SSE event, then closes before the terminating chunk and before the completion event. It asserts that the client keeps `200` and the delivered event, that no status change is attempted after the headers are sent, that the broken connection is not offered to keepalive, that no fallback request is sent after partial output, that no `[DONE]` is synthesized, and that a read error before the first downstream byte still falls back. ### Checklist - [x] I have explained the need for this PR and the problem it solves - [x] I have explained the changes or the new features added to this PR - [x] I have added tests corresponding to this change - [x] I have updated the documentation to reflect this change - [x] I have verified that this change is backward compatible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
