AlinsRan commented on code in PR #13487:
URL: https://github.com/apache/apisix/pull/13487#discussion_r3385965442
##########
apisix/plugins/prometheus/exporter.lua:
##########
@@ -366,16 +416,36 @@ function _M.http_log(conf, ctx)
gen_arr(route_id, service_id, consumer_name, balancer_ip,
vars.request_type, vars.request_llm_model, vars.llm_model,
unpack(extra_labels("llm_latency", ctx))))
+
+ -- Only streaming requests expose a real TTFT; for non-streaming
the
+ -- var holds the total response time, which would pollute the TTFT
+ -- distribution, so record llm_ttft for ai_stream only.
+ if vars.request_type == "ai_stream" then
+ metrics.llm_ttft:observe(tonumber(llm_time_to_first_token),
+ gen_arr(route_id, service_id, consumer_name, balancer_ip,
+ vars.request_type, vars.request_llm_model,
vars.llm_model,
+ unpack(extra_labels("llm_ttft", ctx))))
+ end
Review Comment:
`apisix_llm_latency`:
- `type="total"` — full response latency (`ai_chat` and `ai_stream`). For
streaming this now uses `apisix_upstream_response_time` (time until the whole
response completes) instead of the TTFT.
- `type="ttft"` — time to first token (`ai_stream` only; non-streaming
responses have no first-token moment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
