Copilot commented on code in PR #14028:
URL: https://github.com/apache/apisix/pull/14028#discussion_r4246090347
##########
apisix/plugins/ai-protocols/anthropic-messages.lua:
##########
@@ -29,10 +29,49 @@ local sse = require("apisix.plugins.ai-transport.sse")
local table = table
local type = type
local ipairs = ipairs
+local pairs = pairs
local _M = {}
+-- Anthropic reports cache tokens alongside input_tokens rather than inside it,
+-- so add them back to get a prompt total with the same meaning as OpenAI's
+-- prompt_tokens. Mirrors LiteLLM's AnthropicConfig.calculate_usage.
+local function to_usage(raw)
+ local cache_read = raw.cache_read_input_tokens or 0
+ local cache_creation = raw.cache_creation_input_tokens or 0
+ local prompt = (raw.input_tokens or 0) + cache_read + cache_creation
+ local completion = raw.output_tokens or 0
+ return {
+ prompt_tokens = prompt,
+ completion_tokens = completion,
+ total_tokens = prompt + completion,
+ cache_read_input_tokens = cache_read,
+ cache_creation_input_tokens = cache_creation,
+ reasoning_tokens = 0,
+ }
+end
+
+
+-- message_delta usage is cumulative but may omit fields reported at
+-- message_start (and some providers only report cache tokens there), so
+-- accumulate the raw fields across events before computing usage.
+local function accumulate_usage(state, raw)
+ if type(state) ~= "table" then
+ return raw
+ end
+ local acc = state.anthropic_usage
+ if not acc then
+ acc = {}
+ state.anthropic_usage = acc
+ end
+ for k, v in pairs(raw) do
+ acc[k] = v
+ end
+ return acc
+end
Review Comment:
The function name/comment says “accumulate”, but the implementation is
actually a field-wise merge/overwrite (last value wins) rather than numeric
accumulation. To prevent future contributors from “fixing” it into a sum (which
would break cumulative usage semantics), either rename it to something like
`merge_usage_fields`/`coalesce_usage_fields` or adjust the comment to
explicitly state it overwrites fields to preserve the latest cumulative values.
##########
apisix/plugins/ai-protocols/converters/anthropic-messages-to-openai-chat.lua:
##########
@@ -102,6 +102,33 @@ end
-- SSE event helpers
+-- prompt_tokens_details keys that carry cache-write tokens, in LiteLLM's
+-- order, followed by the cache_creation_input_tokens shape some upstreams use.
+local CACHE_CREATION_DETAIL_KEYS = {
+ "cache_creation_tokens",
+ "cache_write_tokens",
+ "cache_creation_input_tokens",
+}
+
+
+local function cache_creation_tokens(usage)
+ local v = usage.cache_creation_input_tokens
+ if type(v) == "number" and v > 0 then
+ return v
+ end
+ local details = usage.prompt_tokens_details
+ if type(details) == "table" then
+ for _, key in ipairs(CACHE_CREATION_DETAIL_KEYS) do
+ v = details[key]
+ if type(v) == "number" and v > 0 then
+ return v
+ end
+ end
+ end
+ return 0
+end
Review Comment:
`cache_creation_tokens` will raise an error if it’s ever called with a
non-table `usage` (e.g., `nil`). Adding an early `if type(usage) ~= "table"
then return 0 end` guard would make this helper safer and easier to reuse
without relying on all call sites to pre-validate the shape.
##########
docs/zh/latest/plugins/prometheus.md:
##########
@@ -321,6 +321,8 @@ APISIX 在开始 LLM 上游尝试时递增该 gauge，并在请求的日志阶
### `apisix_llm_prompt_tokens` 的标签
+prompt token 包含从缓存读取和写入缓存的 token。对于 Anthropic Messages API，其值为 `input_tokens +
cache_read_input_tokens + cache_creation_input_tokens`，与 OpenAI 的
`prompt_tokens` 含义一致。
Review Comment:
The Chinese docs line mixes singular/plural (“prompt token”) and English
“token” wording in a way that’s inconsistent with surrounding terminology (“令牌”
is used elsewhere in this doc). Consider revising to consistently use either
“prompt tokens” or “提示令牌/提示 tokens”, and keep the cache read/write description
consistent across zh docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
