AlinsRan commented on code in PR #14028:
URL: https://github.com/apache/apisix/pull/14028#discussion_r4247003519
##########
apisix/plugins/ai-protocols/anthropic-messages.lua:
##########
@@ -29,10 +29,49 @@ local sse = require("apisix.plugins.ai-transport.sse")
local table = table
local type = type
local ipairs = ipairs
+local pairs = pairs
local _M = {}
+-- Anthropic reports cache tokens alongside input_tokens rather than inside it,
+-- so add them back to get a prompt total with the same meaning as OpenAI's
+-- prompt_tokens. Mirrors LiteLLM's AnthropicConfig.calculate_usage.
+local function to_usage(raw)
+ local cache_read = raw.cache_read_input_tokens or 0
+ local cache_creation = raw.cache_creation_input_tokens or 0
+ local prompt = (raw.input_tokens or 0) + cache_read + cache_creation
+ local completion = raw.output_tokens or 0
+ return {
+ prompt_tokens = prompt,
+ completion_tokens = completion,
+ total_tokens = prompt + completion,
+ cache_read_input_tokens = cache_read,
+ cache_creation_input_tokens = cache_creation,
+ reasoning_tokens = 0,
+ }
+end
+
+
+-- message_delta usage is cumulative but may omit fields reported at
+-- message_start (and some providers only report cache tokens there), so
+-- accumulate the raw fields across events before computing usage.
+local function accumulate_usage(state, raw)
+ if type(state) ~= "table" then
+ return raw
+ end
+ local acc = state.anthropic_usage
+ if not acc then
+ acc = {}
+ state.anthropic_usage = acc
+ end
+ for k, v in pairs(raw) do
+ acc[k] = v
+ end
+ return acc
+end
Review Comment:
Renamed it to `merge_usage_fields` in 9775c8415. The comment now says that
later events overwrite fields instead of adding to them, because the values are
cumulative.
##########
apisix/plugins/ai-protocols/converters/anthropic-messages-to-openai-chat.lua:
##########
@@ -102,6 +102,33 @@ end
-- SSE event helpers
+-- prompt_tokens_details keys that carry cache-write tokens, in LiteLLM's
+-- order, followed by the cache_creation_input_tokens shape some upstreams use.
+local CACHE_CREATION_DETAIL_KEYS = {
+ "cache_creation_tokens",
+ "cache_write_tokens",
+ "cache_creation_input_tokens",
+}
+
+
+local function cache_creation_tokens(usage)
+ local v = usage.cache_creation_input_tokens
+ if type(v) == "number" and v > 0 then
+ return v
+ end
+ local details = usage.prompt_tokens_details
+ if type(details) == "table" then
+ for _, key in ipairs(CACHE_CREATION_DETAIL_KEYS) do
+ v = details[key]
+ if type(v) == "number" and v > 0 then
+ return v
+ end
+ end
+ end
+ return 0
+end
Review Comment:
Added the guard in 9775c8415.
##########
docs/zh/latest/plugins/prometheus.md:
##########
@@ -321,6 +321,8 @@ APISIX 在开始 LLM 上游尝试时递增该 gauge，并在请求的日志阶
### `apisix_llm_prompt_tokens` 的标签
+prompt token 包含从缓存读取和写入缓存的 token。对于 Anthropic Messages API，其值为 `input_tokens +
cache_read_input_tokens + cache_creation_input_tokens`，与 OpenAI 的
`prompt_tokens` 含义一致。
Review Comment:
Reworded it in 9775c8415. This page uses "prompt token" / "token" everywhere
(for example the metric descriptions and the histogram buckets section), so I
kept those terms here instead of switching to "令牌".
##########
apisix/plugins/ai-protocols/converters/anthropic-messages-to-openai-chat.lua:
##########
@@ -787,15 +814,18 @@ function _M.convert_response(res_body, ctx)
usage.output_tokens = completion_tokens
+ -- Anthropic input_tokens excludes both cache reads and cache writes,
+ -- while OpenAI prompt_tokens includes them (same as LiteLLM).
+ local creation = cache_creation_tokens(res_body.usage)
if type(details) == "table" then
local cached = details.cached_tokens or 0
- usage.input_tokens = math_max(0, prompt_tokens - cached)
+ usage.input_tokens = math_max(0, prompt_tokens - cached - creation)
Review Comment:
The suggested fix of preserving the upstream usage before conversion would
break the moderation plugins (`ai-aws-content-moderation`,
`ai-aliyun-content-moderation`, `ai-lakera-guard`). They pass
`ctx.llm_raw_usage` to `build_deny_response` of the client protocol, so on this
path it has to stay in Anthropic shape. That is also how it worked before this
PR: the converted non-streaming path already stored the converted usage. I took
the second option:
- The PR description now says that `ctx.llm_raw_usage` on the converted
non-streaming path holds the converted Anthropic usage, and lists the change to
`input_tokens` (20 to 15 for the fixture) under "Behavior change".
- `t/plugin/ai-rate-limiting-expression.t` TEST 29/30 cover the `expression`
strategy on this path. `input_tokens + cache_read_input_tokens +
cache_creation_input_tokens` evaluates to 30, which equals the upstream
`prompt_tokens`. Before the converter fix it evaluated to 35, because cache
writes were counted twice. Fixed in 9775c8415.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
