AlinsRan opened a new pull request, #14028: URL: https://github.com/apache/apisix/pull/14028
### Description Two problems in how the Anthropic Messages protocol reports token usage: 1. **Cache tokens in `message_delta` were dropped.** Streaming usage took `cache_read_input_tokens` / `cache_creation_input_tokens` only from `message_start`. Some Anthropic-compatible providers report the final usage, cache tokens included, only in `message_delta`. On those providers `$llm_cache_read_input_tokens` stayed `0`. 2. **Anthropic prompt tokens did not include cache tokens.** Anthropic reports cache tokens alongside `input_tokens`, not inside it. The gateway used `input_tokens` as `prompt_tokens`. So on Anthropic `$llm_prompt_tokens` / `$llm_total_tokens` meant something different than on OpenAI, where `prompt_tokens` already includes cached tokens. Because the field included no cache tokens, ratios such as cache reads / prompt tokens could exceed 100%. #### Changes The behavior follows LiteLLM (`AnthropicConfig.calculate_usage` and the streaming `_handle_usage` in `litellm/llms/anthropic/chat/`): - `prompt_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens` - `total_tokens = prompt_tokens + completion_tokens` - Streaming accumulates the raw usage fields across `message_start` and `message_delta` before computing usage, so cache tokens reported in either event are counted. - The Anthropic-to-OpenAI converter now subtracts `cache_creation_input_tokens` as well as `cached_tokens` from the OpenAI `prompt_tokens` when it builds the Anthropic `input_tokens`. LiteLLM's adapter does the same. Without this change, the converted non-streaming path would count cache-creation tokens twice under the new semantics. The raw upstream usage (`ctx.llm_raw_usage`, which the `expression` strategy of `ai-rate-limiting` and the moderation plugins use) is unchanged. #### Behavior change For requests using the Anthropic Messages protocol with prompt caching: - `$llm_prompt_tokens` and `$llm_total_tokens` (and the matching `llm_summary` log fields) now include cache read and cache creation tokens. - `apisix_llm_prompt_tokens` and `apisix_llm_prompt_tokens_dist` count these tokens too. - With `limit_strategy` set to `total_tokens` or `prompt_tokens`, `ai-rate-limiting` now counts cache tokens toward the quota. This matches how OpenAI-compatible requests are already counted. - In a converted response (Anthropic client, OpenAI-compatible upstream), `input_tokens` excludes `cache_creation_input_tokens` when the upstream reports it. Docs for `ai-proxy`, `ai-proxy-multi`, `ai-rate-limiting` and `prometheus` (en/zh) are updated. #### Which issue(s) this PR fixes: Fixes # ### Checklist - [x] I have explained the need for this PR and the problem it solves - [x] I have explained the changes or the new features added to this PR - [x] I have added tests corresponding to this change - [x] I have updated the documentation to reflect this change - [ ] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first) The change is not backward compatible for Anthropic token accounting, as described in "Behavior change" above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
