sssomeshhh commented on issue #13662:
URL: https://github.com/apache/apisix/issues/13662#issuecomment-4892822326
Thanks @juzhiyuan — appreciate the quick triage, and agreed on the
`base.lua#L82` fix. A couple of notes to close this out accurately, since (as I
noted on Jul 4) my original title overstated it.
**1. Not a live defect on the current API.** I re-verified against a live
`claude-haiku-4-5` stream — both directly against `api.anthropic.com` and
through a proxy, which agreed byte-for-byte on the field set. The
`message_delta` `usage` object repeats `input_tokens` on every delta:
```json
"usage":
{"input_tokens":57,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":328,"output_tokens_details":{"thinking_tokens":155}}
```
(`input_tokens` on the delta == `input_tokens` on `message_start`.) So
`anthropic-messages.lua` computes `total = 57 + 328 = 385` correctly — **no
output-only collapse and no `total_tokens` undercount on today's API**. The
"rate-limit bypass" framing in the title was wrong; this is a
**latent-robustness gap, not a live defect**.
**2. The `base.lua#L82` recompute you linked (from #13477) closes the
accumulation half.** `merge_usage` re-deriving `total = prompt + completion`
and taking the max means that even a delta which *did* arrive output-only would
come out correct after the merge. That leg is solid.
**3. One residual, low-severity — noting it for completeness, entirely your
call whether it's worth tracking.** The *source-minting* leg is unchanged (and
is the same on master): `anthropic-messages.lua:79-83` (and `:101-104`) builds
a per-event `total_tokens = (input_tokens or 0) + (output_tokens or 0)`. That's
only correct because the current API repeats `input_tokens` on every delta —
the `or 0` fallback silently zeroes it if a delta ever omits it. That field set
isn't a documented invariant: the streaming docs' basic-text and tool-use
examples show `message_delta.usage = {"output_tokens": N}` with no
`input_tokens`, while the server-tool-use example includes it — and
`base.lua`'s own comment (`:67-70`) is written for the input-omitting shape. So
it's correct-by-coincidence rather than by construction.
If it's ever worth making robust: since Anthropic's native `usage` has no
`total_tokens`, APISIX has to synthesize one — the only ask is to synthesize it
from a source that can't silently drop the input side (derive from the merged
parts / the cumulative `message_start` `input_tokens` + `message_delta`
`output_tokens`, rather than minting per-frame from a possibly-absent field).
Happy for you to repurpose this as a hardening item or close it and track
the source-minting note separately. Thanks again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
