Hi Pulsar community, I'd like to start the discussion on PIP-490: Zero-downtime token signing-key rotation for the built-in JWT provider.
*PIP PR*: https://github.com/apache/pulsar/pull/26204 *Motivation* Pulsar's built-in token provider (AuthenticationProviderToken) validates every JWT against a single key that is loaded once at broker startup and never reloaded. There is no window in which an old and a new signing key are both accepted, so rotating a signing key today means re-issuing every outstanding token and switching the broker/proxy config in lockstep - in practice a coordinated restart, during which tokens signed with the "other" key are rejected. That turns key-compromise response into an outage, and makes routine rotation painful enough that operators tend to skip it and keep long-lived keys. Pulsar already does multi-key, kid-based resolution - but only in the OIDC provider (AuthenticationProviderOpenID), which is built on a different JWT library and fetches keys from a remote JWKS URI. This PIP brings the same capability to the local token provider, keeping it dependency-free and local. *What it proposes* - Let the provider hold several validation keys at once, each tied to a kid and one algorithm; select the key by the token's kid header. jjwt (which the provider already uses) ships the keyLocator mechanism for exactly this. - Use a local JWKS document as the multi-key format, parsed with jjwt's own Jwks/JwkSet, so each key self-describes its kid, key type, and algorithm. - Hot-reload the key set on a timer, with an atomic swap and last-good-set retention, so a key can be added or removed without a restart. - New config: tokenValidationKeysJwks, tokenValidationKeysRefreshSeconds, tokenRequireKid (mirrored into the proxy); an optional `pulsar tokens create --key-id`; and a new INVALID_TOKEN_KID error code. - Fully backward compatible: with only today's single-key settings and tokens that carry no kid, behavior is byte-for-byte identical to today. The rotation runbook is then purely additive, with no rejection window: add key B -> sign new tokens with kid=B -> wait for A-signed tokens to expire -> remove key A. *Security angle* Going from one key to a "try several keys" model opens the classic JWT algorithm-confusion attack (forge an HS256 token using a public RSA key's bytes as the HMAC secret) unless each key is pinned to one algorithm. The PIP pins the algorithm per key and rejects alg mismatches before any signature check; using a JWKS makes that binding explicit and checked by the code rather than remembered by the operator. This is covered in detail in the Security Considerations section. *Two open questions for this thread* 1. Fallback default. When a token carries no kid, should the provider fall back to trying the configured keys (smoother migration -- my current default), or should we default to strict kid-only matching and make the fallback opt-in? The PIP keeps fallback on, with tokenRequireKid=true to enforce strict matching once every token carries a kid. 2. Key-set format. The PIP uses a local JWKS as the single multi-key format, precisely because each JWK carries its algorithm (which is what prevents the confusion attack above). Is JWKS the right sole interface, or should we also accept a simpler list of kid-tagged key files for operators who find JWKS heavyweight? I'll appreciate the feedback. Thanks, Pavel Zeger
