AndrewJSchofield opened a new pull request, #23234: URL: https://github.com/apache/kafka/pull/23234
A producer with multiple in-flight requests producing to a newly created partition can have its requests reordered around the moment the broker applies the partition metadata. This is particularly evident for tests such as Trogdor which start producing records immediately on topics which have been auto-created. The flow is like this: 1. The producer creates a topic and immediately sends produce request A (sequences 0–16) and request B (sequences 17–21) to the same partition. 2. Request A arrives before the broker has applied the topic creation from the metadata log and fails with a retriable NOT_LEADER_OR_FOLLOWER. 3. Request B arrives after the partition creation has completed. The broker has no producer state and accepted any first sequence in that case, so B is appended. 4. Every retry of request A now fails with OUT_OF_ORDER_SEQUENCE_NUMBER: its sequence range was never written, so it is not in the duplicate cache, and the broker expects sequence 22. The client retries without an epoch bump (it assumes a lower-sequence batch will fill the gap), so A spins until delivery.timeout.ms expires it. The result is silent loss of request A's records despite enable.idempotence=true, acks=all and retries=MAX_INT. The same defect was reported independently in KAFKA-14312, KAFKA-15591 and KAFKA-19880. KAFKA-18202 fixed it for TV2 producers only. If no records have ever been appended to a partition, no producer state can have been lost, so the only valid first sequence for any producer is 0. ProducerAppendInfo now rejects a non-zero first sequence from a client when the producer has no state and the log has never contained a record, with a retriable OutOfOrderSequenceException. The rejected later request is retried, the earlier request's retry (sequence 0) is accepted as soon as the partition is ready, and the later request then lands in order. If the earlier request instead expires, the unresolved-sequence handling bumps the idempotent producer epoch and rewrites the later request to sequence 0, so the producer never gets stuck. Partitions which have ever contained records are unaffected, even for situations where producer state is lost due to retention, record deletion or producer state expiration. Log emptiness is derived from `ProducerStateManager#mapEndOffset`, which tracks the log end offset on the leader and is advanced to the log start offset on reload without a snapshot and when the log start offset is incremented, so it is zero only for a log which has never contained a record. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
