[
https://issues.apache.org/jira/browse/KAFKA-20765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alieh Saeedi reassigned KAFKA-20765:
------------------------------------
Assignee: Alieh Saeedi
> AsyncKafkaConsumer: OffsetFetch retry on STALE_MEMBER_EPOCH can spin in dedup
> loop and never complete the application future
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20765
> URL: https://issues.apache.org/jira/browse/KAFKA-20765
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Reporter: Alieh Saeedi
> Assignee: Alieh Saeedi
> Priority: Critical
>
> AsyncKafkaConsumer: OffsetFetch retry on STALE_MEMBER_EPOCH can spin in dedup
> loop and never complete the application future
> When an in-flight OffsetFetch fails with STALE_MEMBER_EPOCH and the member
> already knows a newer epoch, CommitRequestManager retries the request. If a
> duplicated fetch for the same partitions was chained onto the failed request,
> the retry can be deduplicated against the *already-completed* request:
> PendingRequests.addOffsetFetchRequest matches duplicates only by partitions
> (sameRequest) and does not check whether the matched request is still
> pending. A completed request may still appear in the buffers while its
> completion callbacks run, because buffer removal is itself a completion
> callback and CompletableFuture dependents run LIFO.
> Chaining onto a completed future fails the retry immediately and
> synchronously, which triggers another retry, another dedup, and so on — a
> tight loop with these symptoms:
> - WARN "A duplicated, inflight, request was identified, but unable to find it
> in the outbound buffer" repeated hundreds of times within milliseconds,
> - StackOverflowError surfacing in OffsetFetchRequestState.toString(),
> - no OffsetFetch with the new epoch is ever sent (the epoch is stamped at
> send time, which the loop never reaches),
> - the application-level future is never completed.
> In Kafka Streams with the streams rebalance protocol (KIP-1071) this blocks
> the stream thread in committed-offset initialization until
> default.api.timeout.ms, exceeds max.poll.interval.ms, and gets the member
> evicted. Observed in a system test (streams_broker_down_resilience_test,
> staggered startup: epoch bumped while the fetch was in flight). Plain
> consumer groups (group.protocol=consumer) are exposed too, given concurrent
> same-partition committed-offset fetches during an epoch bump.
> Deterministic reproducer: two concurrent fetchOffsets() for the same
> partition (second gets chained), respond with STALE_MEMBER_EPOCH after
> updating the member epoch.
> Suggested fix: exclude requests whose future is already done from the
> duplicate search, so the retry is enqueued and re-sent with the current epoch.
> Related: the onFailure ERROR for STALE_MEMBER_EPOCH ("cannot be retried and
> will fail") is logged even when a retry follows; worth fixing alongside.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
