Alieh Saeedi created KAFKA-20765:
------------------------------------
Summary: AsyncKafkaConsumer: OffsetFetch retry on
STALE_MEMBER_EPOCH can spin in dedup loop and never complete the application
future
Key: KAFKA-20765
URL: https://issues.apache.org/jira/browse/KAFKA-20765
Project: Kafka
Issue Type: Bug
Components: clients
Reporter: Alieh Saeedi
AsyncKafkaConsumer: OffsetFetch retry on STALE_MEMBER_EPOCH can spin in dedup
loop and never complete the application future
When an in-flight OffsetFetch fails with STALE_MEMBER_EPOCH and the member
already knows a newer epoch, CommitRequestManager retries the request. If a
duplicated fetch for the same partitions was chained onto the failed request,
the retry can be deduplicated against the *already-completed* request:
PendingRequests.addOffsetFetchRequest matches duplicates only by partitions
(sameRequest) and does not check whether the matched request is still pending.
A completed request may still appear in the buffers while its completion
callbacks run, because buffer removal is itself a completion callback and
CompletableFuture dependents run LIFO.
Chaining onto a completed future fails the retry immediately and synchronously,
which triggers another retry, another dedup, and so on — a tight loop with
these symptoms:
- WARN "A duplicated, inflight, request was identified, but unable to find it
in the outbound buffer" repeated hundreds of times within milliseconds,
- StackOverflowError surfacing in OffsetFetchRequestState.toString(),
- no OffsetFetch with the new epoch is ever sent (the epoch is stamped at send
time, which the loop never reaches),
- the application-level future is never completed.
In Kafka Streams with the streams rebalance protocol (KIP-1071) this blocks the
stream thread in committed-offset initialization until default.api.timeout.ms,
exceeds max.poll.interval.ms, and gets the member evicted. Observed in a system
test (streams_broker_down_resilience_test, staggered startup: epoch bumped
while the fetch was in flight). Plain consumer groups (group.protocol=consumer)
are exposed too, given concurrent same-partition committed-offset fetches
during an epoch bump.
Deterministic reproducer: two concurrent fetchOffsets() for the same partition
(second gets chained), respond with STALE_MEMBER_EPOCH after updating the
member epoch.
Suggested fix: exclude requests whose future is already done from the duplicate
search, so the retry is enqueued and re-sent with the current epoch.
Related: the onFailure ERROR for STALE_MEMBER_EPOCH ("cannot be retried and
will fail") is logged even when a retry follows; worth fixing alongside.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
