Blaž Šnuderl created KAFKA-20780:
------------------------------------
Summary: syncKafkaConsumer does not keep a fetch request pending
on the broker, causing latency and throughput regressions vs the classic
consumer
Key: KAFKA-20780
URL: https://issues.apache.org/jira/browse/KAFKA-20780
Project: Kafka
Issue Type: Bug
Components: consumer
Reporter: Blaž Šnuderl
When testing out kip-848 we see worse end to end latencies on realtime topics.
The AsyncKafkaConsumer (KIP-848, group.protocol=consumer) leaves gaps where no
fetch request is pending on the broker. The classic KafkaConsumer keeps one in
flight at all times. On the same low-throughput topic (~1 msg/s), the classic
consumer sees ~3.2 fetches/s and ~28ms latency, while the async consumer sees
~0.47 fetches/s and ~2300ms latency — an ~80x latency regression from switching
the group protocol alone.\{{ConsumerNetworkThread.runOnce()}} sleeps for
\{{maximumTimeToWait()}} between iterations with no fetch pending, so messages
arriving in the gap wait until the next wakeup.
{\{AsyncKafkaConsumer.checkInflightPoll()}} does the \{{inflightPoll == null}}
check before \{{maybeClearCurrentInflightPoll()}}. So on even iterations
\{{inflightPoll}} is still non-null, no new fetch event is submitted, and
\{{pollForFetches}} blocks the full timeout. Net: one fetch per ~2.1s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
