[
https://issues.apache.org/jira/browse/KAFKA-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094299#comment-18094299
]
Lianet Magrans commented on KAFKA-20780:
----------------------------------------
Hi [~snuderl] , thanks for reporting. What client version are you using? (there
were some issues fixed in this area recently). Also please share the client
configs used and we'll look into it. Thanks!
> syncKafkaConsumer does not keep a fetch request pending on the broker,
> causing latency and throughput regressions vs the classic consumer
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20780
> URL: https://issues.apache.org/jira/browse/KAFKA-20780
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Reporter: Blaž Šnuderl
> Priority: Major
>
> When testing out kip-848 we see worse end to end latencies on realtime topics.
> The AsyncKafkaConsumer (KIP-848, group.protocol=consumer) leaves gaps where
> no fetch request is pending on the broker. The classic KafkaConsumer keeps
> one in flight at all times. On the same low-throughput topic (~1 msg/s), the
> classic consumer sees ~3.2 fetches/s and ~28ms latency, while the async
> consumer sees ~0.47 fetches/s and ~2300ms latency — an ~80x latency
> regression from switching the group protocol
> alone.\{{ConsumerNetworkThread.runOnce()}} sleeps for
> \{{maximumTimeToWait()}} between iterations with no fetch pending, so
> messages arriving in the gap wait until the next wakeup.
> {\{AsyncKafkaConsumer.checkInflightPoll()}} does the \{{inflightPoll ==
> null}} check before \{{maybeClearCurrentInflightPoll()}}. So on even
> iterations \{{inflightPoll}} is still non-null, no new fetch event is
> submitted, and \{{pollForFetches}} blocks the full timeout. Net: one fetch
> per ~2.1s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
