[
https://issues.apache.org/jira/browse/KAFKA-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094492#comment-18094492
]
Blaž Šnuderl commented on KAFKA-20780:
--------------------------------------
Client is confluent-kafka 8.2.1 and spring boot 4.0
KIP 848 settings:
group.protocol = consumer
fetch.max.wait.ms = 100 # set ONLY on the KIP-848 path
fetch.min.bytes = 1 # client default (never overridden)
max.partition.fetch.bytes = 10485760 # 10 MiB
max.poll.records = 500 # Kafka default unless env override
enable.auto.commit = false # manual commit (Spring AckMode)
auto.offset.reset = earliest
request.timeout.ms = 30000
security.protocol = SASL_SSL ; sasl.mechanism = PLAIN
key.deserializer = StringDeserializer
value.deserializer = KafkaProtobufDeserializer (Confluent)
# session.timeout.ms / heartbeat.interval.ms NOT set
> syncKafkaConsumer does not keep a fetch request pending on the broker,
> causing latency and throughput regressions vs the classic consumer
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20780
> URL: https://issues.apache.org/jira/browse/KAFKA-20780
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Reporter: Blaž Šnuderl
> Priority: Major
>
> When testing out kip-848 we see worse end to end latencies on realtime topics.
> The AsyncKafkaConsumer (KIP-848, group.protocol=consumer) leaves gaps where
> no fetch request is pending on the broker. The classic KafkaConsumer keeps
> one in flight at all times. On the same low-throughput topic (~1 msg/s), the
> classic consumer sees ~3.2 fetches/s and ~28ms latency, while the async
> consumer sees ~0.47 fetches/s and ~2300ms latency — an ~80x latency
> regression from switching the group protocol
> alone.\{{ConsumerNetworkThread.runOnce()}} sleeps for
> \{{maximumTimeToWait()}} between iterations with no fetch pending, so
> messages arriving in the gap wait until the next wakeup.
> {\{AsyncKafkaConsumer.checkInflightPoll()}} does the \{{inflightPoll ==
> null}} check before \{{maybeClearCurrentInflightPoll()}}. So on even
> iterations \{{inflightPoll}} is still non-null, no new fetch event is
> submitted, and \{{pollForFetches}} blocks the full timeout. Net: one fetch
> per ~2.1s.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
