[
https://issues.apache.org/jira/browse/KAFKA-18139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirk True updated KAFKA-18139:
------------------------------
Description:
We've noticed the AsyncKafkaConsumer polls the NetworkClient at a much higher
cadence than the ClassicKafkaConsumer. Using the
`java-producer-consumer-demo.sh` example, AsyncKafkaConsumer polls
networkClients almost twice as much as ClassicKafkaConsumer and also results in
some side effects such as higher CPU usage (>10% more) and potentially
affecting the request latency. For example, using the example, we measured
ClassicKafkaConsumer's average latency is 10.5ms and AsyncKafkaConsumer is
~12ms.
We suspect the added time delay is mostly caused by the inefficient
NetworkClient poll.
During the demo run produce and consuming 100,000,000 records results in ~400k
network client poll calls for the classic consumer and ~800k for the async
consumer using the default client configuration provided by the example.
was:
We've noticed the AsyncKafkaConsumer polls the NetworkClient at a much higher
cadence than the ClassicKafkaConsumer. Using the
`java-producer-consumer-demo.sh` example, the ClassicKafkaConsumer polls
networkClients almost twice as much as the old consumer and also results in
some side effects such as higher CPU usage (>10% more) and potentially
affecting the request latency. For example, using the example, we measured
ClassicKafkaConsumer's average latency is 10.5ms and AsyncKafkaConsumer is
~12ms.
We suspect the added time delay is mostly caused by the inefficient
NetworkClient poll.
During the demo run produce and consuming 100,000,000 records results in ~400k
network client poll calls for the classic consumer and ~800k for the async
consumer using the default client configuration provided by the example.
> AsyncKafkaConsumer's ConsumerNetworkThread loop requires further optimization
> -----------------------------------------------------------------------------
>
> Key: KAFKA-18139
> URL: https://issues.apache.org/jira/browse/KAFKA-18139
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Reporter: Philip Nee
> Assignee: Kirk True
> Priority: Major
> Labels: async-kafka-consumer-performance,
> consumer-threading-refactor, performance
> Fix For: 4.2.0
>
>
> We've noticed the AsyncKafkaConsumer polls the NetworkClient at a much higher
> cadence than the ClassicKafkaConsumer. Using the
> `java-producer-consumer-demo.sh` example, AsyncKafkaConsumer polls
> networkClients almost twice as much as ClassicKafkaConsumer and also results
> in some side effects such as higher CPU usage (>10% more) and potentially
> affecting the request latency. For example, using the example, we measured
> ClassicKafkaConsumer's average latency is 10.5ms and AsyncKafkaConsumer is
> ~12ms.
>
> We suspect the added time delay is mostly caused by the inefficient
> NetworkClient poll.
>
> During the demo run produce and consuming 100,000,000 records results in
> ~400k network client poll calls for the classic consumer and ~800k for the
> async consumer using the default client configuration provided by the example.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
