[ 
https://issues.apache.org/jira/browse/KAFKA-6743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-6743.
------------------------------------
       Resolution: Fixed
    Fix Version/s: 2.0.0

> ConsumerPerformance fails to consume all messages on topics with large number 
> of partitions
> -------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6743
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6743
>             Project: Kafka
>          Issue Type: Bug
>          Components: core, tools
>    Affects Versions: 0.11.0.2
>            Reporter: Alex Dunayevsky
>            Priority: Minor
>             Fix For: 2.0.0
>
>
> ConsumerPerformance fails to consume all messages on topics with large number 
> of partitions due to a relatively short default polling loop timeout (1000 
> ms) that is not reachable and modifiable by the end user.
> Demo: Create a topic of 10 000 partitions, send a 50 000 000 of 100 byte 
> records using kafka-producer-perf-test and consume them using 
> kafka-consumer-perf-test (ConsumerPerformance). You will likely notice that 
> the number of records returned by the kafka-consumer-perf-test is many times 
> less than expected 50 000 000. This happens due to specific 
> ConsumerPerformance implementation. As the result, in some rough cases it may 
> take a long enough time to process/iterate through the records polled in 
> batches, thus, the time may exceed the default hardcoded polling loop timeout 
> and this is probably not what we want from this utility.
> We have two options: 
> 1) Increasing polling loop timeout in ConsumerPerformance implementation. It 
> defaults to 1000 ms and is hardcoded, thus cannot be changed but we could 
> export it as an OPTIONAL kafka-consumer-perf-test parameter to enable it on a 
> script level configuration and available to the end user.
> 2) Decreasing max.poll.records on a Consumer config level. This is not a fine 
> option though since we do not want to touch the default settings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to