[
https://issues.apache.org/jira/browse/KAFKA-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610383#comment-15610383
]
huxi edited comment on KAFKA-4348 at 10/27/16 2:20 AM:
-------------------------------------------------------
Seems that you do not set 'auto.offset.reset' whose default value is 'latest'.
That means your consumer is awaiting any new produced messages to read. That's
why you see many 'polled 0 records' for the output. If you set
auto.offset.reset to earliest, you should see the consumer starts reading from
scratch very fast.
was (Author: huxi_2b):
Seems that you do not set 'auto.offset.reset' whose default value is 'latest'.
That means your consumer is awaiting the new produced messages to consumer.
That's why you see many 'polled 0 records' for the output. If you set
auto.offset.reset to earliest, you should see the consumer starts reading from
scratch very fast.
> On Mac OS, KafkaConsumer.poll returns 0 when there are still messages on
> Kafka server
> -------------------------------------------------------------------------------------
>
> Key: KAFKA-4348
> URL: https://issues.apache.org/jira/browse/KAFKA-4348
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.9.0.0, 0.9.0.1, 0.10.0.1
> Environment: Mac OS X EI Capitan, Java 1.8.0_111
> Reporter: Yiquan Zhou
> Labels: consumer, mac, polling
>
> Steps to reproduce:
> 1. start the zookeeper and kafka server using the default properties from the
> distribution:
> $ bin/zookeeper-server-start.sh config/zookeeper.properties
> $ bin/kafka-server-start.sh config/server.properties
> 2. create a Kafka consumer using the Java API KafkaConsumer.poll(long
> timeout). It polls the records from the server every second (timeout set to
> 1000) and prints the number of records polled. The code can be found here:
> https://gist.github.com/yiquanzhou/a94569a2c4ec8992444c83f3c393f596
> 3. use bin/kafka-verifiable-producer.sh to generate some messages:
> $ bin/kafka-verifiable-producer.sh --topic connect-test --max-messages 200000
> --broker-list localhost:9092
> wait until all 200k messages are generated and sent to the server.
> 4. Run the consumer Java code. In the output console of the consumer, we can
> see that the consumer starts to poll some records, then it polls 0 records
> for several seconds before polling some more. like this:
> polled 27160 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 26886 records
> polled 26886 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 0 records
> polled 26701 records
> polled 26214 records
> The bug slows down the consumption of messages a lot. And in our use case,
> the consumer wrongly assumes that all messages are read from the topic.
> It is only reproducible on Mac OS X but neither on Linux nor Windows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
