Thanks Tao, it worked. I also played around with my test setting trying to replicate your results, using default settings. But als long as the poll timeout is set to 100ms or larger the only time-out I get are near the start and near the end (when indeed there is nothing to consume). This is with a producer putting out 1000 messages a second. Maybe the load of the producer your using is not constant? Maybe you could run a test with the org.apache.kafka.tools.ProducerPerformance class to see if it makes a difference?
On Tue, Dec 1, 2015 at 11:35 AM tao xiao <xiaotao...@gmail.com> wrote: > Gerard, > > In your case I think you can set fetch.min.bytes=1 so that the server will > answer the fetch request as soon as a single byte of data is available > instead of accumulating enough messages. > > But in my case is I have plenty of messages in broker and I am sure the > size of total message are much larger than the default setting which is > 1024 bytes but still the consumer doesn't return messages for every poll. > > > On Tue, 1 Dec 2015 at 18:29 Gerard Klijs <gerard.kl...@dizzit.com> wrote: > > > I was experimenting with the timeout setting, but as long as messages are > > produced and the consumer(s) keep polling I saw little difference. I did > > see for example that when producing only 1 message a second, still it > > sometimes wait to get three messages. So I also would like to know if > there > > is a faster way. > > > > On Tue, Dec 1, 2015 at 10:35 AM tao xiao <xiaotao...@gmail.com> wrote: > > > > > Hi team, > > > > > > I am using the new consumer with broker version 0.9.0. I notice that > > > poll(time) occasionally returns 0 message even though I have enough > > > messages in broker. The rate of returning 0 message is quite high like > 4 > > > out of 5 polls return 0 message. It doesn't help by increasing the poll > > > timeout from 300ms to 1 second. are there any configurations that I can > > > tune to fetch data as quickly as possible? > > > > > > Both consumer and broker configs are default > > > > > >
