I see the pause regardless of the client's poll timeout. I thought it might be caused by the quota manager (which is supposed to delay fetches if a quota is exceeded), but this is looking more like a problem in the server's network layer. I went ahead and opened KAFKA-3135, so we can move investigation there.
-Jason On Thu, Jan 21, 2016 at 10:52 PM, Samya Maiti <samya.maiti2...@gmail.com> wrote: > Providing a non-Zero value (Zero is default), for > > public ConsumerRecords<K, V> poll(long timeout) > > Works fine for me with no gaps,but said so is just a work around. > Consumer is definitely picking up messages with some delay. > > -Sam > > > > On 22-Jan-2016, at 11:54 am, Jason Gustafson <ja...@confluent.io> wrote: > > > > Hi Krzysztof, > > > > This is definitely weird. I see the data in the broker's send queue, but > > there's a delay of 5 seconds before it's sent to the client. Can you > create > > a JIRA? > > > > Thanks, > > Jason > > > > > > > > On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti <samya.maiti2...@gmail.com > > > > wrote: > > > >> +1, facing same issue. > >> -Sam > >>> On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski < > >> krzysztof.ciesiel...@softwaremill.com> wrote: > >>> > >>> Hello, I'm running into an issue with the new consumer in Kafka > 0.9.0.0. > >>> Here's a runnable gist illustrating the problem: > >>> https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires > >> Kafka on > >>> localhost:9092) > >>> > >>> Scenario description: > >>> First, a producer writes 500000 elements into a topic > >>> Then, a consumer starts to read, polling in a loop. > >>> When "max.partition.fetch.bytes" is set to a relatively small value, > each > >>> "consumer.poll()" returns a batch of messages. > >>> If this value is left as default, the output tends to look like this: > >>> > >>> Poll returned 13793 elements > >>> Poll returned 13793 elements > >>> Poll returned 13793 elements > >>> Poll returned 13793 elements > >>> Poll returned 0 elements > >>> Poll returned 0 elements > >>> Poll returned 0 elements > >>> Poll returned 0 elements > >>> Poll returned 13793 elements > >>> Poll returned 13793 elements > >>> > >>> As we can see, there are weird "gaps" when poll returns 0 elements for > >> some > >>> time. What is the reason for that? Maybe there are some good practices > >>> about setting "max.partition.fetch.bytes" which I don't follow? > >>> > >>> Bests, > >>> Chris > >> > >> > >
