Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
No. With receive.buffer.bytes setting to 64K I am unable to reproduce the error. BTW I set the message size to 10K when testing with 64K buffer size On Thu, 28 Jan 2016 at 01:35 Jason Gustafson wrote: > Hey Tao, > > If you increase "receive.buffer.bytes" to 64K, can you still reproduce the > problem? > > -Jason > > On Tue, Jan 26, 2016 at 11:18 PM, Krzysztof Ciesielski < > krzysztof.ciesiel...@softwaremill.pl> wrote: > > > Jason, > > > > My os/vm is OSX 10.11.3, JDK 1.8.0.40 > > > > — > > Krzysztof > > On 26 January 2016 at 19:04:58, Jason Gustafson (ja...@confluent.io) > > wrote: > > > > Hey Krzysztof, > > > > So far I haven't had any luck figuring out the cause of the 5 second > pause, > > but I've reproduced it with the old consumer on 0.8.2, so that rules out > > anything specific to the new consumer. Can you tell me which os/jvm > you're > > seeing it with? Also, can you try changing the "receive.buffer.bytes" > > consumer configuration to "65536"? This was the default for the old > > consumer and I haven't been able to reproduce the problem when it is set. > > > > -Jason > > >
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Hey Tao, If you increase "receive.buffer.bytes" to 64K, can you still reproduce the problem? -Jason On Tue, Jan 26, 2016 at 11:18 PM, Krzysztof Ciesielski < krzysztof.ciesiel...@softwaremill.pl> wrote: > Jason, > > My os/vm is OSX 10.11.3, JDK 1.8.0.40 > > — > Krzysztof > On 26 January 2016 at 19:04:58, Jason Gustafson (ja...@confluent.io) > wrote: > > Hey Krzysztof, > > So far I haven't had any luck figuring out the cause of the 5 second pause, > but I've reproduced it with the old consumer on 0.8.2, so that rules out > anything specific to the new consumer. Can you tell me which os/jvm you're > seeing it with? Also, can you try changing the "receive.buffer.bytes" > consumer configuration to "65536"? This was the default for the old > consumer and I haven't been able to reproduce the problem when it is set. > > -Jason >
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Jason, My os/vm is OSX 10.11.3, JDK 1.8.0.40 — Krzysztof On 26 January 2016 at 19:04:58, Jason Gustafson (ja...@confluent.io) wrote: Hey Krzysztof, So far I haven't had any luck figuring out the cause of the 5 second pause, but I've reproduced it with the old consumer on 0.8.2, so that rules out anything specific to the new consumer. Can you tell me which os/jvm you're seeing it with? Also, can you try changing the "receive.buffer.bytes" consumer configuration to "65536"? This was the default for the old consumer and I haven't been able to reproduce the problem when it is set. -Jason
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
I managed to reproduce this issue on my mac with receive.buffer.bytes setting to new consumer default value. My JVM is hotspot 64 bit 1.7.0_60 and mac 10.10.5 On Wed, 27 Jan 2016 at 02:54 Krzysztof Ciesielski < krzysztof.ciesiel...@softwaremill.pl> wrote: > Hi Jason, > > Lowering "receive.buffer.bytes” helps, but when the message size gets > bigger - it comes back again. > I will test with 65536 and check how big the message has to be to make the > issue reappear with this value (I suspect that quite big). > > -- > Krzysztof Ciesielski > SoftwareMill > > On 26 January 2016 at 19:04:58, Jason Gustafson (ja...@confluent.io) > wrote: > > Hey Krzysztof, > > So far I haven't had any luck figuring out the cause of the 5 second pause, > but I've reproduced it with the old consumer on 0.8.2, so that rules out > anything specific to the new consumer. Can you tell me which os/jvm you're > seeing it with? Also, can you try changing the "receive.buffer.bytes" > consumer configuration to "65536"? This was the default for the old > consumer and I haven't been able to reproduce the problem when it is set. > > -Jason >
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Hi Jason, Lowering "receive.buffer.bytes” helps, but when the message size gets bigger - it comes back again. I will test with 65536 and check how big the message has to be to make the issue reappear with this value (I suspect that quite big). -- Krzysztof Ciesielski SoftwareMill On 26 January 2016 at 19:04:58, Jason Gustafson (ja...@confluent.io) wrote: Hey Krzysztof, So far I haven't had any luck figuring out the cause of the 5 second pause, but I've reproduced it with the old consumer on 0.8.2, so that rules out anything specific to the new consumer. Can you tell me which os/jvm you're seeing it with? Also, can you try changing the "receive.buffer.bytes" consumer configuration to "65536"? This was the default for the old consumer and I haven't been able to reproduce the problem when it is set. -Jason
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Hey Krzysztof, So far I haven't had any luck figuring out the cause of the 5 second pause, but I've reproduced it with the old consumer on 0.8.2, so that rules out anything specific to the new consumer. Can you tell me which os/jvm you're seeing it with? Also, can you try changing the "receive.buffer.bytes" consumer configuration to "65536"? This was the default for the old consumer and I haven't been able to reproduce the problem when it is set. -Jason
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Hi, On Sun, Jan 24, 2016 at 7:17 PM, Krzysztof Ciesielski < krzysztof.ciesiel...@softwaremill.com> wrote: > Yes, it's exactly 5 seconds on every machine. Sure, I'll open a JIRA > Jason already did so: https://issues.apache.org/jira/browse/KAFKA-3135 Feel free to add relevant information there. Ismael > On Jan 22, 2016 7:24 AM, "Jason Gustafson" wrote: > > > Hi Krzysztof, > > > > This is definitely weird. I see the data in the broker's send queue, but > > there's a delay of 5 seconds before it's sent to the client. Can you > create > > a JIRA? > > > > Thanks, > > Jason > > > > > > > > On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti > > > wrote: > > > > > +1, facing same issue. > > > -Sam > > > > On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski < > > > krzysztof.ciesiel...@softwaremill.com> wrote: > > > > > > > > Hello, I'm running into an issue with the new consumer in Kafka > > 0.9.0.0. > > > > Here's a runnable gist illustrating the problem: > > > > https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires > > > Kafka on > > > > localhost:9092) > > > > > > > > Scenario description: > > > > First, a producer writes 50 elements into a topic > > > > Then, a consumer starts to read, polling in a loop. > > > > When "max.partition.fetch.bytes" is set to a relatively small value, > > each > > > > "consumer.poll()" returns a batch of messages. > > > > If this value is left as default, the output tends to look like this: > > > > > > > > Poll returned 13793 elements > > > > Poll returned 13793 elements > > > > Poll returned 13793 elements > > > > Poll returned 13793 elements > > > > Poll returned 0 elements > > > > Poll returned 0 elements > > > > Poll returned 0 elements > > > > Poll returned 0 elements > > > > Poll returned 13793 elements > > > > Poll returned 13793 elements > > > > > > > > As we can see, there are weird "gaps" when poll returns 0 elements > for > > > some > > > > time. What is the reason for that? Maybe there are some good > practices > > > > about setting "max.partition.fetch.bytes" which I don't follow? > > > > > > > > Bests, > > > > Chris > > > > > > > > >
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Yes, it's exactly 5 seconds on every machine. Sure, I'll open a JIRA On Jan 22, 2016 7:24 AM, "Jason Gustafson" wrote: > Hi Krzysztof, > > This is definitely weird. I see the data in the broker's send queue, but > there's a delay of 5 seconds before it's sent to the client. Can you create > a JIRA? > > Thanks, > Jason > > > > On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti > wrote: > > > +1, facing same issue. > > -Sam > > > On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski < > > krzysztof.ciesiel...@softwaremill.com> wrote: > > > > > > Hello, I'm running into an issue with the new consumer in Kafka > 0.9.0.0. > > > Here's a runnable gist illustrating the problem: > > > https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires > > Kafka on > > > localhost:9092) > > > > > > Scenario description: > > > First, a producer writes 50 elements into a topic > > > Then, a consumer starts to read, polling in a loop. > > > When "max.partition.fetch.bytes" is set to a relatively small value, > each > > > "consumer.poll()" returns a batch of messages. > > > If this value is left as default, the output tends to look like this: > > > > > > Poll returned 13793 elements > > > Poll returned 13793 elements > > > Poll returned 13793 elements > > > Poll returned 13793 elements > > > Poll returned 0 elements > > > Poll returned 0 elements > > > Poll returned 0 elements > > > Poll returned 0 elements > > > Poll returned 13793 elements > > > Poll returned 13793 elements > > > > > > As we can see, there are weird "gaps" when poll returns 0 elements for > > some > > > time. What is the reason for that? Maybe there are some good practices > > > about setting "max.partition.fetch.bytes" which I don't follow? > > > > > > Bests, > > > Chris > > > > >
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
I see the pause regardless of the client's poll timeout. I thought it might be caused by the quota manager (which is supposed to delay fetches if a quota is exceeded), but this is looking more like a problem in the server's network layer. I went ahead and opened KAFKA-3135, so we can move investigation there. -Jason On Thu, Jan 21, 2016 at 10:52 PM, Samya Maiti wrote: > Providing a non-Zero value (Zero is default), for > > public ConsumerRecords poll(long timeout) > > Works fine for me with no gaps,but said so is just a work around. > Consumer is definitely picking up messages with some delay. > > -Sam > > > > On 22-Jan-2016, at 11:54 am, Jason Gustafson wrote: > > > > Hi Krzysztof, > > > > This is definitely weird. I see the data in the broker's send queue, but > > there's a delay of 5 seconds before it's sent to the client. Can you > create > > a JIRA? > > > > Thanks, > > Jason > > > > > > > > On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti > > > wrote: > > > >> +1, facing same issue. > >> -Sam > >>> On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski < > >> krzysztof.ciesiel...@softwaremill.com> wrote: > >>> > >>> Hello, I'm running into an issue with the new consumer in Kafka > 0.9.0.0. > >>> Here's a runnable gist illustrating the problem: > >>> https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires > >> Kafka on > >>> localhost:9092) > >>> > >>> Scenario description: > >>> First, a producer writes 50 elements into a topic > >>> Then, a consumer starts to read, polling in a loop. > >>> When "max.partition.fetch.bytes" is set to a relatively small value, > each > >>> "consumer.poll()" returns a batch of messages. > >>> If this value is left as default, the output tends to look like this: > >>> > >>> Poll returned 13793 elements > >>> Poll returned 13793 elements > >>> Poll returned 13793 elements > >>> Poll returned 13793 elements > >>> Poll returned 0 elements > >>> Poll returned 0 elements > >>> Poll returned 0 elements > >>> Poll returned 0 elements > >>> Poll returned 13793 elements > >>> Poll returned 13793 elements > >>> > >>> As we can see, there are weird "gaps" when poll returns 0 elements for > >> some > >>> time. What is the reason for that? Maybe there are some good practices > >>> about setting "max.partition.fetch.bytes" which I don't follow? > >>> > >>> Bests, > >>> Chris > >> > >> > >
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Providing a non-Zero value (Zero is default), for public ConsumerRecords poll(long timeout) Works fine for me with no gaps,but said so is just a work around. Consumer is definitely picking up messages with some delay. -Sam > On 22-Jan-2016, at 11:54 am, Jason Gustafson wrote: > > Hi Krzysztof, > > This is definitely weird. I see the data in the broker's send queue, but > there's a delay of 5 seconds before it's sent to the client. Can you create > a JIRA? > > Thanks, > Jason > > > > On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti > wrote: > >> +1, facing same issue. >> -Sam >>> On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski < >> krzysztof.ciesiel...@softwaremill.com> wrote: >>> >>> Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0. >>> Here's a runnable gist illustrating the problem: >>> https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires >> Kafka on >>> localhost:9092) >>> >>> Scenario description: >>> First, a producer writes 50 elements into a topic >>> Then, a consumer starts to read, polling in a loop. >>> When "max.partition.fetch.bytes" is set to a relatively small value, each >>> "consumer.poll()" returns a batch of messages. >>> If this value is left as default, the output tends to look like this: >>> >>> Poll returned 13793 elements >>> Poll returned 13793 elements >>> Poll returned 13793 elements >>> Poll returned 13793 elements >>> Poll returned 0 elements >>> Poll returned 0 elements >>> Poll returned 0 elements >>> Poll returned 0 elements >>> Poll returned 13793 elements >>> Poll returned 13793 elements >>> >>> As we can see, there are weird "gaps" when poll returns 0 elements for >> some >>> time. What is the reason for that? Maybe there are some good practices >>> about setting "max.partition.fetch.bytes" which I don't follow? >>> >>> Bests, >>> Chris >> >>
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Hi Krzysztof, This is definitely weird. I see the data in the broker's send queue, but there's a delay of 5 seconds before it's sent to the client. Can you create a JIRA? Thanks, Jason On Thu, Jan 21, 2016 at 11:30 AM, Samya Maiti wrote: > +1, facing same issue. > -Sam > > On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski < > krzysztof.ciesiel...@softwaremill.com> wrote: > > > > Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0. > > Here's a runnable gist illustrating the problem: > > https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires > Kafka on > > localhost:9092) > > > > Scenario description: > > First, a producer writes 50 elements into a topic > > Then, a consumer starts to read, polling in a loop. > > When "max.partition.fetch.bytes" is set to a relatively small value, each > > "consumer.poll()" returns a batch of messages. > > If this value is left as default, the output tends to look like this: > > > > Poll returned 13793 elements > > Poll returned 13793 elements > > Poll returned 13793 elements > > Poll returned 13793 elements > > Poll returned 0 elements > > Poll returned 0 elements > > Poll returned 0 elements > > Poll returned 0 elements > > Poll returned 13793 elements > > Poll returned 13793 elements > > > > As we can see, there are weird "gaps" when poll returns 0 elements for > some > > time. What is the reason for that? Maybe there are some good practices > > about setting "max.partition.fetch.bytes" which I don't follow? > > > > Bests, > > Chris > >
Re: Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
+1, facing same issue. -Sam > On 22-Jan-2016, at 12:16 am, Krzysztof Ciesielski > wrote: > > Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0. > Here's a runnable gist illustrating the problem: > https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires Kafka on > localhost:9092) > > Scenario description: > First, a producer writes 50 elements into a topic > Then, a consumer starts to read, polling in a loop. > When "max.partition.fetch.bytes" is set to a relatively small value, each > "consumer.poll()" returns a batch of messages. > If this value is left as default, the output tends to look like this: > > Poll returned 13793 elements > Poll returned 13793 elements > Poll returned 13793 elements > Poll returned 13793 elements > Poll returned 0 elements > Poll returned 0 elements > Poll returned 0 elements > Poll returned 0 elements > Poll returned 13793 elements > Poll returned 13793 elements > > As we can see, there are weird "gaps" when poll returns 0 elements for some > time. What is the reason for that? Maybe there are some good practices > about setting "max.partition.fetch.bytes" which I don't follow? > > Bests, > Chris
Kafka 0.9 -> consumer.poll() occasionally returns 0 elements
Hello, I'm running into an issue with the new consumer in Kafka 0.9.0.0. Here's a runnable gist illustrating the problem: https://gist.github.com/kciesielski/054bb4359a318aa17561 (requires Kafka on localhost:9092) Scenario description: First, a producer writes 50 elements into a topic Then, a consumer starts to read, polling in a loop. When "max.partition.fetch.bytes" is set to a relatively small value, each "consumer.poll()" returns a batch of messages. If this value is left as default, the output tends to look like this: Poll returned 13793 elements Poll returned 13793 elements Poll returned 13793 elements Poll returned 13793 elements Poll returned 0 elements Poll returned 0 elements Poll returned 0 elements Poll returned 0 elements Poll returned 13793 elements Poll returned 13793 elements As we can see, there are weird "gaps" when poll returns 0 elements for some time. What is the reason for that? Maybe there are some good practices about setting "max.partition.fetch.bytes" which I don't follow? Bests, Chris