Re: Slow Consumer Group Startup

2017-06-16 Thread Eno Thereska
Hi Bryan, So this must be something else since KIP-134 is not in 0.10.2.1, but in the new release 0.11 that hasn't come out yet. Eno > On 14 Jun 2017, at 21:35, Bryan Baugher wrote: > > It does seem like we are in a similar situation described in the KIP ( > https://cwiki.apache.org/confluenc

Re: Slow Consumer Group Startup

2017-06-14 Thread Bryan Baugher
It does seem like we are in a similar situation described in the KIP ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance). For some historical reason we had set our session.timeout.ms to a high value (5 minutes) which corresponds with the amount of

Re: Slow Consumer Group Startup

2017-06-14 Thread Bryan Baugher
While I do have some logs its not trivial to share since the logs are across 16 JVMs and a few different hosts. On Wed, Jun 14, 2017 at 10:34 AM Eno Thereska wrote: > The delay in that KIP is just 3 seconds, not minutes though, right? Would > you have any logs to share? > > Thanks > Eno > > On 1

Re: Slow Consumer Group Startup

2017-06-14 Thread Eno Thereska
The delay in that KIP is just 3 seconds, not minutes though, right? Would you have any logs to share? Thanks Eno > On 14 Jun 2017, at 16:14, Bryan Baugher wrote: > > Our consumer group isn't doing anything stateful and we've seen this > behavior for existing groups as well. It seems like timing

Re: Slow Consumer Group Startup

2017-06-14 Thread Bryan Baugher
Our consumer group isn't doing anything stateful and we've seen this behavior for existing groups as well. It seems like timing could be an issue, thanks for the information. On Tue, Jun 13, 2017 at 7:39 PM James Cheng wrote: > Bryan, > > This sounds related to > https://cwiki.apache.org/conflue

Re: Slow Consumer Group Startup

2017-06-13 Thread James Cheng
Bryan, This sounds related to https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance and https://issues.apache.org/jira/browse/KAFKA-4925. -James > On Jun 13, 2017, at 7:02 AM, Bryan Baugher wrote: > > The topics already exist prior to starting an

Re: Slow Consumer Group Startup

2017-06-13 Thread Bryan Baugher
The topics already exist prior to starting any of the consumers On Mon, Jun 12, 2017 at 9:35 PM J Pai wrote: > When are the topics on which these consumer groups consume, created? > > -Jaikiran > On 13-Jun-2017, at 3:18 AM, Bryan Baugher wrote: > > Hi everyone, > > We are currently experiencing

Re: Slow Consumer Group Startup

2017-06-12 Thread J Pai
When are the topics on which these consumer groups consume, created? -Jaikiran On 13-Jun-2017, at 3:18 AM, Bryan Baugher wrote: Hi everyone, We are currently experiencing slow startup times for our consumer groups (16-32 processes for a hundred or more partitions) in the range of minutes (3-15

Slow Consumer Group Startup

2017-06-12 Thread Bryan Baugher
Hi everyone, We are currently experiencing slow startup times for our consumer groups (16-32 processes for a hundred or more partitions) in the range of minutes (3-15 minutes), where little to no messages are consumed before suddenly everything just starts working at full speed. I'm currently usi