The delay in that KIP is just 3 seconds, not minutes though, right? Would you have any logs to share?
Thanks Eno > On 14 Jun 2017, at 16:14, Bryan Baugher <[email protected]> wrote: > > Our consumer group isn't doing anything stateful and we've seen this > behavior for existing groups as well. It seems like timing could be an > issue, thanks for the information. > > On Tue, Jun 13, 2017 at 7:39 PM James Cheng <[email protected]> wrote: > >> Bryan, >> >> This sounds related to >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance >> and https://issues.apache.org/jira/browse/KAFKA-4925. >> >> -James >> >>> On Jun 13, 2017, at 7:02 AM, Bryan Baugher <[email protected]> wrote: >>> >>> The topics already exist prior to starting any of the consumers >>> >>> On Mon, Jun 12, 2017 at 9:35 PM J Pai <[email protected]> wrote: >>> >>>> When are the topics on which these consumer groups consume, created? >>>> >>>> -Jaikiran >>>> On 13-Jun-2017, at 3:18 AM, Bryan Baugher <[email protected]> wrote: >>>> >>>> Hi everyone, >>>> >>>> We are currently experiencing slow startup times for our consumer groups >>>> (16-32 processes for a hundred or more partitions) in the range of >> minutes >>>> (3-15 minutes), where little to no messages are consumed before suddenly >>>> everything just starts working at full speed. >>>> >>>> I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading >> to >>>> Kafka 0.10.2.1. We also using the newer kafka consumer API and group >>>> management on a simple Apache Storm topology. We don't make use of >> Storm's >>>> kafka spout but instead wrote a simple one ourselves. >>>> >>>> Using the kafka AdminClient I was able to poll for consumer group >> summary >>>> information. What I've found is that the group seems to sit >>>> in PreparingRebalance state for minutes before finally becoming Stable >>>> which then everything starts processing quickly. I've also enabled debug >>>> logging around the consumer's coordinator classes but didn't see >> anything >>>> to indicate the issue. >>>> >>>> I'm hoping that just upgrading to 0.10 or tweaking how we use our >> consumer >>>> in Apache Storm is the problem but are there any pointers on things I >>>> should look at or try? >>>> >>>> Bryan >>>> >>>> >> >>
