When are the topics on which these consumer groups consume, created? -Jaikiran On 13-Jun-2017, at 3:18 AM, Bryan Baugher <bjb...@gmail.com> wrote:
Hi everyone, We are currently experiencing slow startup times for our consumer groups (16-32 processes for a hundred or more partitions) in the range of minutes (3-15 minutes), where little to no messages are consumed before suddenly everything just starts working at full speed. I'm currently using Kafka 0.9.0.1 but we are in the middle of upgrading to Kafka 0.10.2.1. We also using the newer kafka consumer API and group management on a simple Apache Storm topology. We don't make use of Storm's kafka spout but instead wrote a simple one ourselves. Using the kafka AdminClient I was able to poll for consumer group summary information. What I've found is that the group seems to sit in PreparingRebalance state for minutes before finally becoming Stable which then everything starts processing quickly. I've also enabled debug logging around the consumer's coordinator classes but didn't see anything to indicate the issue. I'm hoping that just upgrading to 0.10 or tweaking how we use our consumer in Apache Storm is the problem but are there any pointers on things I should look at or try? Bryan
