Re: Kafka Streams application stuck rebalancing on startup

2020-11-05 Thread Alex Jablonski
Nice! Sorry for not creating the ticket, but I appreciate your investigation (and fix!) Thanks again! On Thu, Nov 5, 2020 at 5:31 PM Sophie Blee-Goldman wrote: > Ok I looked into this a bit and found the bug. I'll open a PR with the fix > sometime today: >

Re: Kafka Streams application stuck rebalancing on startup

2020-11-05 Thread Sophie Blee-Goldman
Ok I looked into this a bit and found the bug. I'll open a PR with the fix sometime today: https://issues.apache.org/jira/browse/KAFKA-10689 I also think we can do a better job of surfacing issues like this, rather than letting the application silently spin without making progress. I left some

Re: Kafka Streams application stuck rebalancing on startup

2020-11-02 Thread Sophie Blee-Goldman
Ok I tried to reproduce the issue with a minimal example, and saw the same results. It seems like there's something weird going on with that exact topology that's causing it to get stuck during the assignment. Maybe it's causing an unexpected cycle in the topology that the assignor can't handle?

Re: Kafka Streams application stuck rebalancing on startup

2020-10-28 Thread Alex Jablonski
This block: @EmbeddedKafka( topics = { "WordCounts", "WordsForNumbers", "OutputTopic" } ) starts up an embedded Kafka in the test and creates the 3 topics (2 input and 1 output). By default it creates them with 2 partitions each, but changing to 1 partition didn't

Re: Kafka Streams application stuck rebalancing on startup

2020-10-28 Thread Sophie Blee-Goldman
Yeah there's definitely something weird going on (assuming this is the full logs over that time period). The last thing we see logged from the StreamThread is this message from around the start of the task assignment process: 2020-10-28 12:22:37.879 DEBUG 27226 --- [-StreamThread-1]

Re: Kafka Streams application stuck rebalancing on startup

2020-10-27 Thread Sophie Blee-Goldman
> > We've been able to get the crucial factors that cause this behavior down to > a particular combination What do you mean by this -- that you only see this when all four of those operators are at play? Or do you see it with any of them. I guess the first thing to narrow down is whether it's

Kafka Streams application stuck rebalancing on startup

2020-10-15 Thread Alex Jablonski
Hey there! My team and I have run across a bit of a jam in our application where, given a particular setup, our Kafka Streams application never seems to start successfully, instead just getting stuck in the REBALANCING state. We've been able to get the crucial factors that cause this behavior