What version of ZooKeeper are you on? There have been a few bugs over the years where ZK has lost ephemeral nodes (and spontaneously de-registered brokers).
On Fri, Apr 29, 2016 at 11:30 AM, Kane Kim <[email protected]> wrote: > Any idea why it's happening? I'm sure rolling restart would fix it. Is it a > bug? > > On Wed, Apr 27, 2016 at 5:42 PM, Kane Kim <[email protected]> wrote: > >> Hello, >> >> Looks like we are hitting leader election bug. I've stopped one broker >> (104224873) on other brokers I see following: >> >> WARN kafka.controller.ControllerChannelManager - [Channel manager on >> controller 104224863]: Not sending request Name: StopReplicaRequest; >> Version: 0; CorrelationId: 843100; ClientId: ; DeletePartitions: false; >> ControllerId: 104224863; ControllerEpoch: 8; Partitions: [mp-auth,169] to >> broker 104224873, since it is offline. >> >> Also describing topics returns this: >> Topic: mp-unknown Partition: 597 Leader: 104224873 Replicas: >> 104224874,104224873,104224875 Isr: 104224873,104224875 >> >> broker 104224873 is shut down, but it's still leader for the partition (at >> least for a couple of hours as I monitor it). >> Zookeeper cluster is healthy. >> >> ls /brokers/ids >> [104224874, 104224875, 104224863, 104224864, 104224871, 104224867, >> 104224868, 104224865, 104224866, 104224876, 104224877, 104224869, >> 104224878, 104224879] >> >> That broker is not registered in ZK. >> -- James Brown Engineer
