Which version of Kafka are you using? The current stable one is 0.8.1.1 On Tue, Oct 14, 2014 at 5:51 PM, Jean-Pascal Billaud <j...@tellapart.com> wrote:
> Hey Neha, > > so I removed another broker like 30mn ago and since then basically the > Producer is dying with: > > Event queue is full of unsent messages, could not send event: > KeyedMessage(my_topic,[B@1b71b7a6,[B@35fdd1e7) > kafka.common.QueueFullException: Event queue is full of unsent messages, > could not send event: KeyedMessage(my_topic,[B@1b71b7a6,[B@35fdd1e7) > at kafka.producer.Producer$$anonfun$asyncSend$1.apply(Unknown Source) > ~[kafka_2.10-0.8.0.jar:0.8.0] > at kafka.producer.Producer$$anonfun$asyncSend$1.apply(Unknown Source) > ~[kafka_2.10-0.8.0.jar:0.8.0] > at scala.collection.Iterator$class.foreach(Iterator.scala:727) > ~[scala-library-2.10.3.jar:na] > at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) > ~[scala-library-2.10.3.jar:na] > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > ~[scala-library-2.10.3.jar:na] > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > ~[scala-library-2.10.3.jar:na] > at kafka.producer.Producer.asyncSend(Unknown Source) > ~[kafka_2.10-0.8.0.jar:0.8.0] > at kafka.producer.Producer.send(Unknown Source) > ~[kafka_2.10-0.8.0.jar:0.8.0] > at kafka.javaapi.producer.Producer.send(Unknown Source) > ~[kafka_2.10-0.8.0.jar:0.8.0] > > It seems like it cannot recover for some reasons. The new leaders were > elected it seems like so it should have picked up the new meta data > information about the partitions. Is this something known from 0.8.0? What > should be looking for to debug/fix this? > > Thanks, > > On Tue, Oct 14, 2014 at 2:22 PM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > > > Regarding (1), I am assuming that it is expected that brokers going down > > will be brought back up soon. At which point, they will pick up from the > > current leader and get back into the ISR. Am I right? > > > > The broker will be added back to the ISR once it is restarted, but it > never > > goes out of the replica list until the admin explicitly moves it using > the > > reassign partitions tool. > > > > Regarding (2), I finally kicked off a reassign_partitions admin task > adding > > broker 7 to the replicas list for partition 0 which finally fixed the > under > > replicated issue: > > Is this therefore expected that the user will fix up the under > replication > > situation? > > > > Yes. Currently, partition reassignment is purely an admin initiated task. > > > > Another thing I'd like to clarify is that for another topic Y, broker 5 > was > > never removed from the ISR array. Note that Y is an unused topic so I am > > guessing that technically broker 5 is not out of sync... though it is > still > > dead. Is this the expected behavior? > > > > Not really. After replica.lag.time.max.ms (which defaults to 10 > seconds), > > the leader should remove the dead broker out of the ISR. > > > > Thanks, > > Neha > > > > On Tue, Oct 14, 2014 at 9:27 AM, Jean-Pascal Billaud <j...@tellapart.com> > > wrote: > > > > > hey folks, > > > > > > I have been testing a kafka cluster of 10 nodes on AWS using version > > > 2.8.0-0.8.0 > > > and see some behavior on failover that I want to make sure I > understand. > > > > > > Initially, I have a topic X with 30 partitions and a replication factor > > of > > > 3. Looking at the partition 0: > > > partition: 0 - leader: 5 preferred leader: 5 brokers: [5, 3, 4] > in-sync: > > > [5, 3, 4] > > > > > > While killing broker 5, the controller immediately grab the next > replica > > in > > > the ISR and assign it as a leader: > > > partition: 0 - leader: 3 preferred leader: 5 brokers: [5, 3, 4] > in-sync: > > > [3, 4] > > > > > > There are couple of things at this point I would like to clarify: > > > > > > (1) Why is broker 5 still in the brokers array for partition 0? Note > this > > > broker array comes from a get of the zookeeper path > > /brokers/topics/[topic] > > > as documented. > > > (2) Partition 0 is now under replicated and the controller does not > seem > > to > > > do anything about. Is this expected? > > > > > > Regarding (1), I am assuming that it is expected that brokers going > down > > > will be brought back up soon. At which point, they will pick up from > the > > > current leader and get back into the ISR. Am I right? > > > > > > Regarding (2), I finally kicked off a reassign_partitions admin task > > adding > > > broker 7 to the replicas list for partition 0 which finally fixed the > > under > > > replicated issue: > > > > > > partition: 0 - leader: 3 expected_leader: 3 brokers: [3, 4, 7] > > in-sync: > > > [3, 4, 7] > > > > > > Is this therefore expected that the user will fix up the under > > replication > > > situation? Or maybe it is expected again that broker 5 will come back > > soon > > > and this whole thing is a non-issue once that's true given that > > > decommissioning brokers is not something supported as of the kafka > > version > > > I am using. > > > > > > Another thing I'd like to clarify is that for another topic Y, broker 5 > > was > > > never removed from the ISR array. Note that Y is an unused topic so I > am > > > guessing that technically broker 5 is not out of sync... though it is > > still > > > dead. Is this the expected behavior? > > > > > > I'd really appreciate somebody to confirm my understanding, > > > > > > Thanks, > > > > > >
