I might give it a try tomorrow. The reason for having so large init and sync limit times is because in the past our ZK cluster was storing large amount of data, and lower values were not enough for the server syncs when restarting zk processes.
On 1 June 2017 at 17:52, Mohammed Manna <[email protected]> wrote: > Cool - I will try and take a look into this - Meanwhile, do you mind awfuly > to change the following and see if things improve? > > tickTime = 1000 > initLimit=3 > syncLimit=5 > > On 1 June 2017 at 16:49, Del Barrio, Alberto < > [email protected]> wrote: > > > Here are the configs you were asking for: > > > > Zookeeper: > > tickTime=1000 > > initLimit=2000 > > syncLimit=1000 > > dataDir=/var/lib/zookeeper > > clientPort=2181 > > server.3=10.0.0.3:2888:3888 > > server.2=10.0.0.2:2888:3888 > > server.1=10.0.0.1:2888:3888 > > > > > > Kafka broker (for one of them): > > broker.id=10 > > listeners=PLAINTEXT://10.0.0.4:9092 > > num.network.threads=3 > > num.io.threads=8 > > socket.send.buffer.bytes=102400 > > socket.receive.buffer.bytes=102400 > > socket.request.max.bytes=104857600 > > log.dirs=/var/lib/kafka > > num.partitions=2 > > num.recovery.threads.per.data.dir=1 > > zookeeper.connect=10.0.0.1:2181,10.0.0.2:2181,10.0.0.3:2181/kafka > > zookeeper.connection.timeout.ms=6000 > > > > In general they're pretty much the default ones. > > I can see in Zookeeper the kafka brokers connected to it and exchanging > > data... > > > > Thanks for your help and time. > > > > On 1 June 2017 at 17:32, Mohammed Manna <[email protected]> wrote: > > > > > Could you please share your broker/zookeeper/topic configs ? > > > > > > On 1 June 2017 at 16:18, Del Barrio, Alberto < > > > [email protected]> wrote: > > > > > > > I tried creating the topic and results are very similar to the > current > > > > situation: there are not ISR and no leader for any of the partitions, > > but > > > > now kafka-topics shows *Leader: none* when for all the other topics, > it > > > > shows *Leader: -1* > > > > > > > > > > > > On 1 June 2017 at 17:05, Mohammed Manna <[email protected]> wrote: > > > > > > > > > I had a similar situation, but only 1 of my ZKs was struggling - > but > > > > since > > > > > the ISR synching time is configurable I was confident to bounce 1 > ZK > > > at a > > > > > time and it worked out. > > > > > does it happen even when you create a new topic with a > > > > > replication:partition ration of 1? > > > > > > > > > > i meant, 3 replicas, 3 partitions :) > > > > > > > > > > On 1 June 2017 at 15:58, Del Barrio, Alberto < > > > > > [email protected]> wrote: > > > > > > > > > > > Hi Mohammed, > > > > > > > > > > > > thanks for your answer. > > > > > > The ZK cluster is not located in the servers where Kafka runs but > > in > > > > > other > > > > > > 3 different machines. This ZK cluster is used by several other > > > services > > > > > > which are not reporting problems. > > > > > > As you suggested, I haven't tried restarting the kafka-server > > > processes > > > > > > because there's no leader for topic partitions, so I don't know > > what > > > > will > > > > > > happen. Never been in a similar situation with Kafka after some > > years > > > > > using > > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 1 June 2017 at 16:29, Mohammed Manna <[email protected]> > > wrote: > > > > > > > > > > > > > Hi Alberto, > > > > > > > > > > > > > > Usually this means that the leader election/replica syncing > > > couldn't > > > > be > > > > > > > successful and the zookeeper logs should be able to show this > > > > > information > > > > > > > too. The leader -1 is what worries me. For your case (3 broker > > > > > cluster), > > > > > > I > > > > > > > am assuming you have done the cluster configuration to have 1 > > > > > > > broker-zookeeper setup ? > > > > > > > If that's the case, you should be able to bounce 1 zookeeper > at a > > > > time > > > > > > and > > > > > > > see if that resolves the issue. > > > > > > > > > > > > > > That said, have you restarted your servers since this issue > > > > surfaced? > > > > > > > > > > > > > > On 1 June 2017 at 14:11, Del Barrio, Alberto < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I'm experiencing an issue which I don't know how to solve, so > > I'm > > > > > > trying > > > > > > > to > > > > > > > > find some guidance on the topic. > > > > > > > > > > > > > > > > I have a cluster composed by 3 servers, one broker per server > > > > running > > > > > > > Kafka > > > > > > > > 0.10.0.1-1 which runs in production with around 100 topics, > > most > > > of > > > > > > them > > > > > > > > divided in several partitions and replicated always between 2 > > > > > servers. > > > > > > > > Suddenly I've notice when looking at my topics (with > > kafka-topics > > > > > tool) > > > > > > > > that no one of them have a leader (Leader: -1) and the list > of > > > ISR > > > > > > > appears > > > > > > > > empty for all the topics. > > > > > > > > So they look something like: > > > > > > > > > > > > > > > > Topic:mytopic PartitionCount:3 ReplicationFactor:2 > > > Configs: > > > > > > > > retention.ms=86400000 > > > > > > > > Topic: mytopic Partition: 0 Leader: -1 Replicas: > > > 30,10 > > > > > > > Isr: > > > > > > > > Topic: mytopic Partition: 1 Leader: -1 Replicas: > > > 10,20 > > > > > > > Isr: > > > > > > > > Topic: mytopic Partition: 2 Leader: -1 Replicas: > > > 20,30 > > > > > > > Isr: > > > > > > > > > > > > > > > > However the applications using it are running normally, > > consumers > > > > as > > > > > > well > > > > > > > > as producers. > > > > > > > > The logs are not showing errors or weird messages with the > > > > exceptions > > > > > > of > > > > > > > > some > > > > > > > > Failed to rename [/var/log/kafka/log-cleaner.log] to > > > > > > > > [/var/log/kafka/log-cleaner.log.2017-05-31-17] > > > > > > > > which appear each few days. > > > > > > > > > > > > > > > > Now I would like to bring back to cluster to a good state. > I'm > > > > afraid > > > > > > of > > > > > > > > restarting brokers because all of them are supposed to be > > leaders > > > > for > > > > > > > some > > > > > > > > partitions, so if I restart them and there's no leader I > might > > > > > > experience > > > > > > > > data loss. > > > > > > > > > > > > > > > > Have you face any similar situation? Can someone give me a > > hint? > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > Alberto. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > News, jobs, product releases, events. > > > > > > Follow 360dialog on LinkedIn <https://www.linkedin.com/ > > > > company/360dialog > > > > > > > > > > > > and Twitter <https://twitter.com/360dialog>. > > > > > > Subscribe to our newsletter <http://www.360dialog.com/ > newsletter/ > > >. > > > > > > > > > > > > > > > > > > *Alberto del Barrio*DevOps Engineer > > > > > > > > > > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_ > > > > > > content=-&utm_medium=email&utm_source=signature&utm_term=logo> > > > > > > > > > > > > > > > > > > > > > > > > *Contact 360dialog*www.360dialog.com > > > > > > <http://www.360dialog.com/?utm_campaign=email-signature&; > > > > > > utm_content=-&utm_medium=email&utm_source=signature& > utm_term=url> > > > > > > [email protected] > > > > > > +49-(0)30-6098-5953-0 > > > > > > > > > > > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany > > > > > > Managing director: Roland Siebert > > > > > > Commercial register: Charlottenbug, HRB 144188 B > > > > > > VAT ID: DE815382679 > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > News, jobs, product releases, events. > > > > Follow 360dialog on LinkedIn <https://www.linkedin.com/ > > company/360dialog > > > > > > > > and Twitter <https://twitter.com/360dialog>. > > > > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. > > > > > > > > > > > > *Alberto del Barrio*DevOps Engineer > > > > > > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_ > > > > content=-&utm_medium=email&utm_source=signature&utm_term=logo> > > > > > > > > > > > > > > > > *Contact 360dialog*www.360dialog.com > > > > <http://www.360dialog.com/?utm_campaign=email-signature&; > > > > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> > > > > [email protected] > > > > +49-(0)30-6098-5953-0 > > > > > > > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany > > > > Managing director: Roland Siebert > > > > Commercial register: Charlottenbug, HRB 144188 B > > > > VAT ID: DE815382679 > > > > > > > > > > > > > > > -- > > News, jobs, product releases, events. > > Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog > > > > and Twitter <https://twitter.com/360dialog>. > > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. > > > > > > *Alberto del Barrio*DevOps Engineer > > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_ > > content=-&utm_medium=email&utm_source=signature&utm_term=logo> > > > > > > > > *Contact 360dialog*www.360dialog.com > > <http://www.360dialog.com/?utm_campaign=email-signature&; > > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> > > [email protected] > > +49-(0)30-6098-5953-0 > > > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany > > Managing director: Roland Siebert > > Commercial register: Charlottenbug, HRB 144188 B > > VAT ID: DE815382679 > > > -- News, jobs, product releases, events. Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog> and Twitter <https://twitter.com/360dialog>. Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. *Alberto del Barrio*DevOps Engineer <http://www.360dialog.com?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=logo> *Contact 360dialog*www.360dialog.com <http://www.360dialog.com/?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> [email protected] +49-(0)30-6098-5953-0 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany Managing director: Roland Siebert Commercial register: Charlottenbug, HRB 144188 B VAT ID: DE815382679
