So, I fixed the problem doing a rolling restart, and after some checks seems there was no data loss.
On 1 June 2017 at 17:57, Del Barrio, Alberto < [email protected]> wrote: > I might give it a try tomorrow. The reason for having so large init and > sync limit times is because in the past our ZK cluster was storing large > amount of data, and lower values were not enough for the server syncs when > restarting zk processes. > > On 1 June 2017 at 17:52, Mohammed Manna <[email protected]> wrote: > >> Cool - I will try and take a look into this - Meanwhile, do you mind >> awfuly >> to change the following and see if things improve? >> >> tickTime = 1000 >> initLimit=3 >> syncLimit=5 >> >> On 1 June 2017 at 16:49, Del Barrio, Alberto < >> [email protected]> wrote: >> >> > Here are the configs you were asking for: >> > >> > Zookeeper: >> > tickTime=1000 >> > initLimit=2000 >> > syncLimit=1000 >> > dataDir=/var/lib/zookeeper >> > clientPort=2181 >> > server.3=10.0.0.3:2888:3888 >> > server.2=10.0.0.2:2888:3888 >> > server.1=10.0.0.1:2888:3888 >> > >> > >> > Kafka broker (for one of them): >> > broker.id=10 >> > listeners=PLAINTEXT://10.0.0.4:9092 >> > num.network.threads=3 >> > num.io.threads=8 >> > socket.send.buffer.bytes=102400 >> > socket.receive.buffer.bytes=102400 >> > socket.request.max.bytes=104857600 >> > log.dirs=/var/lib/kafka >> > num.partitions=2 >> > num.recovery.threads.per.data.dir=1 >> > zookeeper.connect=10.0.0.1:2181,10.0.0.2:2181,10.0.0.3:2181/kafka >> > zookeeper.connection.timeout.ms=6000 >> > >> > In general they're pretty much the default ones. >> > I can see in Zookeeper the kafka brokers connected to it and exchanging >> > data... >> > >> > Thanks for your help and time. >> > >> > On 1 June 2017 at 17:32, Mohammed Manna <[email protected]> wrote: >> > >> > > Could you please share your broker/zookeeper/topic configs ? >> > > >> > > On 1 June 2017 at 16:18, Del Barrio, Alberto < >> > > [email protected]> wrote: >> > > >> > > > I tried creating the topic and results are very similar to the >> current >> > > > situation: there are not ISR and no leader for any of the >> partitions, >> > but >> > > > now kafka-topics shows *Leader: none* when for all the other >> topics, it >> > > > shows *Leader: -1* >> > > > >> > > > >> > > > On 1 June 2017 at 17:05, Mohammed Manna <[email protected]> wrote: >> > > > >> > > > > I had a similar situation, but only 1 of my ZKs was struggling - >> but >> > > > since >> > > > > the ISR synching time is configurable I was confident to bounce 1 >> ZK >> > > at a >> > > > > time and it worked out. >> > > > > does it happen even when you create a new topic with a >> > > > > replication:partition ration of 1? >> > > > > >> > > > > i meant, 3 replicas, 3 partitions :) >> > > > > >> > > > > On 1 June 2017 at 15:58, Del Barrio, Alberto < >> > > > > [email protected]> wrote: >> > > > > >> > > > > > Hi Mohammed, >> > > > > > >> > > > > > thanks for your answer. >> > > > > > The ZK cluster is not located in the servers where Kafka runs >> but >> > in >> > > > > other >> > > > > > 3 different machines. This ZK cluster is used by several other >> > > services >> > > > > > which are not reporting problems. >> > > > > > As you suggested, I haven't tried restarting the kafka-server >> > > processes >> > > > > > because there's no leader for topic partitions, so I don't know >> > what >> > > > will >> > > > > > happen. Never been in a similar situation with Kafka after some >> > years >> > > > > using >> > > > > > it. >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > On 1 June 2017 at 16:29, Mohammed Manna <[email protected]> >> > wrote: >> > > > > > >> > > > > > > Hi Alberto, >> > > > > > > >> > > > > > > Usually this means that the leader election/replica syncing >> > > couldn't >> > > > be >> > > > > > > successful and the zookeeper logs should be able to show this >> > > > > information >> > > > > > > too. The leader -1 is what worries me. For your case (3 broker >> > > > > cluster), >> > > > > > I >> > > > > > > am assuming you have done the cluster configuration to have 1 >> > > > > > > broker-zookeeper setup ? >> > > > > > > If that's the case, you should be able to bounce 1 zookeeper >> at a >> > > > time >> > > > > > and >> > > > > > > see if that resolves the issue. >> > > > > > > >> > > > > > > That said, have you restarted your servers since this issue >> > > > surfaced? >> > > > > > > >> > > > > > > On 1 June 2017 at 14:11, Del Barrio, Alberto < >> > > > > > > [email protected]> wrote: >> > > > > > > >> > > > > > > > Hi all, >> > > > > > > > >> > > > > > > > I'm experiencing an issue which I don't know how to solve, >> so >> > I'm >> > > > > > trying >> > > > > > > to >> > > > > > > > find some guidance on the topic. >> > > > > > > > >> > > > > > > > I have a cluster composed by 3 servers, one broker per >> server >> > > > running >> > > > > > > Kafka >> > > > > > > > 0.10.0.1-1 which runs in production with around 100 topics, >> > most >> > > of >> > > > > > them >> > > > > > > > divided in several partitions and replicated always between >> 2 >> > > > > servers. >> > > > > > > > Suddenly I've notice when looking at my topics (with >> > kafka-topics >> > > > > tool) >> > > > > > > > that no one of them have a leader (Leader: -1) and the list >> of >> > > ISR >> > > > > > > appears >> > > > > > > > empty for all the topics. >> > > > > > > > So they look something like: >> > > > > > > > >> > > > > > > > Topic:mytopic PartitionCount:3 ReplicationFactor:2 >> > > Configs: >> > > > > > > > retention.ms=86400000 >> > > > > > > > Topic: mytopic Partition: 0 Leader: -1 >> Replicas: >> > > 30,10 >> > > > > > > Isr: >> > > > > > > > Topic: mytopic Partition: 1 Leader: -1 >> Replicas: >> > > 10,20 >> > > > > > > Isr: >> > > > > > > > Topic: mytopic Partition: 2 Leader: -1 >> Replicas: >> > > 20,30 >> > > > > > > Isr: >> > > > > > > > >> > > > > > > > However the applications using it are running normally, >> > consumers >> > > > as >> > > > > > well >> > > > > > > > as producers. >> > > > > > > > The logs are not showing errors or weird messages with the >> > > > exceptions >> > > > > > of >> > > > > > > > some >> > > > > > > > Failed to rename [/var/log/kafka/log-cleaner.log] to >> > > > > > > > [/var/log/kafka/log-cleaner.log.2017-05-31-17] >> > > > > > > > which appear each few days. >> > > > > > > > >> > > > > > > > Now I would like to bring back to cluster to a good state. >> I'm >> > > > afraid >> > > > > > of >> > > > > > > > restarting brokers because all of them are supposed to be >> > leaders >> > > > for >> > > > > > > some >> > > > > > > > partitions, so if I restart them and there's no leader I >> might >> > > > > > experience >> > > > > > > > data loss. >> > > > > > > > >> > > > > > > > Have you face any similar situation? Can someone give me a >> > hint? >> > > > > > > > >> > > > > > > > Thanks in advance, >> > > > > > > > Alberto. >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > News, jobs, product releases, events. >> > > > > > Follow 360dialog on LinkedIn <https://www.linkedin.com/ >> > > > company/360dialog >> > > > > > >> > > > > > and Twitter <https://twitter.com/360dialog>. >> > > > > > Subscribe to our newsletter <http://www.360dialog.com/news >> letter/ >> > >. >> > > > > > >> > > > > > >> > > > > > *Alberto del Barrio*DevOps Engineer >> > > > > > >> > > > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_ >> > > > > > content=-&utm_medium=email&utm_source=signature&utm_term=logo> >> > > > > > >> > > > > > >> > > > > > >> > > > > > *Contact 360dialog*www.360dialog.com >> > > > > > <http://www.360dialog.com/?utm_campaign=email-signature& >> > > > > > utm_content=-&utm_medium=email&utm_source=signature&utm_ >> term=url> >> > > > > > [email protected] >> > > > > > +49-(0)30-6098-5953-0 >> > > > > > >> > > > > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany >> > > > > > Managing director: Roland Siebert >> > > > > > Commercial register: Charlottenbug, HRB 144188 B >> > > > > > VAT ID: DE815382679 >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > News, jobs, product releases, events. >> > > > Follow 360dialog on LinkedIn <https://www.linkedin.com/ >> > company/360dialog >> > > > >> > > > and Twitter <https://twitter.com/360dialog>. >> > > > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. >> > > > >> > > > >> > > > *Alberto del Barrio*DevOps Engineer >> > > > >> > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_ >> > > > content=-&utm_medium=email&utm_source=signature&utm_term=logo> >> > > > >> > > > >> > > > >> > > > *Contact 360dialog*www.360dialog.com >> > > > <http://www.360dialog.com/?utm_campaign=email-signature& >> > > > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> >> > > > [email protected] >> > > > +49-(0)30-6098-5953-0 >> > > > >> > > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany >> > > > Managing director: Roland Siebert >> > > > Commercial register: Charlottenbug, HRB 144188 B >> > > > VAT ID: DE815382679 >> > > > >> > > >> > >> > >> > >> > -- >> > News, jobs, product releases, events. >> > Follow 360dialog on LinkedIn <https://www.linkedin.com/comp >> any/360dialog> >> > and Twitter <https://twitter.com/360dialog>. >> > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. >> > >> > >> > *Alberto del Barrio*DevOps Engineer >> > >> > <http://www.360dialog.com?utm_campaign=email-signature&utm_ >> > content=-&utm_medium=email&utm_source=signature&utm_term=logo> >> > >> > >> > >> > *Contact 360dialog*www.360dialog.com >> > <http://www.360dialog.com/?utm_campaign=email-signature& >> > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> >> > [email protected] >> > +49-(0)30-6098-5953-0 >> > >> > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany >> > Managing director: Roland Siebert >> > Commercial register: Charlottenbug, HRB 144188 B >> > VAT ID: DE815382679 >> > >> > > > > -- > News, jobs, product releases, events. > Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog> > and Twitter <https://twitter.com/360dialog>. > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. > > > *Alberto del Barrio*DevOps Engineer > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=logo> > > > > *Contact 360dialog*www.360dialog.com > <http://www.360dialog.com/?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> > [email protected] > +49-(0)30-6098-5953-0 > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany > Managing director: Roland Siebert > Commercial register: Charlottenbug, HRB 144188 B > VAT ID: DE815382679 > -- News, jobs, product releases, events. Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog> and Twitter <https://twitter.com/360dialog>. Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. *Alberto del Barrio*DevOps Engineer <http://www.360dialog.com?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=logo> *Contact 360dialog*www.360dialog.com <http://www.360dialog.com/?utm_campaign=email-signature&utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> [email protected] +49-(0)30-6098-5953-0 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany Managing director: Roland Siebert Commercial register: Charlottenbug, HRB 144188 B VAT ID: DE815382679
