Cool - I will try and take a look into this - Meanwhile, do you mind awfuly to change the following and see if things improve?
tickTime = 1000 initLimit=3 syncLimit=5 On 1 June 2017 at 16:49, Del Barrio, Alberto < [email protected]> wrote: > Here are the configs you were asking for: > > Zookeeper: > tickTime=1000 > initLimit=2000 > syncLimit=1000 > dataDir=/var/lib/zookeeper > clientPort=2181 > server.3=10.0.0.3:2888:3888 > server.2=10.0.0.2:2888:3888 > server.1=10.0.0.1:2888:3888 > > > Kafka broker (for one of them): > broker.id=10 > listeners=PLAINTEXT://10.0.0.4:9092 > num.network.threads=3 > num.io.threads=8 > socket.send.buffer.bytes=102400 > socket.receive.buffer.bytes=102400 > socket.request.max.bytes=104857600 > log.dirs=/var/lib/kafka > num.partitions=2 > num.recovery.threads.per.data.dir=1 > zookeeper.connect=10.0.0.1:2181,10.0.0.2:2181,10.0.0.3:2181/kafka > zookeeper.connection.timeout.ms=6000 > > In general they're pretty much the default ones. > I can see in Zookeeper the kafka brokers connected to it and exchanging > data... > > Thanks for your help and time. > > On 1 June 2017 at 17:32, Mohammed Manna <[email protected]> wrote: > > > Could you please share your broker/zookeeper/topic configs ? > > > > On 1 June 2017 at 16:18, Del Barrio, Alberto < > > [email protected]> wrote: > > > > > I tried creating the topic and results are very similar to the current > > > situation: there are not ISR and no leader for any of the partitions, > but > > > now kafka-topics shows *Leader: none* when for all the other topics, it > > > shows *Leader: -1* > > > > > > > > > On 1 June 2017 at 17:05, Mohammed Manna <[email protected]> wrote: > > > > > > > I had a similar situation, but only 1 of my ZKs was struggling - but > > > since > > > > the ISR synching time is configurable I was confident to bounce 1 ZK > > at a > > > > time and it worked out. > > > > does it happen even when you create a new topic with a > > > > replication:partition ration of 1? > > > > > > > > i meant, 3 replicas, 3 partitions :) > > > > > > > > On 1 June 2017 at 15:58, Del Barrio, Alberto < > > > > [email protected]> wrote: > > > > > > > > > Hi Mohammed, > > > > > > > > > > thanks for your answer. > > > > > The ZK cluster is not located in the servers where Kafka runs but > in > > > > other > > > > > 3 different machines. This ZK cluster is used by several other > > services > > > > > which are not reporting problems. > > > > > As you suggested, I haven't tried restarting the kafka-server > > processes > > > > > because there's no leader for topic partitions, so I don't know > what > > > will > > > > > happen. Never been in a similar situation with Kafka after some > years > > > > using > > > > > it. > > > > > > > > > > > > > > > > > > > > > > > > > On 1 June 2017 at 16:29, Mohammed Manna <[email protected]> > wrote: > > > > > > > > > > > Hi Alberto, > > > > > > > > > > > > Usually this means that the leader election/replica syncing > > couldn't > > > be > > > > > > successful and the zookeeper logs should be able to show this > > > > information > > > > > > too. The leader -1 is what worries me. For your case (3 broker > > > > cluster), > > > > > I > > > > > > am assuming you have done the cluster configuration to have 1 > > > > > > broker-zookeeper setup ? > > > > > > If that's the case, you should be able to bounce 1 zookeeper at a > > > time > > > > > and > > > > > > see if that resolves the issue. > > > > > > > > > > > > That said, have you restarted your servers since this issue > > > surfaced? > > > > > > > > > > > > On 1 June 2017 at 14:11, Del Barrio, Alberto < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I'm experiencing an issue which I don't know how to solve, so > I'm > > > > > trying > > > > > > to > > > > > > > find some guidance on the topic. > > > > > > > > > > > > > > I have a cluster composed by 3 servers, one broker per server > > > running > > > > > > Kafka > > > > > > > 0.10.0.1-1 which runs in production with around 100 topics, > most > > of > > > > > them > > > > > > > divided in several partitions and replicated always between 2 > > > > servers. > > > > > > > Suddenly I've notice when looking at my topics (with > kafka-topics > > > > tool) > > > > > > > that no one of them have a leader (Leader: -1) and the list of > > ISR > > > > > > appears > > > > > > > empty for all the topics. > > > > > > > So they look something like: > > > > > > > > > > > > > > Topic:mytopic PartitionCount:3 ReplicationFactor:2 > > Configs: > > > > > > > retention.ms=86400000 > > > > > > > Topic: mytopic Partition: 0 Leader: -1 Replicas: > > 30,10 > > > > > > Isr: > > > > > > > Topic: mytopic Partition: 1 Leader: -1 Replicas: > > 10,20 > > > > > > Isr: > > > > > > > Topic: mytopic Partition: 2 Leader: -1 Replicas: > > 20,30 > > > > > > Isr: > > > > > > > > > > > > > > However the applications using it are running normally, > consumers > > > as > > > > > well > > > > > > > as producers. > > > > > > > The logs are not showing errors or weird messages with the > > > exceptions > > > > > of > > > > > > > some > > > > > > > Failed to rename [/var/log/kafka/log-cleaner.log] to > > > > > > > [/var/log/kafka/log-cleaner.log.2017-05-31-17] > > > > > > > which appear each few days. > > > > > > > > > > > > > > Now I would like to bring back to cluster to a good state. I'm > > > afraid > > > > > of > > > > > > > restarting brokers because all of them are supposed to be > leaders > > > for > > > > > > some > > > > > > > partitions, so if I restart them and there's no leader I might > > > > > experience > > > > > > > data loss. > > > > > > > > > > > > > > Have you face any similar situation? Can someone give me a > hint? > > > > > > > > > > > > > > Thanks in advance, > > > > > > > Alberto. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > News, jobs, product releases, events. > > > > > Follow 360dialog on LinkedIn <https://www.linkedin.com/ > > > company/360dialog > > > > > > > > > > and Twitter <https://twitter.com/360dialog>. > > > > > Subscribe to our newsletter <http://www.360dialog.com/newsletter/ > >. > > > > > > > > > > > > > > > *Alberto del Barrio*DevOps Engineer > > > > > > > > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_ > > > > > content=-&utm_medium=email&utm_source=signature&utm_term=logo> > > > > > > > > > > > > > > > > > > > > *Contact 360dialog*www.360dialog.com > > > > > <http://www.360dialog.com/?utm_campaign=email-signature&; > > > > > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> > > > > > [email protected] > > > > > +49-(0)30-6098-5953-0 > > > > > > > > > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany > > > > > Managing director: Roland Siebert > > > > > Commercial register: Charlottenbug, HRB 144188 B > > > > > VAT ID: DE815382679 > > > > > > > > > > > > > > > > > > > > > -- > > > News, jobs, product releases, events. > > > Follow 360dialog on LinkedIn <https://www.linkedin.com/ > company/360dialog > > > > > > and Twitter <https://twitter.com/360dialog>. > > > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. > > > > > > > > > *Alberto del Barrio*DevOps Engineer > > > > > > <http://www.360dialog.com?utm_campaign=email-signature&utm_ > > > content=-&utm_medium=email&utm_source=signature&utm_term=logo> > > > > > > > > > > > > *Contact 360dialog*www.360dialog.com > > > <http://www.360dialog.com/?utm_campaign=email-signature&; > > > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> > > > [email protected] > > > +49-(0)30-6098-5953-0 > > > > > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany > > > Managing director: Roland Siebert > > > Commercial register: Charlottenbug, HRB 144188 B > > > VAT ID: DE815382679 > > > > > > > > > -- > News, jobs, product releases, events. > Follow 360dialog on LinkedIn <https://www.linkedin.com/company/360dialog> > and Twitter <https://twitter.com/360dialog>. > Subscribe to our newsletter <http://www.360dialog.com/newsletter/>. > > > *Alberto del Barrio*DevOps Engineer > > <http://www.360dialog.com?utm_campaign=email-signature&utm_ > content=-&utm_medium=email&utm_source=signature&utm_term=logo> > > > > *Contact 360dialog*www.360dialog.com > <http://www.360dialog.com/?utm_campaign=email-signature&; > utm_content=-&utm_medium=email&utm_source=signature&utm_term=url> > [email protected] > +49-(0)30-6098-5953-0 > > 360dialog GmbH, Saarbrücker Str. 36-38, 10405 Berlin, Germany > Managing director: Roland Siebert > Commercial register: Charlottenbug, HRB 144188 B > VAT ID: DE815382679 >
