Re: Supervisord for Kafka 0.8.1
Thanks guys. Will try these two.
On Mar 25, 2016 12:36, "Daniel Tamai" wrote:
> Hi Kashyap,
>
> Using supervisord 3.2, this is the snippet from my supervisord.conf for
> Kafka:
>
> [program:kafka]
> command=%(ENV_KAFKA_HOME)s/bin/kafka-server-start.sh
> %(ENV_CONFIG_DIR)s/server.properties
> stdout_logfile=%(ENV_LOG_DIR)s/kafka_out.log
> stderr_logfile=%(ENV_LOG_DIR)s/kafka_err.log
> umask=022
> autostart=false
> autorestart=true
> startsecs=10
> startretries=3
>
> I don't think this will work with supervisord 3.1.
>
> Em sex, 25 de mar de 2016 às 13:08, Achanta Vamsi Subhash <
> achanta.va...@flipkart.com> escreveu:
>
> > We use daemontools and this is our run file:
> >
> > #!/bin/bash
> >
> > PAC=kafka-0.8.2.x
> > APP_HOME=/usr/share/$PAC
> > # app options
> > APP_CONFIG_HOME=${APP_HOME}/config
> > APP_OPTS="${APP_CONFIG_HOME}/server.properties"
> > JVM_OPTS=""
> >
> > # jvm user options
> > if [ "abc$KAFKA_HEAP_OPTS" == "abc" ]; then
> > export KAFKA_HEAP_OPTS="-Xms8g -Xmx8g"
> > fi
> >
> > if [ "abc$KAFKA_JVM_PERFORMANCE_OPTS" == "abc" ]; then
> > export KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC
> > -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
> > -Djava.awt.headless=true"
> > fi
> >
> > exec setuidgid $APP_USER $APP_HOME/bin/kafka-server-start.sh $JVM_OPTS
> > $APP_OPTS 2>&1
> >
> >
> > On Fri, Mar 25, 2016 at 8:13 PM, Kashyap Mhaisekar
> > wrote:
> >
> > > Hi,
> > > Am having trouble configuring Kafka server starts with supervisord.
> > >
> > > Has anyone from this group succeeded in integrating Kafka server start
> > and
> > > stop via supervisord? Can you please share the snippet of his out of
> > > configured?
> > >
> > > Thanks
> > > Kashyap
> > >
> >
> >
> >
> > --
> > Regards
> > Vamsi Subhash
> >
>
Supervisord for Kafka 0.8.1
Hi, Am having trouble configuring Kafka server starts with supervisord. Has anyone from this group succeeded in integrating Kafka server start and stop via supervisord? Can you please share the snippet of his out of configured? Thanks Kashyap
Re: Keeping Zookeeper and Kafka Server Up
We use supervisord for this. It ensures that the processes are always up and running. Thanks Kashyap On Wednesday, June 17, 2015, Shayne S wrote: > kafka-server-start.sh has a -daemon option, but I don't think Zookeeper has > it. > > On Tue, Jun 16, 2015 at 11:32 PM, Su She > wrote: > > > It seems like nohup has solved this issue, even when the putty window > > becomes inactive the processes are still running (I din't need to > > interact with them). I might look into using screen or tmux as a long > > term solution. > > > > Thanks Terry and Mike! > > > > Best, > > > > Su > > > > > > On Tue, Jun 16, 2015 at 3:42 PM, Terry Bates > > > wrote: > > > Greetings, > > > > > > nohup does the trick, as Mr. Bridge has shared. If you seem to want to > > run > > > these and still have some "interactivity" with > > > the services, consider using "screen" or "tmux" as these will enable > you > > to > > > run these programs in foreground, have added > > > windows you can use to access shell, tail logs, and so on, and enable > you > > > to disconnect from the session, but still have > > > these sessions available for re-attachment. > > > > > > In addition, I using "runit" for service supervision may enable you to > > keep > > > daemons running, but if your services are dying > > > you may need to introspect more deeply on the root cause versus working > > > around it by restarting them. > > > > > > > > > *Terry Bates* > > > > > > *Email: *terryjba...@gmail.com > > > *Phone: (*412) 215-0881 > > > *Skype*: terryjbates > > > *GitHub*: https://github.com/terryjbates > > > *Linkedin*: http://www.linkedin.com/in/terryjbates/ > > > > > > > > > On Tue, Jun 16, 2015 at 3:30 PM, Mike Bridge > > > wrote: > > > > > >> Have you tried using "nohup" > > >> > > >> nohup bin/zookeeper-server-start.sh config/zookeeper.properties & > > >> nohup bin/kafka-server-start.sh config/server.properties & > > >> > > >> > > >> On Tue, Jun 16, 2015 at 3:21 PM, Su She > wrote: > > >> > > >> > Hello Everyone, > > >> > > > >> > I'm wondering how to keep Zookeeper and Kafka Server up even when my > > >> > SSH (using putty) becomes inactive. I've tried running it in the > > >> > background (using &), but it seems like it stops sometimes after a > > >> > couple hours or so and I'll have to restart zookeeper and/or the > kafka > > >> > server. > > >> > > > >> > The only remediation i've found is to export TMOUT=[big number], but > > >> > there must be another solution. > > >> > > > >> > Thank you! > > >> > > > >> > Best, > > >> > > > >> > Su > > >> > > > >> > > >
Re: Security in 0.8.2 beta
Thanks. We use the encryption approach as well. But the 2 topic approach is unique. Thank you. Kashyap. On Nov 12, 2014 1:54 AM, "Joe Stein" wrote: > I know a few implements that do this "encrypt your messages with a PSK > between producers and consumers". One of them actually writes the > "encrypted " on a different topic foreach downstream > consumer private key that can read the message. This way when you are > consuming you consume from two topics 1) the topic with the message (which > is encrypted) you want 2) the topic that you can use your private key > to decrypt (because your public key was used) the symmetric key and then > use that to decrypt the message (which you join from the two streams by the > uuid so each message has a different secrete key encrypted with your public > key) The other ones I can't talk about =8^) but this one I mention is > interesting solution to this problem with Kafka I really like. > > /*** > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > / > > On Wed, Nov 12, 2014 at 2:41 AM, Mathias Herberts < > mathias.herbe...@gmail.com> wrote: > > > Simply encrypt your messages with a PSK between producers and consumers. > > On Nov 12, 2014 4:38 AM, "Kashyap Mhaisekar" > wrote: > > > > > Hi, > > > Is there a way to secure the topics created in Kafka 0.8.2 beta? The > need > > > is to ensure no one is asked to read data from the topic without > > > authorization. > > > > > > Regards > > > Kashyap > > > > > >
Security in 0.8.2 beta
Hi, Is there a way to secure the topics created in Kafka 0.8.2 beta? The need is to ensure no one is asked to read data from the topic without authorization. Regards Kashyap
Re: Partitions per Machine for a topic
Great. Thanks all. But there is no easy way to change partitions once the topic is created right? I run kafka 0.8.2 and its a difficult job to reassign partitions. On Fri, Jul 25, 2014 at 3:10 PM, Clark Haskins < chask...@linkedin.com.invalid> wrote: > You can have more partitions than machines in the cluster, you cannot > however have a replication factor that is great than the number of > machines in the cluster. > > You could easily have a topic with 100 partitions on a 3 node cluster. > > -Clark > > > Clark Elliott Haskins III > LinkedIn DDS Site Reliability Engineer > Kafka, Zookeeper, Samza SRE > Mobile: 505.385.1484 > BlueJeans: https://www.bluejeans.com/chaskins > > > chask...@linkedin.com > https://www.linkedin.com/in/clarkhaskins > There is no place like 127.0.0.1 > > > > > On 7/23/14, 2:14 PM, "Kashyap Mhaisekar" wrote: > > >HI, > >Is the maximum no. of partitions for a topic dependent on the no. of > >machines in a kafka cluster? > >For e.g., if I have 3 machines in a cluster, can I have 5 partitions with > >a > >caveat that one machine can host multiple partitions for a given topic? > > > >Regards, > >Kashyap > >
Partitions per Machine for a topic
HI, Is the maximum no. of partitions for a topic dependent on the no. of machines in a kafka cluster? For e.g., if I have 3 machines in a cluster, can I have 5 partitions with a caveat that one machine can host multiple partitions for a given topic? Regards, Kashyap
Partition reassign Kafka 0.8.1.1
Hi,
I was testing out Kafka 0.8.1.1 and found that i get the following
exception during partition re-assignment :
*./kafka-reassign-partitions.sh --path-to-json-file ritest.json --zookeeper
localhost:2181*
*Partitions reassignment failed due to Partition reassignment currently in
progress for Map(). Aborting operation*
*kafka.common.AdminCommandFailedException: Partition reassignment currently
in progress for Map(). Aborting operation*
*at
kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:91)*
*at
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:65)*
*at
kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)*
*My JSOn file is as follows *
*{"partitions":[{"topic": "ritest1","partition": 3,
"replicas": [1,2,3] }]}*
What am I doing wrong?
Regards,
Kashyap
Re: Loss of Leader in Kafka
Hi Neha/Jun, A kafa topic deletion involves - 1. Zookeeper clean up (Nodes in zookeeper clusters to be removed - one under brokers/ and other where the offset info is stored) 2. kafka data logs to cleared on each server in cluster for the topic to be deleted 3. partition and replication data to be removed on each server in cluster for the topic to be deleted So to delete a topic, if we can use kafka-delete-topic.sh and manually ensure the above 3, we should be able to restart the kafka servers and confirm that topic is indeed deleted and can be recreated. Am I right with the above? Am still on 0.8.0. and trying to understand what is in 0.8.1 that is missing in 0.8.0 Regards, Kashyap On Tue, May 13, 2014 at 11:09 AM, Neha Narkhede wrote: > >> 3. Deleted the topic. Checked only the zookeeper to see if the > /brokers/topics DOES NOT have the topic > > You are seeing this problem since delete topic is not supported in Kafka. > Any attempt to delete a topic may leave your cluster in an unstable state. > We plan to release 0.8.2 with delete topic support or you could try the > latest trunk. > > > On Mon, May 12, 2014 at 9:39 AM, Kashyap Mhaisekar >wrote: > > > Hi, > > I am hitting a strange exception while creating a topic in Kafka - > > Steps to generate this- > > 1. Created a topic multipartition_test with 2 partitions and 2 replicas > > 2. Added some data to this topics and verified data is coming up for both > > partitions > > 3. Deleted the topic. Checked only the zookeeper to see if the > > /brokers/topics DOES NOT have the topic > > 4. Recreated the topic in exactly the same way as in point 1. > > > > After this, when I list topics using ./kafka-list-topic.sh, i see that > > *leader:* none and *isr:* for this topic. State change logs give the > > following exception. > > > > kafka.common.StateChangeFailedException: encountered error while electing > > leader for partition [multipartition_test,1] due to: LeaderAndIsr > > information doesn't exist for partition [multipartition_test,1] in > > OnlinePartition state. > > at > > > > > kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:327) > > at > > > > > kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:154) > > at > > > > > kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:110) > > at > > > > > kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:109) > > at scala.collection.immutable.Set$Set2.foreach(Set.scala:101) > > at > > > > > kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:109) > > at > > > > > kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:325) > > at > > > > > kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:312) > > at > > > > > kafka.controller.PartitionStateMachine$TopicChangeListener.liftedTree1$1(PartitionStateMachine.scala:376) > > at > > > > > kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:361) > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) > > at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) > > Caused by: kafka.common.StateChangeFailedException: LeaderAndIsr > > information doesn't exist for partition [multipartition_test,1] in > > OnlinePartition state > > at > > > > > kafka.controller.PartitionStateMachine.getLeaderIsrAndEpochOrThrowException(PartitionStateMachine.scala:347) > > at > > > > > kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:291) > > ... 11 more > > > > Can you please help what am I doing wrong? > > > > Regards, > > kashyap > > >
Re: Loss of Leader in Kafka
After topic deletion, I deleted the physical folders in data logs and then restarted Kafka. That resolved. Thanks! Will try out 0.8.1 and let the group know. Regards, Kashyap On Monday, May 12, 2014, Jun Rao wrote: > Delete topic doesn't quite work in 0.8.1. We recently fixed it in trunk. > Could you give it a try and see if you see the same issue? > > Thanks, > > Jun > > > On Mon, May 12, 2014 at 9:39 AM, Kashyap Mhaisekar > > >wrote: > > > Hi, > > I am hitting a strange exception while creating a topic in Kafka - > > Steps to generate this- > > 1. Created a topic multipartition_test with 2 partitions and 2 replicas > > 2. Added some data to this topics and verified data is coming up for both > > partitions > > 3. Deleted the topic. Checked only the zookeeper to see if the > > /brokers/topics DOES NOT have the topic > > 4. Recreated the topic in exactly the same way as in point 1. > > > > After this, when I list topics using ./kafka-list-topic.sh, i see that > > *leader:* none and *isr:* for this topic. State change logs give the > > following exception. > > > > kafka.common.StateChangeFailedException: encountered error while electing > > leader for partition [multipartition_test,1] due to: LeaderAndIsr > > information doesn't exist for partition [multipartition_test,1] in > > OnlinePartition state. > > at > > > > > kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:327) > > at > > > > > kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:154) > > at > > > > > kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:110) > > at > > > > > kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:109) > > at scala.collection.immutable.Set$Set2.foreach(Set.scala:101) > > at > > > > > kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:109) > > at > > > > > kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:325) > > at > > > > > kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:312) > > at > > > > > kafka.controller.PartitionStateMachine$TopicChangeListener.liftedTree1$1(PartitionStateMachine.scala:376) > > at > > > > > kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:361) > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) > > at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) > > Caused by: kafka.common.StateChangeFailedException: LeaderAndIsr > > information doesn't exist for partition [multipartition_test,1] in > > OnlinePartition state > > at > > > > > kafka.controller.PartitionStateMachine.getLeaderIsrAndEpochOrThrowException(PartitionStateMachine.scala:347) > > at > > > > > kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:291) > > ... 11 more > > > > Can you please help what am I doing wrong? > > > > Regards, > > kashyap > > >
Loss of Leader in Kafka
Hi, I am hitting a strange exception while creating a topic in Kafka - Steps to generate this- 1. Created a topic multipartition_test with 2 partitions and 2 replicas 2. Added some data to this topics and verified data is coming up for both partitions 3. Deleted the topic. Checked only the zookeeper to see if the /brokers/topics DOES NOT have the topic 4. Recreated the topic in exactly the same way as in point 1. After this, when I list topics using ./kafka-list-topic.sh, i see that *leader:* none and *isr:* for this topic. State change logs give the following exception. kafka.common.StateChangeFailedException: encountered error while electing leader for partition [multipartition_test,1] due to: LeaderAndIsr information doesn't exist for partition [multipartition_test,1] in OnlinePartition state. at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:327) at kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:154) at kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:110) at kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:109) at scala.collection.immutable.Set$Set2.foreach(Set.scala:101) at kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:109) at kafka.controller.KafkaController.onNewPartitionCreation(KafkaController.scala:325) at kafka.controller.KafkaController.onNewTopicCreation(KafkaController.scala:312) at kafka.controller.PartitionStateMachine$TopicChangeListener.liftedTree1$1(PartitionStateMachine.scala:376) at kafka.controller.PartitionStateMachine$TopicChangeListener.handleChildChange(PartitionStateMachine.scala:361) at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568) at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71) Caused by: kafka.common.StateChangeFailedException: LeaderAndIsr information doesn't exist for partition [multipartition_test,1] in OnlinePartition state at kafka.controller.PartitionStateMachine.getLeaderIsrAndEpochOrThrowException(PartitionStateMachine.scala:347) at kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:291) ... 11 more Can you please help what am I doing wrong? Regards, kashyap
Re: Log Retention in Kafka
Thanks Jun. I am using kafka_2.8.0-0.8.0-beta1.jar. I dont see this script kafka-topic.sh in the bin folder. Is there way to do this in the version mentioned? --Kashyap On Mon, May 5, 2014 at 11:18 AM, Jun Rao wrote: > See http://kafka.apache.org/documentation.html#basic_ops_modify_topic > > Thanks, > > Jun > > > On Sun, May 4, 2014 at 10:11 PM, Kashyap Mhaisekar >wrote: > > > Is there a way to do this at runtime using some available scripts in > > kafka/bin? If so, any pointers on which script? > > > > Regards, > > Kashyap > > > > > > On Tue, Apr 22, 2014 at 11:11 PM, Kashyap Mhaisekar > >wrote: > > > > > Thanks Joel. Am using version 2.8.0. > > > > > > Thanks, > > > Kashyap > > > > > > > > > On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy > wrote: > > > > > >> Which version of Kafka are you using? > > >> > > >> You can read up on the configuration options here: > > >> http://kafka.apache.org/documentation.html#configuration > > >> > > >> You can specify time-based retention using log.retention.minutes which > > >> will apply to all topics. You can override that on per-topic basis - > > >> see further down in the above page under "topic-level configuration" > > >> > > >> On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote: > > >> > Hi, > > >> > I wanted to set the message expiry for a message on a kafka topic. > Is > > >> there > > >> > anything like this in kafka? > > >> > I came across a property - *log.retention.hours* and > > >> > *topic.log.retention.hours* > > >> > Had some queries around it.And it was mentioned that > > >> > topic.log.retention.hours is per topic configuration. > > >> > Had some queries around it - > > >> > 1. Does it mean that I need to specific the > > >> .log.retention.hours > > >> > in the kafka config? > > >> > 2. Can this property be overriden anywhere? > > >> > 3. Is it possible for the producer to set a message expiry so that > the > > >> > message expires after a configurable period of time? > > >> > > > >> > Regards, > > >> > Kashyap > > >> > > >> > > > > > >
Re: Log Retention in Kafka
Is there a way to do this at runtime using some available scripts in kafka/bin? If so, any pointers on which script? Regards, Kashyap On Tue, Apr 22, 2014 at 11:11 PM, Kashyap Mhaisekar wrote: > Thanks Joel. Am using version 2.8.0. > > Thanks, > Kashyap > > > On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy wrote: > >> Which version of Kafka are you using? >> >> You can read up on the configuration options here: >> http://kafka.apache.org/documentation.html#configuration >> >> You can specify time-based retention using log.retention.minutes which >> will apply to all topics. You can override that on per-topic basis - >> see further down in the above page under "topic-level configuration" >> >> On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote: >> > Hi, >> > I wanted to set the message expiry for a message on a kafka topic. Is >> there >> > anything like this in kafka? >> > I came across a property - *log.retention.hours* and >> > *topic.log.retention.hours* >> > Had some queries around it.And it was mentioned that >> > topic.log.retention.hours is per topic configuration. >> > Had some queries around it - >> > 1. Does it mean that I need to specific the >> .log.retention.hours >> > in the kafka config? >> > 2. Can this property be overriden anywhere? >> > 3. Is it possible for the producer to set a message expiry so that the >> > message expires after a configurable period of time? >> > >> > Regards, >> > Kashyap >> >> >
Re: Log Retention in Kafka
Thanks Joel. Am using version 2.8.0. Thanks, Kashyap On Tue, Apr 22, 2014 at 5:53 PM, Joel Koshy wrote: > Which version of Kafka are you using? > > You can read up on the configuration options here: > http://kafka.apache.org/documentation.html#configuration > > You can specify time-based retention using log.retention.minutes which > will apply to all topics. You can override that on per-topic basis - > see further down in the above page under "topic-level configuration" > > On Tue, Apr 22, 2014 at 02:34:24PM -0500, Kashyap Mhaisekar wrote: > > Hi, > > I wanted to set the message expiry for a message on a kafka topic. Is > there > > anything like this in kafka? > > I came across a property - *log.retention.hours* and > > *topic.log.retention.hours* > > Had some queries around it.And it was mentioned that > > topic.log.retention.hours is per topic configuration. > > Had some queries around it - > > 1. Does it mean that I need to specific the > .log.retention.hours > > in the kafka config? > > 2. Can this property be overriden anywhere? > > 3. Is it possible for the producer to set a message expiry so that the > > message expires after a configurable period of time? > > > > Regards, > > Kashyap > >
Log Retention in Kafka
Hi, I wanted to set the message expiry for a message on a kafka topic. Is there anything like this in kafka? I came across a property - *log.retention.hours* and *topic.log.retention.hours* Had some queries around it.And it was mentioned that topic.log.retention.hours is per topic configuration. Had some queries around it - 1. Does it mean that I need to specific the .log.retention.hours in the kafka config? 2. Can this property be overriden anywhere? 3. Is it possible for the producer to set a message expiry so that the message expires after a configurable period of time? Regards, Kashyap
Re: Leader: -1 on Kafka
No. Is this error common? How to overcome this? Regards, Kashyap On Mon, Apr 21, 2014 at 4:02 PM, Joel Koshy wrote: > So only one of the brokers have the data? i.e., with one partition. > > Do you see errors in your broker log around that time? What do you see > in the kafka-state-change.log for that topic/partition? > > On Mon, Apr 21, 2014 at 03:36:52PM -0500, Kashyap Mhaisekar wrote: > > Yes. > > I have 3 kafka brokers and I have created one topic with 1 partition and > 1 > > replication (default options). All params are default. I was testing it > > with heavy load and had a storm reading from this and suddenly the > messages > > stop. kafka's list topics script files then show the leader as -1. > > > > I got the same error even with 2 partitions and default replication. > > > > Regards, > > Kashyap > > > > > > On Mon, Apr 21, 2014 at 1:26 PM, Joel Koshy wrote: > > > > > Can you describe your set up in more detail and also if you can > > > reproduce this easily? This can happen when none of the replicas for a > > > partition are available, but cannot comment further without details. > > > > > > On Mon, Apr 21, 2014 at 10:54:59AM -0500, Kashyap Mhaisekar wrote: > > > > Hi, > > > > At times, some of kafka topics end up showing the leader as -1. After > > > this, > > > > the messages dont get added to the topic nor consumed. I tried > digging > > > into > > > > why the leader turns -1 *(leader: -1)* > > > > > > > > Is there a reason why this happens and how it can be resolved? > > > > > > > > Regards, > > > > kashyap > > > > > > > >
Re: Leader: -1 on Kafka
Yes. I have 3 kafka brokers and I have created one topic with 1 partition and 1 replication (default options). All params are default. I was testing it with heavy load and had a storm reading from this and suddenly the messages stop. kafka's list topics script files then show the leader as -1. I got the same error even with 2 partitions and default replication. Regards, Kashyap On Mon, Apr 21, 2014 at 1:26 PM, Joel Koshy wrote: > Can you describe your set up in more detail and also if you can > reproduce this easily? This can happen when none of the replicas for a > partition are available, but cannot comment further without details. > > On Mon, Apr 21, 2014 at 10:54:59AM -0500, Kashyap Mhaisekar wrote: > > Hi, > > At times, some of kafka topics end up showing the leader as -1. After > this, > > the messages dont get added to the topic nor consumed. I tried digging > into > > why the leader turns -1 *(leader: -1)* > > > > Is there a reason why this happens and how it can be resolved? > > > > Regards, > > kashyap > >
Leader: -1 on Kafka
Hi, At times, some of kafka topics end up showing the leader as -1. After this, the messages dont get added to the topic nor consumed. I tried digging into why the leader turns -1 *(leader: -1)* Is there a reason why this happens and how it can be resolved? Regards, kashyap
