Can we archive our kafka-requestlog in kafka0.8??
Hello All, I am using kafka 0.8. I want to archive my kafka-request.log, server.log, controller.log, staet-change.log. Is there any property in log4j by which i can archive my logs in zip folders or files. Please suggest. Regards, Nishant Kumar
kafka build error scala 2.10
While doing dev setup as described in https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup im getting following build errors. immutable is already defined as class immutable Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 38 Scala Problem threadsafe is already defined as class threadsafe Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 28 Scala Problem nonthreadsafe is already defined as class nonthreadsafe Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 33 Scala Problem This error is coming from a file Util /kafka/src/main/scala/kafka/utils/Annotations_2.9+.scala Please note that i had to install scala 2.10 eclipse plugin as Juno had some problem with 2.9. Regards, Pushkar
Re: regarding run-simulator.sh
i see many tools mentioned for perf here https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing of all these what all already exist in 0.8 release? e.g. i was not able to find jmx-dump.sh , R script etc anywhere. On Wed, Dec 18, 2013 at 11:01 AM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: thanks Jun. On Wed, Dec 18, 2013 at 10:47 AM, Jun Rao jun...@gmail.com wrote: You can run kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh. Thanks, Jun On Tue, Dec 17, 2013 at 8:44 PM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: i am not able to find run-simulator.sh in 0.8 even after building perf.if this tool has been deprecated what are other alternatives available now for perf testing? Regards, Pushkar
Re: Kafka producer behavior
Thanks for response Gerrit and Guozhang !! Hi Gerrit, I am trying to use same round robin partitioner shared by you but hard luck, still round robin partitioning not working. I have successfully registered RoundRobinPartitioner in kafka producer. Code of RoundRobinPartitioner class as: public RoundRobinPartitioner(VerifiableProperties props){ log.info(Using Round Robin Partitioner class...); } @Override public int partition(String key, int partitions) { log.info(Inside partition method); int i = counter.getAndIncrement(); if(i == Integer.MAX_VALUE){ counter.set(0); return 0; }else return i % partitions; } When i produce the data, first log message Using Round Robin Partitioner class... is printed and second message Inside partition method is not printed. From that we can ensure that RoundRobinPartitioner has been successfully registered but logic of round robin is not getting called. Any help to resolve what i am missing ? Thanks in advance !! On Tue, Dec 17, 2013 at 5:59 PM, Guozhang Wang wangg...@gmail.com wrote: Hello, This issue is known as in this JIRA: https://issues.apache.org/jira/browse/KAFKA-1067 Guozhang On Tue, Dec 17, 2013 at 8:48 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: hi, I've had the same issue with the kafka producer. you need to use a different partitioner than the default one provided for kafka. I've created a round robin partitioner that works well for equally distributing data across partitions. https://github.com/gerritjvv/pseidon/blob/master/pseidon-kafka/java/pseidon/kafka/util/RoundRobinPartitioner.java On Tue, Dec 17, 2013 at 5:32 PM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have created a topic test-topic1 in kafka. When i am listing status of that topic using bin/kafka-list-topic.sh, the status is: topic: test-topic1partition: 0leader: 0 replicas: 0,1 isr: 0,1 topic: test-topic1partition: 1leader: 1 replicas: 1,0 isr: 1,0 As both partition are on two separate nodes so when we produce the data it should be go to both nodes. But when i insert the data, it is going to only one node. For example if i insert 1000 messages then all 1000 messages will go either node1 or node2. Data is not evenly distributed on both nodes. Expected: 500 messages should go to node1 and 500 messages should go to node2. Any suggestion why i am facing this behavior? -- *Thanks Regards* *Hanish Bansal* -- -- Guozhang -- *Thanks Regards* *Hanish Bansal*
Re: Kafka producer behavior
Hi, this is a gotcha about kafka producer partitioning, you much send the messages with a non null key. If the key is null kafka will not call the partitioner. Because with this partitioner the key does not matter you can pass in a constant string like 1 etc. Oh one more thing, on performance: The produce's send method has a synchronized block on the producer instance, which means performance goes down the drain. I could only get (on a 12 core, 72 gig ram) machine 13K tps out of the producer. A way to solve this is to instantiate an array/list of N producers and then in your send code round robin over the producers. I got to 80K tps (for my use case) using 6 producer instances from a single box sending to 3 kafka servers. e.g. send ( msg ) { producers[ producer-index.getAndIncrement() % producer_count ].send(msg) } Regards, Gerrit On Wed, Dec 18, 2013 at 11:24 AM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Thanks for response Gerrit and Guozhang !! Hi Gerrit, I am trying to use same round robin partitioner shared by you but hard luck, still round robin partitioning not working. I have successfully registered RoundRobinPartitioner in kafka producer. Code of RoundRobinPartitioner class as: public RoundRobinPartitioner(VerifiableProperties props){ log.info(Using Round Robin Partitioner class...); } @Override public int partition(String key, int partitions) { log.info(Inside partition method); int i = counter.getAndIncrement(); if(i == Integer.MAX_VALUE){ counter.set(0); return 0; }else return i % partitions; } When i produce the data, first log message Using Round Robin Partitioner class... is printed and second message Inside partition method is not printed. From that we can ensure that RoundRobinPartitioner has been successfully registered but logic of round robin is not getting called. Any help to resolve what i am missing ? Thanks in advance !! On Tue, Dec 17, 2013 at 5:59 PM, Guozhang Wang wangg...@gmail.com wrote: Hello, This issue is known as in this JIRA: https://issues.apache.org/jira/browse/KAFKA-1067 Guozhang On Tue, Dec 17, 2013 at 8:48 AM, Gerrit Jansen van Vuuren gerrit...@gmail.com wrote: hi, I've had the same issue with the kafka producer. you need to use a different partitioner than the default one provided for kafka. I've created a round robin partitioner that works well for equally distributing data across partitions. https://github.com/gerritjvv/pseidon/blob/master/pseidon-kafka/java/pseidon/kafka/util/RoundRobinPartitioner.java On Tue, Dec 17, 2013 at 5:32 PM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have created a topic test-topic1 in kafka. When i am listing status of that topic using bin/kafka-list-topic.sh, the status is: topic: test-topic1partition: 0leader: 0 replicas: 0,1 isr: 0,1 topic: test-topic1partition: 1leader: 1 replicas: 1,0 isr: 1,0 As both partition are on two separate nodes so when we produce the data it should be go to both nodes. But when i insert the data, it is going to only one node. For example if i insert 1000 messages then all 1000 messages will go either node1 or node2. Data is not evenly distributed on both nodes. Expected: 500 messages should go to node1 and 500 messages should go to node2. Any suggestion why i am facing this behavior? -- *Thanks Regards* *Hanish Bansal* -- -- Guozhang -- *Thanks Regards* *Hanish Bansal*
Data loss in case of request.required.acks set to -1
Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have configured request.required.acks in producer configuration to -1. As mentioned in documentation http://kafka.apache.org/documentation.html#producerconfigs, setting this value to -1 provides guarantee that no messages will be lost. I am getting below behaviour: If kafka is running as foreground process and i am shutting down the kafka leader node using ctrl+C then no data is lost. But if i abnormally terminate the kafka using kill -9 pid then still facing data loss even after configuring request.required.acks to -1. Any suggestions? -- *Thanks Regards* *Hanish Bansal*
Re: Data loss in case of request.required.acks set to -1
You can try setting a higher value for message.send.max.retries in producer config. Regards, Pushkar On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have configured request.required.acks in producer configuration to -1. As mentioned in documentation http://kafka.apache.org/documentation.html#producerconfigs, setting this value to -1 provides guarantee that no messages will be lost. I am getting below behaviour: If kafka is running as foreground process and i am shutting down the kafka leader node using ctrl+C then no data is lost. But if i abnormally terminate the kafka using kill -9 pid then still facing data loss even after configuring request.required.acks to -1. Any suggestions? -- *Thanks Regards* *Hanish Bansal*
Re: problem with high-level consumer stream filter regex....
Joe, I think the java code I listed in the Jira ticket should reproduce the issue directly, does that not work? Jason On Tue, Dec 17, 2013 at 9:49 AM, Joe Stein joe.st...@stealth.ly wrote: Hi Jason, I just replied on the ticket. If it is a bug the update to create new filter or fix as bug, same. Can you post some code to help reproduce the problem? so apples to apples and such, thanks! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Dec 17, 2013 at 1:16 AM, Jason Rosenberg j...@squareup.com wrote: Ping Any thoughts on this? Seems like a bug, but then again, we're not sure what the expected behavior for regexes should be here (e.g. is there a way to whitelist topics with a filter that looks for a leading substring, but then blocks subsequent substrings)? E.g. apply a blacklist to a whitelist :). Jason On Thu, Dec 12, 2013 at 1:01 PM, Jason Rosenberg j...@squareup.com wrote: All, I've filed: https://issues.apache.org/jira/browse/KAFKA-1180 We are needing to create a stream selector that essentially combines the logic of the BlackList and WhiteList classes. That is, we want to select a topic that contains a certain prefix, as long as it doesn't also contain a secondary string. This should be easy to do with ordinary java Regex's, but we're running into some issues, trying to do this with the WhiteList class only. We have a pattern that uses negative lookahead, like this: test-(?!bad\\b)[\\w]+ So this should select a topic like: test-good, but exclude a topic like test-bad, and also exclude a topic without the test prefix, like foo-bar. Instead, what we see is a NullPointerException in the ConsumerIterator, and the consumer just hangs, after sending a topic of 'test-topic' followed by 'test-bad': 21700 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683] ERROR kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683], Error due to kafka.common.KafkaException: error processing data for partition [test-bad,0] offset 0 at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:137) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109) at scala.collection.immutable.Map$Map1.foreach(Map.scala:105) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) at kafka.utils.Utils$.inLock(Utils.scala:565) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Caused by: java.lang.NullPointerException at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60) at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128) ... 9 more
Re: Data loss in case of request.required.acks set to -1
Hi pushkar, I tried with configuring message.send.max.retries to 10. Default value for this is 3. But still facing data loss. On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: You can try setting a higher value for message.send.max.retries in producer config. Regards, Pushkar On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have configured request.required.acks in producer configuration to -1. As mentioned in documentation http://kafka.apache.org/documentation.html#producerconfigs, setting this value to -1 provides guarantee that no messages will be lost. I am getting below behaviour: If kafka is running as foreground process and i am shutting down the kafka leader node using ctrl+C then no data is lost. But if i abnormally terminate the kafka using kill -9 pid then still facing data loss even after configuring request.required.acks to -1. Any suggestions? -- *Thanks Regards* *Hanish Bansal* -- *Thanks Regards* *Hanish Bansal*
Re: problem with high-level consumer stream filter regex....
Hey Jason, I have someone looking into it now (they just started). I can look at it on Friday or if I finish up what I am working on for tomorrow then sooner. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Dec 18, 2013 at 8:15 AM, Jason Rosenberg j...@squareup.com wrote: Joe, I think the java code I listed in the Jira ticket should reproduce the issue directly, does that not work? Jason On Tue, Dec 17, 2013 at 9:49 AM, Joe Stein joe.st...@stealth.ly wrote: Hi Jason, I just replied on the ticket. If it is a bug the update to create new filter or fix as bug, same. Can you post some code to help reproduce the problem? so apples to apples and such, thanks! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Dec 17, 2013 at 1:16 AM, Jason Rosenberg j...@squareup.com wrote: Ping Any thoughts on this? Seems like a bug, but then again, we're not sure what the expected behavior for regexes should be here (e.g. is there a way to whitelist topics with a filter that looks for a leading substring, but then blocks subsequent substrings)? E.g. apply a blacklist to a whitelist :). Jason On Thu, Dec 12, 2013 at 1:01 PM, Jason Rosenberg j...@squareup.com wrote: All, I've filed: https://issues.apache.org/jira/browse/KAFKA-1180 We are needing to create a stream selector that essentially combines the logic of the BlackList and WhiteList classes. That is, we want to select a topic that contains a certain prefix, as long as it doesn't also contain a secondary string. This should be easy to do with ordinary java Regex's, but we're running into some issues, trying to do this with the WhiteList class only. We have a pattern that uses negative lookahead, like this: test-(?!bad\\b)[\\w]+ So this should select a topic like: test-good, but exclude a topic like test-bad, and also exclude a topic without the test prefix, like foo-bar. Instead, what we see is a NullPointerException in the ConsumerIterator, and the consumer just hangs, after sending a topic of 'test-topic' followed by 'test-bad': 21700 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683] ERROR kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683], Error due to kafka.common.KafkaException: error processing data for partition [test-bad,0] offset 0 at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:137) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109) at scala.collection.immutable.Map$Map1.foreach(Map.scala:105) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) at kafka.utils.Utils$.inLock(Utils.scala:565) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Caused by: java.lang.NullPointerException at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60) at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128) ... 9 more
Re: Data loss in case of request.required.acks set to -1
How many replicas do you have? On Wed, Dec 18, 2013 at 8:57 AM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi pushkar, I tried with configuring message.send.max.retries to 10. Default value for this is 3. But still facing data loss. On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: You can try setting a higher value for message.send.max.retries in producer config. Regards, Pushkar On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have configured request.required.acks in producer configuration to -1. As mentioned in documentation http://kafka.apache.org/documentation.html#producerconfigs, setting this value to -1 provides guarantee that no messages will be lost. I am getting below behaviour: If kafka is running as foreground process and i am shutting down the kafka leader node using ctrl+C then no data is lost. But if i abnormally terminate the kafka using kill -9 pid then still facing data loss even after configuring request.required.acks to -1. Any suggestions? -- *Thanks Regards* *Hanish Bansal* -- *Thanks Regards* *Hanish Bansal*
Re: Can we archive our kafka-requestlog in kafka0.8??
Take a look at http://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppenderand search for compress. Thanks, Jun On Tue, Dec 17, 2013 at 11:58 PM, Nishant Kumar nish.a...@gmail.com wrote: Hello All, I am using kafka 0.8. I want to archive my kafka-request.log, server.log, controller.log, staet-change.log. Is there any property in log4j by which i can archive my logs in zip folders or files. Please suggest. Regards, Nishant Kumar
Re: kafka build error scala 2.10
You may have to exclude Annotations.scala. Thanks, Jun On Wed, Dec 18, 2013 at 12:16 AM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: While doing dev setup as described in https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup im getting following build errors. immutable is already defined as class immutable Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 38 Scala Problem threadsafe is already defined as class threadsafe Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 28 Scala Problem nonthreadsafe is already defined as class nonthreadsafe Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 33 Scala Problem This error is coming from a file Util /kafka/src/main/scala/kafka/utils/Annotations_2.9+.scala Please note that i had to install scala 2.10 eclipse plugin as Juno had some problem with 2.9. Regards, Pushkar
Re: regarding run-simulator.sh
Most tools can be found in the kafka.tools package. For dumping jmx, use JmxTool. Thanks, Jun On Wed, Dec 18, 2013 at 1:03 AM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: i see many tools mentioned for perf here https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing of all these what all already exist in 0.8 release? e.g. i was not able to find jmx-dump.sh , R script etc anywhere. On Wed, Dec 18, 2013 at 11:01 AM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: thanks Jun. On Wed, Dec 18, 2013 at 10:47 AM, Jun Rao jun...@gmail.com wrote: You can run kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh. Thanks, Jun On Tue, Dec 17, 2013 at 8:44 PM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: i am not able to find run-simulator.sh in 0.8 even after building perf.if this tool has been deprecated what are other alternatives available now for perf testing? Regards, Pushkar
Re: Data loss in case of request.required.acks set to -1
my doubt was they are dropping off at producer level only.so suggested playing with paramaters like retries and backoff.ms and also with refreshinterval on producer side. Regards, Pushkar On Wed, Dec 18, 2013 at 10:01 PM, Guozhang Wang wangg...@gmail.com wrote: Hanish, Did you kill -9 one of the brokers only or bouncing them iteratively? Guozhang On Wed, Dec 18, 2013 at 8:02 AM, Joe Stein joe.st...@stealth.ly wrote: How many replicas do you have? On Wed, Dec 18, 2013 at 8:57 AM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi pushkar, I tried with configuring message.send.max.retries to 10. Default value for this is 3. But still facing data loss. On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: You can try setting a higher value for message.send.max.retries in producer config. Regards, Pushkar On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have configured request.required.acks in producer configuration to -1. As mentioned in documentation http://kafka.apache.org/documentation.html#producerconfigs, setting this value to -1 provides guarantee that no messages will be lost. I am getting below behaviour: If kafka is running as foreground process and i am shutting down the kafka leader node using ctrl+C then no data is lost. But if i abnormally terminate the kafka using kill -9 pid then still facing data loss even after configuring request.required.acks to -1. Any suggestions? -- *Thanks Regards* *Hanish Bansal* -- *Thanks Regards* *Hanish Bansal* -- -- Guozhang
Re: Migrating a cluster from 0.8.0 to 0.8.1
Thanks Neha, I rolled upgrades and completed a rebalance! I ran into a few small issues I figured I would share. On a few Brokers, there were some log directories left over from some failed rebalances which prevented the 0.8.1 brokers from starting once I completed the upgrade. These directories contained an index file and a zero size log file, once I cleaned those out the brokers were able to start up fine. If anyone else runs into the same problem, and is running RHEL, this is the bash script I used to clean them out: du --max-depth=1 -h /data/kafka/logs | grep K | sed s/.*K.// | sudo rm -r On Tue, Dec 17, 2013 at 10:42 AM, Neha Narkhede neha.narkh...@gmail.comwrote: There are no compatibility issues. You can roll upgrades through the cluster one node at a time. Thanks Neha On Tue, Dec 17, 2013 at 9:15 AM, Drew Goya d...@gradientx.com wrote: So I'm going to be going through the process of upgrading a cluster from 0.8.0 to the trunk (0.8.1). I'm going to be expanding this cluster several times and the problems with reassigning partitions in 0.8.0 mean I have to move to trunk(0.8.1) asap. Will it be safe to roll upgrades through the cluster one by one? Also are there any client compatibility issues I need to worry about? Am I going to need to pause/upgrade all my consumers/producers at once or can I roll upgrades through the cluster and then upgrade my clients one by one? Thanks in advance!
Re: Data loss in case of request.required.acks set to -1
Wouldn't you want to set the controlled.shutdown.enable to true so the broker would do this for you before ending itself? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Dec 18, 2013 at 11:36 AM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: my doubt was they are dropping off at producer level only.so suggested playing with paramaters like retries and backoff.ms and also with refreshinterval on producer side. Regards, Pushkar On Wed, Dec 18, 2013 at 10:01 PM, Guozhang Wang wangg...@gmail.com wrote: Hanish, Did you kill -9 one of the brokers only or bouncing them iteratively? Guozhang On Wed, Dec 18, 2013 at 8:02 AM, Joe Stein joe.st...@stealth.ly wrote: How many replicas do you have? On Wed, Dec 18, 2013 at 8:57 AM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi pushkar, I tried with configuring message.send.max.retries to 10. Default value for this is 3. But still facing data loss. On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: You can try setting a higher value for message.send.max.retries in producer config. Regards, Pushkar On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal hanish.bansal.agar...@gmail.com wrote: Hi All, We are having kafka cluster of 2 nodes. (using 0.8.0 final release) Replication Factor: 2 Number of partitions: 2 I have configured request.required.acks in producer configuration to -1. As mentioned in documentation http://kafka.apache.org/documentation.html#producerconfigs, setting this value to -1 provides guarantee that no messages will be lost. I am getting below behaviour: If kafka is running as foreground process and i am shutting down the kafka leader node using ctrl+C then no data is lost. But if i abnormally terminate the kafka using kill -9 pid then still facing data loss even after configuring request.required.acks to -1. Any suggestions? -- *Thanks Regards* *Hanish Bansal* -- *Thanks Regards* *Hanish Bansal* -- -- Guozhang
Re: Consumer Group Rebalance Issues
Thanks for the help with this Jun, really appreciate it! So I found this in the logs for consumer 007 about an hour previous. Besides that no real activity. It looks like 007 rebalanced and successfully claimed partition 24-27. Shortly after that its zookeeper client timed out and reconnected. It didn't rebalance again after this. 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], begin rebalancing consumer trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 try #0 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] [ConsumerFetcherManager-1387249529483] Stopping leader finder thread 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], Shutting down 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], Stopped 2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] [ConsumerFetcherManager-1387249529483] Stopping all fetchers 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], Shutting down 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], Stopped 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], Shutting down 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error: null 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], Stopped 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], Shutting down 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error: null 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], Stopped 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], Shutting down 2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error: null 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], Stopped 2013-12-17 15:51:06 ConsumerFetcherThread [INFO] [ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12], Shutdown completed 2013-12-17 15:51:06 ConsumerFetcherManager [INFO] [ConsumerFetcherManager-1387249529483] All connections stopped 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Cleared all relevant queues for this fetcher 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Cleared the data chunks in all the consumer message iterators 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Committing all offsets after clearing the fetcher queues 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Releasing partition ownership 2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO] [trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Consumer trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 rebalancing the following partitions: List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127) for topic Events2 with consumers: List(trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-0,
Re: problem with high-level consumer stream filter regex....
thanks Joe! On Wed, Dec 18, 2013 at 11:05 AM, Joe Stein joe.st...@stealth.ly wrote: Hey Jason, I have someone looking into it now (they just started). I can look at it on Friday or if I finish up what I am working on for tomorrow then sooner. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Dec 18, 2013 at 8:15 AM, Jason Rosenberg j...@squareup.com wrote: Joe, I think the java code I listed in the Jira ticket should reproduce the issue directly, does that not work? Jason On Tue, Dec 17, 2013 at 9:49 AM, Joe Stein joe.st...@stealth.ly wrote: Hi Jason, I just replied on the ticket. If it is a bug the update to create new filter or fix as bug, same. Can you post some code to help reproduce the problem? so apples to apples and such, thanks! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Dec 17, 2013 at 1:16 AM, Jason Rosenberg j...@squareup.com wrote: Ping Any thoughts on this? Seems like a bug, but then again, we're not sure what the expected behavior for regexes should be here (e.g. is there a way to whitelist topics with a filter that looks for a leading substring, but then blocks subsequent substrings)? E.g. apply a blacklist to a whitelist :). Jason On Thu, Dec 12, 2013 at 1:01 PM, Jason Rosenberg j...@squareup.com wrote: All, I've filed: https://issues.apache.org/jira/browse/KAFKA-1180 We are needing to create a stream selector that essentially combines the logic of the BlackList and WhiteList classes. That is, we want to select a topic that contains a certain prefix, as long as it doesn't also contain a secondary string. This should be easy to do with ordinary java Regex's, but we're running into some issues, trying to do this with the WhiteList class only. We have a pattern that uses negative lookahead, like this: test-(?!bad\\b)[\\w]+ So this should select a topic like: test-good, but exclude a topic like test-bad, and also exclude a topic without the test prefix, like foo-bar. Instead, what we see is a NullPointerException in the ConsumerIterator, and the consumer just hangs, after sending a topic of 'test-topic' followed by 'test-bad': 21700 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683] ERROR kafka.consumer.ConsumerFetcherThread - [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683], Error due to kafka.common.KafkaException: error processing data for partition [test-bad,0] offset 0 at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:137) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109) at scala.collection.immutable.Map$Map1.foreach(Map.scala:105) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) at kafka.utils.Utils$.inLock(Utils.scala:565) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Caused by: java.lang.NullPointerException at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60) at kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49) at kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128) ... 9 more
RE: a consumer question
Thanks, Jun. That is also my guess:) If the exception is caught, I can easily convert hasNext() from blocking to nonblocking. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, December 18, 2013 12:09 AM To: users@kafka.apache.org Subject: Re: a consumer question Actually, hasNext() only returns false when the consumer connector is shutdown. Typically, you either set consumer.timeout.ms to -1 or a value larger than 0. If it's set to 0, my guess is that it throws a timeout exception immediately if there is no more message. Thanks, Jun On Tue, Dec 17, 2013 at 4:57 PM, Guozhang Wang wangg...@gmail.com wrote: If there is no more messages, hasNext will return false instead of throwing an exception. Guozhang On Tue, Dec 17, 2013 at 11:53 AM, Yu, Libo libo...@citi.com wrote: Sorry, a typo. Correct my question. When consumer.timeout.ms is set to 0, if there is no message available, hasNext() throws a timeout exception, otherwise it returns true. Is that the right behavior? Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Tuesday, December 17, 2013 12:40 AM To: users@kafka.apache.org Subject: Re: a consumer question If there is a message, hasNext() returns true, not throwing an exception. Thanks, Jun On Mon, Dec 16, 2013 at 11:29 AM, Yu, Libo libo...@citi.com wrote: Hi folks, For this parameters, if consumer.timeout.ms is set to 0, whenever I call ConsumerIterator's hasNext(), if there is a message available, a timeout exception will be thrown. Is my understanding correct? Thanks. consumer.timeout.ms -1 Throw a timeout exception to the consumer if no message is available for consumption after the specified interval Regards, Libo -- -- Guozhang
Re: a consumer question
Jun is right. Just checked the code. If you set consumer.timeout.ms to 0 then if there is no message a ConsumerTimeoutException will be thrown right away. On Tue, Dec 17, 2013 at 9:08 PM, Jun Rao jun...@gmail.com wrote: Actually, hasNext() only returns false when the consumer connector is shutdown. Typically, you either set consumer.timeout.ms to -1 or a value larger than 0. If it's set to 0, my guess is that it throws a timeout exception immediately if there is no more message. Thanks, Jun On Tue, Dec 17, 2013 at 4:57 PM, Guozhang Wang wangg...@gmail.com wrote: If there is no more messages, hasNext will return false instead of throwing an exception. Guozhang On Tue, Dec 17, 2013 at 11:53 AM, Yu, Libo libo...@citi.com wrote: Sorry, a typo. Correct my question. When consumer.timeout.ms is set to 0, if there is no message available, hasNext() throws a timeout exception, otherwise it returns true. Is that the right behavior? Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Tuesday, December 17, 2013 12:40 AM To: users@kafka.apache.org Subject: Re: a consumer question If there is a message, hasNext() returns true, not throwing an exception. Thanks, Jun On Mon, Dec 16, 2013 at 11:29 AM, Yu, Libo libo...@citi.com wrote: Hi folks, For this parameters, if consumer.timeout.ms is set to 0, whenever I call ConsumerIterator's hasNext(), if there is a message available, a timeout exception will be thrown. Is my understanding correct? Thanks. consumer.timeout.ms -1 Throw a timeout exception to the consumer if no message is available for consumption after the specified interval Regards, Libo -- -- Guozhang -- -- Guozhang
Re: ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition
Hi Joe, I'm trying to reproduce it with the Vagrant setup you provided. Thanks for setting that up! I should also need to run the sbt commands from the README to build Kafka, right? You included the output from bin/kafka-list-topic.sh. Based on the problem I've described, this wouldn't show the issue, would it? If I'm reading the source right, this command only queries ZooKeeper, while the problem that I'm seeing is in the metadata reported by the brokers. I am using the Oracle JDK, version 1.6.0_45. I'm not sure what you mean by having one topic for the 15 partitions. The single topic I used as an example has 15 partitions. I have two other topics with the same number of partitions and replicas, and they exhibit the same problem. I'll keep trying to reproduce it with the Vagrant setup. Thanks! Ryan On Tue, Dec 17, 2013 at 9:39 PM, Joe Stein joe.st...@stealth.ly wrote: Hi Ryan, can you help re-reproduce the issue on virtual machines? If so, I added two more brokers (so five in total now) in a vagrant file https://github.com/stealthly/kafka/tree/0.8_hubspot_testing_1 git clone https://github.com/stealthly/kafka/tree/0.8_hubspot_testing_1 cd 0.8_hubspot_testing_1 vagrant up you need vagrant http://www.vagrantup.com/downloads.html and virtual box installed https://www.virtualbox.org/ I tried to reproduce and not sure what steps to take or is there issue when it launches? Joes-MacBook-Air:kafka joestein$ bin/kafka-create-topic.sh --zookeeper 192.168.50.5:2181 --replica 2 --partition 15 --topic hubspot_testing creation succeeded! Joes-MacBook-Air:kafka joestein$ bin/kafka-list-topic.sh --zookeeper 192.168.50.5:2181 topic: hubspot_testing partition: 0 leader: 3 replicas: 3,1 isr: 3,1 topic: hubspot_testing partition: 1 leader: 4 replicas: 4,2 isr: 4,2 topic: hubspot_testing partition: 2 leader: 1 replicas: 1,3 isr: 1,3 topic: hubspot_testing partition: 3 leader: 2 replicas: 2,4 isr: 2,4 topic: hubspot_testing partition: 4 leader: 3 replicas: 3,2 isr: 3,2 topic: hubspot_testing partition: 5 leader: 4 replicas: 4,3 isr: 4,3 topic: hubspot_testing partition: 6 leader: 1 replicas: 1,4 isr: 1,4 topic: hubspot_testing partition: 7 leader: 2 replicas: 2,1 isr: 2,1 topic: hubspot_testing partition: 8 leader: 3 replicas: 3,4 isr: 3,4 topic: hubspot_testing partition: 9 leader: 4 replicas: 4,1 isr: 4,1 topic: hubspot_testing partition: 10 leader: 1 replicas: 1,2 isr: 1,2 topic: hubspot_testing partition: 11 leader: 2 replicas: 2,3 isr: 2,3 topic: hubspot_testing partition: 12 leader: 3 replicas: 3,1 isr: 3,1 topic: hubspot_testing partition: 13 leader: 4 replicas: 4,2 isr: 4,2 topic: hubspot_testing partition: 14 leader: 1 replicas: 1,3 isr: 1,3 Are you using the Oracle JDK? Do you have one topic for the 15 partitions? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Dec 17, 2013 at 7:09 PM, Ryan Berdeen rberd...@hubspot.com wrote: Sorry it's taken so long to reply, the issue went away after I reassigned partitions. Now it's back. I haven't checked JMX, because the brokers and zookeeper have been reporting the same ISR for several hours. Some more details: The cluster/topic has 5 brokers (1, 4, 5, 7, 8) 15 partitions (0...14) 2 replicas A single broker, 4, is the one missing from the ISR in every case. For partitions where 4 is the leader (1, 6, 11), it is present in the ISR. For partitions where 4 is not the leader (4, 8, 12), it is not present in the ISR. Here's the output of my tool, showing assignment and ISR: https://gist.github.com/also/8012383#file-from-brokers-txt I haven't seen anything interesting in the logs, but I'm not entirely sure what to look for. The cluster is currently in this state, and if it goes like last time, this will persist until I reassign partitions. What can I do in the meantime to track down the issue? Thanks, Ryan On Thu, Dec 5, 2013 at 12:55 AM, Jun Rao jun...@gmail.com wrote: Do you see any ISR churns on the brokers? You can check the ISR expand/shrink rate jmx. Thanks, Jun On Wed, Dec 4, 2013 at 3:53 PM, Ryan Berdeen rberd...@hubspot.com wrote: I'm working on some monitoring tools for Kafka, and I've seen a couple of clusters get into a state where ClientUtils.fetchTopicMetadata will show that not all replicas are in the ISR. At the same time, ZkUtils.getLeaderIsrAndEpochForPartition will show that all all partitions are in the ISR, and the kafka.server:name=UnderReplicatedPartitions,type=ReplicaManager MBean will report 0. What's going on? Is there something wrong with my controller, or should I not be paying attention to ClientUtils.fetchTopicMetadata?
Re: ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition
Yes, you need to build kafka first ./sbt update ./sbt package ./sbt assembly-package-dependency Once you can make it reproducible please make a pull request if you could to that repo and I can run or do whatever you did. The /vagrant directory is shared as the folder that you did the vagrant up in (if you didn't know that wanted to mention it is helpful to get at file from the vm and commit them). Thanks! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Dec 18, 2013 at 7:22 PM, Ryan Berdeen rberd...@hubspot.com wrote: Hi Joe, I'm trying to reproduce it with the Vagrant setup you provided. Thanks for setting that up! I should also need to run the sbt commands from the README to build Kafka, right? You included the output from bin/kafka-list-topic.sh. Based on the problem I've described, this wouldn't show the issue, would it? If I'm reading the source right, this command only queries ZooKeeper, while the problem that I'm seeing is in the metadata reported by the brokers. I am using the Oracle JDK, version 1.6.0_45. I'm not sure what you mean by having one topic for the 15 partitions. The single topic I used as an example has 15 partitions. I have two other topics with the same number of partitions and replicas, and they exhibit the same problem. I'll keep trying to reproduce it with the Vagrant setup. Thanks! Ryan On Tue, Dec 17, 2013 at 9:39 PM, Joe Stein joe.st...@stealth.ly wrote: Hi Ryan, can you help re-reproduce the issue on virtual machines? If so, I added two more brokers (so five in total now) in a vagrant file https://github.com/stealthly/kafka/tree/0.8_hubspot_testing_1 git clone https://github.com/stealthly/kafka/tree/0.8_hubspot_testing_1 cd 0.8_hubspot_testing_1 vagrant up you need vagrant http://www.vagrantup.com/downloads.html and virtual box installed https://www.virtualbox.org/ I tried to reproduce and not sure what steps to take or is there issue when it launches? Joes-MacBook-Air:kafka joestein$ bin/kafka-create-topic.sh --zookeeper 192.168.50.5:2181 --replica 2 --partition 15 --topic hubspot_testing creation succeeded! Joes-MacBook-Air:kafka joestein$ bin/kafka-list-topic.sh --zookeeper 192.168.50.5:2181 topic: hubspot_testing partition: 0 leader: 3 replicas: 3,1 isr: 3,1 topic: hubspot_testing partition: 1 leader: 4 replicas: 4,2 isr: 4,2 topic: hubspot_testing partition: 2 leader: 1 replicas: 1,3 isr: 1,3 topic: hubspot_testing partition: 3 leader: 2 replicas: 2,4 isr: 2,4 topic: hubspot_testing partition: 4 leader: 3 replicas: 3,2 isr: 3,2 topic: hubspot_testing partition: 5 leader: 4 replicas: 4,3 isr: 4,3 topic: hubspot_testing partition: 6 leader: 1 replicas: 1,4 isr: 1,4 topic: hubspot_testing partition: 7 leader: 2 replicas: 2,1 isr: 2,1 topic: hubspot_testing partition: 8 leader: 3 replicas: 3,4 isr: 3,4 topic: hubspot_testing partition: 9 leader: 4 replicas: 4,1 isr: 4,1 topic: hubspot_testing partition: 10 leader: 1 replicas: 1,2 isr: 1,2 topic: hubspot_testing partition: 11 leader: 2 replicas: 2,3 isr: 2,3 topic: hubspot_testing partition: 12 leader: 3 replicas: 3,1 isr: 3,1 topic: hubspot_testing partition: 13 leader: 4 replicas: 4,2 isr: 4,2 topic: hubspot_testing partition: 14 leader: 1 replicas: 1,3 isr: 1,3 Are you using the Oracle JDK? Do you have one topic for the 15 partitions? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Dec 17, 2013 at 7:09 PM, Ryan Berdeen rberd...@hubspot.com wrote: Sorry it's taken so long to reply, the issue went away after I reassigned partitions. Now it's back. I haven't checked JMX, because the brokers and zookeeper have been reporting the same ISR for several hours. Some more details: The cluster/topic has 5 brokers (1, 4, 5, 7, 8) 15 partitions (0...14) 2 replicas A single broker, 4, is the one missing from the ISR in every case. For partitions where 4 is the leader (1, 6, 11), it is present in the ISR. For partitions where 4 is not the leader (4, 8, 12), it is not present in the ISR. Here's the output of my tool, showing assignment and ISR: https://gist.github.com/also/8012383#file-from-brokers-txt I haven't seen anything interesting in the logs, but I'm not entirely sure what to look for. The cluster is currently in this state, and if it goes like last time, this will persist until I reassign partitions. What can I do in the meantime to track down the issue? Thanks, Ryan
Writing unit tests for Kafka code
I'm writing some Kafka client code and I'd like to unit test it. I've found some resources that say this is possible. For example: http://ransilberman.wordpress.com/2013/07/19/how-to-unit-test-kafka/ but all the information I've found seems a little bit incorrect. For example, the above link uses enable.zookeeper which seems to be an ignored property. Similarly, other properties mentioned in that file have changed names. I found a few posts on this mailing list describing how it might be done (sorry - this was a few days ago and I don't have links handy), but they didn't quite work either. I tried digging into the scala code but it wasn't immediately obvious how to do things, probably partially because I don't know Scala. If somebody could post working Java code that constructs a Kafka broker that can be used for unit testing that would be fantastic. My preference would be something that doesn't require ZooKeeper, but there's some talk that in 0.8 that may no longer be an option. If not, using the Curator framework's TestingServer would be great (I'm using Curator everyone). But ultimately, I'll take anything that works! Thanks in advance, Oliver
Re: kafka build error scala 2.10
i see two files name Annotation_2.8.scala and Annotation_2.9.scala. Excluding them does not help.Is this what you were referring to? Regards, Pushkar On Wed, Dec 18, 2013 at 9:52 PM, Jun Rao jun...@gmail.com wrote: You may have to exclude Annotations.scala. Thanks, Jun On Wed, Dec 18, 2013 at 12:16 AM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: While doing dev setup as described in https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup im getting following build errors. immutable is already defined as class immutable Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 38 Scala Problem threadsafe is already defined as class threadsafe Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 28 Scala Problem nonthreadsafe is already defined as class nonthreadsafe Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils line 33 Scala Problem This error is coming from a file Util /kafka/src/main/scala/kafka/utils/Annotations_2.9+.scala Please note that i had to install scala 2.10 eclipse plugin as Juno had some problem with 2.9. Regards, Pushkar
Re: Kafka 0.8.0 server-stop.sh does not stop broker
Maybe we should just change it to kill -15. Could you try if that works? Could you also file a jira? Thanks, Jun On Tue, Dec 17, 2013 at 9:18 PM, Bryan Baugher bjb...@gmail.com wrote: RHEL 6.4 64bit Java 6u35 On Tue, Dec 17, 2013 at 10:57 PM, Jun Rao jun...@gmail.com wrote: Which OS are you on? Thanks, Jun On Tue, Dec 17, 2013 at 11:15 AM, Bryan Baugher bjb...@gmail.com wrote: Hi, We have been trying out the kafka 0.8.0 beta1 for awhile and recently attempted to upgrade to 0.8.0 but noticed that the stop server script doesn't seem to stop the broker anymore. I noticed here[1] that a commit was made before the release to change the signal sent to stop the broker from SIGTERM to SIGINT. Changing this script back to using SIGTERM seems to fix the issue for me. Has anyone else noticed this, is there a bug or should I log one? [1] - https://github.com/apache/kafka/commit/51de7c55d2b3107b79953f401fc8c9530bd0eea0 -Bryan -- -Bryan
Re: Writing unit tests for Kafka code
You can take a look at Kafka's unit tests. See how kafka.integration.KafkaServerTestHarness is being used. Thanks, Jun On Wed, Dec 18, 2013 at 11:17 AM, Oliver Dain od...@3cinteractive.comwrote: Found some of the other references I'd used that also don't quite work: https://gist.github.com/fjavieralba/7930018 https://gist.github.com/mardambey/2650743 I know, the obvious question is what didn't work. I'm afraid I no longer recall. I took each example, tried it, got errors, tried to fix them (e.g. change properties names), etc. but it was on a deadline and several days ago and I no longer recall exactly what I did. I can try to recreate if people think one of those should work, but I suspect there have been known changes. It would be fantastic if this was all wrapped up in a single method or class so creating an embedded broker and/or cluster for testing was really easy. From: New User od...@3cinteractive.commailto:od...@3cinteractive.com Date: Wednesday, December 18, 2013 at 11:10 AM To: users@kafka.apache.orgmailto:users@kafka.apache.org users@kafka.apache.orgmailto:users@kafka.apache.org Subject: Writing unit tests for Kafka code I'm writing some Kafka client code and I'd like to unit test it. I've found some resources that say this is possible. For example: http://ransilberman.wordpress.com/2013/07/19/how-to-unit-test-kafka/ but all the information I've found seems a little bit incorrect. For example, the above link uses enable.zookeeper which seems to be an ignored property. Similarly, other properties mentioned in that file have changed names. I found a few posts on this mailing list describing how it might be done (sorry - this was a few days ago and I don't have links handy), but they didn't quite work either. I tried digging into the scala code but it wasn't immediately obvious how to do things, probably partially because I don't know Scala. If somebody could post working Java code that constructs a Kafka broker that can be used for unit testing that would be fantastic. My preference would be something that doesn't require ZooKeeper, but there's some talk that in 0.8 that may no longer be an option. If not, using the Curator framework's TestingServer would be great (I'm using Curator everyone). But ultimately, I'll take anything that works! Thanks in advance, Oliver