Can we archive our kafka-requestlog in kafka0.8??

2013-12-18 Thread Nishant Kumar
Hello All,

I am using kafka 0.8.
I want to archive my kafka-request.log, server.log, controller.log,
staet-change.log.

Is there any property in log4j by which i can archive my logs in zip
folders or files.

Please suggest.

Regards,
Nishant Kumar


kafka build error scala 2.10

2013-12-18 Thread pushkar priyadarshi
While doing dev setup as described in
https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup

im getting following build errors.

immutable is already defined as class immutable Annotations_2.9+.scala
/KafkaEclipse/core/src/main/scala/kafka/utils line 38 Scala Problem

threadsafe is already defined as class threadsafe Annotations_2.9+.scala
/KafkaEclipse/core/src/main/scala/kafka/utils line 28 Scala Problem

nonthreadsafe is already defined as class nonthreadsafe
Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils
line 33 Scala
Problem


This error is coming from  a file
Util /kafka/src/main/scala/kafka/utils/Annotations_2.9+.scala

Please note that i had to install scala 2.10 eclipse plugin as Juno had
some problem with 2.9.


Regards,

Pushkar


Re: regarding run-simulator.sh

2013-12-18 Thread pushkar priyadarshi
i see many tools mentioned for perf here

https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing

of all these what all already exist in 0.8 release?
e.g. i was not able to find jmx-dump.sh , R script etc anywhere.


On Wed, Dec 18, 2013 at 11:01 AM, pushkar priyadarshi 
priyadarshi.push...@gmail.com wrote:

 thanks Jun.


 On Wed, Dec 18, 2013 at 10:47 AM, Jun Rao jun...@gmail.com wrote:

 You can run kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh.

 Thanks,

 Jun


 On Tue, Dec 17, 2013 at 8:44 PM, pushkar priyadarshi 
 priyadarshi.push...@gmail.com wrote:

  i am not able to find run-simulator.sh in 0.8 even after building
 perf.if
  this tool has been deprecated what are other alternatives available now
 for
  perf testing?
 
  Regards,
  Pushkar
 





Re: Kafka producer behavior

2013-12-18 Thread Hanish Bansal
Thanks for response Gerrit and Guozhang !!

Hi Gerrit,

I am trying to use  same round robin partitioner shared by you but hard
luck, still round robin partitioning not working.

I have successfully registered RoundRobinPartitioner in kafka producer.

Code of RoundRobinPartitioner class as:

public RoundRobinPartitioner(VerifiableProperties props){
 log.info(Using Round Robin Partitioner class...);
}

@Override
public int partition(String key, int partitions) {
log.info(Inside partition method);
int i = counter.getAndIncrement();
if(i == Integer.MAX_VALUE){
counter.set(0);
 return 0;
}else
 return i % partitions;
}

When i produce the data, first log message Using Round Robin Partitioner
class... is printed and second message Inside partition method is not
printed.

From that we can ensure that RoundRobinPartitioner has been successfully
registered but logic of round robin is not getting called.

Any help to resolve what i am missing ?

Thanks in advance !!



On Tue, Dec 17, 2013 at 5:59 PM, Guozhang Wang wangg...@gmail.com wrote:

 Hello,

 This issue is known as in this JIRA:

 https://issues.apache.org/jira/browse/KAFKA-1067

 Guozhang


 On Tue, Dec 17, 2013 at 8:48 AM, Gerrit Jansen van Vuuren 
 gerrit...@gmail.com wrote:

  hi,
 
  I've had the same issue with the kafka producer.
 
  you need to use a different partitioner than the default one provided for
  kafka.
  I've created a round robin partitioner that works well for equally
  distributing data across partitions.
 
 
 
 https://github.com/gerritjvv/pseidon/blob/master/pseidon-kafka/java/pseidon/kafka/util/RoundRobinPartitioner.java
 
 
 
 
 
  On Tue, Dec 17, 2013 at 5:32 PM, Hanish Bansal 
  hanish.bansal.agar...@gmail.com wrote:
 
   Hi All,
  
   We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
   Replication Factor: 2
   Number of partitions: 2
  
   I have created a topic test-topic1 in kafka.
  
   When i am listing status of that topic using bin/kafka-list-topic.sh,
 the
   status is:
  
   topic: test-topic1partition: 0leader: 0   replicas: 0,1
  isr:
   0,1
   topic: test-topic1partition: 1leader: 1   replicas: 1,0
  isr:
   1,0
  
   As both partition are on two separate nodes so when we produce the data
  it
   should be go to both nodes.
  
   But when i insert the data, it is going to only one node.
  
   For example if i insert 1000 messages then all 1000 messages will go
  either
   node1 or node2. Data is not evenly distributed on both nodes.
  
   Expected: 500 messages should go to node1 and 500 messages should go to
   node2.
  
   Any suggestion why i am facing this behavior?
  
   --
   *Thanks  Regards*
   *Hanish Bansal*
  
 



 --
 -- Guozhang




-- 
*Thanks  Regards*
*Hanish Bansal*


Re: Kafka producer behavior

2013-12-18 Thread Gerrit Jansen van Vuuren
Hi,

this is a gotcha about kafka producer partitioning, you much send the
messages with a non null key.
If the key is null kafka will not call the partitioner.

Because with this partitioner the key does not matter you can pass in a
constant string like 1 etc.

Oh one more thing, on performance:

The produce's send method has a synchronized block on the producer
instance, which means performance goes down the drain.
I could only get (on a 12 core, 72 gig ram) machine 13K tps out of the
producer. A way to solve this is to instantiate an array/list of N
producers and then in your send code round robin over the producers.
I got to 80K tps (for my use case) using 6 producer instances from a single
box sending to 3 kafka servers.

e.g.


send ( msg ) {
  producers[ producer-index.getAndIncrement() % producer_count ].send(msg)
}

Regards,
 Gerrit


On Wed, Dec 18, 2013 at 11:24 AM, Hanish Bansal 
hanish.bansal.agar...@gmail.com wrote:

 Thanks for response Gerrit and Guozhang !!

 Hi Gerrit,

 I am trying to use  same round robin partitioner shared by you but hard
 luck, still round robin partitioning not working.

 I have successfully registered RoundRobinPartitioner in kafka producer.

 Code of RoundRobinPartitioner class as:

 public RoundRobinPartitioner(VerifiableProperties props){
  log.info(Using Round Robin Partitioner class...);
 }

 @Override
 public int partition(String key, int partitions) {
 log.info(Inside partition method);
 int i = counter.getAndIncrement();
 if(i == Integer.MAX_VALUE){
 counter.set(0);
  return 0;
 }else
  return i % partitions;
 }

 When i produce the data, first log message Using Round Robin Partitioner
 class... is printed and second message Inside partition method is not
 printed.

 From that we can ensure that RoundRobinPartitioner has been successfully
 registered but logic of round robin is not getting called.

 Any help to resolve what i am missing ?

 Thanks in advance !!



 On Tue, Dec 17, 2013 at 5:59 PM, Guozhang Wang wangg...@gmail.com wrote:

  Hello,
 
  This issue is known as in this JIRA:
 
  https://issues.apache.org/jira/browse/KAFKA-1067
 
  Guozhang
 
 
  On Tue, Dec 17, 2013 at 8:48 AM, Gerrit Jansen van Vuuren 
  gerrit...@gmail.com wrote:
 
   hi,
  
   I've had the same issue with the kafka producer.
  
   you need to use a different partitioner than the default one provided
 for
   kafka.
   I've created a round robin partitioner that works well for equally
   distributing data across partitions.
  
  
  
 
 https://github.com/gerritjvv/pseidon/blob/master/pseidon-kafka/java/pseidon/kafka/util/RoundRobinPartitioner.java
  
  
  
  
  
   On Tue, Dec 17, 2013 at 5:32 PM, Hanish Bansal 
   hanish.bansal.agar...@gmail.com wrote:
  
Hi All,
   
We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
Replication Factor: 2
Number of partitions: 2
   
I have created a topic test-topic1 in kafka.
   
When i am listing status of that topic using bin/kafka-list-topic.sh,
  the
status is:
   
topic: test-topic1partition: 0leader: 0   replicas: 0,1
   isr:
0,1
topic: test-topic1partition: 1leader: 1   replicas: 1,0
   isr:
1,0
   
As both partition are on two separate nodes so when we produce the
 data
   it
should be go to both nodes.
   
But when i insert the data, it is going to only one node.
   
For example if i insert 1000 messages then all 1000 messages will go
   either
node1 or node2. Data is not evenly distributed on both nodes.
   
Expected: 500 messages should go to node1 and 500 messages should go
 to
node2.
   
Any suggestion why i am facing this behavior?
   
--
*Thanks  Regards*
*Hanish Bansal*
   
  
 
 
 
  --
  -- Guozhang
 



 --
 *Thanks  Regards*
 *Hanish Bansal*



Data loss in case of request.required.acks set to -1

2013-12-18 Thread Hanish Bansal
Hi All,

We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
Replication Factor: 2
Number of partitions: 2


I have configured request.required.acks in producer configuration to -1.

As mentioned in documentation
http://kafka.apache.org/documentation.html#producerconfigs, setting this
value to -1 provides guarantee that no messages will be lost.

I am getting below behaviour:

If kafka is running as foreground process and i am shutting down the kafka
leader node using ctrl+C then no data is lost.

But if i abnormally terminate the kafka using kill -9 pid then still
facing data loss even after configuring request.required.acks to -1.

Any suggestions?
-- 
*Thanks  Regards*
*Hanish Bansal*


Re: Data loss in case of request.required.acks set to -1

2013-12-18 Thread pushkar priyadarshi
You can try setting a higher value for message.send.max.retries in
producer config.

Regards,
Pushkar


On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal 
hanish.bansal.agar...@gmail.com wrote:

 Hi All,

 We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
 Replication Factor: 2
 Number of partitions: 2


 I have configured request.required.acks in producer configuration to -1.

 As mentioned in documentation
 http://kafka.apache.org/documentation.html#producerconfigs, setting this
 value to -1 provides guarantee that no messages will be lost.

 I am getting below behaviour:

 If kafka is running as foreground process and i am shutting down the kafka
 leader node using ctrl+C then no data is lost.

 But if i abnormally terminate the kafka using kill -9 pid then still
 facing data loss even after configuring request.required.acks to -1.

 Any suggestions?
 --
 *Thanks  Regards*
 *Hanish Bansal*



Re: problem with high-level consumer stream filter regex....

2013-12-18 Thread Jason Rosenberg
Joe,

I think the java code I listed in the Jira ticket should reproduce the
issue directly, does that not work?

Jason


On Tue, Dec 17, 2013 at 9:49 AM, Joe Stein joe.st...@stealth.ly wrote:

 Hi Jason, I just replied on the ticket.  If it is a bug the update to
 create new filter or fix as bug, same.

 Can you post some code to help reproduce the problem?  so apples to apples
 and such, thanks!

 /***
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
 /


 On Tue, Dec 17, 2013 at 1:16 AM, Jason Rosenberg j...@squareup.com wrote:

  Ping
 
  Any thoughts on this?
 
  Seems like a bug, but then again, we're not sure what the expected
 behavior
  for regexes should be here (e.g. is there a way to whitelist topics with
 a
  filter that looks for a leading substring, but then blocks subsequent
  substrings)?  E.g. apply a blacklist to a whitelist :).
 
  Jason
 
 
  On Thu, Dec 12, 2013 at 1:01 PM, Jason Rosenberg j...@squareup.com
 wrote:
 
   All, I've filed:  https://issues.apache.org/jira/browse/KAFKA-1180
  
   We are needing to create a stream selector that essentially combines
 the
   logic of the BlackList and WhiteList classes.  That is, we want to
  select a
   topic that contains a certain prefix, as long as it doesn't also
 contain
  a
   secondary string.
  
   This should be easy to do with ordinary java Regex's, but we're running
   into some issues, trying to do this with the WhiteList class only.
  
   We have a pattern that uses negative lookahead, like this:
  
   test-(?!bad\\b)[\\w]+
  
   So this should select a topic like: test-good, but exclude a topic
 like
   test-bad, and also exclude a topic without the test prefix, like
   foo-bar.
  
   Instead, what we see is a NullPointerException in the ConsumerIterator,
   and the consumer just hangs, after sending a topic of 'test-topic'
  followed
   by 'test-bad':
  
   21700
  
 
 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683]
   ERROR kafka.consumer.ConsumerFetcherThread  -
  
 
 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683],
   Error due to
   kafka.common.KafkaException: error processing data for partition
   [test-bad,0] offset 0
   at
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:137)
at
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109)
   at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
at
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109)
   at
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
at
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
   at kafka.utils.Utils$.inLock(Utils.scala:565)
at
  
 
 kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108)
   at
  
 kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
   Caused by: java.lang.NullPointerException
   at
 kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60)
at
  
 
 kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49)
   at
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128)
... 9 more
  
 



Re: Data loss in case of request.required.acks set to -1

2013-12-18 Thread Hanish Bansal
Hi pushkar,

I tried with configuring  message.send.max.retries to 10. Default value
for this is 3.

But still facing data loss.


On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi 
priyadarshi.push...@gmail.com wrote:

 You can try setting a higher value for message.send.max.retries in
 producer config.

 Regards,
 Pushkar


 On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal 
 hanish.bansal.agar...@gmail.com wrote:

  Hi All,
 
  We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
  Replication Factor: 2
  Number of partitions: 2
 
 
  I have configured request.required.acks in producer configuration to -1.
 
  As mentioned in documentation
  http://kafka.apache.org/documentation.html#producerconfigs, setting this
  value to -1 provides guarantee that no messages will be lost.
 
  I am getting below behaviour:
 
  If kafka is running as foreground process and i am shutting down the
 kafka
  leader node using ctrl+C then no data is lost.
 
  But if i abnormally terminate the kafka using kill -9 pid then still
  facing data loss even after configuring request.required.acks to -1.
 
  Any suggestions?
  --
  *Thanks  Regards*
  *Hanish Bansal*
 




-- 
*Thanks  Regards*
*Hanish Bansal*


Re: problem with high-level consumer stream filter regex....

2013-12-18 Thread Joe Stein
Hey Jason, I have someone looking into it now (they just started).

I can look at it on Friday or if I finish up what I am working on for
tomorrow then sooner.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/


On Wed, Dec 18, 2013 at 8:15 AM, Jason Rosenberg j...@squareup.com wrote:

 Joe,

 I think the java code I listed in the Jira ticket should reproduce the
 issue directly, does that not work?

 Jason


 On Tue, Dec 17, 2013 at 9:49 AM, Joe Stein joe.st...@stealth.ly wrote:

  Hi Jason, I just replied on the ticket.  If it is a bug the update to
  create new filter or fix as bug, same.
 
  Can you post some code to help reproduce the problem?  so apples to
 apples
  and such, thanks!
 
  /***
   Joe Stein
   Founder, Principal Consultant
   Big Data Open Source Security LLC
   http://www.stealth.ly
   Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
  /
 
 
  On Tue, Dec 17, 2013 at 1:16 AM, Jason Rosenberg j...@squareup.com
 wrote:
 
   Ping
  
   Any thoughts on this?
  
   Seems like a bug, but then again, we're not sure what the expected
  behavior
   for regexes should be here (e.g. is there a way to whitelist topics
 with
  a
   filter that looks for a leading substring, but then blocks subsequent
   substrings)?  E.g. apply a blacklist to a whitelist :).
  
   Jason
  
  
   On Thu, Dec 12, 2013 at 1:01 PM, Jason Rosenberg j...@squareup.com
  wrote:
  
All, I've filed:  https://issues.apache.org/jira/browse/KAFKA-1180
   
We are needing to create a stream selector that essentially combines
  the
logic of the BlackList and WhiteList classes.  That is, we want to
   select a
topic that contains a certain prefix, as long as it doesn't also
  contain
   a
secondary string.
   
This should be easy to do with ordinary java Regex's, but we're
 running
into some issues, trying to do this with the WhiteList class only.
   
We have a pattern that uses negative lookahead, like this:
   
test-(?!bad\\b)[\\w]+
   
So this should select a topic like: test-good, but exclude a topic
  like
test-bad, and also exclude a topic without the test prefix, like
foo-bar.
   
Instead, what we see is a NullPointerException in the
 ConsumerIterator,
and the consumer just hangs, after sending a topic of 'test-topic'
   followed
by 'test-bad':
   
21700
   
  
 
 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683]
ERROR kafka.consumer.ConsumerFetcherThread  -
   
  
 
 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683],
Error due to
kafka.common.KafkaException: error processing data for partition
[test-bad,0] offset 0
at
   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:137)
 at
   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
 at
   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109)
at
   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
 at
   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
at kafka.utils.Utils$.inLock(Utils.scala:565)
 at
   
  
 
 kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108)
at
   
  kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
Caused by: java.lang.NullPointerException
at
  kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60)
 at
   
  
 
 kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49)
at
   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128)
 ... 9 more
   
  
 



Re: Data loss in case of request.required.acks set to -1

2013-12-18 Thread Joe Stein
How many replicas do you have?


On Wed, Dec 18, 2013 at 8:57 AM, Hanish Bansal 
hanish.bansal.agar...@gmail.com wrote:

 Hi pushkar,

 I tried with configuring  message.send.max.retries to 10. Default value
 for this is 3.

 But still facing data loss.


 On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi 
 priyadarshi.push...@gmail.com wrote:

  You can try setting a higher value for message.send.max.retries in
  producer config.
 
  Regards,
  Pushkar
 
 
  On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal 
  hanish.bansal.agar...@gmail.com wrote:
 
   Hi All,
  
   We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
   Replication Factor: 2
   Number of partitions: 2
  
  
   I have configured request.required.acks in producer configuration to
 -1.
  
   As mentioned in documentation
   http://kafka.apache.org/documentation.html#producerconfigs, setting
 this
   value to -1 provides guarantee that no messages will be lost.
  
   I am getting below behaviour:
  
   If kafka is running as foreground process and i am shutting down the
  kafka
   leader node using ctrl+C then no data is lost.
  
   But if i abnormally terminate the kafka using kill -9 pid then
 still
   facing data loss even after configuring request.required.acks to -1.
  
   Any suggestions?
   --
   *Thanks  Regards*
   *Hanish Bansal*
  
 



 --
 *Thanks  Regards*
 *Hanish Bansal*



Re: Can we archive our kafka-requestlog in kafka0.8??

2013-12-18 Thread Jun Rao
Take a look at
http://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppenderand
search for compress.

Thanks,

Jun


On Tue, Dec 17, 2013 at 11:58 PM, Nishant Kumar nish.a...@gmail.com wrote:

 Hello All,

 I am using kafka 0.8.
 I want to archive my kafka-request.log, server.log, controller.log,
 staet-change.log.

 Is there any property in log4j by which i can archive my logs in zip
 folders or files.

 Please suggest.

 Regards,
 Nishant Kumar



Re: kafka build error scala 2.10

2013-12-18 Thread Jun Rao
You may have to exclude Annotations.scala.

Thanks,

Jun


On Wed, Dec 18, 2013 at 12:16 AM, pushkar priyadarshi 
priyadarshi.push...@gmail.com wrote:

 While doing dev setup as described in
 https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup

 im getting following build errors.

 immutable is already defined as class immutable Annotations_2.9+.scala
 /KafkaEclipse/core/src/main/scala/kafka/utils line 38 Scala Problem

 threadsafe is already defined as class threadsafe Annotations_2.9+.scala
 /KafkaEclipse/core/src/main/scala/kafka/utils line 28 Scala Problem

 nonthreadsafe is already defined as class nonthreadsafe
 Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils
 line 33 Scala
 Problem


 This error is coming from  a file
 Util /kafka/src/main/scala/kafka/utils/Annotations_2.9+.scala

 Please note that i had to install scala 2.10 eclipse plugin as Juno had
 some problem with 2.9.


 Regards,

 Pushkar



Re: regarding run-simulator.sh

2013-12-18 Thread Jun Rao
Most tools can be found in the kafka.tools package. For dumping jmx,
use JmxTool.

Thanks,

Jun


On Wed, Dec 18, 2013 at 1:03 AM, pushkar priyadarshi 
priyadarshi.push...@gmail.com wrote:

 i see many tools mentioned for perf here

 https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing

 of all these what all already exist in 0.8 release?
 e.g. i was not able to find jmx-dump.sh , R script etc anywhere.


 On Wed, Dec 18, 2013 at 11:01 AM, pushkar priyadarshi 
 priyadarshi.push...@gmail.com wrote:

  thanks Jun.
 
 
  On Wed, Dec 18, 2013 at 10:47 AM, Jun Rao jun...@gmail.com wrote:
 
  You can run kafka-producer-perf-test.sh and kafka-consumer-perf-test.sh.
 
  Thanks,
 
  Jun
 
 
  On Tue, Dec 17, 2013 at 8:44 PM, pushkar priyadarshi 
  priyadarshi.push...@gmail.com wrote:
 
   i am not able to find run-simulator.sh in 0.8 even after building
  perf.if
   this tool has been deprecated what are other alternatives available
 now
  for
   perf testing?
  
   Regards,
   Pushkar
  
 
 
 



Re: Data loss in case of request.required.acks set to -1

2013-12-18 Thread pushkar priyadarshi
my doubt was they are dropping off at producer level only.so suggested
playing with paramaters like retries and backoff.ms and also with
refreshinterval on producer side.

Regards,
Pushkar


On Wed, Dec 18, 2013 at 10:01 PM, Guozhang Wang wangg...@gmail.com wrote:

 Hanish,

 Did you kill -9 one of the brokers only or bouncing them iteratively?

 Guozhang


 On Wed, Dec 18, 2013 at 8:02 AM, Joe Stein joe.st...@stealth.ly wrote:

  How many replicas do you have?
 
 
  On Wed, Dec 18, 2013 at 8:57 AM, Hanish Bansal 
  hanish.bansal.agar...@gmail.com wrote:
 
   Hi pushkar,
  
   I tried with configuring  message.send.max.retries to 10. Default
 value
   for this is 3.
  
   But still facing data loss.
  
  
   On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi 
   priyadarshi.push...@gmail.com wrote:
  
You can try setting a higher value for message.send.max.retries in
producer config.
   
Regards,
Pushkar
   
   
On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal 
hanish.bansal.agar...@gmail.com wrote:
   
 Hi All,

 We are having kafka cluster of 2 nodes. (using 0.8.0 final release)
 Replication Factor: 2
 Number of partitions: 2


 I have configured request.required.acks in producer configuration
 to
   -1.

 As mentioned in documentation
 http://kafka.apache.org/documentation.html#producerconfigs,
 setting
   this
 value to -1 provides guarantee that no messages will be lost.

 I am getting below behaviour:

 If kafka is running as foreground process and i am shutting down
 the
kafka
 leader node using ctrl+C then no data is lost.

 But if i abnormally terminate the kafka using kill -9 pid then
   still
 facing data loss even after configuring request.required.acks to
 -1.

 Any suggestions?
 --
 *Thanks  Regards*
 *Hanish Bansal*

   
  
  
  
   --
   *Thanks  Regards*
   *Hanish Bansal*
  
 



 --
 -- Guozhang



Re: Migrating a cluster from 0.8.0 to 0.8.1

2013-12-18 Thread Drew Goya
Thanks Neha, I rolled upgrades and completed a rebalance!

I ran into a few small issues I figured I would share.

On a few Brokers, there were some log directories left over from some
failed rebalances which prevented the 0.8.1 brokers from starting once I
completed the upgrade.  These directories contained an index file and a
zero size log file, once I cleaned those out the brokers were able to start
up fine.  If anyone else runs into the same problem, and is running RHEL,
this is the bash script I used to clean them out:

du --max-depth=1 -h /data/kafka/logs | grep K | sed s/.*K.// | sudo rm -r


On Tue, Dec 17, 2013 at 10:42 AM, Neha Narkhede neha.narkh...@gmail.comwrote:

 There are no compatibility issues. You can roll upgrades through the
 cluster one node at a time.

 Thanks
 Neha


 On Tue, Dec 17, 2013 at 9:15 AM, Drew Goya d...@gradientx.com wrote:

  So I'm going to be going through the process of upgrading a cluster from
  0.8.0 to the trunk (0.8.1).
 
  I'm going to be expanding this cluster several times and the problems
 with
  reassigning partitions in 0.8.0 mean I have to move to trunk(0.8.1) asap.
 
  Will it be safe to roll upgrades through the cluster one by one?
 
  Also are there any client compatibility issues I need to worry about?
  Am I
  going to need to pause/upgrade all my consumers/producers at once or can
 I
  roll upgrades through the cluster and then upgrade my clients one by one?
 
  Thanks in advance!
 



Re: Data loss in case of request.required.acks set to -1

2013-12-18 Thread Joe Stein
Wouldn't you want to set the controlled.shutdown.enable to true so the
broker would do this for you before ending itself?

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/


On Wed, Dec 18, 2013 at 11:36 AM, pushkar priyadarshi 
priyadarshi.push...@gmail.com wrote:

 my doubt was they are dropping off at producer level only.so suggested
 playing with paramaters like retries and backoff.ms and also with
 refreshinterval on producer side.

 Regards,
 Pushkar


 On Wed, Dec 18, 2013 at 10:01 PM, Guozhang Wang wangg...@gmail.com
 wrote:

  Hanish,
 
  Did you kill -9 one of the brokers only or bouncing them iteratively?
 
  Guozhang
 
 
  On Wed, Dec 18, 2013 at 8:02 AM, Joe Stein joe.st...@stealth.ly wrote:
 
   How many replicas do you have?
  
  
   On Wed, Dec 18, 2013 at 8:57 AM, Hanish Bansal 
   hanish.bansal.agar...@gmail.com wrote:
  
Hi pushkar,
   
I tried with configuring  message.send.max.retries to 10. Default
  value
for this is 3.
   
But still facing data loss.
   
   
On Wed, Dec 18, 2013 at 12:44 PM, pushkar priyadarshi 
priyadarshi.push...@gmail.com wrote:
   
 You can try setting a higher value for message.send.max.retries
 in
 producer config.

 Regards,
 Pushkar


 On Wed, Dec 18, 2013 at 5:34 PM, Hanish Bansal 
 hanish.bansal.agar...@gmail.com wrote:

  Hi All,
 
  We are having kafka cluster of 2 nodes. (using 0.8.0 final
 release)
  Replication Factor: 2
  Number of partitions: 2
 
 
  I have configured request.required.acks in producer configuration
  to
-1.
 
  As mentioned in documentation
  http://kafka.apache.org/documentation.html#producerconfigs,
  setting
this
  value to -1 provides guarantee that no messages will be lost.
 
  I am getting below behaviour:
 
  If kafka is running as foreground process and i am shutting down
  the
 kafka
  leader node using ctrl+C then no data is lost.
 
  But if i abnormally terminate the kafka using kill -9 pid
 then
still
  facing data loss even after configuring request.required.acks to
  -1.
 
  Any suggestions?
  --
  *Thanks  Regards*
  *Hanish Bansal*
 

   
   
   
--
*Thanks  Regards*
*Hanish Bansal*
   
  
 
 
 
  --
  -- Guozhang
 



Re: Consumer Group Rebalance Issues

2013-12-18 Thread Drew Goya
Thanks for the help with this Jun, really appreciate it!  So I found this
in the logs for consumer 007 about an hour previous.  Besides that no real
activity.

It looks like 007 rebalanced and successfully claimed partition 24-27.
 Shortly after that its zookeeper client timed out and reconnected.  It
didn't rebalance again after this.

2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], begin
rebalancing consumer
trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 try #0
2013-12-17 15:51:06 ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1387249529483] Stopping leader finder thread
2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread],
Shutting down
2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread],
Stopped
2013-12-17 15:51:06 ConsumerFetcherManager$LeaderFinderThread [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-leader-finder-thread],
Shutdown completed
2013-12-17 15:51:06 ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1387249529483] Stopping all fetchers
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13],
Shutting down
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13],
Stopped
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-13],
Shutdown completed
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11],
Shutting down
2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error:
null
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11],
Stopped
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-11],
Shutdown completed
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10],
Shutting down
2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error:
null
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10],
Stopped
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-10],
Shutdown completed
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12],
Shutting down
2013-12-17 15:51:06 SimpleConsumer [INFO] Reconnect due to socket error:
null
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12],
Stopped
2013-12-17 15:51:06 ConsumerFetcherThread [INFO]
[ConsumerFetcherThread-trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8-0-12],
Shutdown completed
2013-12-17 15:51:06 ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1387249529483] All connections stopped
2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Cleared all
relevant queues for this fetcher
2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Cleared the
data chunks in all the consumer message iterators
2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Committing
all offsets after clearing the fetcher queues
2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Releasing
partition ownership
2013-12-17 15:51:06 ZookeeperConsumerConnector [INFO]
[trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8], Consumer
trackingGroup_prod-storm-sup-trk007-1387249529436-fb79e4c8 rebalancing the
following partitions: List(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51,
52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70,
71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89,
90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106,
107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121,
122, 123, 124, 125, 126, 127) for topic Events2 with consumers:
List(trackingGroup_prod-storm-sup-trk001-1387249529775-2a8484f1-0,

Re: problem with high-level consumer stream filter regex....

2013-12-18 Thread Jason Rosenberg
thanks Joe!


On Wed, Dec 18, 2013 at 11:05 AM, Joe Stein joe.st...@stealth.ly wrote:

 Hey Jason, I have someone looking into it now (they just started).

 I can look at it on Friday or if I finish up what I am working on for
 tomorrow then sooner.

 /***
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
 /


 On Wed, Dec 18, 2013 at 8:15 AM, Jason Rosenberg j...@squareup.com wrote:

  Joe,
 
  I think the java code I listed in the Jira ticket should reproduce the
  issue directly, does that not work?
 
  Jason
 
 
  On Tue, Dec 17, 2013 at 9:49 AM, Joe Stein joe.st...@stealth.ly wrote:
 
   Hi Jason, I just replied on the ticket.  If it is a bug the update to
   create new filter or fix as bug, same.
  
   Can you post some code to help reproduce the problem?  so apples to
  apples
   and such, thanks!
  
   /***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
   /
  
  
   On Tue, Dec 17, 2013 at 1:16 AM, Jason Rosenberg j...@squareup.com
  wrote:
  
Ping
   
Any thoughts on this?
   
Seems like a bug, but then again, we're not sure what the expected
   behavior
for regexes should be here (e.g. is there a way to whitelist topics
  with
   a
filter that looks for a leading substring, but then blocks subsequent
substrings)?  E.g. apply a blacklist to a whitelist :).
   
Jason
   
   
On Thu, Dec 12, 2013 at 1:01 PM, Jason Rosenberg j...@squareup.com
   wrote:
   
 All, I've filed:  https://issues.apache.org/jira/browse/KAFKA-1180

 We are needing to create a stream selector that essentially
 combines
   the
 logic of the BlackList and WhiteList classes.  That is, we want to
select a
 topic that contains a certain prefix, as long as it doesn't also
   contain
a
 secondary string.

 This should be easy to do with ordinary java Regex's, but we're
  running
 into some issues, trying to do this with the WhiteList class only.

 We have a pattern that uses negative lookahead, like this:

 test-(?!bad\\b)[\\w]+

 So this should select a topic like: test-good, but exclude a
 topic
   like
 test-bad, and also exclude a topic without the test prefix,
 like
 foo-bar.

 Instead, what we see is a NullPointerException in the
  ConsumerIterator,
 and the consumer just hangs, after sending a topic of 'test-topic'
followed
 by 'test-bad':

 21700

   
  
 
 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683]
 ERROR kafka.consumer.ConsumerFetcherThread  -

   
  
 
 [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683],
 Error due to
 kafka.common.KafkaException: error processing data for partition
 [test-bad,0] offset 0
 at

   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:137)
  at

   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109)
 at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
  at

   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109)
 at

   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
  at

   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109)
 at kafka.utils.Utils$.inLock(Utils.scala:565)
  at

   
  
 
 kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108)
 at

  
 kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
 Caused by: java.lang.NullPointerException
 at
   kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60)
  at

   
  
 
 kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49)
 at

   
  
 
 kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128)
  ... 9 more

   
  
 



RE: a consumer question

2013-12-18 Thread Yu, Libo
Thanks, Jun. That is also my guess:) If the exception is caught, I can easily
convert hasNext() from blocking to nonblocking.

Regards,

Libo


-Original Message-
From: Jun Rao [mailto:jun...@gmail.com] 
Sent: Wednesday, December 18, 2013 12:09 AM
To: users@kafka.apache.org
Subject: Re: a consumer question

Actually, hasNext() only returns false when the consumer connector is shutdown. 
Typically, you either set consumer.timeout.ms to -1 or a value larger than 0. 
If it's set to 0, my guess is that it throws a timeout exception immediately if 
there is no more message.

Thanks,

Jun


On Tue, Dec 17, 2013 at 4:57 PM, Guozhang Wang wangg...@gmail.com wrote:

 If there is no more messages, hasNext will return false instead of 
 throwing an exception.

 Guozhang


 On Tue, Dec 17, 2013 at 11:53 AM, Yu, Libo libo...@citi.com wrote:

  Sorry, a typo. Correct my question. When consumer.timeout.ms is set 
  to
 0,
   if there is no
  message available, hasNext() throws a timeout exception, otherwise 
  it returns true.
  Is that the right behavior?
 
  Regards,
 
  Libo
 
 
  -Original Message-
  From: Jun Rao [mailto:jun...@gmail.com]
  Sent: Tuesday, December 17, 2013 12:40 AM
  To: users@kafka.apache.org
  Subject: Re: a consumer question
 
  If there is a message, hasNext() returns true, not throwing an exception.
 
  Thanks,
 
  Jun
 
 
  On Mon, Dec 16, 2013 at 11:29 AM, Yu, Libo libo...@citi.com wrote:
 
   Hi folks,
  
   For this parameters, if consumer.timeout.ms is set to 0, whenever 
   I call ConsumerIterator's hasNext(), if there is a message 
   available, a timeout exception will be thrown. Is my understanding 
   correct? Thanks.
  
   consumer.timeout.ms
  
   -1
  
   Throw a timeout exception to the consumer if no message is 
   available for consumption after the specified interval
  
  
  
   Regards,
  
   Libo
  
  
 



 --
 -- Guozhang



Re: a consumer question

2013-12-18 Thread Guozhang Wang
Jun is right. Just checked the code. If you set consumer.timeout.ms to 0
then if there is no message a ConsumerTimeoutException will be thrown right
away.


On Tue, Dec 17, 2013 at 9:08 PM, Jun Rao jun...@gmail.com wrote:

 Actually, hasNext() only returns false when the consumer connector is
 shutdown. Typically, you either set consumer.timeout.ms to -1 or a value
 larger than 0. If it's set to 0, my guess is that it throws a timeout
 exception immediately if there is no more message.

 Thanks,

 Jun


 On Tue, Dec 17, 2013 at 4:57 PM, Guozhang Wang wangg...@gmail.com wrote:

  If there is no more messages, hasNext will return false instead of
 throwing
  an exception.
 
  Guozhang
 
 
  On Tue, Dec 17, 2013 at 11:53 AM, Yu, Libo libo...@citi.com wrote:
 
   Sorry, a typo. Correct my question. When consumer.timeout.ms is set to
  0,
if there is no
   message available, hasNext() throws a timeout exception, otherwise it
   returns true.
   Is that the right behavior?
  
   Regards,
  
   Libo
  
  
   -Original Message-
   From: Jun Rao [mailto:jun...@gmail.com]
   Sent: Tuesday, December 17, 2013 12:40 AM
   To: users@kafka.apache.org
   Subject: Re: a consumer question
  
   If there is a message, hasNext() returns true, not throwing an
 exception.
  
   Thanks,
  
   Jun
  
  
   On Mon, Dec 16, 2013 at 11:29 AM, Yu, Libo libo...@citi.com wrote:
  
Hi folks,
   
For this parameters, if consumer.timeout.ms is set to 0, whenever I
call ConsumerIterator's hasNext(), if there is a message available, a
timeout exception will be thrown. Is my understanding correct?
 Thanks.
   
consumer.timeout.ms
   
-1
   
Throw a timeout exception to the consumer if no message is available
for consumption after the specified interval
   
   
   
Regards,
   
Libo
   
   
  
 
 
 
  --
  -- Guozhang
 




-- 
-- Guozhang


Re: ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition

2013-12-18 Thread Ryan Berdeen
Hi Joe,

I'm trying to reproduce it with the Vagrant setup you provided. Thanks for
setting that up! I should also need to run the sbt commands from the README
to build Kafka, right?

You included the output from bin/kafka-list-topic.sh. Based on the
problem I've described, this wouldn't show the issue, would it? If I'm
reading the source right, this command only queries ZooKeeper, while the
problem that I'm seeing is in the metadata reported by the brokers.

I am using the Oracle JDK, version 1.6.0_45.

I'm not sure what you mean by having one topic for the 15 partitions. The
single topic I used as an example has 15 partitions. I have two other
topics with the same number of partitions and replicas, and they exhibit
the same problem.

I'll keep trying to reproduce it with the Vagrant setup.

Thanks!

Ryan


On Tue, Dec 17, 2013 at 9:39 PM, Joe Stein joe.st...@stealth.ly wrote:

 Hi Ryan, can you help re-reproduce the issue on virtual machines?  If so, I
 added two more brokers (so five in total now) in a vagrant file
 https://github.com/stealthly/kafka/tree/0.8_hubspot_testing_1

 git clone https://github.com/stealthly/kafka/tree/0.8_hubspot_testing_1
 cd 0.8_hubspot_testing_1
 vagrant up

 you need vagrant http://www.vagrantup.com/downloads.html and virtual box
 installed https://www.virtualbox.org/

 I tried to reproduce and not sure what steps to take or is there issue when
 it launches?

 Joes-MacBook-Air:kafka joestein$ bin/kafka-create-topic.sh --zookeeper
 192.168.50.5:2181 --replica 2 --partition 15 --topic hubspot_testing
 creation succeeded!

 Joes-MacBook-Air:kafka joestein$ bin/kafka-list-topic.sh --zookeeper
 192.168.50.5:2181
 topic: hubspot_testing partition: 0 leader: 3 replicas: 3,1 isr: 3,1
 topic: hubspot_testing partition: 1 leader: 4 replicas: 4,2 isr: 4,2
 topic: hubspot_testing partition: 2 leader: 1 replicas: 1,3 isr: 1,3
 topic: hubspot_testing partition: 3 leader: 2 replicas: 2,4 isr: 2,4
 topic: hubspot_testing partition: 4 leader: 3 replicas: 3,2 isr: 3,2
 topic: hubspot_testing partition: 5 leader: 4 replicas: 4,3 isr: 4,3
 topic: hubspot_testing partition: 6 leader: 1 replicas: 1,4 isr: 1,4
 topic: hubspot_testing partition: 7 leader: 2 replicas: 2,1 isr: 2,1
 topic: hubspot_testing partition: 8 leader: 3 replicas: 3,4 isr: 3,4
 topic: hubspot_testing partition: 9 leader: 4 replicas: 4,1 isr: 4,1
 topic: hubspot_testing partition: 10 leader: 1 replicas: 1,2 isr: 1,2
 topic: hubspot_testing partition: 11 leader: 2 replicas: 2,3 isr: 2,3
 topic: hubspot_testing partition: 12 leader: 3 replicas: 3,1 isr: 3,1
 topic: hubspot_testing partition: 13 leader: 4 replicas: 4,2 isr: 4,2
 topic: hubspot_testing partition: 14 leader: 1 replicas: 1,3 isr: 1,3

 Are you using the Oracle JDK?

 Do you have one topic for the 15 partitions?

 /***
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
 /


 On Tue, Dec 17, 2013 at 7:09 PM, Ryan Berdeen rberd...@hubspot.com
 wrote:

  Sorry it's taken so long to reply, the issue went away after I reassigned
  partitions. Now it's back.
 
  I haven't checked JMX, because the brokers and zookeeper have been
  reporting the same ISR for several hours.
 
  Some more details:
 
  The cluster/topic has
5 brokers (1, 4, 5, 7, 8)
15 partitions (0...14)
2 replicas
 
  A single broker, 4, is the one missing from the ISR in every case. For
  partitions where 4 is the leader (1, 6, 11), it is present in the ISR.
 For
  partitions where 4 is not the leader (4, 8, 12), it is not present in the
  ISR. Here's the output of my tool, showing assignment and ISR:
  https://gist.github.com/also/8012383#file-from-brokers-txt
 
  I haven't seen anything interesting in the logs, but I'm not entirely
 sure
  what to look for. The cluster is currently in this state, and if it goes
  like last time, this will persist until I reassign partitions.
 
  What can I do in the meantime to track down the issue?
 
  Thanks,
 
  Ryan
 
  On Thu, Dec 5, 2013 at 12:55 AM, Jun Rao jun...@gmail.com wrote:
 
   Do you see any ISR churns on the brokers? You can check the ISR
   expand/shrink rate jmx.
  
   Thanks,
  
   Jun
  
  
   On Wed, Dec 4, 2013 at 3:53 PM, Ryan Berdeen rberd...@hubspot.com
  wrote:
  
I'm working on some monitoring tools for Kafka, and I've seen a
 couple
  of
clusters get into a state where ClientUtils.fetchTopicMetadata will
  show
that not all replicas are in the ISR.
   
At the same time, ZkUtils.getLeaderIsrAndEpochForPartition will show
  that
all all partitions are in the ISR, and
the
  kafka.server:name=UnderReplicatedPartitions,type=ReplicaManager
MBean will report 0.
   
What's going on? Is there something wrong with my controller, or
  should I
not be paying attention to ClientUtils.fetchTopicMetadata?
   
 

Re: ClientUtils.fetchTopicMetadata reports smaller ISR than ZkUtils.getLeaderIsrAndEpochForPartition

2013-12-18 Thread Joe Stein
Yes, you need to build kafka first

./sbt update
./sbt package
./sbt assembly-package-dependency

Once you can make it reproducible please make a pull request if you could
to that repo and I can run or do whatever you did.  The /vagrant directory
is shared as the folder that you did the vagrant up in (if you didn't know
that wanted to mention it is helpful to get at file from the vm and commit
them).

Thanks!

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
/


On Wed, Dec 18, 2013 at 7:22 PM, Ryan Berdeen rberd...@hubspot.com wrote:

 Hi Joe,

 I'm trying to reproduce it with the Vagrant setup you provided. Thanks for
 setting that up! I should also need to run the sbt commands from the README
 to build Kafka, right?

 You included the output from bin/kafka-list-topic.sh. Based on the
 problem I've described, this wouldn't show the issue, would it? If I'm
 reading the source right, this command only queries ZooKeeper, while the
 problem that I'm seeing is in the metadata reported by the brokers.

 I am using the Oracle JDK, version 1.6.0_45.

 I'm not sure what you mean by having one topic for the 15 partitions. The
 single topic I used as an example has 15 partitions. I have two other
 topics with the same number of partitions and replicas, and they exhibit
 the same problem.

 I'll keep trying to reproduce it with the Vagrant setup.

 Thanks!

 Ryan


 On Tue, Dec 17, 2013 at 9:39 PM, Joe Stein joe.st...@stealth.ly wrote:

  Hi Ryan, can you help re-reproduce the issue on virtual machines?  If
 so, I
  added two more brokers (so five in total now) in a vagrant file
  https://github.com/stealthly/kafka/tree/0.8_hubspot_testing_1
 
  git clone https://github.com/stealthly/kafka/tree/0.8_hubspot_testing_1
  cd 0.8_hubspot_testing_1
  vagrant up
 
  you need vagrant http://www.vagrantup.com/downloads.html and virtual box
  installed https://www.virtualbox.org/
 
  I tried to reproduce and not sure what steps to take or is there issue
 when
  it launches?
 
  Joes-MacBook-Air:kafka joestein$ bin/kafka-create-topic.sh --zookeeper
  192.168.50.5:2181 --replica 2 --partition 15 --topic hubspot_testing
  creation succeeded!
 
  Joes-MacBook-Air:kafka joestein$ bin/kafka-list-topic.sh --zookeeper
  192.168.50.5:2181
  topic: hubspot_testing partition: 0 leader: 3 replicas: 3,1 isr: 3,1
  topic: hubspot_testing partition: 1 leader: 4 replicas: 4,2 isr: 4,2
  topic: hubspot_testing partition: 2 leader: 1 replicas: 1,3 isr: 1,3
  topic: hubspot_testing partition: 3 leader: 2 replicas: 2,4 isr: 2,4
  topic: hubspot_testing partition: 4 leader: 3 replicas: 3,2 isr: 3,2
  topic: hubspot_testing partition: 5 leader: 4 replicas: 4,3 isr: 4,3
  topic: hubspot_testing partition: 6 leader: 1 replicas: 1,4 isr: 1,4
  topic: hubspot_testing partition: 7 leader: 2 replicas: 2,1 isr: 2,1
  topic: hubspot_testing partition: 8 leader: 3 replicas: 3,4 isr: 3,4
  topic: hubspot_testing partition: 9 leader: 4 replicas: 4,1 isr: 4,1
  topic: hubspot_testing partition: 10 leader: 1 replicas: 1,2 isr: 1,2
  topic: hubspot_testing partition: 11 leader: 2 replicas: 2,3 isr: 2,3
  topic: hubspot_testing partition: 12 leader: 3 replicas: 3,1 isr: 3,1
  topic: hubspot_testing partition: 13 leader: 4 replicas: 4,2 isr: 4,2
  topic: hubspot_testing partition: 14 leader: 1 replicas: 1,3 isr: 1,3
 
  Are you using the Oracle JDK?
 
  Do you have one topic for the 15 partitions?
 
  /***
   Joe Stein
   Founder, Principal Consultant
   Big Data Open Source Security LLC
   http://www.stealth.ly
   Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop
  /
 
 
  On Tue, Dec 17, 2013 at 7:09 PM, Ryan Berdeen rberd...@hubspot.com
  wrote:
 
   Sorry it's taken so long to reply, the issue went away after I
 reassigned
   partitions. Now it's back.
  
   I haven't checked JMX, because the brokers and zookeeper have been
   reporting the same ISR for several hours.
  
   Some more details:
  
   The cluster/topic has
 5 brokers (1, 4, 5, 7, 8)
 15 partitions (0...14)
 2 replicas
  
   A single broker, 4, is the one missing from the ISR in every case. For
   partitions where 4 is the leader (1, 6, 11), it is present in the ISR.
  For
   partitions where 4 is not the leader (4, 8, 12), it is not present in
 the
   ISR. Here's the output of my tool, showing assignment and ISR:
   https://gist.github.com/also/8012383#file-from-brokers-txt
  
   I haven't seen anything interesting in the logs, but I'm not entirely
  sure
   what to look for. The cluster is currently in this state, and if it
 goes
   like last time, this will persist until I reassign partitions.
  
   What can I do in the meantime to track down the issue?
  
   Thanks,
  
   Ryan
  
 

Writing unit tests for Kafka code

2013-12-18 Thread Oliver Dain
I'm writing some Kafka client code and I'd like to unit test it. I've found 
some resources that say this is possible. For example:

http://ransilberman.wordpress.com/2013/07/19/how-to-unit-test-kafka/

but all the information I've found seems a little bit incorrect. For example, 
the above link uses enable.zookeeper which seems to be an ignored property. 
Similarly, other properties mentioned in that file have changed names. I found 
a few posts on this mailing list describing how it might be done (sorry - this 
was a few days ago and I don't have links handy), but they didn't quite work 
either. I tried digging into the scala code but it wasn't immediately obvious 
how to do things, probably partially because I don't know Scala.

If somebody could post working Java code that constructs a Kafka broker that 
can be used for unit testing that would be fantastic. My preference would be 
something that doesn't require ZooKeeper, but there's some talk that in 0.8 
that may no longer be an option. If not, using the Curator framework's 
TestingServer would be great (I'm using Curator everyone). But ultimately, I'll 
take anything that works!

Thanks in advance,
Oliver


Re: kafka build error scala 2.10

2013-12-18 Thread pushkar priyadarshi
i see two files name Annotation_2.8.scala and Annotation_2.9.scala.
Excluding them does not help.Is this what you were referring to?

Regards,
Pushkar


On Wed, Dec 18, 2013 at 9:52 PM, Jun Rao jun...@gmail.com wrote:

 You may have to exclude Annotations.scala.

 Thanks,

 Jun


 On Wed, Dec 18, 2013 at 12:16 AM, pushkar priyadarshi 
 priyadarshi.push...@gmail.com wrote:

  While doing dev setup as described in
  https://cwiki.apache.org/confluence/display/KAFKA/Developer+Setup
 
  im getting following build errors.
 
  immutable is already defined as class immutable Annotations_2.9+.scala
  /KafkaEclipse/core/src/main/scala/kafka/utils line 38 Scala Problem
 
  threadsafe is already defined as class threadsafe Annotations_2.9+.scala
  /KafkaEclipse/core/src/main/scala/kafka/utils line 28 Scala Problem
 
  nonthreadsafe is already defined as class nonthreadsafe
  Annotations_2.9+.scala /KafkaEclipse/core/src/main/scala/kafka/utils
  line 33 Scala
  Problem
 
 
  This error is coming from  a file
  Util /kafka/src/main/scala/kafka/utils/Annotations_2.9+.scala
 
  Please note that i had to install scala 2.10 eclipse plugin as Juno had
  some problem with 2.9.
 
 
  Regards,
 
  Pushkar
 



Re: Kafka 0.8.0 server-stop.sh does not stop broker

2013-12-18 Thread Jun Rao
Maybe we should just change it to kill -15. Could you try if that works?
Could you also file a jira?

Thanks,

Jun


On Tue, Dec 17, 2013 at 9:18 PM, Bryan Baugher bjb...@gmail.com wrote:

 RHEL 6.4 64bit
 Java 6u35


 On Tue, Dec 17, 2013 at 10:57 PM, Jun Rao jun...@gmail.com wrote:

  Which OS are you on?
 
  Thanks,
 
  Jun
 
 
  On Tue, Dec 17, 2013 at 11:15 AM, Bryan Baugher bjb...@gmail.com
 wrote:
 
   Hi,
  
   We have been trying out the kafka 0.8.0 beta1 for awhile and recently
   attempted to upgrade to 0.8.0 but noticed that the stop server script
   doesn't seem to stop the broker anymore. I noticed here[1] that a
 commit
   was made before the release to change the signal sent to stop the
 broker
   from SIGTERM to SIGINT. Changing this script back to using SIGTERM
 seems
  to
   fix the issue for me. Has anyone else noticed this, is there a bug or
   should I log one?
  
   [1] -
  
  
 
 https://github.com/apache/kafka/commit/51de7c55d2b3107b79953f401fc8c9530bd0eea0
  
   -Bryan
  
 



 --
 -Bryan



Re: Writing unit tests for Kafka code

2013-12-18 Thread Jun Rao
You can take a look at Kafka's unit tests. See how
kafka.integration.KafkaServerTestHarness is being used.

Thanks,

Jun


On Wed, Dec 18, 2013 at 11:17 AM, Oliver Dain od...@3cinteractive.comwrote:

 Found some of the other references I'd used that also don't quite work:

 https://gist.github.com/fjavieralba/7930018
 https://gist.github.com/mardambey/2650743

 I know, the obvious question is what didn't work. I'm afraid I no longer
 recall. I took each example, tried it, got errors, tried to fix them (e.g.
 change properties names), etc. but it was on a deadline and several days
 ago and I no longer recall exactly what I did. I can try to recreate if
 people think one of those should work, but I suspect there have been known
 changes.

 It would be fantastic if this was all wrapped up in a single method or
 class so creating an embedded broker and/or cluster for testing was really
 easy.


 From: New User od...@3cinteractive.commailto:od...@3cinteractive.com
 Date: Wednesday, December 18, 2013 at 11:10 AM
 To: users@kafka.apache.orgmailto:users@kafka.apache.org 
 users@kafka.apache.orgmailto:users@kafka.apache.org
 Subject: Writing unit tests for Kafka code

 I'm writing some Kafka client code and I'd like to unit test it. I've
 found some resources that say this is possible. For example:

 http://ransilberman.wordpress.com/2013/07/19/how-to-unit-test-kafka/

 but all the information I've found seems a little bit incorrect. For
 example, the above link uses enable.zookeeper which seems to be an
 ignored property. Similarly, other properties mentioned in that file have
 changed names. I found a few posts on this mailing list describing how it
 might be done (sorry - this was a few days ago and I don't have links
 handy), but they didn't quite work either. I tried digging into the scala
 code but it wasn't immediately obvious how to do things, probably partially
 because I don't know Scala.

 If somebody could post working Java code that constructs a Kafka broker
 that can be used for unit testing that would be fantastic. My preference
 would be something that doesn't require ZooKeeper, but there's some talk
 that in 0.8 that may no longer be an option. If not, using the Curator
 framework's TestingServer would be great (I'm using Curator everyone). But
 ultimately, I'll take anything that works!

 Thanks in advance,
 Oliver