zookeeper upgrade or remove zookeeper dependency

2014-09-17 Thread Mingtao Zhang
Hi, I could see kafka is using zookeeper 3.3.4. For my integration purpose, I want to use curator, which requires a higher version than 3.3.4 even in its lowest version. I there any plan to bump up zookeeper dependency? Or is there any plan to remove zookeeper dependency? Best Regards, Mingtao

Re: zookeeper upgrade or remove zookeeper dependency

2014-09-17 Thread Neha Narkhede
Kafka trunk is on a later zookeeper version (3.4.6). So the next release (0.8.2) will depend on zookeeper 3.4.6 On Wed, Sep 17, 2014 at 8:55 AM, Mingtao Zhang mail2ming...@gmail.com wrote: Hi, I could see kafka is using zookeeper 3.3.4. For my integration purpose, I want to use curator,

Re: MBeans, dashes, underscores, and KAFKA-1481

2014-09-17 Thread Jun Rao
Bhavesh, Yes, allowing dot in clientId and topic makes it a bit harder to define the JMX bean names. I see a couple of solutions here. 1. Disable dot in clientId and topic names. The issue is that dot may already be used in existing deployment. 2. We can represent the JMX bean name differently

Re: zookeeper upgrade or remove zookeeper dependency

2014-09-17 Thread Mingtao Zhang
Could you also share a rough time point of 0.8.2 release? Best Regards, Mingtao On Wed, Sep 17, 2014 at 12:10 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Kafka trunk is on a later zookeeper version (3.4.6). So the next release (0.8.2) will depend on zookeeper 3.4.6 On Wed, Sep 17, 2014

Different partitioning between new producer and old producer

2014-09-17 Thread Bae, Jae Hyeon
The major motivation of adopting new producer before it's released, old producer is showing terrible throughput of cross-regional kafka mirroring in EC2. Let me share numbers. Using iperf, network bandwidth between us-west-2 AWS EC2 and us-east-1 AWS EC2 is more than 40 MB/sec. But old

Disactivating Yammer Metrics Monitoring

2014-09-17 Thread François Langelier
Hi all! We are using yammer metrics to monitor some parts of our system. Since we upgrade from kafka 0.7.2 to 0.8.1.1, we saw a lot more data getting in our graphite server and from what I saw, it looks like it all come from our producers. From what i understand, since we already use graphite,

Re: MBeans, dashes, underscores, and KAFKA-1481

2014-09-17 Thread Bhavesh Mistry
Sure we can do the option 2 for JMX beans. But same solution should be applied to producer.metrics() method for new producer. Regardless of metric is access (JMX or via producer), it has to be consistent naming convention. For example, I get following metric name when my topic is topic.dot. So

[Java New Producer] CPU Usage Spike to 100% when network connection is lost

2014-09-17 Thread Bhavesh Mistry
Hi Kafka Dev team, I see my CPU spike to 100% when network connection is lost for while. It seems network IO thread are very busy logging following error message. Is this expected behavior ? 2014-09-17 14:06:16.830 [kafka-producer-network-thread] ERROR

Re: MBeans, dashes, underscores, and KAFKA-1481

2014-09-17 Thread Otis Gospodnetic
Hi Jun, On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao jun...@gmail.com wrote: Bhavesh, Yes, allowing dot in clientId and topic makes it a bit harder to define the JMX bean names. I see a couple of solutions here. 1. Disable dot in clientId and topic names. The issue is that dot may already be

Re: Migrating data from old brokers to new borkers question

2014-09-17 Thread Alexis Midon
we would be very happy to contribute. However a description of the current plan and status regarding tooling would be helpful. It would speed up the learning curve. You mentioned some jira tickets? (maybe I should sign up to the developer mailing list and take the conversation over there) On

Re: zookeeper upgrade or remove zookeeper dependency

2014-09-17 Thread Neha Narkhede
You can track the list of open bugs here https://issues.apache.org/jira/browse/KAFKA-1558?jql=project%20%3D%20Kafka%20and%20fixVersion%20%3D%200.8.2%20and%20status%20!%3D%20Resolved%20and%20status%20!%3D%20Closed . On Wed, Sep 17, 2014 at 10:00 AM, Mingtao Zhang mail2ming...@gmail.com wrote:

Re: Different partitioning between new producer and old producer

2014-09-17 Thread Neha Narkhede
Could you make them same logic? Otherwise, I have to change implementation of kafka producer container. The new producer is much more flexible and allows the user to use custom partitioning logic and provide the partition number in the ProducerRecord. That way it is broadly applicable to a

Re: [Java New Producer] CPU Usage Spike to 100% when network connection is lost

2014-09-17 Thread Neha Narkhede
This seems like a problem. Please can you file a JIRA and attach the log4j output there? On Wed, Sep 17, 2014 at 2:14 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi Kafka Dev team, I see my CPU spike to 100% when network connection is lost for while. It seems network IO thread are

Re: Migrating data from old brokers to new borkers question

2014-09-17 Thread Neha Narkhede
The idea is to bake the functionality of such a tool in Kafka itself. In an ideal world, a Kafka cluster would automatically detect leader and data imbalance and trigger a rebalance operation that leads to optimal performance. I'm not sure if we have a JIRA for this though. So feel free to create

Re: Disactivating Yammer Metrics Monitoring

2014-09-17 Thread Daniel Compton
Hi Francois I didn't quite understand how you've set up your metrics reporting. Are you using the https://github.com/criteo/kafka-ganglia metrics reporter? If so then you should be able to adjust the config to exclude the metrics you don't want, with kafka.ganglia.metrics.exclude.regex. On 18