Re: Setting log.default.flush.interval.ms and log.default.flush.scheduler.interval.ms

2014-09-16 Thread Hemanth Yamijala
Neha, I got some time today to test the JMX values you mentioned. My test setup includes a Kafka broker, a PHP based load generator that generates messages at a steady rate, and a Storm Kafka consumer. The PHP load generator continuously pumps in a standard 3KB message with a sleep interval of

Re: KafkaSpout

2014-09-16 Thread Harsha
Are you running this using storm LocalCluster and you want to use external zookeeper for LocalCluster?. The latest code has those changes you can pass params to LocalCluster(localhost,2181). -Harsha On Mon, Sep 15, 2014, at 10:30 PM, siddharth ubale wrote: Hi harsha, Yes i did check with

Re: ZkClient bug can bring down broker/consumer on zookeeper push in EC2 environment

2014-09-16 Thread Neha Narkhede
Would you mind sharing your workaround with the community? On Mon, Sep 15, 2014 at 10:17 PM, Bae, Jae Hyeon metac...@gmail.com wrote: The above pull request didn't work perfectly. After a bunch of testing experiment, we decided that fixing zkclient itself isn't easy. So we decided to go with

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

2014-09-16 Thread Michael G. Noll
Many thanks for the clarification, Jun! Michael On 16.09.2014, at 02:11, Jun Rao jun...@gmail.com wrote: Yes, that description is not precise. We do allow dots in general. However, a topic can't be just . or ... Thanks, Jun On Mon, Sep 15, 2014 at 9:31 AM, Michael G. Noll

Re: ZkClient bug can bring down broker/consumer on zookeeper push in EC2 environment

2014-09-16 Thread Bae, Jae Hyeon
I am not sure how many companies are running zookeeper like we are doing and I haven't seen any other companies terminating all EC2 instances to update zookeeper cluster. But in the worst case, zookeeper EC2 instances can be replaced before restarting kafka broker and consumer, sharing the

Re: MBeans, dashes, underscores, and KAFKA-1481

2014-09-16 Thread Otis Gospodnetic
Hi, So maybe I should I should have asked the Q explicitly: Could we commit the patch from https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, it's clear what problems the current MBean names can cause? Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log

Why is Scala 2.9.2 version of kafka recommended?

2014-09-16 Thread Ash G
On Kafka download page we saw - We build for multiple versions of Scala. This only matters if you are using Scala and you want a version built for the same Scala version you use. Otherwise any version should work (2.9.2 is recommended). Why is Scala 2.9.2 version of kafka recommended?

Re: MBeans, dashes, underscores, and KAFKA-1481

2014-09-16 Thread Bhavesh Mistry
HI Otis, What is migration path ? If topic with special chars exists already( .,-,| etc) in previous version of producer/consumer of Kafka, what happens after the upgrade new producer or consumer (kafka version) ? Also, in new producer API (Kafka Trunk), does this enforce the rule about client

Re: Migrating data from old brokers to new borkers question

2014-09-16 Thread Gwen Shapira
Since these tools are so useful, I wonder what it requires (from both Airbnb and Kafka) to merge this into Kafka project. I think there are couple of Jira regarding improved tool usability that this resolved. On Mon, Sep 15, 2014 at 11:45 AM, Alexis Midon alexis.mi...@airbedandbreakfast.com

Re: Non-blocking High-Level Consumer

2014-09-16 Thread Gwen Shapira
For Fluffka, I created a wrapping function: IterStatus timedHasNext() { try { long startTime = System.currentTimeMillis(); it.hasNext(); long endTime = System.currentTimeMillis(); return new IterStatus(true,endTime-startTime); } catch (ConsumerTimeoutException e)

Re: Why is Scala 2.9.2 version of kafka recommended?

2014-09-16 Thread Joe Stein
That is for the broker. It is what was most used in production (by LinkedIn) prior to the official release. The client is binary and language independent. You can run 2.10 (for the broker) too, no probs. /*** Joe Stein Founder, Principal Consultant Big

Re: KafkaSpout

2014-09-16 Thread siddharth ubale
Hi harsha, Do you mean that o.9.2 incubating has those changes?? i am using the latest release of storm i.e 0.9.2 incubating... i am unable to pass on : public static LocalCluster lc = new LocalCluster(localhost,2181); please do let me know.. Thanks, Sidharth Ubale On Tue, Sep 16, 2014 at