no zk connection timeout in consumer config of kafka 0.7?

Hi, In http://kafka.apache.org/08/configuration.html, there are 2 parameter about zk in consumer. zookeeper.session.timeout.ms 6000 Zookeeper session timeout. If the consumer fails to heartbeat to zookeeper for this period of time it is considered dead and a rebalance will occur.

Re: Setting log.default.flush.interval.ms and log.default.flush.scheduler.interval.ms

Thanks Neha and Jun for the pointers. We will try and evaluate this as well. Hemanth On Sat, Sep 13, 2014 at 4:45 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Hemanth, Specifically, you'd want to monitor kafka:type=kafka.SocketServerStats:getMaxProduceRequestMs and

KafkaSpout

Hi all, I am using a kafka spout for reading data from a producer into my storm topology. On the kafkaconfig : 1. When i set Kafka.forcefromstart = true .. it read from the beginning , however if i close my topology midway and run the topology again by setting Kafka.forcefromstart = false , it

Non-blocking High-Level Consumer

Hi, Is it possible to read all available messages with HLC in a non-blocking way? E.g. read all messages and not wait for more messages to appear in the topic. As far as I understand, currently one has to keep high-level consumer in a separate thread until it is shut down explicitly, but how can

Re: Non-blocking High-Level Consumer

There isn't a very clean way to do this in the 0.8.x high level consumer. You can configure consumer.timeout.ms to a certain value so that the consumer's blocking iterator returns if no message arrives for consumer.timeout.ms. Thanks Neha On Mon, Sep 15, 2014 at 6:33 AM, Ivan Balashov

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Ah -- I only tested with dots . However, haven't you said earlier in this thread that only - and _ are allowed as special chars? This would exclude dots, although in the error message below they (dots) are said to be fine. Is this a bug or a feature? ;-) --Michael On 14.09.2014, at

Re: KafkaSpout

Siddharth, Did you try using SpoutConfig spoutConfig.forceFromStart = false spoutConfig.startOffsetTime = kafka.api.OffsetRequest.LatestTime() There are some good examples here on integrating storm and kafka https://github.com/miguno/kafka-storm-starter

Re: Migrating data from old brokers to new borkers question

distribution will be even based on the number of partitions. It is the same logic as AdminUtils. see https://github.com/airbnb/kafkat/blob/master/lib/kafkat/command/reassign.rb#L39 On Sun, Sep 14, 2014 at 6:05 PM, Neha Narkhede neha.narkh...@gmail.com wrote: This is great. Thanks for sharing!

Re: Kafka High Level Consumer

Thanks Joe Stein This worked :) On Fri, Sep 12, 2014 at 3:19 PM, Rahul Mittal rahulmitta...@gmail.com wrote: Hi , Is there a way in kafka to read data from all topics, from a consumer group without specifying topics in a dynamic way. That is if new topics are created on kafka brokers the

Re: no zk connection timeout in consumer config of kafka 0.7?

We have that in 0.7 too. It's called zk.connectiontimeout.ms. Thanks, Jun On Mon, Sep 15, 2014 at 12:55 AM, Yonghui Zhao zhaoyong...@gmail.com wrote: Hi, In http://kafka.apache.org/08/configuration.html, there are 2 parameter about zk in consumer. zookeeper.session.timeout.ms 6000

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Yes, that description is not precise. We do allow dots in general. However, a topic can't be just . or ... Thanks, Jun On Mon, Sep 15, 2014 at 9:31 AM, Michael G. Noll michael+st...@michael-noll.com wrote: Ah -- I only tested with dots . However, haven't you said earlier in this thread

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Hi Jun, I also wanted to highlight the implication of having a dot . as part of the topic and client.id with .. Both topic name and client id need rule enforcement. JMX Bean names are not correct: eg: I have topics called topic.dot and client.id as my.client. JMX Bean name is:

Re: Need Document and Explanation Of New Metrics Name in New Java Producer on Kafka Trunk

Hi Bhavesh, Yes, if you look at https://issues.apache.org/jira/browse/KAFKA-1481 it describes this sort of a problem with various delimiters used by Kafka. As Jun suggested, I'll start a new thread and try to clearly explain the problem. Otis -- Monitoring * Alerting * Anomaly Detection *

MBeans, dashes, underscores, and KAFKA-1481

Hi, *Problem:* Some Kafka 0.8.x MBeans have names composed of things like consumer group-topic-metric name. Note how dashes are used as delimiters. When consumer group and topic don't contain the delimiter character all is good if you want to extract parts of this MBean name by simply splitting

Re: ZkClient bug can bring down broker/consumer on zookeeper push in EC2 environment

Thanks for reporting this issue. Agree that this is a problem for Kafka users using AWS. Please can you open a JIRA so we can keep track of this? On Sun, Aug 17, 2014 at 11:41 AM, Bae, Jae Hyeon metac...@gmail.com wrote: Recently, we found the serious ZkClient bug, actual Apache Zookeeper

Re: ZkClient bug can bring down broker/consumer on zookeeper push in EC2 environment

The above pull request didn't work perfectly. After a bunch of testing experiment, we decided that fixing zkclient itself isn't easy. So we decided to go with another workaround. We're expecting zookeeper 3.5.0 will be stabilized as soon as possible with the feature to refresh connections and the

Re: KafkaSpout

Hi harsha, Yes i did check with the above mentioned spoutconfig setup I think i need to bypass the in memory zookeeper which i am unable to do in local mode of storm. Any thoughts on that? Thanks, Siddharth ubale