kafka message expiry question

hi, as far as i understand, log retention time in kafka will delete message that's older than the retention time. i'm wonder what it implies for consumer since i'm using simple consumer to manage offsets in predefined consumer group. say i have a list of messages for a partition of topic:

Re: offset commit api

the offset commit and fetch requests to any broker. Kafka-backed consumer offsets is currently in trunk and will be released in 0.8.2. Thanks, Joel On Mon, Aug 04, 2014 at 02:57:02PM -0700, Weide Zhang wrote: Hi It seems to me that 0.8.1.1 doesn't have the ConsumerMetadata API. So

Re: offset commit api

Hi It seems to me that 0.8.1.1 doesn't have the ConsumerMetadata API. So what broker I should choose in order to commit and fetch offset information ? Shall I use zookeeper for offset to manage it manually instead ? Thanks, Weide On Sun, Aug 3, 2014 at 4:34 PM, Weide Zhang weo...@gmail.com

Re: kafka simpleconsumer question

at 5:14 PM, Weide Zhang weo...@gmail.com wrote: Hi , Just want to ask some basic question about kafka simple consumer. 1. if I'm using simple consumer and doesn't really depend on zookeeper to manage partition offset. (application manage offset themselves). Will that remove

kafka simpleconsumer question

Hi , Just want to ask some basic question about kafka simple consumer. 1. if I'm using simple consumer and doesn't really depend on zookeeper to manage partition offset. (application manage offset themselves). Will that remove the zookeeper dependency for consumer ? 2. if zookeeper dies, will

topics load balancing within a consumer group

Hi, I have a question regarding load balancing within a consumer group. Say I have a consumer group of 4 consumers which subscribe to 4 topics , each of which have one partition. Will there be rebalancing happening on topic level ? Or I will expect consumer 1 have all the data ? Weide

kafka 0.8.1.1 works under java 1.6 ?

Hi, According to the Kafka documentation, seems Java 1.7 is recommended. But for our production environment we are still using java 1.6. Will that be a problem of using java 1.6 and use Kafka 0.8.1.1 ? Thanks a lot, Weide

Re: What happens to Kafka when ZK lost its quorum?

Hi Guozhang, In worst case zookeeper dies for say 1 hour and come back up, will things still be recovered automatically after 1 hour ? Weide On Wed, May 14, 2014 at 8:28 AM, Guozhang Wang wangg...@gmail.com wrote: In 0.8, the servers and consumers are heavily dependent on ZK to function.

zookeeper down

Can kafka survive when zookeeper is down and not connectable ? Will the consumer or producer still work in that case ? Weide

Re: question about mirror maker

to produce messages, for example, if the network goes down. If it can still consume messages, but cannot produce them, you will lose messages as the consumer will continue to commit offsets with no knowledge that the producer is failing. -Todd On 5/8/14, 11:20 AM, Weide Zhang weo...@gmail.com wrote

question about mirror maker

Hi, I have a question about mirror maker. say I have 3 data centers each producing topic 'A' with separate kafka cluster running. if 3 of the data need to be kept in sync with each other, shall i create 3 mirror maker in each data center to get the data from the other two ? also, it mentioned