Is there a way to change Kafka's data path in Zookeeper transparently to Kafka client?

Hi group, We are running Kafka clusters in production, using Zookeeper without chroot, unwisely. Then there are several Kafka-related dirs in Zookeeper's root path with some general names, such as */consumers*. And recently we have another application coincidently using same */consumers *path,

Re: Auto preferred leader elections cause data loss?

Hi group, I think I've hit the KAFKA-1561 and KAFKA-1211, that is follower's HW is always <= leader's HW, then when leader transits, data loss can happen... So what we could do is try to do every thing to avoid leader transition, right? 2015-09-14 11:59 GMT+08:00 Zhao Weinan <zhaow...@gma

Re: Auto preferred leader elections cause data loss?

, then the lag between this new leader and old-gone-leader is lost, right? Thanks! 2015-09-12 18:15 GMT+08:00 Zhao Weinan <zhaow...@gmail.com>: > Hi group, > > I've been through some data loss on kafka cluster, one case maybe caused > by the auto preferred leader election. > >

Auto preferred leader elections cause data loss?

Hi group, I've been through some data loss on kafka cluster, one case maybe caused by the auto preferred leader election. Here is the situation: 3 brokers = {0,1,2}, 1 partition with 3 replicas on 0/1/2, all in sync while 0 is leader and 1 is controller, current offset is 100. And here is my

Re: Possible DEAD LOCK for one day at broker controller?

Senji <kse...@gmail.com>: > Hi Zhao, Do you see any other errors regarding checkpoint file? Is this > reproducible by you and if you can you enable debug log level to get more > info.​ > > On Thu, Aug 20, 2015 at 7:44 AM, Zhao Weinan <zhaow...@gmail.com> wrote: > >

Re: Possible DEAD LOCK for one day at broker controller?

from 10 days and see if the brokers are fine. On Tue, Aug 18, 2015 at 12:48 AM Zhao Weinan zhaow...@gmail.com wrote: Hi Kishore Senji, Did you constantly send messages to your test topic? Or just one time send? I've just did some test, the log.lastModified is updated with every message

Re: Problem with node after restart no partitions?

Hi, I've been through some similar problems, posted in anthor thread http://mail-archives.apache.org/mod_mbox/kafka-users/201508.mbox/%3CCAJY8EoeqCpxFirQzLYiKk5Gz4bqQrVv3Z_WGx8T5qPnk3VMAbA%40mail.gmail.com%3E(I just pasted some broker logs, please check the latest message which the mail archiever

Re: Possible DEAD LOCK for one day at broker controller?

segment file is huge and the OS may be taking a long time to destage the file cache on to disk before letting it to be deleted. This could be the reason for long pause which could be causing the Zk connections to be timed out. On Mon, Aug 17, 2015 at 6:59 AM Zhao Weinan zhaow...@gmail.com

Re: Possible DEAD LOCK for one day at broker controller?

be the fix we might need. retention.ms config parameter should not have commas in the value. Are you just using it here to clarify for us. It so happened in your On Sun, Aug 16, 2015 at 1:52 AM Zhao Weinan zhaow...@gmail.com wrote: Hi guys, I got this problem, after changing one topic's config

Possible DEAD LOCK for one day at broker controller?

Hi guys, I got this problem, after changing one topic's config to retention.ms=86,400,000 from 864,000,000, the brokers start to shedule and do deletions of outdated index of that topic. Then for some reason some brokers' connection with zookeeper were expired, suddenly lots of ERRORs showed up

high level consumer api blocked forever

encountered this, or it's this a known issue? Any information will be helpful. Thanks a lot! -- Zhao Weinan

Is there acl or other things on kafka?

Hi, i've searched the mailing list archive, but nothing found. I'm wondering how to prevent test producer from sending dirty messages to a production broker? -- zhaown

Re: Is there acl or other things on kafka?

Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Mon, Nov 4, 2013 at 4:19 AM, zhao weinan xcvisc...@gmail.com wrote: Hi, i've searched the mailing list archive, but nothing found