Re: Consumer Group Rebalance Issues

Our cluster is currently running 3.4.4. I see Kafka is currently using the 3.3.4 client, is there a potential conflict there? On Wed, Dec 18, 2013 at 9:12 PM, Jun Rao wrote: > The issue is that consumer 007 didn't see consumer 006 during rebalancing. > So, it made a decision in conflict with c

Re: Data loss in case of request.required.acks set to -1

Hi Guozhang, I have tried with Kafka-0.8.1 after applying patch 1188 but thats not helping in this case. Also controlled.shutdown.enable is also not helpful in case of abnormally shutdown (i.e. SIGKILL (-9)). Any other suggestion? On Thu, Dec 19, 2013 at 3:59 PM, Guozhang Wang wrote: > Yes,

Re: upgrade from beta1 to 0.81

We migrated from 0.8.0 to 0.8.1 last week. We have a 15 broker cluster so it took a while to roll through them one by one. Once I finished I was finally able to complete a partition reassignment. I also had to do some manual cleanup, but Neha says it will be fixed soon: https://issues.apache.or

Re: Migrating a cluster from 0.8.0 to 0.8.1

Hi Drew, That problem will be fixed by https://issues.apache.org/jira/browse/KAFKA-1074. I think we are close to checking that in to trunk. Thanks, Neha On Wed, Dec 18, 2013 at 9:02 AM, Drew Goya wrote: > Thanks Neha, I rolled upgrades and completed a rebalance! > > I ran into a few small iss

Re: Connection Timeouts

The fetch.wait.max.ms is unchanged. The request log is empty. I'll turn it up to TRACE (it's currently on INFO) and see what it contains. Thanks --- Did you change fetch.wait.max.ms in the consumer config? If so, did you make

Re: Kafka 0.8.0 server-stop.sh does not stop broker

Bryan, The server's shutdown hook should be able to trigger with any SIG except SIGKILL. I just tried the following process and the stop script works for me: 1. bin/zookeeper-server-start.sh config/zookeeper.properties 2. bin/kafka-server-start.sh config/server.properties 3. bin/kafka-server-s

Re: upgrade from beta1 to 0.81

Libo, As I have replied in the previous email, it can be done in place. Guozhang On Thu, Dec 19, 2013 at 10:51 AM, Yu, Libo wrote: > Hi Guozhang, > > What I'd like to know is whether the upgrade from 0.8-beta1 to 0.81 will > be in place. > > Regards, > > Libo > > > -Original Message-

Re: upgrade from beta1 to 0.81

0.8.1 is working in stable at LinkedIn now. Guozhang On Thu, Dec 19, 2013 at 10:52 AM, Yu, Libo wrote: > I also want to know how stable the 0.81 will be, compared with 0.8 or > 0.8-beta1. > > Regards, > > Libo > > > -Original Message- > From: Jason Rosenberg [mailto:j...@squareup.com]

RE: upgrade from beta1 to 0.81

I also want to know how stable the 0.81 will be, compared with 0.8 or 0.8-beta1. Regards, Libo -Original Message- From: Jason Rosenberg [mailto:j...@squareup.com] Sent: Thursday, December 19, 2013 12:54 PM To: users@kafka.apache.org Subject: Re: upgrade from beta1 to 0.81 How stable i

RE: upgrade from beta1 to 0.81

Hi Guozhang, What I'd like to know is whether the upgrade from 0.8-beta1 to 0.81 will be in place. Regards, Libo -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Thursday, December 19, 2013 12:18 PM To: users@kafka.apache.org Subject: Re: upgrade from beta1 to

Re: upgrade from beta1 to 0.81

Jason, We are targeting at releasing 0.8.1 next month. Guozhang On Thu, Dec 19, 2013 at 9:53 AM, Jason Rosenberg wrote: > How stable is 0.8.1, will there be a 'release' of this soon, or are there > still significant open issues? > > Thanks, > > Jason > > > On Thu, Dec 19, 2013 at 12:17 PM, Gu

Re: doubt regarding the metadata.brokers.list parameter in producer properties

Since 0.8, broker discovery is not through zk anymore. Guozhang On Thu, Dec 19, 2013 at 8:45 AM, pushkar priyadarshi < priyadarshi.push...@gmail.com> wrote: > is the auto broker discovery not possible through zk.connect? > > Regards, > Pushkar > > > On Thu, Dec 19, 2013 at 9:29 PM, Jun Rao wro

Re: upgrade from beta1 to 0.81

How stable is 0.8.1, will there be a 'release' of this soon, or are there still significant open issues? Thanks, Jason On Thu, Dec 19, 2013 at 12:17 PM, Guozhang Wang wrote: > Libo, yes the upgrade from 0.8 to 0.8.1 can be done in place. > > Guozhang > > > On Thu, Dec 19, 2013 at 8:57 AM, Yu,

Re: upgrade from beta1 to 0.81

Libo, yes the upgrade from 0.8 to 0.8.1 can be done in place. Guozhang On Thu, Dec 19, 2013 at 8:57 AM, Yu, Libo wrote: > Hi folks, > > As the tools in 0.8 are not stable and we don't want to take the risk. we > want > to skip 0.8 and upgrade from beta1 to 0.81 directly. So my question is > wh

upgrade from beta1 to 0.81

Hi folks, As the tools in 0.8 are not stable and we don't want to take the risk. we want to skip 0.8 and upgrade from beta1 to 0.81 directly. So my question is whether we can do an in place upgrade and let 0.81 use beta1's zk and kf data. Assume that we will disable log compaction. Thanks. Regard

Re: doubt regarding the metadata.brokers.list parameter in producer properties

is the auto broker discovery not possible through zk.connect? Regards, Pushkar On Thu, Dec 19, 2013 at 9:29 PM, Jun Rao wrote: > metadata.broker.list is also used when there are leader failures. Could you > get a vip in ec2 and put the brokers' ip behind the vip? > > Thanks, > > Jun > > > On T

Re: Writing unit tests for Kafka code

If your developing in Scala then check out my Scala Example code repository https://github.com/stealthly/scala-kafka. It spins up two virtual machines (zookeeper and a kafka broker) and has a sample producer and consumer with passing test cases (against the vms). If your developing in Java, the Va

Re: Can we archive our kafka-requestlog in kafka0.8??

Never used the gz part. So don't know how it works. Another way is to just use the regular rolling file appender and have a background process to compress the rolled logs. Thanks, Jun On Thu, Dec 19, 2013 at 1:44 AM, Nishant Kumar wrote: > Thanks Jun, > > Thanks Jun for your quick reply. But

Re: Data loss in case of request.required.acks set to -1

Yes, please go ahead. On Thu, Dec 19, 2013 at 2:30 AM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi Guazhang, > > Can I try it now with trunk HEAD kafka code after applying the patch > KAFKA-1188.patch< > https://issues.apache.org/jira/secure/attachment/12619475/KAFKA-1188.patch

Re: doubt regarding the metadata.brokers.list parameter in producer properties

metadata.broker.list is also used when there are leader failures. Could you get a vip in ec2 and put the brokers' ip behind the vip? Thanks, Jun On Thu, Dec 19, 2013 at 4:07 AM, Arjun wrote: > So from your reply what i understood is this particular property iis used > only when starting the p

Re: doubt regarding the metadata.brokers.list parameter in producer properties

metadata.broker.list is used to discover brokers within the kafka cluster on producer startup and on produce request gets error responses (that is when the producer needs to refresh its memory of the available brokers). While the real broker list discovered is a list of host:port, this list used fo

Re: doubt regarding the metadata.brokers.list parameter in producer properties

correct On Thu, Dec 19, 2013 at 5:37 PM, Arjun wrote: > So from your reply what i understood is this particular property iis used > only when starting the producers. > > is that right? can you please confirm. > > Thanks > Arjun Narasimha Kota > > > On Thursday 19 December 2013 05:33 PM, pushkar

Re: doubt regarding the metadata.brokers.list parameter in producer properties

So from your reply what i understood is this particular property iis used only when starting the producers. is that right? can you please confirm. Thanks Arjun Narasimha Kota On Thursday 19 December 2013 05:33 PM, pushkar priyadarshi wrote: 1.When you start producing : at this time if any of

Re: doubt regarding the metadata.brokers.list parameter in producer properties

1.When you start producing : at this time if any of your supplied broker is alive system will continue to work. 2.Broker going down and coming up with new IP : producer API refreshes metadata information on failures(configurable) so they should be able to detect new brokers. But i dont think it's p

doubt regarding the metadata.brokers.list parameter in producer properties

Hi, I am running kafka 0.8 and zoo keeper along with it. The problem i have is with the metadata.broker.list which is present in the producer properties. As i am using the zookeeper can i just ignore this property and kafka will work fine? We run kafka on the ec2 nodes, lets say we are running

Re: Data loss in case of request.required.acks set to -1

Hi Guazhang, Can I try it now with trunk HEAD kafka code after applying the patch KAFKA-1188.patch? On Wed, Dec 18, 2013 at 9:49 PM, Guozhang Wang wrote: > Kafka server's shutdown hook should capture all SIG but NOT

Re: Kafka 8.0 performance

Hi Jun, Magnus, thanks for the suggestions, actually we implemented our own producer in erlang to make sure that this is not the bottleneck. In the mean time we decided to stay with our current installation of kafka 0.6 and improve our existing failover mechanism. Fortunately, we can tolerate

Re: Kafka producer behavior

Thanks Gerrit !! I am able to use that now. On Wed, Dec 18, 2013 at 10:44 AM, Gerrit Jansen van Vuuren < gerrit...@gmail.com> wrote: > Hi, > > this is a gotcha about kafka producer partitioning, you much send the > messages with a non null key. > If the key is null kafka will not call the parti

Re: Can we archive our kafka-requestlog in kafka0.8??

Thanks Jun, Thanks Jun for your quick reply. But i am still not able to make archive files in kafka/log folder. I am using following properties in kafka/config/log4j.properties file log4j.appender.requestAppender=org.apache.log4j.RollingFileAppender log4j.appender.requestAppender.policy= org.apa