Re: Kafka broker stopped send/receive messages

Do you see ZK session expiration? If so, take a look at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whypartitionleadersmigratethemselvessometimes ? Thanks, Jun On Fri, May 30, 2014 at 2:48 PM, joe smith wrote: > > Hi, > > We are doing load testing. Ran into some exception - nee

Kafka broker stopped send/receive messages

Hi, We are doing load testing.  Ran into some exception - need some help clarifying what the problem is and how to bring the broker back on-line (hoping w/o the need to shutdown other brokers).  Each broker was handling about 100k msgs/sec.  Payload is small - less than 500 bytes About 2 hours

Re: mBean to monitor message per partitions in topic

Ok, if you want to track this on the broker, you can use the consumer offset check tool that Guozhang mentioned. Thanks, Jun On Fri, May 30, 2014 at 7:44 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > Thanks ... but I think its really not good for my case... > > This is beca

Re: question about synchronous producer

I should mention that before each retry the producer will try to refresh its metadata, so with replication factor >1 and when leader of the partition has failed, it may find a different leader next time since it is migrated by the controller, and retry will send to a different broker. Guozhang O

Re: question about synchronous producer

That is correct. On Fri, May 30, 2014 at 6:20 AM, Libo Yu wrote: > Digged into kafka code and did some tests. Let me try to answer it by > myself. Please correct me if I am wrong. > While send() is being called, if brokers are down, send will throw > FailedToSendMessageException. Users can catc

HA: mBean to monitor message per partitions in topic

Thanks ... but I think its really not good for my case... This is because I must connect to all consumers for such stat...And I may have a lot of consumers... from 5k to 15k... So why not to add getNumFetchRequests & getNumProduceRequests in kafka.BrokerTopicStat.[topic]? (I can use getBytesI

RE: question about synchronous producer

Digged into kafka code and did some tests. Let me try to answer it by myself. Please correct me if I am wrong. While send() is being called, if brokers are down, send will throw FailedToSendMessageException. Users can catch it and retry. > From: yu_l...@hotmail.com > To: users@kafka.apache.org >

RE: question about synchronous producer

Thanks for your help, Guozhen. Assume request.required.acks is set to 1 for the scenario I mentioned. While send() is being called, brokers are shutdown. What should I expect? 1 send() will retry 3 times with a 100ms interval and return. 2 send will not return until the brokers are back online.

Re: log.retention.size

Sorry for the delay on this. Yes, that's right -- it'd be just another term in the chain of 'or' conditions. Currently it's OR . With the global condition, it would be OR OR In my view, that's fairly simple and intuitive, hence a fine piece of logic. Regards, András On 5/27/2014 4:34

Re: running on scala 2.11

cool. thanks, Joe. On Fri, May 30, 2014 at 6:54 AM, Joe Stein wrote: > It is possible that when I tried this ticket initially there was something > environmental in my build process causing it to appear to work. > > I reopened the ticket. > > I will go back through it again on a few different

HA: how to control assign policy for consumers

Well for my task you are right... I just need to make sure that a producer would connect to as few brokers as possible, and to achieve this you just needs to know the partition assignment map, which exist on any broker's metadata cache But if we try (for example) to rebalance the hole syste