producer consumer fail if the leader failed-over

Hi List, I'm new to kafka, feel sorry if this has been asked, I didn't figure out my answer by googling, so asked here. Thanks in advance! I'm following the kafka quick start http://kafka.apache.org/documentation.html#quickstart and managed setup a kafka cluster with two brokers, which

Re: running on scala 2.11

just simply tried artifact kafka_2.10:0.8.1.1 as suggested in https://issues.apache.org/jira/browse/KAFKA-1454 but got the error written above now I've cloned the repo and compiled with ./gradlew jar = success ./gradlew -PscalaVersion=2.10.0 jar = success ./gradlew -PscalaVersion=2.11.0 jar =

Re: producer consumer fail if the leader failed-over

Hi Chengwei, What do you see in the Kafka server logs? This may help you diagnose the failure. Cheers, Robert On Thu, May 29, 2014 at 2:38 AM, Chengwei Yang chengwei.yang...@gmail.com wrote: Hi List, I'm new to kafka, feel sorry if this has been asked, I didn't figure out my answer by

Re: Geeting Payload back from Kafka message

Have you looked at https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example ? Thanks, Jun On Wed, May 28, 2014 at 10:19 PM, Kumar Pradeep kprad...@novell.com wrote: Hi, I am having trouble in getting my payload message back from the Kafka Message in my consumer. My

Re: How to get last message

There is no direct way. You can try getting the last offset (the offset of the next to be appended message) using the getOffsetBefore api and then using that offset - 1 to fetch. Thanks, Jun On Wed, May 28, 2014 at 5:24 PM, hsy...@gmail.com hsy...@gmail.com wrote: Is there a way to get the

HA: mBean to monitor message per partitions in topic

Do you mean getOffsetLag ? (https://cwiki.apache.org/confluence/display/KAFKA/Operations) But its all about offset not message... So for example to create real load balance between broker I can: 1. Calculate average unread message in topic per broker: -

HA: how to control assign policy for consumers

May be right place to add such optimization within https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ? -Исходное сообщение- От: Guozhang Wang [mailto:wangg...@gmail.com] Отправлено: Wednesday, May 28, 2014 8:57 PM Кому: users@kafka.apache.org Тема: Re: how to control

Job opportunity for someone learning Kafka

Hi All, I'm looking to connect with someone who's got about 6 months of Kafka experience looking to build on it with a contract opportunity I have. Please let me know if you'd be interested. Thanks, Jason Gordon Technical Recruiter Red Oak Technologies, Inc. 2105 S. Bascom Ave;

Re: how to control assign policy for consumers

In the new consumer design the partitioning algorithm can be customized by the user: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design Would that fit your use case? Guozhang On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич a.ryab...@ntc-vulkan.ru

HA: how to control assign policy for consumers

Well may be yes and may no I think that in design we must point out that co-ordinator also must analize connection graph and must map consumers to broker with minimal network overhead... -Исходное сообщение- От: Guozhang Wang [mailto:wangg...@gmail.com] Отправлено: Thursday, May

question about synchronous producer

Hi team, Assume I am using a synchronous producer and it has the following default properties: message.send.max.retries 3 retry.backoff.ms 100 I use java api Producer.send(message) to send a message. While send() is being called, if the brokers are shutdown, what happens? send()

another question about Synchronous producer

Hi, Assume I am using a synchronous producer and request.required.acks is set to 0 (default). When java api Producer.send(message) returns, is the message acknowledged? Is there any chance of message loss in the case? Thanks. Libo

Re: Producer Side Metric Details

We have 4 producers (for 4 topics) within one JVM and all of them configured with same broker list. Each one of the producer has client.id configured as topic name. When I see the JMX console, I see that *ProducerRequestMetric* JMX have issue, it takes the last client.id config and report the

Exception kafka.common.NotLeaderForPartitionException

Hi Kafka User Group, We have recently changed the “message.max.bytes” to 2MB and rebooted all the brokers at once. After this, we are getting “kafka.common.NotLeaderForPartitionException” on both Producer side and consumer side. How do I fix this, I have restarted producer and consumer

Re: producer consumer fail if the leader failed-over

Hi Jun, Thanks for your reply. On Thu, May 29, 2014 at 07:52:11AM -0700, Jun Rao wrote: You need to pass in multiple brokers in --broker-list in the producer. To obtain the new leaders, the producer needs to talk to a live broker to get the new metadata. Then I did tried with with 3 broker,

Message lost during leader fail-over

Hi Jun, I think I was scared by these exceptions, the producer and consumer are still alive, seems reconnect to another broker though it through a lot of exceptions. However I find another issue seems if the consumer write a message just after the leader broker killed, the message will lost,

Get a list of consumer groups (for ConsumerOffsetChecker)

I'm doing some initial exploration of kafka and the system tools. With the console producer and consumers running, I'd like to run the ConsumerOffsetChecker, but it requires a group parameter. I don't know what to put there. Is there a tool I can run to get a list of consumer groups? I see a

Re: Get a list of consumer groups (for ConsumerOffsetChecker)

Ryan, We do not have this support in zk tools yet, could you file a JIRA for this? Guozhang On Thu, May 29, 2014 at 7:22 PM, Ryan Williams rwilli...@gmail.com wrote: I'm doing some initial exploration of kafka and the system tools. With the console producer and consumers running, I'd like

Re: another question about Synchronous producer

Libo, No there will be no ACKs sent back, messages may failed to reach the broker but producers will not be notified in this case. Guozhang On Thu, May 29, 2014 at 5:24 PM, Libo Yu yu_l...@hotmail.com wrote: Hi, Assume I am using a synchronous producer and request.required.acks is set to

Re: Exception kafka.common.NotLeaderForPartitionException

Bhavesh, This exception is usually normal and transient upon broker restarts. Do you still see them after restarting the brokers for a while? Guozhang On Thu, May 29, 2014 at 6:02 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi Kafka User Group, We have recently changed the

Re: mBean to monitor message per partitions in topic

If you open up jmx (e.g. jconsole) in a consumer instance, you will see the jmx of name *-ConsumerLag. Thanks, Jun On Thu, May 29, 2014 at 8:53 AM, Рябков Алексей Николаевич a.ryab...@ntc-vulkan.ru wrote: Do you mean getOffsetLag ? (

Re: producer consumer fail if the leader failed-over

That's just a warning and the producer will retry. Was there an error (which means all retries failed)? Thanks, Jun On Thu, May 29, 2014 at 6:06 PM, Chengwei Yang chengwei.yang...@gmail.com wrote: Hi Jun, Thanks for your reply. On Thu, May 29, 2014 at 07:52:11AM -0700, Jun Rao wrote:

Re: Message lost during leader fail-over

Have you looked at request.required.acks in http://kafka.apache.org/documentation.html#producerconfigs ? Thanks, Jun On Thu, May 29, 2014 at 6:43 PM, Chengwei Yang chengwei.yang...@gmail.com wrote: Hi Jun, I think I was scared by these exceptions, the producer and consumer are still

Re: Exception kafka.common.NotLeaderForPartitionException

Hi Guozhang Wang, I restarted the Broker yesterday, and I am still seeing this exception on consumer and producer sides. Thanks, Bhavesh On Thu, May 29, 2014 at 8:14 PM, Guozhang Wang wangg...@gmail.com wrote: Bhavesh, This exception is usually normal and transient upon broker restarts. Do

Re: running on scala 2.11

It is possible that when I tried this ticket initially there was something environmental in my build process causing it to appear to work. I reopened the ticket. I will go back through it again on a few different environments and in more detail and post results either way.

Re: Message lost during leader fail-over

On Thu, May 29, 2014 at 08:39:47PM -0700, Jun Rao wrote: Have you looked at request.required.acks in http://kafka.apache.org/documentation.html#producerconfigs ? Thanks Jun, that's make sense to me. -- Thanks, Chengwei Thanks, Jun