Getting Simple consumer details using MBean

nsumer details. I am using Kafka 0.8.1.1 version. Thanks and Regards, Madhukar Bharti

Re: Getting Simple consumer details using MBean

: > Those are for 0.7. In 0.8, you should see sth > like FetchRequestRateAndTimeMs in SimpleConsumer. > > Thanks, > > Jun > > On Wed, Nov 12, 2014 at 5:14 AM, Madhukar Bharti > > wrote: > > > Hi, > > > > I want to get the simple con

Re: Getting Simple consumer details using MBean

type="FetchRequestAndResponseMetrics",name="SimpleConsumerShell-AllBrokersFetchRequestRateAndTimeMs" > > Thanks, > > Jun > > On Wed, Nov 12, 2014 at 9:57 PM, Madhukar Bharti > > wrote: > > > Hi Jun Rao, > > > > Thanks for your quick rep

Re: Getting Simple consumer details using MBean

un Rao wrote: > > > So, you want to monitor the mbeans on the broker side? Take a look at > > http://kafka.apache.org/documentation.html#monitoring > > > > Thanks, > > > > Jun > > > > On Thu, Nov 13, 2014 at 10:58 PM, Madhukar Bharti < > &g

How to Setup MirrorMaker in Generalized way

-0 in Datacenter2*. Please suggest me a solution for doing so. If MirrorMaker <http://kafka.apache.org/documentation.html#basic_ops_mirror_maker> tool provide any configurations which solve this use-case please let me know. Regards, Madhukar Bharti

Re: How to Setup MirrorMaker in Generalized way

the message key for partitioning so > > order is preserved on a per-key basis. > > > There is no way to setup an *exact* Kafka mirror yet. > > Thanks, > Neha > > On Mon, Dec 8, 2014 at 7:47 AM, Madhukar Bharti > wrote: > > > Hi, > > > > I am go

Re: How To Specify partioner.class for Mirrormaker?

Dear Alex Melville, You can put the you partitioner class in an jar file inside kafka/libs. Now if you will start Mirror maker it will load the jar and you will not get any exception. On Sun, Dec 14, 2014 at 9:41 AM, Alex Melville wrote: > > Sorry, the email sent before I typed my question. Ple

How to setup inter DC replication in Kafka 0.8.1.1

Hi, I want to setup inter DC replication b/w Kafka clusters. Is there any inbuilt tool to do this? I already have tried MirrorMaker tool but the problem is, if MM killed then some messages get duplicated. I don't want to duplicate the messages. Please suggest a way to do this. Please share your

Re: How to setup inter DC replication in Kafka 0.8.1.1

FKA-1650 / 1839 / 1840. > > For your concern, by saying "MM killed" do you mean it is shutdown or > hard-failed / crashed that can cause duplicates? > > Guozhang > > On Mon, Jan 19, 2015 at 10:51 PM, Madhukar Bharti < > bhartimadhu...@gmail.com> > wrote:

How to fetch consumer group names of a Topic from Kafka offset manager in Kafka 0.8.2.1

Hi, I am using Kafka 0.8.2.1. I have two topics with 10 partitions each. Noticed that one more topic exist named as "__consumer_offset" with 50 partitions. My questions are: 1. Why this topic is created with 50 partition? 2. How to get consumer group names for a topic? Is there any document or A

Async producer using Sync producer for send

Hi All, I am using *async *producer to send the data. When I checked the log it is showing as below: [2015-03-31 11:09:55,915] INFO Verifying properties (kafka.utils.VerifiableProperties) [2015-03-31 11:09:55,946] INFO Property key.serializer.class is overridden to kafka.serializer.StringEncoder

Re: Async producer using Sync producer for send

sender thread, it still > uses SyncProducer to send ProducerRequests to brokers. > > Jiangjie (Becket) Qin > > On 3/30/15, 10:44 PM, "Madhukar Bharti" wrote: > > >Hi All, > > > >I am using *async *producer to send the data. When I checked the log it is >

Re: delete.retention.ms in 0.8.1

Hi Gaurav, What is your "log.retention.check.interval.ms" ? There might be a chance it will be high so it is not able to delete in specified interval. And also in Kafka 0.8.1 it will be "retention.ms". Please check this Regards, Madh

Issue with high level consumer in 8.1.1 after restart

Hi All, We are facing issue with Kafka high Level consumers. We are using 0.8.1.1 version. Sometimes after restart consumers picks the messages sometimes it starts but not receives any messages. Is high level consumer is not reliable? I have checked with the log. Even if re-balance succeed, consu

Re: Issue with high level consumer in 8.1.1 after restart

: > It actually depends on how many consumers you have in the same group and > how many partitions the particular topic has. > > Can you elaborate on your configuration? > > Thanks, > > Mayuresh > > On Mon, Apr 6, 2015 at 3:35 AM, Madhukar Bharti > wrote: > >

Re: Producer does not recognize new brokers

Hi, Simply adding the brokers to the cluster will not reassign or redistribute topic partitions to newly added brokers. As it is also mentioned in documentation. "*However these new servers will not automatically be assigned any data partitions, so unless partitions are moved to them they won't b

Re: Consumer offsets in offsets topic 0.8.2

Hi Vamsi, You can also see the example here if you want to use Java API to get the offset from topic. Regards, Madhukar On Mon, Apr 13, 2015 at 10:25 PM, 4mayank <4may...@gmail.com> wr

Why fetching meta-data for topic is done three times?

Hi All, I came across a problem, If we use broker IP which is not reachable or out of network. Then it takes more than configured time(request.timeout.ms). After checking the log got to know that it is trying to fetch topic meta-data three times by changing correlation id. Due to this even though

Re: Issue with high level consumer in 8.1.1 after restart

Hi All, Any update on this? We are facing same issue all the time when ever we re-start consumers. On Mon, Apr 6, 2015 at 11:19 PM, Madhukar Bharti wrote: > Hi Mayuresh, > > We are having only one consumer in the group and only one partition for > that topic. > We have set aut

Re: Why fetching meta-data for topic is done three times?

response time and if all threads will be blocked due to producer send then whole application will be blocked for 3 sec. So we want to reduce this time to 1 sec. in overall to throw Exception. What is the possible way to do this? Thanks Madhukar On Thu, Apr 16, 2015 at 8:10 PM, Madhukar Bharti

Re: Why fetching meta-data for topic is done three times?

these set to defaults? > > message.send.max.retries > retry.backoff.ms > topic.metadata.refresh.interval.ms > > Thanks > Zakee > > > > > On Apr 23, 2015, at 11:48 PM, Madhukar Bharti > wrote: > > > > Hi All, > > > > Once gone through code f

Re: Why fetching meta-data for topic is done three times?

Hi Zakee, >message.send.max.retries is 1 Regards, Madhukar On Tue, Apr 28, 2015 at 6:17 PM, Madhukar Bharti wrote: > Hi Zakee, > > Thanks for your reply. > > >message.send.max.retries > 3 > > >retry.backoff.ms > 100 > > >topic.metadata.refresh.int

Re: Why fetching meta-data for topic is done three times?

wrote: > With reties 1 you still see the 3 secs delay? The idea is, you can change > these property to reduce the time to throw exception to 1 secs or below. > Does that help? > > Thanks > Zakee > > > > > On Apr 28, 2015, at 10:29 PM, Madhukar

Doubt regarding new Producer and old Producer API

Hi all, What are the possible use cases for using new producer API? - Is this only provides async call with callback feature? - Is partitioner class has been removed from new Producer API? if not then how to implement it if I want to use only client APIs? Regards, Madhukar

New Producer keep on trying to connect even after metadata timeout occured

Hi, I am testing Kafka-0.8.2.1 new producer API. For synchronous sending, I am calling future.get() just after producer send. I killed my broker and started Produce, noticed that it is throwing ExecutionException but after that It is still trying to re-connect to broker and this is keep on going

Re: Error handling in New AsyncProducer

gt; only triggered when the cluster is reachable . > > --SK > -- Thanks and Regards, Madhukar Bharti Mob: 7845755539

Re: Error handling in New AsyncProducer

as of right now you need to manually set the min.isr's with > alter > > topic due to KAFKA-2114 > > Either way, you will eventually get an error in the producer thread that > > isn't passed to the callback. > > > > And I think that's the point. > >

Doubts regarding KafkaProducer implemetation

't see any exception like *failed to produce after 3 retries*. Regards, Madhukar Bharti

Re: Kafka 0.11 broker running out of file descriptors

g normal operation is about 8k, so there is a lot of > headroom. > > I'm not sure if it's the other brokers trying to replicate that kills > it, or whether it's clients trying to publish messages. > > Has anyone seen a behavior like this? I'd appreciate any pointers. > > Thanks, > > Lukas > -- Thanks and Regards, Madhukar Bharti Mob: 7845755539

How to Downgrade Kafka from 0.11.0.1 to 0.10.0.1

by one. Re-install with Kafka 0.10.0.1. But it dint worked properly for all partitions. Regards, Madhukar -- Thanks and Regards, Madhukar Bharti Mob: 7845755539

Re: How to Downgrade Kafka from 0.11.0.1 to 0.10.0.1

ues you saw when you > downgraded (e.g. error logs, behavior, etc) it might help folks help you. > At the moment, I wouldn't know where to begin with the issue you've > described as there's not a ton of detail about what, exactly, went wrong. > > On Sun, Nov 5, 2017 at

Producer TimeoutException while accessing with domain and L3 Load balancer.

Hi, We have Kafka cluster with three brokers(0.10.0.1). We are accessing this cluster using domain name say common.test. Also, L3 load balancer has been configured for this domain name, so that request will be passed to brokers in RR way. The Producer has been implemented in java client(KafkaProd

0.8.2.1 Client not able to connect with Kafka 0.10.0.1 cluster even the cluster has message format version 0.8.2.

ol changes has been made in Kafka-0.10.X.X but don't want to update our client to 0.10.0.1 as of now. Is there any way without updating client the same code will still give the consumer offset. -- Thanks and Regards, Madhukar Bharti Mob: 7845755539

Re: 0.8.2.1 Client not able to connect with Kafka 0.10.0.1 cluster even the cluster has message format version 0.8.2.

> OffsetFetchRequest should have api key = 9, but for some reason your broker > is receiving a request with api key = 10 (GroupCoordinatorRequest). Can you > provide the stacktrace from the client as well? > > Ismael > > On Tue, Nov 1, 2016 at 12:13 PM, Madhukar Bharti > &

Re: 0.8.2.1 Client not able to connect with Kafka 0.10.0.1 cluster even the cluster has message format version 0.8.2.

Hi, After checking we found that there was an issue with version id passed in ConsumerMetadataRequest after setting it to 0(ConsumerMetadataRequest.currentVersion()). It started working! Thanks! Madhukar On Tue, Nov 1, 2016 at 10:29 PM, Madhukar Bharti wrote: > Hi Ismael, > > Bel

What can be reason for fetcher thread for slow response.

Hi, We are having 3 brokers in a cluster. Producer request is getting failed for broker 2. We are frequently getting below exception: 15/09/09 22:09:06 WARN async.DefaultEventHandler: Failed to send producer request with* correlation id 1455 to broker 2* with data for partitions [UserEvents,0] >

Re: What can be reason for fetcher thread for slow response.

used? Which Producer API is being used > > (http://kafka.apache.org/documentation.html#producerapi)? And what are > > the configs for this producer? > > > > Also, because I know little about tomcat, is there a semantic for a > > singleton, or a server singleton? > > -Erik >

Kafka Async producer throwing Exception before configured timeout.

Hi , We are getting below exception frequently even, we have configured timeout as 1ms. Does anyone also has faced this issue. org.apache.kafka.common.errors.TimeoutException: Failed to update metadata > after 28 ms We are using Kafka-0.8.2.1 and producer configurations are: > message.se

Zookeeper offset commit/Consumer Re-balance issues

Hi, Recently we started facing some issue in our stage/production setup, where we are running 3 brokers in a cluster and ~800 partitions with 130 Topics. Most of the topics has High level consumers. In total more than 200 Consumers groups are running and listening for the topics. 1. We stared ge

__consumer_offsets Topic files are not getting deleted after log retention hrs

Dear Team, We are using Kafka-0.8.2.1 and having log.retention.hours=168 but files of __consumer_offsets are not getting deleted, due to this lots of disc spaces are used. Please help how to delete file of offset storage topic after specified time. Thanks and Regards, Madhukar

Re: __consumer_offsets Topic files are not getting deleted after log retention hrs

heck > what > > are the values? > > > > Note that only old log segments will be considered for deletion so if > your > > __consumer_offsets topic only have one segment file, it will not be > > deleted. > > > > Guozhang > > > > > > On

Data loss in Kafka.

Hi All, We have seen data loss in Kafka, once we restarted the cluster. We have 3 Brokers in a cluster. In order to prevent data loss we have configured *min.insync.replicas=2* for all topics. Initially, when all brokers(1,2,3) are live we produced few messages(say 50 messages). Now we killed br

Re: Kafka broker crash

Hi Rahul, Whether the path is "/tmp/kafka-logs/" or "/temp/kafka-logs" ? Mostly if path is set to "/tmp/" then in case machine restart it may delete the files. So it is throwing FileNotFoundException. you can change the file location to some other path and restart all broker. This might fix the

Re: Data loss in Kafka.

alse if you'd like to try and avoid data loss in this scenario. By > default it is set to true. > > Reference: http://kafka.apache.org/documentation.html#design_uncleanleader > > Regards, > > On Wed, Jun 22, 2016 at 10:49 AM, Madhukar Bharti < > bhartimadhu...@gm