So to make JMX accurate, we need to tweak the frequency of commits? What setting would that be?
On Mon, Nov 16, 2015 at 8:40 AM, Scott Reynolds <sreyno...@twilio.com> wrote: > On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama < > osama.abu-ob...@morganstanley.com> wrote: > > > I can observe the same thing: > > > > - Lag values read through the Kafka consumer JMX is 0 > > > This metric includes *uncommitted* offsets > > - Lag values read through kafka-run-class.sh > > kafka.tools.ConsumerOffsetChecker is on average 200K-400K > > > * This is just the *committed* offsets > > > > When the Lag value in the Kafka consumer JMX is high (for example 5M), > > ConsumerOffsetChecker shows a matching number. > > > > I am running kafka_2.10-0.8.2.1 > > > > Osama > > > > -----Original Message----- > > From: allen chan [mailto:allen.michael.c...@gmail.com] > > Sent: Saturday, November 14, 2015 8:34 PM > > To: users@kafka.apache.org > > Subject: Re: consumer offset tool and JMX metrics do not match > > > > I believe producers / brokers / and consumers has been restarted at > > different times. > > What do you think the issue is? > > > > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <prabhbha...@gmail.com> > > wrote: > > > > > Hi, > > > > > > Did any of your brokers/consumers undergo a restart between the time > > > after you had started the consumption and until you see this issue ? > > > > > > Thanks, > > > Prabhjot > > > > > > > > > > > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan > > > <allen.michael.c...@gmail.com> > > > wrote: > > > > > > > I also looked at this metric in JMX and it is also 0 > > > > > > > *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst > > > ash* > > > > > > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan < > > > allen.michael.c...@gmail.com> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker > > > > > vs JMX > > > > > > > > > > > > (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas > > > h,topic=logstash_fdm,partition=*) > > > > > and they do not match. > > > > > > > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX > > > > > shows 0 > > > > for > > > > > all partitions. > > > > > > > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to > JMX. > > > > > > > > > > Has anyone seen this issue before? I am trying to use JMX to pull > > > > > stats into monitoring system > > > > > > > > > > > > > > > -- > > > > > Allen Michael Chan > > > > > > > > > > > > > > > > > > > > > -- > > > > Allen Michael Chan > > > > > > > > > > > > > > > > -- > > > --------------------------------------------------------- > > > "There are only 10 types of people in the world: Those who understand > > > binary, and those who don't" > > > > > > > > > > > -- > > Allen Michael Chan > > > > > > > > > -------------------------------------------------------------------------------- > > > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the > > opinions or views contained herein are not intended to be, and do not > > constitute, advice within the meaning of Section 975 of the Dodd-Frank > Wall > > Street Reform and Consumer Protection Act. If you have received this > > communication in error, please destroy all electronic and paper copies; > do > > not disclose, use or act upon the information; and notify the sender > > immediately. Mistransmission is not intended to waive confidentiality or > > privilege. Morgan Stanley reserves the right, to the extent permitted > under > > applicable law, to monitor electronic communications. This message is > > subject to terms available at the following link: > > http://www.morganstanley.com/disclaimers. If you cannot access these > > links, please notify us by reply message and we will send the contents to > > you. By messaging with Morgan Stanley you consent to the foregoing. > -- Allen Michael Chan
