On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama < osama.abu-ob...@morganstanley.com> wrote:
> I can observe the same thing: > > - Lag values read through the Kafka consumer JMX is 0 > This metric includes *uncommitted* offsets - Lag values read through kafka-run-class.sh > kafka.tools.ConsumerOffsetChecker is on average 200K-400K > * This is just the *committed* offsets > When the Lag value in the Kafka consumer JMX is high (for example 5M), > ConsumerOffsetChecker shows a matching number. > > I am running kafka_2.10-0.8.2.1 > > Osama > > -----Original Message----- > From: allen chan [mailto:allen.michael.c...@gmail.com] > Sent: Saturday, November 14, 2015 8:34 PM > To: users@kafka.apache.org > Subject: Re: consumer offset tool and JMX metrics do not match > > I believe producers / brokers / and consumers has been restarted at > different times. > What do you think the issue is? > > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <prabhbha...@gmail.com> > wrote: > > > Hi, > > > > Did any of your brokers/consumers undergo a restart between the time > > after you had started the consumption and until you see this issue ? > > > > Thanks, > > Prabhjot > > > > > > > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan > > <allen.michael.c...@gmail.com> > > wrote: > > > > > I also looked at this metric in JMX and it is also 0 > > > > > *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst > > ash* > > > > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan < > > allen.michael.c...@gmail.com> > > > wrote: > > > > > > > Hi All, > > > > > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker > > > > vs JMX > > > > > > > > > (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas > > h,topic=logstash_fdm,partition=*) > > > > and they do not match. > > > > > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX > > > > shows 0 > > > for > > > > all partitions. > > > > > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX. > > > > > > > > Has anyone seen this issue before? I am trying to use JMX to pull > > > > stats into monitoring system > > > > > > > > > > > > -- > > > > Allen Michael Chan > > > > > > > > > > > > > > > > -- > > > Allen Michael Chan > > > > > > > > > > > -- > > --------------------------------------------------------- > > "There are only 10 types of people in the world: Those who understand > > binary, and those who don't" > > > > > > -- > Allen Michael Chan > > > > -------------------------------------------------------------------------------- > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the > opinions or views contained herein are not intended to be, and do not > constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall > Street Reform and Consumer Protection Act. If you have received this > communication in error, please destroy all electronic and paper copies; do > not disclose, use or act upon the information; and notify the sender > immediately. Mistransmission is not intended to waive confidentiality or > privilege. Morgan Stanley reserves the right, to the extent permitted under > applicable law, to monitor electronic communications. This message is > subject to terms available at the following link: > http://www.morganstanley.com/disclaimers. If you cannot access these > links, please notify us by reply message and we will send the contents to > you. By messaging with Morgan Stanley you consent to the foregoing.
