Hi Sunil, Looks like Metricbeats has a Jolokia module that will capture JMX exposed metrics for you: https://www.elastic.co/blog/brewing-in-beats-add-support-for-jolokia-lmx
Kind regards, Liam Clarke On Fri, Feb 21, 2020 at 6:16 PM Sunil CHAUDHARI <sunilchaudh...@dbs.com.invalid> wrote: > Hi Liam Clarke, > Thanks for this elaboration. > Surely I will google. > One more question about Mbeans. If I am capturing system matrix on kafka > broker using metricbeat, then is it possible that I will get those Mbeans? > I know this is out of the topic, but in general, if I capture JVM metric > with any of the monitoring tools, then will I get it? > > > > -----Original Message----- > From: Liam Clarke <liam.cla...@adscale.co.nz> > Sent: Friday, February 21, 2020 1:03 AM > To: users@kafka.apache.org > Subject: Re: [External] Re: Urgent helep please! How to measure producer > and consumer throughput for single partition? > > This mail originated from an external party outside DBS - > users-return-39570-sunilchaudhari=dbs....@kafka.apache.org. Do not click > on links or open attachments unless you recognize the sender and know the > content is safe. > > The metrics are exposed in the JVM the producer is running within as > Mbeans. The long string I gave you is the relevant MBean object name. You > can connect to the JVM using JConsole to view the MBeans. There are also > multiple libraries that will scrape a JVM via JMX to extract values from > MBeans. > > If you're not familiar with JMX or JConsole, there's plenty of great > documentation on the Internet, have a Google :) > > On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI, > <sunilchaudh...@dbs.com.invalid> > wrote: > > > Hi Liam Clarke, > > Sorry but this is bit unclear for me. > > Can you please elaborate your answer? I am beginner to Kafka. > > " Producers emit metrics via JMX ": > > - How to enable this? I have kafka-Manager. Can I make use of > > kafka-manager? How? > > > “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate” > > please help to explain this. > > > > Regards, > > Sunil. > > > > -----Original Message----- > > From: Liam Clarke <liam.cla...@adscale.co.nz> > > Sent: Thursday, February 20, 2020 11:16 AM > > To: users@kafka.apache.org > > Subject: [External] Re: Urgent helep please! How to measure producer > > and consumer throughput for single partition? > > > > This mail originated from an external party outside DBS - mailto: > > users-return-39554-sunilchaudhari=dbs....@kafka.apache.org. Do not > > click on links or open attachments unless you recognize the sender and > > know the content is safe. > > > > Hi Sunil, > > > > Producers emit metrics via JMX that will help you, assuming that your > > producers are using a round robin partition assignment strategy, you > > could divide this metric by your number of partitions, > > > > > > kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-s > > end-rate > > > > Kind regards, > > > > Liam Clarke > > > > On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto: > > sunilchaudh...@dbs.com.invalid> wrote: > > > > > Hi > > > I was referring to the article by Mr. June Rao about partitions in > > > kafka cluster. > > > https://www.confluent.io/blog/how-choose-number-topics-partitions-ka > > > fk > > > a-cluster/ > > > > > > "A rough formula for picking the number of partitions is based on > > > throughput. You measure the throughout that you can achieve on a > > > single partition for production (call it p) and consumption (call it > > > c). Let's say your target throughput is t. Then you need to have at > > > least max(t/p, t/c) partitions." > > > > > > I have the data pipeline as below. > > > > > > Filebeat-->Kafka-->Logstash-->Elasticsearch > > > There are many filebeat agents sending data to kafka. I want to > > > understand , how can I measure the events per seconds getting > > > written to Kafka? This will help me to know 'p' in above formula. > > > I can measure the consumer throughput by monitoring logsatsh > > > pipelines on Kibana. So it will give me 'c' in above formula. > > > > > > I know target throughput in my cluster, that is 't'. 30k events/s. > > > > > > Please let me know if I am going wrong? > > > > > > Regards, > > > Sunil. > > > CONFIDENTIAL NOTE: > > > The information contained in this email is intended only for the use > > > of the individual or entity named above and may contain information > > > that is privileged, confidential and exempt from disclosure under > > applicable law. > > > If the reader of this message is not the intended recipient, you are > > > hereby notified that any dissemination, distribution or copying of > > > this communication is strictly prohibited. If you have received this > > > message in error, please immediately notify the sender and delete > > > the > > mail. Thank you. > > > > > > > > > CONFIDENTIAL NOTE: > > The information contained in this email is intended only for the use > > of the individual or entity named above and may contain information > > that is privileged, confidential and exempt from disclosure under > applicable law. > > If the reader of this message is not the intended recipient, you are > > hereby notified that any dissemination, distribution or copying of > > this communication is strictly prohibited. If you have received this > > message in error, please immediately notify the sender and delete the > mail. Thank you. > > > > CONFIDENTIAL NOTE: > The information contained in this email is intended only for the use of > the individual or entity named above and may contain information that is > privileged, confidential and exempt from disclosure under applicable law. > If the reader of this message is not the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you have received this message in > error, please immediately notify the sender and delete the mail. Thank you. >
