Re: Java API to read metrics via JMX

I don’t know of Java-based solution but I have successfully used: Kafka JMX Exporter for Prometheus <———> Kafka to collect JMX metrics from Kafka. > On 10 Aug 2018, at 11:10 am, Raghav wrote: > > Hi > > I found > https://github.com/kafka-dev/kafka/blob/master/perf/src/main/java/kafka/perf/jmx

Re: Java API to read metrics via JMX

Hi I found https://github.com/kafka-dev/kafka/blob/master/perf/src/main/java/kafka/perf/jmx/BrokerJmxClient.java code written by Neha to pull JMX metrics via MBean. In here: https://github.com/kafka-dev/kafka/blob/master/perf/src/main/java/kafka/perf/jmx/BrokerJmxClient.java#L37 there is a mentio

Re: Looking for help with a question on the consumer API

Actually there is very minimal work done by the thread doing the polling. However it is possible that the thread doing the polling may not get schedule due to other things happening on the system - like a long GC pause. From what you all are saying it sounds like a poll will do a heartbeat which

Re: Kafka Streams - Merge vs. Join

Hi John, Sorry for the confusion! I just noticed that we failed to document the merge operator. I've created https://issues.apache.org/jira/browse/KAFKA-7269 to fix it. But in the mean time, * merge: interleave the records from two streams to produce one collated stream * join: compute a new stre

Kafka Streams - Merge vs. Join

Hi All, I am a little confused on the difference between the KStreamBuilder merge() function and doing a KStream-to-KStream Join operation. I understand the difference between Inner, Left and Outer joins, but I don't understand exactly what the difference is between the two. It

Kafka Streams - Merge vs. Join

Hi All, I am a little confused on the difference between the KStreamBuilder merge() function and doing a KStream-to-KStream Join operation. I understand the difference between Inner, Left and Outer joins, but I don't understand exactly what the difference is between the two. It ap

Re: Looking for help with a question on the consumer API

Hi, Yes , if you don'nt call poll within the configured timeouts, the broker thinks the consumer is gone. But increasing the timeout is not a sustainable design. In general the code in the consumer poll loop should be very fast and do minimal work. Any heavy duty work should be done by handing

RE: [External] Re: Java API to read metrics via JMX

We use Jolokia (which has a java agent you can load with kafka to expose metrics via HTTP) and Influx/Telegraf which has support for Jolokia. There is a fair bit of configuration but it can be done without any coding. -Dave -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com]

Re: Looking for help with a question on the consumer API

Consumer gets kicked out if it fails to send heart beat in designated time period. Every call to poll sends one heart beat to consumer group coordinator. You need to look at *how much time is it taking to process your single record*. *Maybe it is exceeding session.timeout.ms

Re: Very long consumer rebalances

Hi, Yes my consumer application works like below 1. Reads how many workers are required to process each topics from properties file 2. As many threads are spawned as there are workers mentioned in properties file, topic name is passed to this thread. FixedThreadPool implementation

Re: Very long consumer rebalances

In v0.10.0.1, consumer heartbeat background thread feature is not available. Lot of users faced similar errors. So, KIP-62 is proposed. You have to update your Kafka version

(tentative) release date for next 8.5.x

Hello, Would it be possible to get a tentative timeline for next 8.5.x release? Or is it confidential. We wanted to move to 8.5.32 but saw a latest bugfix for URLEncoder reported about a week back, so wondering if it's worth the wait. Regards,

Re: Very long consumer rebalances

In the simplest way, how have you implemented your consumer? 1) Does your consumers join a designated group, process messages, and then closes all connection? Or does it stay open perpetually until server shutdown? 2) Have you configured the session timeouts for client and zookeeper accordingly?

Re: Very long consumer rebalances

I am facing too many problems these days. Now one of our consumer groups is rebalancing every now and then. And rebalance takes very low, more than 5-10 minutes. Even after re-balancing I see that only half of the consumers are active/receive assignment. Its all going haywire. I am seeing these l