Kafka support for IPV6 only Networking stack
Dear Team , We would like to check if Kafka support IPv6 only configuration . In this configuration customer will have only IPv6 Networking for the operating system where Kafka is going to be installed . Regards, Senjoorian
How to get async commit callbacks in the rebalance listener?
Hi, I am trying to do async commits from the rebalance listener, to be precise, in method onPartitionsRevoked. The idea is to wait until all commits for the current epoch are done before the rebalance barrier and by doing so prevent duplicate processing. It is not so hard to call commitAsync(offsets, callback), but what method should be used so that the Kafka client gets a chance to call the callback? I tried the following: *1. Call **commitAsync(Collections.emptyMap, callback)** * Unfortunately when you call commitAsync with an empty offsets map, it doesn't call the callbacks of previous commits. There is a PR from 2020 that would fix this issue: https://github.com/apache/kafka/pull/9111. This PR was closed without merging. Should this PR be reconsidered? *2. Pause all partitions and call **poll(0)* Doesn't work; you'll get a "KafkaConsumer is not safe for multi-threaded access" exception. *3. Call commitSync(**Collections.emptyMap, callback)** * Behaves the same as under point 1. *4. Repeated calls to **commitAsync(offsets, callback)** * This time we keep calling commitAsync with the same offsets until these offsets are committed. Unfortunately, this never ends. Either because commitAsync doesn't call the callbacks, or because this just stacks up more commits to complete. I looked at the other methods on the consumer API but I didn't find anything that looked suitable for this use case. So to repeat the question: What method should I invoke (from the onPartitionsRevoked callback), to make the Kafka client invoke the callback of earlier async commits? Some context: I am working on zio-kafka; a completely async library that provides a concurrent streaming layer on top of the Java client. Thanks, Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com
Consumer offset value -Apache kafka 3.2.3
Dear Kafka Experts How can we check for a particular offset number in Apache kafka 3.2.3 version.Could you please share some light. The kafka_console_consumer tool is throwing class not found error. ./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic your-topic --group your-consumer-group --zookeeper localhost:2181
Re: Consumer Lag Metrics/ Topic level metrics
Many thanks Samuel. Will go thru this. On Tue, Apr 25, 2023 at 9:03 PM Samuel Delepiere < samuel.delepi...@celer-tech.com> wrote: > Hi, > > I use a combination of the Prometheus JMX exporter ( > https://github.com/prometheus/jmx_exporter) and the Prometheus Kafka > exporter (https://github.com/danielqsj/kafka_exporter). > The consumer lag metrics come from the latter. > > I can then output the data in Grafana > > > Regards, > > Sam. > > > > On 25 Apr 2023, at 16:26, Kafka Life wrote: > > Dear Kafka Experts > > Could you please suggest good metrics exporter for consumer lag and topic > level metrics apart from Linkedin kafka burrow for the kafka broker > cluster. > > > > *This message, including any attachments, may include private, privileged > and confidential information and is intended only for the personal and > confidential use of the intended recipient(s). If the reader of this > message is not an intended recipient, you are hereby notified that any > review, use, dissemination, distribution, printing or copying of this > message or its contents is strictly prohibited and may be unlawful. If you > are not an intended recipient or have received this communication in error, > please immediately notify the sender by telephone and/or a reply email and > permanently delete the original message, including any attachments, without > making a copy.* >
