Re: kafka metric to monitor for consumer FETCH using disk caching and not going to disk

2021-05-16 Thread Alexandre Dupriez
Not that I know of - but others may advise otherwise. The change from KIP-551 is fairly self-contained and can be backported well though. Thanks, Alexandre Le dim. 16 mai 2021 à 14:51, Pushkar Deole a écrit : > > thanks Alexandre... currently we are using kafka 2.5.0, so is there any > metric th

Re: kafka metric to monitor for consumer FETCH using disk caching and not going to disk

2021-05-16 Thread Pushkar Deole
thanks Alexandre... currently we are using kafka 2.5.0, so is there any metric that can be used from 2.5.0? On Sun, May 16, 2021 at 6:02 PM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Hi Pushkar, > > If you are using Linux and Kafka 2.6.0+, the closest metric to what > you are looki

Re: kafka metric to monitor for consumer FETCH using disk caching and not going to disk

2021-05-16 Thread Alexandre Dupriez
Hi Pushkar, If you are using Linux and Kafka 2.6.0+, the closest metric to what you are looking for is TotalDiskReadBytes [1], which measures data transfer at the block layer. Assuming your consumers are doing tail reads and there is no other activity which requires loading pages from the disk on

Re: Last Stable Offset (LSO) stuck for specific topic partition after Broker issues

2021-05-16 Thread Alexandre Dupriez
Hi Pieter, FWIW, you may have encountered the following bug: https://issues.apache.org/jira/browse/KAFKA-12671 . Thanks, Alexandre Le ven. 12 juin 2020 à 00:43, D C a écrit : > > Hey peeps, > > Anyone else encountered this and got to the bottom of it? > > I'm facing a similar issue, having LSO

Re: Kafka Definitive guide v2 states auto.leader.rebalance.enable = true is not recommended

2021-05-16 Thread Alexandre Dupriez
Hi Liam, The property you referred to corresponds to partition leadership, not ownership from consumers. See https://issues.apache.org/jira/browse/KAFKA-4084 for a discussion about why post-incident leader rebalance can sometimes impact foreground traffic. Thanks, Alexandre Le lun. 12 avr. 2021

Re: Unable to run kafka on ec2 free tier instance

2021-05-16 Thread Alexandre Dupriez
Hi Satendra, The JVM core indicates you are running out of system memory. Increasing the heap size of your JVM will not help; if anything, it will make things worse. You need to check which processes occupy system memory (look for their resident set size) and work on reducing memory consumption ac