What is the best way to detect consumer lag? We are running each consumer as a separate group and I am running the ConsumerOffsetChecker to assess the partitions and the lag for each group/consumer. I run this every 5 minutes. In some cases I run this command up to 75 times on each 5 min polling cycle (once for each group/consuer). An example of the command is (bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group consumer-group1 --zkconnect zkhost:zkport)
The problem I am running into is CPU usage on the broker when these commands run. We have a dedicated broker that has no leader partitions, but the high CPU still concerns me. Is there a better way to detect consumer lag? Preferably one that is less impactful? Gene Robichaux Manager, Database Operations Match.com 8300 Douglas Avenue I Suite 800 I Dallas, TX 75225
