Hi, I'm trying to debug why one particular partition has a huge lag for our consumers, but while debugging I ran the following 2 commands which seem to contradict each other. I snipped the output for brevity. I'm focused on partition 80. I'm also running Kafka 0.8.2.2 with the high level consumer in the java process connected to it.
$ kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group mycgroup --zookeeper $zoo --topic maxwell Group Topic Pid Offset logSize Lag Owner ... mycgroup maxwell 75 0 0 0 mycgroup_myhost-1464168755165-8c0adb35-0 mycgroup maxwell 76 196963610 196963611 1 mycgroup_myhost-1464168755165-8c0adb35-0 mycgroup maxwell 77 1326987677 1326987950 273 mycgroup_myhost-1464168755165-8c0adb35-0 mycgroup maxwell 78 255727662 255727843 181 mycgroup_myhost-1464168755165-8c0adb35-0 mycgroup maxwell 79 0 0 0 mycgroup_myhost-1464168755165-8c0adb35-0 mycgroup maxwell 80 20450706001 20700099163 249393162 mycgroup_myhost-1464168755399-66315e10-0 mycgroup maxwell 81 940659152 940667598 8446 mycgroup_myhost-1464168755399-66315e10-0 mycgroup maxwell 82 607085553 607087055 1502 mycgroup_myhost-1464168755399-66315e10-0 mycgroup maxwell 83 0 0 0 mycgroup_myhost-1464168755399-66315e10-0 mycgroup maxwell 84 61361682 61361682 0 mycgroup_myhost-1464168755399-66315e10-0 mycgroup maxwell 85 343862579 343866453 3874 mycgroup_myhost-1464168755399-66315e10-0 ... $ kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group mycgroup --zookeeper.connect $zoo ... [2016-05-25 13:03:19,851] ERROR No owner for partition [maxwell,75] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,853] ERROR Owner mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,75] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,853] ERROR No owner for partition [maxwell,76] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,855] ERROR Owner mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,76] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,855] ERROR No owner for partition [maxwell,77] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,857] ERROR Owner mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,77] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,857] ERROR No owner for partition [maxwell,78] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,860] ERROR Owner mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,78] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,860] ERROR No owner for partition [maxwell,79] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,861] ERROR Owner mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,79] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,862] ERROR No owner for partition [maxwell,80] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,864] ERROR Owner mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,80] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,864] ERROR No owner for partition [maxwell,81] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,866] ERROR Owner mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,81] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,866] ERROR No owner for partition [maxwell,82] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,868] ERROR Owner mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,82] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,868] ERROR No owner for partition [maxwell,83] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,870] ERROR Owner mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,83] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,870] ERROR No owner for partition [maxwell,84] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,871] ERROR Owner mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,84] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,871] ERROR No owner for partition [maxwell,85] (kafka.tools.VerifyConsumerRebalance$) [2016-05-25 13:03:19,873] ERROR Owner mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,85] is not a valid member of consumer group mycgroup (kafka.tools.VerifyConsumerRebalance$) ... Thanks, Shane.