Hi, I'm trying to debug why one particular partition has a huge lag for our
consumers, but while debugging I ran the following 2 commands which seem to
contradict each other. I snipped the output for brevity. I'm focused on
partition 80. I'm also running Kafka 0.8.2.2 with the high level consumer
in the java process connected to it.
$ kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group
mycgroup --zookeeper $zoo --topic maxwell
Group Topic Pid Offset
logSize Lag Owner
...
mycgroup maxwell 75 0 0
0 mycgroup_myhost-1464168755165-8c0adb35-0
mycgroup maxwell 76 196963610
196963611 1
mycgroup_myhost-1464168755165-8c0adb35-0
mycgroup maxwell 77 1326987677
1326987950 273
mycgroup_myhost-1464168755165-8c0adb35-0
mycgroup maxwell 78 255727662
255727843 181
mycgroup_myhost-1464168755165-8c0adb35-0
mycgroup maxwell 79 0 0
0 mycgroup_myhost-1464168755165-8c0adb35-0
mycgroup maxwell 80 20450706001
20700099163 249393162
mycgroup_myhost-1464168755399-66315e10-0
mycgroup maxwell 81 940659152
940667598 8446
mycgroup_myhost-1464168755399-66315e10-0
mycgroup maxwell 82 607085553
607087055 1502
mycgroup_myhost-1464168755399-66315e10-0
mycgroup maxwell 83 0 0
0 mycgroup_myhost-1464168755399-66315e10-0
mycgroup maxwell 84 61361682
61361682 0
mycgroup_myhost-1464168755399-66315e10-0
mycgroup maxwell 85 343862579
343866453 3874
mycgroup_myhost-1464168755399-66315e10-0
...
$ kafka-run-class.sh kafka.tools.VerifyConsumerRebalance --group
mycgroup --zookeeper.connect $zoo
...
[2016-05-25 13:03:19,851] ERROR No owner for partition [maxwell,75]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,853] ERROR Owner
mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,75] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,853] ERROR No owner for partition [maxwell,76]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,855] ERROR Owner
mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,76] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,855] ERROR No owner for partition [maxwell,77]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,857] ERROR Owner
mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,77] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,857] ERROR No owner for partition [maxwell,78]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,860] ERROR Owner
mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,78] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,860] ERROR No owner for partition [maxwell,79]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,861] ERROR Owner
mycgroup_myhost-1464168755165-8c0adb35-0 for partition [maxwell,79] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,862] ERROR No owner for partition [maxwell,80]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,864] ERROR Owner
mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,80] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,864] ERROR No owner for partition [maxwell,81]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,866] ERROR Owner
mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,81] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,866] ERROR No owner for partition [maxwell,82]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,868] ERROR Owner
mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,82] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,868] ERROR No owner for partition [maxwell,83]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,870] ERROR Owner
mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,83] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,870] ERROR No owner for partition [maxwell,84]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,871] ERROR Owner
mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,84] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,871] ERROR No owner for partition [maxwell,85]
(kafka.tools.VerifyConsumerRebalance$)
[2016-05-25 13:03:19,873] ERROR Owner
mycgroup_myhost-1464168755399-66315e10-0 for partition [maxwell,85] is
not a valid member of consumer group mycgroup
(kafka.tools.VerifyConsumerRebalance$)
...
Thanks,
Shane.
