We have a case where one of our brokers running Kafka 0.10.1.1 is telling
another one to shut down. There is nothing obvious (to me) in the logs of
either source or destination of this message to explain why it would happen.
Does anybody see anything in the logs (below) that I have missed, or know why
it might have happened? The important log line seems to be "[2017-05-20
00:15:11,282] INFO [Controller 3]: Shutting down broker 2".
Thanks for taking a look.
Neil
Logs from source of message:
[2017-05-20 00:14:26,980] TRACE [Controller 3]: leader imbalance ratio for
broker 2 is 0.000000 (kafka.controller.KafkaController)
[2017-05-20 00:14:26,980] DEBUG [Controller 3]: topics not in preferred replica
Map() (kafka.controller.KafkaController)
[2017-05-20 00:14:26,980] TRACE [Controller 3]: leader imbalance ratio for
broker 1 is 0.000000 (kafka.controller.KafkaController)
[2017-05-20 00:14:26,980] DEBUG [Controller 3]: topics not in preferred replica
Map() (kafka.controller.KafkaController)
[2017-05-20 00:14:26,980] TRACE [Controller 3]: leader imbalance ratio for
broker 3 is 0.000000 (kafka.controller.KafkaController)
[2017-05-20 00:15:11,282] INFO [Controller 3]: Shutting down broker 2
(kafka.controller.KafkaController)
[2017-05-20 00:15:11,283] DEBUG [Controller 3]: All shutting down brokers: 2
(kafka.controller.KafkaController)
[2017-05-20 00:15:11,283] DEBUG [Controller 3]: Live brokers: 1,3
(kafka.controller.KafkaController)
[2017-05-20 00:15:11,290] DEBUG The stop replica request (delete = true) sent
to broker 2 is (kafka.controller.ControllerBrokerRequestBatch)
[2017-05-20 00:15:11,291] DEBUG The stop replica request (delete = false) sent
to broker 2 is
[Topic=56bd99ee-fe4c-4928-9404-4661ee6658d0,Partition=0,Replica=2]
(kafka.controller.ControllerBrokerRequestBatch)
[2017-05-20 00:15:11,291] INFO [Replica state machine on controller 3]:
Invoking state change to OfflineReplica for replicas
[Topic=56bd99ee-fe4c-4928-9404-4661ee6658d0,Partition=0,Replica=2]
(kafka.controller.ReplicaStateMachine)
[2017-05-20 00:15:11,291] DEBUG [Controller 3]: Removing replica 2 from ISR 1,2
for partition [56bd99ee-fe4c-4928-9404-4661ee6658d0,0].
(kafka.controller.KafkaController)
[2017-05-20 00:15:11,292] INFO [Controller-3-to-broker-2-send-thread],
Controller 3 connected to 10.2.91.3:9092 (id: 2 rack: null) for sending state
change requests (kafka.controller.RequestSendThread)
[2017-05-20 00:15:11,295] INFO [Controller 3]: New leader and ISR for partition
[56bd99ee-fe4c-4928-9404-4661ee6658d0,0] is
{"leader":1,"leader_epoch":4,"isr":[1]} (kafka.controller.KafkaController)
[2017-05-20 00:15:11,296] DEBUG The stop replica request (delete = true) sent
to broker 2 is (kafka.controller.ControllerBrokerRequestBatch)
Logs from destination of message:
[2017-05-20 00:05:15,111] INFO [Group Metadata Manager on Broker 2]: Group
SubscriptionChangeEventListenerSupport1791696017 transitioned to Dead in
generation 0 (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:05:15,113] INFO [Group Metadata Manager on Broker 2]: Group
SubscriptionChangeEventListenerSupport1613331553 transitioned to Dead in
generation 0 (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:05:15,116] INFO [Group Metadata Manager on Broker 2]: Group
TopicChangeEventListenerSupport783132489 transitioned to Dead in generation 0
(kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:05:15,117] INFO [Group Metadata Manager on Broker 2]: Removed 4
expired offsets in 9 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:06:39,628] INFO [Group Metadata Manager on Broker 2]: Removed 0
expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:10:52,818] INFO [Group Metadata Manager on Broker 2]: Removed 0
expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:11:12,972] INFO Rolled new log segment for
'__consumer_offsets-15' in 1 ms. (kafka.log.Log)
[2017-05-20 00:12:21,053] INFO Deleting segment 0 from log
__consumer_offsets-15. (kafka.log.Log)
[2017-05-20 00:12:21,053] INFO Deleting segment 1362179 from log
__consumer_offsets-15. (kafka.log.Log)
[2017-05-20 00:12:21,054] INFO Deleting index
/kafka/kafka-logs/__consumer_offsets-15/00000000000000000000.index.deleted
(kafka.log.OffsetIndex)
[2017-05-20 00:12:21,054] INFO Deleting index
/kafka/kafka-logs/__consumer_offsets-15/00000000000000000000.timeindex.deleted
(kafka.log.TimeIndex)
[2017-05-20 00:12:21,062] INFO Deleting index
/kafka/kafka-logs/__consumer_offsets-15/00000000000001362179.index.deleted
(kafka.log.OffsetIndex)
[2017-05-20 00:12:21,062] INFO Deleting index
/kafka/kafka-logs/__consumer_offsets-15/00000000000001362179.timeindex.deleted
(kafka.log.TimeIndex)
[2017-05-20 00:15:11,245] INFO [Kafka Server 2], shutting down
(kafka.server.KafkaServer)
[2017-05-20 00:15:11,253] INFO [Kafka Server 2], Starting controlled shutdown
(kafka.server.KafkaServer)
[2017-05-20 00:15:15,108] INFO [Group Metadata Manager on Broker 2]: Group
TopicChangeEventListenerSupport118711122 transitioned to Dead in generation 0
(kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,109] INFO [Group Metadata Manager on Broker 2]: Group
SubscriptionChangeEventListenerSupport331556082 transitioned to Dead in
generation 0 (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,109] INFO [Group Metadata Manager on Broker 2]: Group
SJbpcclRdKX24ufaZJKVq2zGG5o transitioned to Dead in generation 0
(kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,110] INFO [Group Metadata Manager on Broker 2]: Group
SubscriptionChangeEventListenerSupport1449809648 transitioned to Dead in
generation 0 (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,112] INFO [Group Metadata Manager on Broker 2]: Group
TopicChangeEventListenerSupport1382661554 transitioned to Dead in generation 0
(kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,113] INFO [Group Metadata Manager on Broker 2]: Group
HooOsvTltcBTv0bRtQBV8VoPjMb transitioned to Dead in generation 0
(kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,114] INFO [Group Metadata Manager on Broker 2]: Group
TopicChangeEventListenerSupport146271880 transitioned to Dead in generation 0
(kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,114] INFO [Group Metadata Manager on Broker 2]: Group
ea832581-b029-4702-b14b-b3abfd1c108c transitioned to Dead in generation 0
(kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,115] INFO [Group Metadata Manager on Broker 2]: Group
26oOEEuiPtfvKb7yubawEEaIUrf transitioned to Dead in generation 0
(kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:15:15,116] INFO [Group Metadata Manager on Broker 2]: Removed 9
expired offsets in 8 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:16:39,628] INFO [Group Metadata Manager on Broker 2]: Removed 0
expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-05-20 00:24:12,128] INFO Rolled new log segment for
'__consumer_offsets-8' in 1 ms. (kafka.log.Log)
[2017-05-20 00:25:15,117] INFO [Group Metadata Manager on Broker 2]: Removed 0
expired offsets in 9 milliseconds. (kafka.coordinator.GroupMetadataManager)
