Hi team,
We are running confluent 0.9.0.1 on a cluster with 6 brokers. These days one of
our broker(broker 1) frequently shrink the ISR and expand it immediately every
about 20 minutes and I couldn’t find out why. Based on the log, I can kick out
any of other brokers, not just a specific one. Here’s one log sample:
[2017-02-19 13:01:01,741] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.ArrayIndexOutOfBoundsException
[2017-02-19 13:01:01,777] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.ArrayIndexOutOfBoundsException
[2017-02-19 13:01:01,781] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.ArrayIndexOutOfBoundsException
[2017-02-19 13:01:01,796] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.ArrayIndexOutOfBoundsException
[2017-02-19 13:01:17,230] INFO Partition [khloe-updates-prod-els,0] on broker
1: Shrinking ISR for partition [khloe-updates-prod-els,0] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,231] INFO Partition [jtop-migration-prod-els,37] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,37] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,232] INFO Partition [hummus-asset-v2-prod-els,0] on broker
1: Shrinking ISR for partition [hummus-asset-v2-prod-els,0] from 1,5,3 to 1,3
(kafka.cluster.Partition)
[2017-02-19 13:01:17,233] INFO Partition [jtop-migration-prod-els,57] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,57] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,234] INFO Partition [jtop-migration-smoke-els,6] on broker
1: Shrinking ISR for partition [jtop-migration-smoke-els,6] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,234] INFO Partition [jtop-migration-prod-els,12] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,12] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,235] INFO Partition [__consumer_offsets,39] on broker 1:
Shrinking ISR for partition [__consumer_offsets,39] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,236] INFO Partition [jtop-migration-smoke-els,51] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,51] from 1,5,6
to 1,6 (kafka.cluster.Partition)
[2017-02-19 13:01:17,237] INFO Partition [hummus-entity-merged-v2-prod-els,0]
on broker 1: Shrinking ISR for partition [hummus-entity-merged-v2-prod-els,0]
from 1,5,4 to 1,4 (kafka.cluster.Partition)
[2017-02-19 13:01:17,238] INFO Partition
[resolver-notification-profiles-staging-els,1] on broker 1: Shrinking ISR for
partition [resolver-notification-profiles-staging-els,1] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,239] INFO Partition [jtop-migration-prod-els,52] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,52] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,240] INFO Partition [compass-home-record-prod-els,0] on
broker 1: Shrinking ISR for partition [compass-home-record-prod-els,0] from
1,5,4 to 1,4 (kafka.cluster.Partition)
[2017-02-19 13:01:17,241] INFO Partition [jtop-migration-smoke-els,46] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,46] from 1,5,4
to 1,4 (kafka.cluster.Partition)
[2017-02-19 13:01:17,242] INFO Partition [stolmen-entity-prod-els,2] on broker
1: Shrinking ISR for partition [stolmen-entity-prod-els,2] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,243] INFO Partition [hummus-boundary-v2-prod-els,0] on
broker 1: Shrinking ISR for partition [hummus-boundary-v2-prod-els,0] from
1,5,3 to 1,3 (kafka.cluster.Partition)
[2017-02-19 13:01:17,244] INFO Partition [jtop-migration-smoke-els,11] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,11] from 1,5,6
to 1,6 (kafka.cluster.Partition)
[2017-02-19 13:01:17,244] INFO Partition [flintan-asset-view-smoke-els,4] on
broker 1: Shrinking ISR for partition [flintan-asset-view-smoke-els,4] from
1,5,4 to 1,4 (kafka.cluster.Partition)
[2017-02-19 13:01:17,245] INFO Partition [stolmen-bundle-prod-els,3] on broker
1: Shrinking ISR for partition [stolmen-bundle-prod-els,3] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,246] INFO Partition [flintan-asset-view-prod-els,23] on
broker 1: Shrinking ISR for partition [flintan-asset-view-prod-els,23] from
1,5,6 to 1,6 (kafka.cluster.Partition)
[2017-02-19 13:01:17,247] INFO Partition
[resolver-notifcation-requests-prod-els,0] on broker 1: Shrinking ISR for
partition [resolver-notifcation-requests-prod-els,0] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,248] INFO Partition [mj,0] on broker 1: Shrinking ISR for
partition [mj,0] from 1,5,6 to 1,6 (kafka.cluster.Partition)
[2017-02-19 13:01:17,249] INFO Partition [flintan-asset-view-prod-els,17] on
broker 1: Shrinking ISR for partition [flintan-asset-view-prod-els,17] from
1,5,4 to 1,4 (kafka.cluster.Partition)
[2017-02-19 13:01:17,250] INFO Partition [flintan-asset-view-prod-els,47] on
broker 1: Shrinking ISR for partition [flintan-asset-view-prod-els,47] from
1,5,4 to 1,4 (kafka.cluster.Partition)
[2017-02-19 13:01:17,251] INFO Partition [jtop-migration-smoke-els,26] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,26] from 1,5,4
to 1,4 (kafka.cluster.Partition)
[2017-02-19 13:01:17,252] INFO Partition [__consumer_offsets,24] on broker 1:
Shrinking ISR for partition [__consumer_offsets,24] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,253] INFO Partition [eds-incre-prod-els,0] on broker 1:
Shrinking ISR for partition [eds-incre-prod-els,0] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,254] INFO Partition [jtop-migration-prod-els,17] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,17] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,254] INFO Partition [__consumer_offsets,44] on broker 1:
Shrinking ISR for partition [__consumer_offsets,44] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,255] INFO Partition [jtop-migration-smoke-els,31] on
broker 1: Shrinking ISR for partition [jtop-migration-smoke-els,31] from 1,5,6
to 1,6 (kafka.cluster.Partition)
[2017-02-19 13:01:17,256] INFO Partition [__consumer_offsets,4] on broker 1:
Shrinking ISR for partition [__consumer_offsets,4] from 1,5,6 to 1,6
(kafka.cluster.Partition)
[2017-02-19 13:01:17,257] INFO Partition [jtop-migration-prod-els,32] on broker
1: Shrinking ISR for partition [jtop-migration-prod-els,32] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,258] INFO Partition [flintan-asset-view-smoke-els,40] on
broker 1: Shrinking ISR for partition [flintan-asset-view-smoke-els,40] from
1,5,6 to 1,6 (kafka.cluster.Partition)
[2017-02-19 13:01:17,259] INFO Partition [__consumer_offsets,19] on broker 1:
Shrinking ISR for partition [__consumer_offsets,19] from 1,5,4 to 1,4
(kafka.cluster.Partition)
[2017-02-19 13:01:17,260] INFO Partition [flintan-asset-view-smoke-els,10] on
broker 1: Shrinking ISR for partition [flintan-asset-view-smoke-els,10] from
1,5,6 to 1,6 (kafka.cluster.Partition)
[2017-02-19 13:01:17,261] INFO Partition [flintan-asset-view-smoke-els,34] on
broker 1: Shrinking ISR for partition [flintan-asset-view-smoke-els,34] from
1,5,4 to 1,4 (kafka.cluster.Partition)
[2017-02-19 13:01:27,320] INFO [GroupCoordinator 1]: Preparing to restabilize
group flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 with old
generation 1 (kafka.coordinator.GroupCoordinator)
[2017-02-19 13:01:27,320] INFO [GroupCoordinator 1]: Group
flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 generation 1 is dead
and removed (kafka.coordinator.GroupCoordinator)
[2017-02-19 13:01:28,263] INFO [GroupCoordinator 1]: Preparing to restabilize
group flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 with old
generation 0 (kafka.coordinator.GroupCoordinator)
[2017-02-19 13:01:28,263] INFO [GroupCoordinator 1]: Stabilized group
flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 generation 1
(kafka.coordinator.GroupCoordinator)
[2017-02-19 13:01:28,263] INFO [GroupCoordinator 1]: Assignment received from
leader for group flintan-metadatacache-264f0076-644b-4c7f-ae21-70526c6dd642 for
generation 1 (kafka.coordinator.GroupCoordinator)
[2017-02-19 13:01:33,773] INFO Partition [stolmen-entity-prod-els,2] on broker
1: Expanding ISR for partition [stolmen-entity-prod-els,2] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,774] INFO Partition [jtop-migration-prod-els,57] on broker
1: Expanding ISR for partition [jtop-migration-prod-els,57] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,775] INFO Partition [jtop-migration-smoke-els,31] on
broker 1: Expanding ISR for partition [jtop-migration-smoke-els,31] from 1,6 to
1,6,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,776] INFO Partition [jtop-migration-smoke-els,26] on
broker 1: Expanding ISR for partition [jtop-migration-smoke-els,26] from 1,4 to
1,4,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,777] INFO Partition [compass-home-record-prod-els,0] on
broker 1: Expanding ISR for partition [compass-home-record-prod-els,0] from 1,4
to 1,4,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,778] INFO Partition [flintan-asset-view-smoke-els,4] on
broker 1: Expanding ISR for partition [flintan-asset-view-smoke-els,4] from 1,4
to 1,4,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,779] INFO Partition [flintan-asset-view-smoke-els,10] on
broker 1: Expanding ISR for partition [flintan-asset-view-smoke-els,10] from
1,6 to 1,6,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,780] INFO Partition
[resolver-notification-profiles-staging-els,1] on broker 1: Expanding ISR for
partition [resolver-notification-profiles-staging-els,1] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,780] INFO Partition [hummus-asset-v2-prod-els,0] on broker
1: Expanding ISR for partition [hummus-asset-v2-prod-els,0] from 1,3 to 1,3,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,781] INFO Partition [jtop-migration-smoke-els,6] on broker
1: Expanding ISR for partition [jtop-migration-smoke-els,6] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,782] INFO Partition
[resolver-notifcation-requests-prod-els,0] on broker 1: Expanding ISR for
partition [resolver-notifcation-requests-prod-els,0] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,782] INFO Partition [hummus-boundary-v2-prod-els,0] on
broker 1: Expanding ISR for partition [hummus-boundary-v2-prod-els,0] from 1,3
to 1,3,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,783] INFO Partition [jtop-migration-smoke-els,51] on
broker 1: Expanding ISR for partition [jtop-migration-smoke-els,51] from 1,6 to
1,6,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,784] INFO Partition [stolmen-bundle-prod-els,3] on broker
1: Expanding ISR for partition [stolmen-bundle-prod-els,3] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,785] INFO Partition [eds-incre-prod-els,0] on broker 1:
Expanding ISR for partition [eds-incre-prod-els,0] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,785] INFO Partition [jtop-migration-prod-els,52] on broker
1: Expanding ISR for partition [jtop-migration-prod-els,52] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,786] INFO Partition [mj,0] on broker 1: Expanding ISR for
partition [mj,0] from 1,6 to 1,6,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,787] INFO Partition [jtop-migration-smoke-els,46] on
broker 1: Expanding ISR for partition [jtop-migration-smoke-els,46] from 1,4 to
1,4,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,788] INFO Partition [jtop-migration-prod-els,37] on broker
1: Expanding ISR for partition [jtop-migration-prod-els,37] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,789] INFO Partition [jtop-migration-prod-els,12] on broker
1: Expanding ISR for partition [jtop-migration-prod-els,12] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,790] INFO Partition [flintan-asset-view-smoke-els,40] on
broker 1: Expanding ISR for partition [flintan-asset-view-smoke-els,40] from
1,6 to 1,6,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,790] INFO Partition [khloe-updates-prod-els,0] on broker
1: Expanding ISR for partition [khloe-updates-prod-els,0] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,791] INFO Partition [jtop-migration-prod-els,17] on broker
1: Expanding ISR for partition [jtop-migration-prod-els,17] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,792] INFO Partition [hummus-entity-merged-v2-prod-els,0]
on broker 1: Expanding ISR for partition [hummus-entity-merged-v2-prod-els,0]
from 1,4 to 1,4,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,793] INFO Partition [flintan-asset-view-smoke-els,34] on
broker 1: Expanding ISR for partition [flintan-asset-view-smoke-els,34] from
1,4 to 1,4,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,794] INFO Partition [jtop-migration-prod-els,32] on broker
1: Expanding ISR for partition [jtop-migration-prod-els,32] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,795] INFO Partition [jtop-migration-smoke-els,11] on
broker 1: Expanding ISR for partition [jtop-migration-smoke-els,11] from 1,6 to
1,6,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,804] INFO Partition [__consumer_offsets,44] on broker 1:
Expanding ISR for partition [__consumer_offsets,44] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,805] INFO Partition [__consumer_offsets,4] on broker 1:
Expanding ISR for partition [__consumer_offsets,4] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,806] INFO Partition [__consumer_offsets,24] on broker 1:
Expanding ISR for partition [__consumer_offsets,24] from 1,6 to 1,6,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,807] INFO Partition [__consumer_offsets,39] on broker 1:
Expanding ISR for partition [__consumer_offsets,39] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,808] INFO Partition [__consumer_offsets,19] on broker 1:
Expanding ISR for partition [__consumer_offsets,19] from 1,4 to 1,4,5
(kafka.cluster.Partition)
[2017-02-19 13:01:33,809] INFO Partition [flintan-asset-view-prod-els,17] on
broker 1: Expanding ISR for partition [flintan-asset-view-prod-els,17] from 1,4
to 1,4,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,811] INFO Partition [flintan-asset-view-prod-els,23] on
broker 1: Expanding ISR for partition [flintan-asset-view-prod-els,23] from 1,6
to 1,6,5 (kafka.cluster.Partition)
[2017-02-19 13:01:33,812] INFO Partition [flintan-asset-view-prod-els,47] on
broker 1: Expanding ISR for partition [flintan-asset-view-prod-els,47] from 1,4
to 1,4,5 (kafka.cluster.Partition)
This time it kicks out broker 5 but it can kick all other brokers, and this
only happens to broker 1, I don’t see any other broker has such scenario. I
also take a look at controllers log, here’s the log at corresponding time:
[2017-02-19 13:01:19,731] DEBUG [IsrChangeNotificationListener] Fired!!!
(kafka.controller.IsrChangeNotificationListener)
[2017-02-19 13:01:19,765] DEBUG Sending MetadataRequest to
Brokers:ArrayBuffer(5, 1, 6, 2, 3, 4) for
TopicAndPartitions:Set([__consumer_offsets,44], [stolmen-entity-prod-els,2],
[jtop-migration-prod-els,57], [jtop-migration-smoke-els,31],
[jtop-migration-smoke-els,26], [__consumer_offsets,4],
[compass-home-record-prod-els,0], [flintan-asset-view-smoke-els,4],
[flintan-asset-view-smoke-els,10], [__consumer_offsets,24],
[resolver-notification-profiles-staging-els,1],
[flintan-asset-view-prod-els,17], [hummus-asset-v2-prod-els,0],
[jtop-migration-smoke-els,6], [resolver-notifcation-requests-prod-els,0],
[hummus-boundary-v2-prod-els,0], [__consumer_offsets,39],
[jtop-migration-smoke-els,51], [stolmen-bundle-prod-els,3],
[eds-incre-prod-els,0], [flintan-asset-view-prod-els,23],
[jtop-migration-prod-els,52], [mj,0], [jtop-migration-smoke-els,46],
[flintan-asset-view-prod-els,47], [jtop-migration-prod-els,37],
[jtop-migration-prod-els,12], [flintan-asset-view-smoke-els,40],
[__consumer_offsets,19], [khloe-updates-prod-els,0],
[jtop-migration-prod-els,17], [hummus-entity-merged-v2-prod-els,0],
[flintan-asset-view-smoke-els,34], [jtop-migration-prod-els,32],
[jtop-migration-smoke-els,11]) (kafka.controller.IsrChangeNotificationListener)
[2017-02-19 13:01:19,770] DEBUG [IsrChangeNotificationListener] Fired!!!
(kafka.controller.IsrChangeNotificationListener)
[2017-02-19 13:01:39,731] DEBUG [IsrChangeNotificationListener] Fired!!!
(kafka.controller.IsrChangeNotificationListener)
[2017-02-19 13:01:39,766] DEBUG Sending MetadataRequest to
Brokers:ArrayBuffer(5, 1, 6, 2, 3, 4) for
TopicAndPartitions:Set([__consumer_offsets,44], [stolmen-entity-prod-els,2],
[jtop-migration-prod-els,57], [jtop-migration-smoke-els,31],
[jtop-migration-smoke-els,26], [__consumer_offsets,4],
[compass-home-record-prod-els,0], [flintan-asset-view-smoke-els,4],
[flintan-asset-view-smoke-els,10], [__consumer_offsets,24],
[resolver-notification-profiles-staging-els,1],
[flintan-asset-view-prod-els,17], [hummus-asset-v2-prod-els,0],
[jtop-migration-smoke-els,6], [resolver-notifcation-requests-prod-els,0],
[hummus-boundary-v2-prod-els,0], [__consumer_offsets,39],
[jtop-migration-smoke-els,51], [stolmen-bundle-prod-els,3],
[eds-incre-prod-els,0], [flintan-asset-view-prod-els,23],
[jtop-migration-prod-els,52], [mj,0], [jtop-migration-smoke-els,46],
[flintan-asset-view-prod-els,47], [jtop-migration-prod-els,37],
[jtop-migration-prod-els,12], [flintan-asset-view-smoke-els,40],
[__consumer_offsets,19], [khloe-updates-prod-els,0],
[jtop-migration-prod-els,17], [hummus-entity-merged-v2-prod-els,0],
[flintan-asset-view-smoke-els,34], [jtop-migration-prod-els,32],
[jtop-migration-smoke-els,11]) (kafka.controller.IsrChangeNotificationListener)
[2017-02-19 13:01:39,771] DEBUG [IsrChangeNotificationListener] Fired!!!
(kafka.controller.IsrChangeNotificationListener)
Sometimes when broker 1 shrink/expand ISR, controller will throw IOException
saying that all other brokers was disconnected before the response was read,
but this doesn’t happen on all shrink/expand so I’m not sure if this is also
related.
[2017-02-19 13:19:19,740] WARN [Controller-4-to-broker-2-send-thread],
Controller 4 epoch 35 fails to send request
{controller_id=4,controller_epoch=35,partition_states=[{topic=flintan-asset-view-prod-els,partition=29,controller_epoch=34,leader=1,leader_epoch=2,isr=[1,6],zk_version=888,replicas=[1,6,2]},{topic=flintan-asset-view-prod-els,partition=59,controller_epoch=34,leader=1,leader_epoch=2,isr=[1,6],zk_version=888,replicas=[1,6,2]},{topic=flintan-asset-view-smoke-els,partition=22,controller_epoch=34,leader=1,leader_epoch=3,isr=[1,3],zk_version=910,replicas=[1,2,3]},{topic=resolver-notification-profiles-staging-els,partition=1,controller_epoch=34,leader=-2,leader_epoch=0,isr=[1,4,5],zk_version=0,replicas=[1,4,5]},{topic=flintan-asset-view-smoke-els,partition=52,controller_epoch=34,leader=1,leader_epoch=3,isr=[1,3],zk_version=910,replicas=[1,2,3]},{topic=flintan-asset-view-smoke-els,partition=16,controller_epoch=34,leader=1,leader_epoch=2,isr=[1,6],zk_version=888,replicas=[1,6,2]},{topic=resolver-notification-profiles-staging-els,partition=0,controller_epoch=34,leader=-2,leader_epoch=0,isr=[6,4,3],zk_version=0,replicas=[6,3,4]},{topic=flintan-asset-view-smoke-els,partition=46,controller_epoch=34,leader=1,leader_epoch=2,isr=[1,6],zk_version=888,replicas=[1,6,2]},{topic=flintan-asset-view-prod-els,partition=35,controller_epoch=34,leader=1,leader_epoch=3,isr=[1,3],zk_version=910,replicas=[1,2,3]},{topic=flintan-asset-view-prod-els,partition=5,controller_epoch=34,leader=1,leader_epoch=3,isr=[1,3],zk_version=910,replicas=[1,2,3]}],live_brokers=[{id=6,end_points=[{port=9092,host=elsconfluent006.server.hulu.com,security_protocol_type=0}]},{id=5,end_points=[{port=9092,host=elsconfluent005.server.hulu.com,security_protocol_type=0}]},{id=3,end_points=[{port=9092,host=elsconfluent003.server.hulu.com,security_protocol_type=0}]},{id=4,end_points=[{port=9092,host=elsconfluent004.server.hulu.com,security_protocol_type=0}]},{id=2,end_points=[{port=9092,host=elsconfluent002.server.hulu.com,security_protocol_type=0}]},{id=1,end_points=[{port=9092,host=elsconfluent001.server.hulu.com,security_protocol_type=0}]}]}
to broker Node(2, elsconfluent002.server.hulu.com, 9092). Reconnecting to
broker. (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 2 was disconnected before the response was
read
at
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
at
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
at scala.Option.foreach(Option.scala:257)
at
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
at
kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
at
kafka.utils.NetworkClientBlockingOps$.recurse$1(NetworkClientBlockingOps.scala:129)
at
kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntilFound$extension(NetworkClientBlockingOps.scala:139)
at
kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
at
kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:180)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:171)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
My understand on shrink/expand ISR is that broker n fails to sent fetch request
to broker 1 in replica.lag.time.max.ms(10000), then broker 1 will kick broker n
out of ISR. But from the log it doesn’t look like this. So what other reason
could it be?
Can someone help me with why this is happening and how can I react to this?
Thanks!
Jun
