vishal patel has shared a OneDrive file with you. To view it, click the link below.
<https://1drv.ms/t/s!AhS5CvIRnaQCbAyL7HxyxR6CQOU> [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<https://1drv.ms/t/s!AhS5CvIRnaQCbAyL7HxyxR6CQOU> GC_log.txt<https://1drv.ms/t/s!AhS5CvIRnaQCbAyL7HxyxR6CQOU> We have 2 shards and 2 replicas with 7 zookeepers in our live environment. unexpectedly shard got down. From logs, we can not identify why shard got down. I have attached the GC and solr log. ********************************************* My solr.xml data ************** <solr> <solrcloud> <str name="host">${host:localhost}</str> <int name="hostPort">${jetty.port:8983}</int> <str name="hostContext">${hostContext:solr}</str> <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool> <int name="zkClientTimeout">${zkClientTimeout:600000}</int> <int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int> <int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int> <str name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}</str> <str name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}</str> </solrcloud> <shardHandlerFactory name="shardHandlerFactory" class="HttpShardHandlerFactory"> <int name="socketTimeout">${socketTimeout:600000}</int> <int name="connTimeout">${connTimeout:60000}</int> </shardHandlerFactory> </solr> ********************************************* My zoo.cfg data ************** tickTime=2000 initLimit=10 syncLimit=5 ********************************************* Can you suggest me how can we find out this issue?
2019-05-10 13:00:54.559 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3a0b9e39 name:ZooKeeperConnection Watcher:10.200.312.80:11111,10.200.312.81:11112,10.200.312.82:11113,10.200.312.83:11114,10.200.312.84:11115,10.200.312.85:11116,10.200.312.86:11117 got event WatchedEvent state:Expired type:None path:null path:null type:None 2019-05-10 13:00:54.559 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... 2019-05-10 13:00:54.559 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.Overseer Overseer (id=246176007594049637-localhost:8983_solr-n_0000000048) closing 2019-05-10 13:00:54.621 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new one... 2019-05-10 13:00:54.691 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper 2019-05-10 13:00:54.691 ERROR (OverseerExitThread) [ ] o.a.s.c.Overseer could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348) at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:309) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:89) at org.apache.solr.cloud.Overseer$ClusterStateUpdater$2.run(Overseer.java:268) 2019-05-10 13:00:57.662 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr-EventThread) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3a0b9e39 name:ZooKeeperConnection Watcher:10.200.312.80:11111,10.200.312.81:11112,10.200.312.82:11113,10.200.312.83:11114,10.200.312.84:11115,10.200.312.85:11116,10.200.312.86:11117 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 2019-05-10 13:00:57.666 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2019-05-10 13:00:57.668 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished. 2019-05-10 13:00:57.668 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core states after session expiration. 2019-05-10 13:00:57.700 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 2019-05-10 13:01:01.138 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties 2019-05-10 13:01:03.382 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3) 2019-05-10 13:01:03.400 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection1] 2019-05-10 13:01:03.406 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection2] 2019-05-10 13:01:03.409 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection3] 2019-05-10 13:01:03.410 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection4] 2019-05-10 13:01:03.411 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection5] 2019-05-10 13:01:03.412 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection6] 2019-05-10 13:01:03.413 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection7] 2019-05-10 13:01:03.414 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection8] 2019-05-10 13:01:03.415 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection9] 2019-05-10 13:01:08.449 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection10] 2019-05-10 13:01:08.518 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection11] 2019-05-10 13:01:08.537 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection12] 2019-05-10 13:01:08.544 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection13] 2019-05-10 13:01:08.545 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection14] 2019-05-10 13:01:09.672 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection15] 2019-05-10 13:01:09.674 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection16] 2019-05-10 13:01:09.676 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection17] 2019-05-10 13:01:09.679 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection18] 2019-05-10 13:01:09.681 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection19] 2019-05-10 13:01:09.682 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection20] 2019-05-10 13:01:09.684 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection21] 2019-05-10 13:01:09.686 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection22] 2019-05-10 13:01:09.687 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection23] 2019-05-10 13:01:09.689 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection24] 2019-05-10 13:01:09.691 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection25] 2019-05-10 13:01:09.693 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection26] 2019-05-10 13:01:09.695 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader Deleting data for [collection27] 2019-05-10 13:01:09.697 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ElectionContext Canceling election /overseer_elect/election/246176007594049637-localhost:8983_solr-n_0000000048 2019-05-10 13:01:09.702 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ElectionContext cancelElection did not find election node to remove /overseer_elect/election/246176007594049637-localhost:8983_solr-n_0000000048 2019-05-10 13:01:09.704 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/534406384210149573-localhost:8983_solr-n_0000000059 2019-05-10 13:01:09.704 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/246176007594049669-10.200.312.77:8983_solr-n_0000000058 to know if I could be the leader 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection13] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection10] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection3] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection23] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection17] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection21] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection25] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection2] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection20] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection8] coreNodeName=[core_node1] 2019-05-10 13:01:09.705 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection26] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection22] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection16] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection18] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection12] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection7] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection24] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection11] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection14] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection6] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection19] coreNodeName=[core_node1] 2019-05-10 13:01:09.706 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection15] coreNodeName=[core_node1] 2019-05-10 13:01:09.707 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection4] coreNodeName=[core_node1] 2019-05-10 13:01:09.707 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection1] coreNodeName=[core_node1] 2019-05-10 13:01:09.707 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection5] coreNodeName=[core_node1] 2019-05-10 13:01:09.707 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection9] coreNodeName=[core_node1] 2019-05-10 13:01:09.707 WARN (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection27] coreNodeName=[core_node1] 2019-05-10 13:01:09.707 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Publish node=localhost:8983_solr as DOWN 2019-05-10 13:01:09.768 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.071 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.079 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.088 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.096 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.103 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.111 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.119 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.126 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.134 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.143 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.151 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.158 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.165 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.172 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.178 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.186 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.192 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.199 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.207 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.215 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.225 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.234 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.245 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.264 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.272 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.277 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Replica core_node1 NOT in leader-initiated recovery, need to wait for leader to see down state. 2019-05-10 13:01:11.283 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/localhost:8983_solr 2019-05-10 13:01:11.330 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/localhost:8983_solr 2019-05-10 13:01:11.554 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper 2019-05-10 13:01:11.569 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ConnectionManager Connected:true 2019-05-10 13:01:11.787 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:12.135 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:12.623 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:16.195 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:16.862 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:19.834 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:20.135 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:22.575 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:26.173 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:26.404 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:28.293 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:28.619 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:29.253 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:31.966 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:32.047 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:40.511 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:46.548 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:46.831 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:47.997 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:51.953 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:53.355 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:54.298 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:55.344 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:56.269 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:58.487 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4]) 2019-05-10 13:01:58.634 INFO (zkCallback-4-thread-632-processing-n:localhost:8983_solr) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
