[ https://issues.apache.org/jira/browse/SOLR-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809799#comment-13809799 ]
Nathan Neulinger commented on SOLR-5407: ---------------------------------------- Digging further, it looks like it all keys around some sort of communications problem with zookeeper - looks like it all started at the end of this log snippet below (reverse time order) when it's reporting that 'Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper'. 2013-10-29T16:25:50.344Z Going to wait for coreNodeName: core_node2, state: down, checkLive: null, onlyIfLeader: null 2013-10-29T16:25:50.329Z publishing core=myappqa-master_v8_shard1_replica1 state=down 2013-10-29T16:25:50.329Z Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false 2013-10-29T16:25:50.328Z Waited coreNodeName: core_node1, state: down, checkLive: null, onlyIfLeader: null for: 1 seconds. 2013-10-29T16:25:49.884Z A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) 2013-10-29T16:25:49.825Z Updating cloud state from ZooKeeper... 2013-10-29T16:25:49.825Z Update state numShards=1 message={ "operation":"state", "state":"down", "base_url":"http://10.170.2.54:8983/solr", "core":"hiv... 2013-10-29T16:25:49.324Z Going to wait for coreNodeName: core_node1, state: down, checkLive: null, onlyIfLeader: null 2013-10-29T16:25:49.309Z Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false 2013-10-29T16:25:49.308Z publishing core=myappqa-master_v6_shard1_replica2 state=down 2013-10-29T16:25:49.308Z Waited coreNodeName: core_node1, state: down, checkLive: null, onlyIfLeader: null for: 2 seconds. 2013-10-29T16:25:48.302Z A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) 2013-10-29T16:25:48.239Z Updating cloud state from ZooKeeper... 2013-10-29T16:25:48.239Z Update state numShards=1 message={ "operation":"state", "state":"down", "base_url":"http://10.170.2.54:8983/solr", "core":"hiv... 2013-10-29T16:25:47.304Z Going to wait for coreNodeName: core_node1, state: down, checkLive: null, onlyIfLeader: null 2013-10-29T16:25:47.289Z Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false 2013-10-29T16:25:47.289Z publishing core=myappstaging-profile_v7_shard1_replica1 state=down 2013-10-29T16:25:47.287Z Waited coreNodeName: core_node2, state: down, checkLive: null, onlyIfLeader: null for: 2 seconds. 2013-10-29T16:25:46.469Z A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) 2013-10-29T16:25:46.406Z Update state numShards=1 message={ "operation":"state", "state":"down", "base_url":"http://10.170.2.54:8983/solr", "core":"hiv... 2013-10-29T16:25:45.925Z Updating cloud state from ZooKeeper... 2013-10-29T16:25:45.286Z Going to wait for coreNodeName: core_node2, state: down, checkLive: null, onlyIfLeader: null 2013-10-29T16:25:45.270Z publishing core=myappstaging-profile_v8_shard1_replica1 state=down 2013-10-29T16:25:45.270Z Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false 2013-10-29T16:25:45.269Z Waited coreNodeName: core_node2, state: down, checkLive: null, onlyIfLeader: null for: 2 seconds. 2013-10-29T16:25:45.039Z A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) 2013-10-29T16:25:44.994Z makePath: /collections/myappproduction-production_v8/leaders/shard1 2013-10-29T16:25:44.994Z I am the new leader: http://10.136.6.24:8983/solr/myappproduction-production_v8_shard1_replica1/ shard1 2013-10-29T16:25:44.994Z http://10.136.6.24:8983/solr/myappproduction-production_v8_shard1_replica1/ has no replicas 2013-10-29T16:25:44.991Z Sync replicas to http://10.136.6.24:8983/solr/myappproduction-production_v8_shard1_replica1/ 2013-10-29T16:25:44.991Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.991Z Running the leader process for shard shard1 2013-10-29T16:25:44.991Z I may be the new leader - try and sync 2013-10-29T16:25:44.991Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.991Z Checking if I should try and be the leader. 2013-10-29T16:25:44.940Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-feature-completion_v9_shard1_replica1/ shard1 2013-10-29T16:25:44.940Z makePath: /collections/myappstaging-feature-completion_v9/leaders/shard1 2013-10-29T16:25:44.939Z http://10.136.6.24:8983/solr/myappstaging-feature-completion_v9_shard1_replica1/ has no replicas 2013-10-29T16:25:44.939Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.939Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.939Z I may be the new leader - try and sync 2013-10-29T16:25:44.939Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-feature-completion_v9_shard1_replica1/ 2013-10-29T16:25:44.938Z Running the leader process for shard shard1 2013-10-29T16:25:44.938Z Checking if I should try and be the leader. 2013-10-29T16:25:44.880Z makePath: /collections/myappstaging-profile_v7/leaders/shard1 2013-10-29T16:25:44.873Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-profile_v7_shard1_replica2/ shard1 2013-10-29T16:25:44.872Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-profile_v7_shard1_replica2/ 2013-10-29T16:25:44.872Z Stopping recovery for zkNodeName=core_node2core=myappstaging-profile_v7_shard1_replica2 2013-10-29T16:25:44.872Z I may be the new leader - try and sync 2013-10-29T16:25:44.872Z Checking if I should try and be the leader. 2013-10-29T16:25:44.872Z http://10.136.6.24:8983/solr/myappstaging-profile_v7_shard1_replica2/ has no replicas 2013-10-29T16:25:44.872Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.872Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.863Z Running the leader process for shard shard1 2013-10-29T16:25:44.809Z makePath: /collections/myappqa-master_v6/leaders/shard1 2013-10-29T16:25:44.808Z Stopping recovery for zkNodeName=core_node2core=myappqa-master_v6_shard1_replica1 2013-10-29T16:25:44.808Z http://10.136.6.24:8983/solr/myappqa-master_v6_shard1_replica1/ has no replicas 2013-10-29T16:25:44.808Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.808Z I am the new leader: http://10.136.6.24:8983/solr/myappqa-master_v6_shard1_replica1/ shard1 2013-10-29T16:25:44.808Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.808Z I may be the new leader - try and sync 2013-10-29T16:25:44.808Z Sync replicas to http://10.136.6.24:8983/solr/myappqa-master_v6_shard1_replica1/ 2013-10-29T16:25:44.808Z Checking if I should try and be the leader. 2013-10-29T16:25:44.807Z Running the leader process for shard shard1 2013-10-29T16:25:44.749Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-feature-completion_v8_shard1_replica1/ shard1 2013-10-29T16:25:44.749Z makePath: /collections/myappstaging-feature-completion_v8/leaders/shard1 2013-10-29T16:25:44.749Z http://10.136.6.24:8983/solr/myappstaging-feature-completion_v8_shard1_replica1/ has no replicas 2013-10-29T16:25:44.747Z I may be the new leader - try and sync 2013-10-29T16:25:44.747Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.747Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.747Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-feature-completion_v8_shard1_replica1/ 2013-10-29T16:25:44.747Z Checking if I should try and be the leader. 2013-10-29T16:25:44.746Z Running the leader process for shard shard1 2013-10-29T16:25:44.694Z makePath: /collections/myappproduction-production_v9/leaders/shard1 2013-10-29T16:25:44.693Z http://10.136.6.24:8983/solr/myappproduction-production_v9_shard1_replica2/ has no replicas 2013-10-29T16:25:44.693Z I am the new leader: http://10.136.6.24:8983/solr/myappproduction-production_v9_shard1_replica2/ shard1 2013-10-29T16:25:44.692Z I may be the new leader - try and sync 2013-10-29T16:25:44.692Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.692Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.692Z Sync replicas to http://10.136.6.24:8983/solr/myappproduction-production_v9_shard1_replica2/ 2013-10-29T16:25:44.692Z Checking if I should try and be the leader. 2013-10-29T16:25:44.691Z Running the leader process for shard shard1 2013-10-29T16:25:44.647Z makePath: /collections/myappstaging-alpha4_v9/leaders/shard1 2013-10-29T16:25:44.647Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-alpha4_v9_shard1_replica1/ shard1 2013-10-29T16:25:44.647Z http://10.136.6.24:8983/solr/myappstaging-alpha4_v9_shard1_replica1/ has no replicas 2013-10-29T16:25:44.632Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.632Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-alpha4_v9_shard1_replica1/ 2013-10-29T16:25:44.632Z I may be the new leader - try and sync 2013-10-29T16:25:44.632Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.631Z Checking if I should try and be the leader. 2013-10-29T16:25:44.630Z Running the leader process for shard shard1 2013-10-29T16:25:44.572Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-master_v9_shard1_replica1/ shard1 2013-10-29T16:25:44.572Z makePath: /collections/myappstaging-master_v9/leaders/shard1 2013-10-29T16:25:44.571Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.571Z http://10.136.6.24:8983/solr/myappstaging-master_v9_shard1_replica1/ has no replicas 2013-10-29T16:25:44.571Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.571Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-master_v9_shard1_replica1/ 2013-10-29T16:25:44.571Z I may be the new leader - try and sync 2013-10-29T16:25:44.571Z Checking if I should try and be the leader. 2013-10-29T16:25:44.568Z Running the leader process for shard shard1 2013-10-29T16:25:44.511Z makePath: /collections/myappstaging-preetalpha4_v9/leaders/shard1 2013-10-29T16:25:44.511Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v9_shard1_replica2/ shard1 2013-10-29T16:25:44.511Z http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v9_shard1_replica2/ has no replicas 2013-10-29T16:25:44.510Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.509Z Checking if I should try and be the leader. 2013-10-29T16:25:44.509Z Running the leader process for shard shard1 2013-10-29T16:25:44.509Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v9_shard1_replica2/ 2013-10-29T16:25:44.509Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.509Z I may be the new leader - try and sync 2013-10-29T16:25:44.450Z makePath: /collections/collection1/leaders/shard1 2013-10-29T16:25:44.450Z I am the new leader: http://10.136.6.24:8983/solr/collection1/ shard1 2013-10-29T16:25:44.444Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:44.444Z http://10.136.6.24:8983/solr/collection1/ has no replicas 2013-10-29T16:25:44.444Z Sync Success - now sync replicas to me 2013-10-29T16:25:44.444Z Sync replicas to http://10.136.6.24:8983/solr/collection1/ 2013-10-29T16:25:44.444Z I may be the new leader - try and sync 2013-10-29T16:25:44.443Z Checking if I should try and be the leader. 2013-10-29T16:25:44.434Z Running the leader process for shard shard1 2013-10-29T16:25:44.418Z A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1) 2013-10-29T16:25:44.418Z Updating live nodes... (1) 2013-10-29T16:25:43.715Z makePath: /collections/myappqa-master_v9/leaders/shard1 2013-10-29T16:25:43.714Z Checking if I should try and be the leader. 2013-10-29T16:25:43.714Z I may be the new leader - try and sync 2013-10-29T16:25:43.714Z I am the new leader: http://10.136.6.24:8983/solr/myappqa-master_v9_shard1_replica1/ shard1 2013-10-29T16:25:43.714Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:43.714Z Sync replicas to http://10.136.6.24:8983/solr/myappqa-master_v9_shard1_replica1/ 2013-10-29T16:25:43.714Z http://10.136.6.24:8983/solr/myappqa-master_v9_shard1_replica1/ has no replicas 2013-10-29T16:25:43.714Z Sync Success - now sync replicas to me 2013-10-29T16:25:43.713Z Running the leader process for shard shard1 2013-10-29T16:25:43.654Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-alpha3_v8_shard1_replica2/ shard1 2013-10-29T16:25:43.654Z makePath: /collections/myappstaging-alpha3_v8/leaders/shard1 2013-10-29T16:25:43.654Z http://10.136.6.24:8983/solr/myappstaging-alpha3_v8_shard1_replica2/ has no replicas 2013-10-29T16:25:43.653Z I may be the new leader - try and sync 2013-10-29T16:25:43.653Z Checking if I should try and be the leader. 2013-10-29T16:25:43.653Z Sync Success - now sync replicas to me 2013-10-29T16:25:43.653Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-alpha3_v8_shard1_replica2/ 2013-10-29T16:25:43.653Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:43.652Z Running the leader process for shard shard1 2013-10-29T16:25:43.591Z makePath: /collections/myappqa-master_v8/leaders/shard1 2013-10-29T16:25:43.591Z I am the new leader: http://10.136.6.24:8983/solr/myappqa-master_v8_shard1_replica2/ shard1 2013-10-29T16:25:43.591Z http://10.136.6.24:8983/solr/myappqa-master_v8_shard1_replica2/ has no replicas 2013-10-29T16:25:43.575Z I may be the new leader - try and sync 2013-10-29T16:25:43.575Z Checking if I should try and be the leader. 2013-10-29T16:25:43.575Z Sync Success - now sync replicas to me 2013-10-29T16:25:43.575Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:43.575Z Sync replicas to http://10.136.6.24:8983/solr/myappqa-master_v8_shard1_replica2/ 2013-10-29T16:25:43.574Z Running the leader process for shard shard1 2013-10-29T16:25:43.505Z makePath: /collections/myappstaging-profile_v8/leaders/shard1 2013-10-29T16:25:43.498Z http://10.136.6.24:8983/solr/myappstaging-profile_v8_shard1_replica2/ has no replicas 2013-10-29T16:25:43.498Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-profile_v8_shard1_replica2/ 2013-10-29T16:25:43.498Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-profile_v8_shard1_replica2/ shard1 2013-10-29T16:25:43.498Z Sync Success - now sync replicas to me 2013-10-29T16:25:43.497Z Stopping recovery for zkNodeName=core_node1core=myappstaging-profile_v8_shard1_replica2 2013-10-29T16:25:43.497Z I may be the new leader - try and sync 2013-10-29T16:25:43.497Z Checking if I should try and be the leader. 2013-10-29T16:25:43.497Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:43.489Z Running the leader process for shard shard1 2013-10-29T16:25:43.472Z Update state numShards=1 message={ "operation":"state", "state":"down", "base_url":"http://10.170.2.54:8983/solr", "core":"hiv... 2013-10-29T16:25:43.456Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v9_shard1_replica2/ shard1 2013-10-29T16:25:43.456Z makePath: /collections/myappstaging-alpha4-adam_v9/leaders/shard1 2013-10-29T16:25:43.456Z [myappstaging-alpha4-adam_v9_shard1_replica1] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=... 2013-10-29T16:25:43.440Z Sync Success - now sync replicas to me 2013-10-29T16:25:43.440Z PeerSync: core=myappstaging-alpha4-adam_v9_shard1_replica2 url=http://10.136.6.24:8983/solr DONE. sync succeeded 2013-10-29T16:25:43.440Z http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v9_shard1_replica2/ has no replicas 2013-10-29T16:25:43.440Z PeerSync: core=myappstaging-alpha4-adam_v9_shard1_replica2 url=http://10.136.6.24:8983/solr Our versions are newer. ourLowThreshold=14495694169781043... 2013-10-29T16:25:43.439Z PeerSync: core=myappstaging-alpha4-adam_v9_shard1_replica2 url=http://10.136.6.24:8983/solr Received 17 versions from 10.170.2.54:8983/solr/myappstag... 2013-10-29T16:25:43.330Z Updating cloud state from ZooKeeper... 2013-10-29T16:25:43.322Z PeerSync: core=myappstaging-alpha4-adam_v9_shard1_replica2 url=http://10.136.6.24:8983/solr START replicas=[http://10.170.2.54:8983/solr/myappstaging-... 2013-10-29T16:25:43.321Z I may be the new leader - try and sync 2013-10-29T16:25:43.321Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v9_shard1_replica2/ 2013-10-29T16:25:43.321Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:43.321Z Checking if I should try and be the leader. 2013-10-29T16:25:43.314Z Running the leader process for shard shard1 2013-10-29T16:25:43.314Z Process current queue of collection creations 2013-10-29T16:25:43.314Z Starting to work on the main queue 2013-10-29T16:25:43.264Z Going to wait for coreNodeName: core_node2, state: down, checkLive: null, onlyIfLeader: null 2013-10-29T16:25:43.262Z Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false 2013-10-29T16:25:43.247Z Overseer (id=234546968080679008-10.136.6.24:8983_solr-n_0000000021) starting 2013-10-29T16:25:43.247Z makePath: /overseer_elect/leader 2013-10-29T16:25:43.203Z PeerSync: core=myappstaging-preetalpha4_v8_shard1_replica1 url=http://10.170.2.54:8983/solr DONE. sync succeeded 2013-10-29T16:25:43.203Z [myappstaging-preetalpha4_v8_shard1_replica1] webapp=/solr path=/get params={sync=http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_repl... 2013-10-29T16:25:43.203Z makePath: /collections/myappstaging-preetalpha4_v8/leaders/shard1 2013-10-29T16:25:43.189Z PeerSync: core=myappstaging-preetalpha4_v8_shard1_replica1 url=http://10.170.2.54:8983/solr Our versions are newer. ourLowThreshold=14490017764527308... 2013-10-29T16:25:43.189Z http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_replica2/: sync completed with http://10.170.2.54:8983/solr/myappstaging-preetalpha4_... 2013-10-29T16:25:43.189Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_replica2/ shard1 2013-10-29T16:25:43.187Z PeerSync: core=myappstaging-preetalpha4_v8_shard1_replica1 url=http://10.170.2.54:8983/solr Received 10 versions from 10.136.6.24:8983/solr/myappstag... 2013-10-29T16:25:43.187Z [myappstaging-preetalpha4_v8_shard1_replica2] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=... 2013-10-29T16:25:43.187Z PeerSync: core=myappstaging-preetalpha4_v8_shard1_replica1 url=http://10.170.2.54:8983/solr START replicas=[http://10.136.6.24:8983/solr/myappstaging-... 2013-10-29T16:25:43.187Z [myappstaging-preetalpha4_v8_shard1_replica1] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=... 2013-10-29T16:25:43.187Z http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_replica2/: try and ask http://10.170.2.54:8983/solr/myappstaging-preetalpha4_v8_shard1... 2013-10-29T16:25:43.187Z Sync Success - now sync replicas to me 2013-10-29T16:25:43.170Z PeerSync: core=myappstaging-preetalpha4_v8_shard1_replica2 url=http://10.136.6.24:8983/solr DONE. sync succeeded 2013-10-29T16:25:43.170Z PeerSync: core=myappstaging-preetalpha4_v8_shard1_replica2 url=http://10.136.6.24:8983/solr Received 10 versions from 10.170.2.54:8983/solr/myappstag... 2013-10-29T16:25:43.170Z PeerSync: core=myappstaging-preetalpha4_v8_shard1_replica2 url=http://10.136.6.24:8983/solr Our versions are newer. ourLowThreshold=14490017764527308... 2013-10-29T16:25:43.100Z PeerSync: core=myappstaging-preetalpha4_v8_shard1_replica2 url=http://10.136.6.24:8983/solr START replicas=[http://10.170.2.54:8983/solr/myappstaging-... 2013-10-29T16:25:43.100Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-preetalpha4_v8_shard1_replica2/ 2013-10-29T16:25:43.100Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:43.100Z I may be the new leader - try and sync 2013-10-29T16:25:43.100Z Checking if I should try and be the leader. 2013-10-29T16:25:43.099Z Running the leader process for shard shard1 2013-10-29T16:25:43.056Z makePath: /collections/myappstaging-alpha3_v7/leaders/shard1 2013-10-29T16:25:43.044Z [myappstaging-alpha3_v7_shard1_replica2] webapp=/solr path=/get params={sync=http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/&... 2013-10-29T16:25:43.044Z PeerSync: core=myappstaging-alpha3_v7_shard1_replica2 url=http://10.170.2.54:8983/solr DONE. sync succeeded 2013-10-29T16:25:43.041Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/ shard1 2013-10-29T16:25:43.041Z http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/: sync completed with http://10.170.2.54:8983/solr/myappstaging-alpha3_v7_shard1_... 2013-10-29T16:25:43.041Z [myappstaging-alpha3_v7_shard1_replica1] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} st... 2013-10-29T16:25:43.040Z PeerSync: core=myappstaging-alpha3_v7_shard1_replica2 url=http://10.170.2.54:8983/solr Our versions are newer. ourLowThreshold=1448183627488690176 ot... 2013-10-29T16:25:43.040Z PeerSync: core=myappstaging-alpha3_v7_shard1_replica2 url=http://10.170.2.54:8983/solr Received 98 versions from 10.136.6.24:8983/solr/myappstaging-a... 2013-10-29T16:25:43.027Z PeerSync: core=myappstaging-alpha3_v7_shard1_replica2 url=http://10.170.2.54:8983/solr START replicas=[http://10.136.6.24:8983/solr/myappstaging-alpha... 2013-10-29T16:25:43.011Z [myappstaging-alpha3_v7_shard1_replica2] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} st... 2013-10-29T16:25:42.989Z http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/: try and ask http://10.170.2.54:8983/solr/myappstaging-alpha3_v7_shard1_replica2/... 2013-10-29T16:25:42.989Z Sync Success - now sync replicas to me 2013-10-29T16:25:42.988Z PeerSync: core=myappstaging-alpha3_v7_shard1_replica1 url=http://10.136.6.24:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.982Z PeerSync: core=myappstaging-alpha3_v7_shard1_replica1 url=http://10.136.6.24:8983/solr Our versions are newer. ourLowThreshold=1448183627488690176 ot... 2013-10-29T16:25:42.982Z PeerSync: core=myappstaging-alpha3_v7_shard1_replica1 url=http://10.136.6.24:8983/solr Received 98 versions from 10.170.2.54:8983/solr/myappstaging-a... 2013-10-29T16:25:42.955Z PeerSync: core=myappstaging-alpha3_v7_shard1_replica1 url=http://10.136.6.24:8983/solr START replicas=[http://10.170.2.54:8983/solr/myappstaging-alpha... 2013-10-29T16:25:42.954Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-alpha3_v7_shard1_replica1/ 2013-10-29T16:25:42.954Z Stopping recovery for zkNodeName=core_node1core=myappstaging-alpha3_v7_shard1_replica1 2013-10-29T16:25:42.953Z I may be the new leader - try and sync 2013-10-29T16:25:42.953Z Running the leader process for shard shard1 2013-10-29T16:25:42.953Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:42.953Z Checking if I should try and be the leader. 2013-10-29T16:25:42.897Z [myappstaging-alpha4-adam_v8_shard1_replica1] webapp=/solr path=/get params={sync=http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_repl... 2013-10-29T16:25:42.896Z makePath: /collections/myappstaging-alpha4-adam_v8/leaders/shard1 2013-10-29T16:25:42.882Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_replica2/ shard1 2013-10-29T16:25:42.881Z http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_replica2/: sync completed with http://10.170.2.54:8983/solr/myappstaging-alpha4-adam_... 2013-10-29T16:25:42.881Z [myappstaging-alpha4-adam_v8_shard1_replica2] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=... 2013-10-29T16:25:42.881Z PeerSync: core=myappstaging-alpha4-adam_v8_shard1_replica1 url=http://10.170.2.54:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.880Z PeerSync: core=myappstaging-alpha4-adam_v8_shard1_replica1 url=http://10.170.2.54:8983/solr Received 10 versions from 10.136.6.24:8983/solr/myappstag... 2013-10-29T16:25:42.880Z PeerSync: core=myappstaging-alpha4-adam_v8_shard1_replica1 url=http://10.170.2.54:8983/solr Our versions are newer. ourLowThreshold=14489007407885189... 2013-10-29T16:25:42.798Z [myappstaging-alpha4-adam_v8_shard1_replica1] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=... 2013-10-29T16:25:42.798Z PeerSync: core=myappstaging-alpha4-adam_v8_shard1_replica1 url=http://10.170.2.54:8983/solr START replicas=[http://10.136.6.24:8983/solr/myappstaging-... 2013-10-29T16:25:42.782Z Sync Success - now sync replicas to me 2013-10-29T16:25:42.782Z PeerSync: core=myappstaging-alpha4-adam_v8_shard1_replica2 url=http://10.136.6.24:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.782Z http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_replica2/: try and ask http://10.170.2.54:8983/solr/myappstaging-alpha4-adam_v8_shard1... 2013-10-29T16:25:42.782Z PeerSync: core=myappstaging-alpha4-adam_v8_shard1_replica2 url=http://10.136.6.24:8983/solr Our versions are newer. ourLowThreshold=14489007407885189... 2013-10-29T16:25:42.781Z PeerSync: core=myappstaging-alpha4-adam_v8_shard1_replica2 url=http://10.136.6.24:8983/solr Received 10 versions from 10.170.2.54:8983/solr/myappstag... 2013-10-29T16:25:42.765Z PeerSync: core=myappstaging-alpha4-adam_v8_shard1_replica2 url=http://10.136.6.24:8983/solr START replicas=[http://10.170.2.54:8983/solr/myappstaging-... 2013-10-29T16:25:42.754Z Checking if I should try and be the leader. 2013-10-29T16:25:42.754Z I may be the new leader - try and sync 2013-10-29T16:25:42.754Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-alpha4-adam_v8_shard1_replica2/ 2013-10-29T16:25:42.754Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:42.754Z Stopping recovery for zkNodeName=core_node2core=myappstaging-alpha4-adam_v8_shard1_replica2 2013-10-29T16:25:42.749Z Running the leader process for shard shard1 2013-10-29T16:25:42.693Z makePath: /collections/myappstaging-preetalpha5_v9/leaders/shard1 2013-10-29T16:25:42.693Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_replica2/ shard1 2013-10-29T16:25:42.692Z PeerSync: core=myappstaging-preetalpha5_v9_shard1_replica1 url=http://10.170.2.54:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.692Z http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_replica2/: sync completed with http://10.170.2.54:8983/solr/myappstaging-preetalpha5_... 2013-10-29T16:25:42.692Z [myappstaging-preetalpha5_v9_shard1_replica1] webapp=/solr path=/get params={sync=http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_repl... 2013-10-29T16:25:42.618Z [myappstaging-preetalpha5_v9_shard1_replica2] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=... 2013-10-29T16:25:42.603Z PeerSync: core=myappstaging-preetalpha5_v9_shard1_replica1 url=http://10.170.2.54:8983/solr Received 10 versions from 10.136.6.24:8983/solr/myappstag... 2013-10-29T16:25:42.603Z PeerSync: core=myappstaging-preetalpha5_v9_shard1_replica1 url=http://10.170.2.54:8983/solr Our versions are newer. ourLowThreshold=14499085375829442... 2013-10-29T16:25:42.596Z PeerSync: core=myappstaging-preetalpha5_v9_shard1_replica1 url=http://10.170.2.54:8983/solr START replicas=[http://10.136.6.24:8983/solr/myappstaging-... 2013-10-29T16:25:42.579Z [myappstaging-preetalpha5_v9_shard1_replica1] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=... 2013-10-29T16:25:42.565Z http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_replica2/: try and ask http://10.170.2.54:8983/solr/myappstaging-preetalpha5_v9_shard1... 2013-10-29T16:25:42.565Z Sync Success - now sync replicas to me 2013-10-29T16:25:42.565Z PeerSync: core=myappstaging-preetalpha5_v9_shard1_replica2 url=http://10.136.6.24:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.553Z PeerSync: core=myappstaging-preetalpha5_v9_shard1_replica2 url=http://10.136.6.24:8983/solr Our versions are newer. ourLowThreshold=14499085375829442... 2013-10-29T16:25:42.553Z PeerSync: core=myappstaging-preetalpha5_v9_shard1_replica2 url=http://10.136.6.24:8983/solr Received 10 versions from 10.170.2.54:8983/solr/myappstag... 2013-10-29T16:25:42.549Z PeerSync: core=myappstaging-preetalpha5_v9_shard1_replica2 url=http://10.136.6.24:8983/solr START replicas=[http://10.170.2.54:8983/solr/myappstaging-... 2013-10-29T16:25:42.549Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-preetalpha5_v9_shard1_replica2/ 2013-10-29T16:25:42.549Z I may be the new leader - try and sync 2013-10-29T16:25:42.548Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:42.547Z Running the leader process for shard shard1 2013-10-29T16:25:42.547Z Checking if I should try and be the leader. 2013-10-29T16:25:42.505Z [myappstaging-alpha4_v8_shard1_replica2] webapp=/solr path=/get params={sync=http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/&... 2013-10-29T16:25:42.502Z makePath: /collections/myappstaging-alpha4_v8/leaders/shard1 2013-10-29T16:25:42.489Z [myappstaging-alpha4_v8_shard1_replica1] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} st... 2013-10-29T16:25:42.489Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/ shard1 2013-10-29T16:25:42.489Z http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/: sync completed with http://10.170.2.54:8983/solr/myappstaging-alpha4_v8_shard1_... 2013-10-29T16:25:42.488Z PeerSync: core=myappstaging-alpha4_v8_shard1_replica2 url=http://10.170.2.54:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.488Z PeerSync: core=myappstaging-alpha4_v8_shard1_replica2 url=http://10.170.2.54:8983/solr Our versions are newer. ourLowThreshold=1449121459772325888 ot... 2013-10-29T16:25:42.488Z PeerSync: core=myappstaging-alpha4_v8_shard1_replica2 url=http://10.170.2.54:8983/solr Received 1 versions from 10.136.6.24:8983/solr/myappstaging-al... 2013-10-29T16:25:42.411Z http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/: try and ask http://10.170.2.54:8983/solr/myappstaging-alpha4_v8_shard1_replica2/... 2013-10-29T16:25:42.411Z Sync Success - now sync replicas to me 2013-10-29T16:25:42.410Z PeerSync: core=myappstaging-alpha4_v8_shard1_replica2 url=http://10.170.2.54:8983/solr START replicas=[http://10.136.6.24:8983/solr/myappstaging-alpha... 2013-10-29T16:25:42.410Z [myappstaging-alpha4_v8_shard1_replica2] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} st... 2013-10-29T16:25:42.394Z PeerSync: core=myappstaging-alpha4_v8_shard1_replica1 url=http://10.136.6.24:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.394Z PeerSync: core=myappstaging-alpha4_v8_shard1_replica1 url=http://10.136.6.24:8983/solr Our versions are newer. ourLowThreshold=1449121459772325888 ot... 2013-10-29T16:25:42.394Z PeerSync: core=myappstaging-alpha4_v8_shard1_replica1 url=http://10.136.6.24:8983/solr Received 1 versions from 10.170.2.54:8983/solr/myappstaging-al... 2013-10-29T16:25:42.388Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) 2013-10-29T16:25:42.388Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322) 2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252) 2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65) 2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249) 2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556) 2013-10-29T16:25:42.387Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249) 2013-10-29T16:25:42.383Z <13>Oct 29 16:25:49 solr-p1 solr: [qtp739893596-11195] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:org.apache.zookeeper.KeeperExcept... 2013-10-29T16:25:42.371Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) 2013-10-29T16:25:42.371Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322) 2013-10-29T16:25:42.371Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556) 2013-10-29T16:25:42.370Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252) 2013-10-29T16:25:42.370Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249) 2013-10-29T16:25:42.370Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65) 2013-10-29T16:25:42.370Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249) 2013-10-29T16:25:42.369Z <13>Oct 29 16:25:49 solr-p1 solr: [qtp739893596-11186] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:org.apache.zookeeper.KeeperExcept... 2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65) 2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322) 2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) 2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556) 2013-10-29T16:25:42.362Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249) 2013-10-29T16:25:42.361Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249) 2013-10-29T16:25:42.361Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252) 2013-10-29T16:25:42.360Z <13>Oct 29 16:25:49 solr-p1 solr: [qtp739893596-11194] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:org.apache.zookeeper.KeeperExcept... 2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249) 2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65) 2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322) 2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) 2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249) 2013-10-29T16:25:42.352Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556) 2013-10-29T16:25:42.351Z <13>Oct 29 16:25:49 solr-p1 solr: [qtp739893596-11196] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:org.apache.zookeeper.KeeperExcept... 2013-10-29T16:25:42.351Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252) 2013-10-29T16:25:42.347Z According to ZK I (id=162489378735849528-10.170.2.54:8983_solr-n_0000000019) am no longer a leader. 2013-10-29T16:25:42.346Z null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json at org.apache.zookeeper.Ke... 2013-10-29T16:25:42.339Z null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json at org.apache.zookeeper.Ke... 2013-10-29T16:25:42.327Z PeerSync: core=myappstaging-alpha4_v8_shard1_replica1 url=http://10.136.6.24:8983/solr START replicas=[http://10.170.2.54:8983/solr/myappstaging-alpha... 2013-10-29T16:25:42.326Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-alpha4_v8_shard1_replica1/ 2013-10-29T16:25:42.324Z I may be the new leader - try and sync 2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) 2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: [qtp739893596-11192] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:org.apache.zookeeper.KeeperExcept... 2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322) 2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249) 2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249) 2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252) 2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556) 2013-10-29T16:25:42.320Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65) 2013-10-29T16:25:42.316Z Checking if I should try and be the leader. 2013-10-29T16:25:42.316Z null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json at org.apache.zookeeper.Ke... 2013-10-29T16:25:42.316Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:42.310Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkStateReader.updateAliases(ZkStateReader.java:556) 2013-10-29T16:25:42.310Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:322) 2013-10-29T16:25:42.310Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195) 2013-10-29T16:25:42.309Z null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json at org.apache.zookeeper.Ke... 2013-10-29T16:25:42.309Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249) 2013-10-29T16:25:42.309Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65) 2013-10-29T16:25:42.307Z Running the leader process for shard shard1 2013-10-29T16:25:42.306Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249) 2013-10-29T16:25:42.306Z <13>Oct 29 16:25:49 solr-p1 solr: #011at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252) 2013-10-29T16:25:42.300Z Overseer cannot talk to ZK 2013-10-29T16:25:42.300Z null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json at org.apache.zookeeper.Ke... 2013-10-29T16:25:42.299Z <13>Oct 29 16:25:49 solr-p1 solr: [qtp739893596-11191] ERROR org.apache.solr.servlet.SolrDispatchFilter - null:org.apache.zookeeper.KeeperExcept... 2013-10-29T16:25:42.299Z null:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /aliases.json at org.apache.zookeeper.Ke... 2013-10-29T16:25:42.298Z 2013-10-29T16:25:42.268Z makePath: /collections/myappstaging-feature-rulecompletion_v9/leaders/shard1 2013-10-29T16:25:42.263Z I am the new leader: http://10.136.6.24:8983/solr/myappstaging-feature-rulecompletion_v9_shard1_replica1/ shard1 2013-10-29T16:25:42.259Z [myappstaging-feature-rulecompletion_v9_shard1_replica2] webapp=/solr path=/get params={sync=http://10.136.6.24:8983/solr/myappstaging-feature-rulecom... 2013-10-29T16:25:42.259Z PeerSync: core=myappstaging-feature-rulecompletion_v9_shard1_replica2 url=http://10.170.2.54:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.251Z [myappstaging-feature-rulecompletion_v9_shard1_replica1] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&a... 2013-10-29T16:25:42.251Z PeerSync: core=myappstaging-feature-rulecompletion_v9_shard1_replica2 url=http://10.170.2.54:8983/solr Our versions are newer. ourLowThreshold=144962... 2013-10-29T16:25:42.251Z PeerSync: core=myappstaging-feature-rulecompletion_v9_shard1_replica2 url=http://10.170.2.54:8983/solr Received 4 versions from 10.136.6.24:8983/solr... 2013-10-29T16:25:42.251Z http://10.136.6.24:8983/solr/myappstaging-feature-rulecompletion_v9_shard1_replica1/: sync completed with http://10.170.2.54:8983/solr/myappstaging-f... 2013-10-29T16:25:42.247Z http://10.136.6.24:8983/solr/myappstaging-feature-rulecompletion_v9_shard1_replica1/: try and ask http://10.170.2.54:8983/solr/myappstaging-feature-ru... 2013-10-29T16:25:42.246Z Sync Success - now sync replicas to me 2013-10-29T16:25:42.246Z PeerSync: core=myappstaging-feature-rulecompletion_v9_shard1_replica1 url=http://10.136.6.24:8983/solr DONE. sync succeeded 2013-10-29T16:25:42.239Z PeerSync: core=myappstaging-feature-rulecompletion_v9_shard1_replica2 url=http://10.170.2.54:8983/solr START replicas=[http://10.136.6.24:8983/solr/hi... 2013-10-29T16:25:42.236Z [myappstaging-feature-rulecompletion_v9_shard1_replica2] webapp=/solr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&a... 2013-10-29T16:25:42.236Z Connection with ZooKeeper reestablished. 2013-10-29T16:25:42.236Z publishing core=myappstaging-alpha3_v7_shard1_replica2 state=down 2013-10-29T16:25:42.232Z PeerSync: core=myappstaging-feature-rulecompletion_v9_shard1_replica1 url=http://10.136.6.24:8983/solr Our versions are newer. ourLowThreshold=144962... 2013-10-29T16:25:42.232Z PeerSync: core=myappstaging-feature-rulecompletion_v9_shard1_replica1 url=http://10.136.6.24:8983/solr Received 4 versions from 10.170.2.54:8983/solr... 2013-10-29T16:25:42.231Z PeerSync: core=myappstaging-feature-rulecompletion_v9_shard1_replica1 url=http://10.136.6.24:8983/solr START replicas=[http://10.170.2.54:8983/solr/hi... 2013-10-29T16:25:42.230Z Client is connected to ZooKeeper 2013-10-29T16:25:42.230Z Sync replicas to http://10.136.6.24:8983/solr/myappstaging-feature-rulecompletion_v9_shard1_replica1/ 2013-10-29T16:25:42.230Z I may be the new leader - try and sync 2013-10-29T16:25:42.230Z My last published State was Active, it's okay to be the leader. 2013-10-29T16:25:42.229Z Watcher org.apache.solr.common.cloud.ConnectionManager@52e80740 name:ZooKeeperConnection Watcher:solr-z1.domain.com:2181,solr-z2.domain.com:2181,sol... 2013-10-29T16:25:42.223Z Checking if I should try and be the leader. 2013-10-29T16:25:42.222Z Waiting for client to connect to ZooKeeper 2013-10-29T16:25:42.211Z Running the leader process for shard shard1 2013-10-29T16:25:42.186Z Connection expired - starting a new one... 2013-10-29T16:25:42.186Z Our previous ZooKeeper session was expired. Attempting to reconnect to recover relationship with ZooKeeper... 2013-10-29T16:25:42.179Z Watcher fired on path: null state: Expired type None 2013-10-29T16:25:42.179Z Watcher fired on path: null state: Expired type None 2013-10-29T16:25:42.179Z Watcher fired on path: null state: Expired type None 2013-10-29T16:25:42.179Z Watcher fired on path: null state: Expired type None 2013-10-29T16:25:42.179Z Watcher org.apache.solr.common.cloud.ConnectionManager@52e80740 name:ZooKeeperConnection Watcher:solr-z1.domain.com:2181,solr-z2.domain.com:2181,sol... 2013-10-29T16:25:42.169Z [myappstaging-feature-rulecompletion2_v9_shard1_replica2] webapp=/solr path=/get params={sync=http://10.136.6.24:8983/solr/myappstaging-feature-ruleco... 2013-10-29T16:25:42.169Z PeerSync: core=myappstaging-feature-rulecompletion2_v9_shard1_replica2 url=http://10.170.2.54:8983/solr DONE. sync succeeded > Strange error condition with cloud replication not working quite right > ---------------------------------------------------------------------- > > Key: SOLR-5407 > URL: https://issues.apache.org/jira/browse/SOLR-5407 > Project: Solr > Issue Type: Bug > Affects Versions: 4.5 > Reporter: Nathan Neulinger > Labels: cloud, replication > > I have a clodu deployment of 4.5 on EC2. Architecture is 3 dedicated ZK > nodes, and a pair of solr nodes. I'll apologize in advance that this error > report is not going to have a lot of detail, I'm really hoping that the > scenario/description will trigger some "likely" possible explanation. > The situation I got into was that the server had decided to fail over, so my > app servers were all taking to what should have been the primary for most of > the shards/collections, but actually was the replica. > Here's where it gets odd - no errors being returned to the client code for > any of the searches or document updates - and the current primary server was > definitely receiving all of the updates - even though they were being > submitted to the inactive/replica node. (clients talking to solr-p1, which > was not primary at the time, and writes were being passed through to solr-r1, > which was primary at the time.) > All sounds good so far right? Except - the replica server at the time, > through which the writes were passing - never got any of those content > updates. It had an old unmodified copy of the index. > I restarted solr-p1 (was the replica at the time) - no change in behavior. > Behavior did not change until I killed and restarted the current primary > (solr-r1) to force it to fail over. > At that point, everything was all happy again and working properly. > Until this morning, when one of the developers provisioned a new collection, > which happened to put it's primary on solr-r1. Again, clients all pointing at > solr-p1. The developer reported that the documents were going into the index, > but not visible on the replica server. -- This message was sent by Atlassian JIRA (v6.1#6144) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org