Hi,
I run solr cloud on GKE; and I’m trying to move my pods to a new disk type. In
doing so the disk will be brand new. I’ve landed in a position that I’m unsure
how to recover from, where the new node is not syncing data from the leader.
To explain exactly what’s happening, lets say I have two nodes:
*
solr-0
*
solr-1
And both are active and fully replicated.
I take solr-1 down, and point it at the new disk (which is empty), and bring it
back up.
The server starts fine, I can access solr-1 via the UI, but it never recovers,
in the “Cloud -> Graph” UI, I can see the shard on solr-1 is down.
I can see it in the “Cloud -> Nodes” GUI as up, however its collections have a
funny state, for example: "postcodes-006_s1r9_(down): undefined", vs solr-0
which shows "postcodes-006_s1r11: 847.3Mb”.
I was expecting the node to come up and see its disk was empty, and resync its
data from the leader, but instead it’s just sat doing, nothing….
The fact I’m moving to new disks is somewhat moot, more broadly this is showing
me that if we lost data on a node for whatever reason, it doesn’t “fix itself”
- which I always (maybe blindly) assumed it would, because when I bring up
brand new nodes (different name) it does.
Could anyone advise what I’ve done wrong here, and what the process should be
to get a node to resend its data entirely?
This is what the API shows:
shard1":{
"range":"80000000-7fffffff",
"replicas":{
"core_node10":{
"core":"postcodes-006_shard1_replica_n9",
"node_name":"solr-1.search-solr-next.svc.cluster.local:80_solr",
"type":"NRT",
"state":"down",
"force_set_state":"false",
"base_url":"http://solr-1.search-solr-next.svc.cluster.local:80/solr";
},
"core_node12":{
"core":"postcodes-006_shard1_replica_n11",
"node_name":"solr-0.search-solr-next.svc.cluster.local:80_solr",
"type":"NRT",
"state":"active",
"leader":"true",
"force_set_state":"false",
"base_url":"http://solr-0.search-solr-next.svc.cluster.local:80/solr";,
"property.preferredleader":"true"
}
},
"state":"active",
"health":"ORANGE"
}
Unless expressly stated otherwise in this email, this e-mail is sent on behalf
of Auto Trader Limited Registered Office: 1 Tony Wilson Place, Manchester,
Lancashire, M15 4FN (Registered in England No. 03909628). Auto Trader Limited
is part of the Auto Trader Group Plc group. This email and any files
transmitted with it are confidential and may be legally privileged, and
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this email in error please notify the sender.
This email message has been swept for the presence of computer viruses.
