autokctrl-team opened a new issue, #847:
URL: https://github.com/apache/solr-operator/issues/847
# Environment
- solr-operator built from `ed5c5c7d28a4c1189d19f581259e05385c0d4b20`
- Solr 9.7.0
- Kubernetes: kind
- SolrCloud: 2 pods, `dataStorage: ephemeral`, `updateStrategy.method:
Managed`
# What happened
I edited one field on a healthy SolrCloud:
```yaml
spec:
solrAddressability:
podPort: 8983 # -> 8984
```
That is a pod template change, so the operator correctly started a managed
rolling update. But it deleted the pods **without migrating any replicas
first**.
The operator log shows the delete happening five times:
```
ManagedUpdateSelector ... Deleting solr pod for update ... "pod":
"t4-solrcloud-1"
```
and zero migration activity for those pods — no `Migrating all replicas off
of pod before deletion`, no `ReplicaMigrationStarted` event, no
`move-replicas-*` async request. Nothing was logged as an error. As far as the
operator was concerned the rolling update succeeded.
# Where the source code is wrong
A running pod registers its Solr node name in ZooKeeper using the `podPort`
value from the pod template at creation time. After the CR is updated, the pod
remains registered as `...:8983_solr`.
On each reconcile, the operator constructs the lookup key from the current
CR using
[`SolrNodeName`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/util/solr_update_util.go#L527-L529).
After the update, this function returns `...:8984_solr`. The registered node
name and the operator's lookup key therefore no longer match.
So the lookup in
[`PodContents`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/util/solr_update_util.go#L112-L115)
misses every running pod:
```go
// controllers/util/solr_update_util.go:112-115
contents, isInClusterState = state.NodeContents[SolrNodeName(cloud, podName)]
```
That failed lookup is immediately converted to `podHasReplicas=false`:
```go
// controllers/util/solr_update_util.go:118-121
func (state NodeReplicaState) PodHasReplicas(cloud *solr.SolrCloud, podName
string) bool {
contents, isInClusterState := state.PodContents(cloud, podName)
return isInClusterState && contents.replicas > 0
}
```
[`EvictReplicasForPodIfNecessary`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/util/solr_update_util.go#L551-L602)
treats this `false` as proof that there are no replicas. It skips
`REPLACENODE` and allows the pod to be deleted:
```go
// controllers/util/solr_update_util.go:569-591
if podHasReplicas {
// Submit new Replace Node request
...
} else {
canDeletePod = true
}
```
[`DeletePodForUpdate`](https://github.com/apache/solr-operator/blob/ed5c5c7d28a4c1189d19f581259e05385c0d4b20/controllers/solr_pod_lifecycle_util.go#L49-L112)
then deletes the pod without rechecking the cluster state:
```go
// controllers/solr_pod_lifecycle_util.go:90-106
if podHasReplicas {
requeueAfterDuration = time.Millisecond * 10
} else {
deletePod = true
}
...
if deletePod {
err = r.Delete(ctx, pod, client.Preconditions{UID: &pod.UID})
}
```
The bug is that "node name not found" is treated as "pod has no replicas."
After `podPort` changes, the lookup key is wrong, but the operator still skips
migration and deletes the pod.
Because the same edit is a pod template change, this repeats for every pod
in the StatefulSet.
# Expected behavior
A missed lookup should not be treated as proof that the pod has no replicas.
The operator should distinguish between these cases:
- The pod is present in the cluster state and has no replicas: it is safe to
delete.
- The pod is not found in the cluster state: do not delete it until its
replica state can be determined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
