pyttel commented on PR #10: URL: https://github.com/apache/ozone-helm-charts/pull/10#issuecomment-3299392217
Sorry for the late response — I was overloaded with work. ^^ Yes, this is the default behavior of Kubernetes when handling scaling of ReplicaSets. I’m currently not aware of any way to manage this manually. The behavior seems to be implemented [here](https://github.com/kubernetes/kubernetes/blob/release-1.11/pkg/controller/controller_utils.go#L737). I’ve simply adapted to this strategy. There are some edge cases where it doesn’t fit (for example, with unready pods). To address this dynamically within Kubernetes, the only real solution would probably be a Kubernetes operator. The ports should be configurable dynamically — I can change that. I’ve been working with my changes in our own deployments for a few months now. Using Ozone there has helped me understand the implementation in more detail. The need for running Ozone Manager (OM) in HA may actually be obsolete. From my understanding: there is one master OM, while the others are just waiting for failover. But Kubernetes already takes care of this. If the OM pod/container crashes and good health checks are in place, Kubernetes will detect this and restart OM as a new pod/container. StatefulSets take care of persistence. Even more: Raft-based leader election is problematic in Kubernetes because services act as load balancers for the pods. If traffic is distributed round-robin, most requests to OM will fail with “not the master.” Of course, this can be mitigated with health checks and readiness probes. But it feels like Raft and Kubernetes are two concepts that don’t align well in this scenario. In our deployments, I’m currently running only a single OM. So we should discuss whether it really makes sense to have HA for OM (and possibly also for SCM) when running in Kubernetes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
