shuan1026 opened a new pull request, #48: URL: https://github.com/apache/ozone-helm-charts/pull/48
## What changes were proposed in this pull request? Follow-up to [PR #20 discussion](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2632054499): add a preStop hook / longer graceful termination so OM/SCM do not stop while still holding Ratis leadership. * Add `lifecycle.preStop` on OM/SCM StatefulSets (`replicas > 1`): if this pod is leader, run `ozone admin om|scm transfer -r` before shutdown. * Add `terminationGracePeriodSeconds` for HA OM/SCM (default path used **150s**) so a slow transfer + short sleep still finishes before SIGKILL. * The preStop script only does the leader transfer; it does **not** send `kill -TERM 1` as in the [PR #20 discussion](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2632054499) sample. kubelet sends SIGTERM to the container automatically once preStop returns, and the image's PID 1 (`dumb-init`) forwards it to the OM/SCM process, so an explicit `kill` would just duplicate that and adds a dependency on the entrypoint's PID 1. * Complements the existing pre-upgrade `om-leader-transfer-job` (scale-down + persistence); this covers ordinary pod delete / roll / drain. * Set `publishNotReadyAddresses: true` on the OM/SCM headless Services. During termination a terminating endpoint always has `ready: false`, so by default it is dropped from the headless Service DNS ([Kubernetes docs](https://kubernetes.io/docs/tutorials/services/pods-and-endpoint-termination-flow/)). The transferee then cannot resolve the outgoing leader's address to confirm the handoff, and `transfer -r` blocks until `TIMED_OUT(60s)`. Publishing not-ready addresses keeps the terminating pod resolvable so the transfer completes quickly (see [discussion](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r3619734581)). ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-14351 ## How was this patch tested? - Green CI run, and tested manually with leader transfer and basic ozone commands. * kind, OM HA (`om.replicas=3`): over repeated leader pod-delete cycles, preStop transfers leadership and quorum consistently recovers to a single LEADER. * kind, SCM HA (`scm.replicas=3`): same repeated leader pod-delete cycles, with the same result. * Compared with `main` (both OM and SCM): no preStop / default 30s grace; deleting the leader does **not** attempt transfer (failover is only Raft election after SIGTERM). Manually Test: ```bash kind create cluster --name ozone-helm-prestop # OM HA (default) helm upgrade --install ozone charts/ozone --wait --timeout 15m kubectl exec ozone-om-0 -c om -- ozone admin om roles -id cluster1 kubectl delete pod ozone-om-1 # delete current LEADER kubectl get events --sort-by='.lastTimestamp' | grep -iE 'preStop|Killing' kubectl exec ozone-om-0 -c om -- ozone admin om roles -id cluster1 # expect: single LEADER; transfer done in preStop (no TIMED_OUT) # SCM HA helm upgrade --install ozone charts/ozone --set scm.replicas=3 --wait --timeout 20m kubectl exec ozone-scm-0 -c scm -- ozone admin scm roles -id cluster1 kubectl delete pod ozone-scm-1 # delete current LEADER kubectl get events --sort-by='.lastTimestamp' | grep -iE 'preStop|Killing' kubectl exec ozone-scm-0 -c scm -- ozone admin scm roles -id cluster1 # expect: single LEADER; transfer done in preStop ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
