shuan1026 opened a new pull request, #42:
URL: https://github.com/apache/ozone-helm-charts/pull/42
## What changes were proposed in this pull request?
This change adds a `startupProbe` for HA OM/SCM deployments so longer
initialization is not treated as a liveness failure and does not restart the
container.
The chart currently relies on an HTTP `livenessProbe` against the UI port.
Under HA (`replicas > 1`), OM/SCM startup can take longer; a short liveness
window can kill the process while it is still bootstrapping.
A `startupProbe` addresses that failure mode: until startup succeeds,
kubelet does not apply liveness-based restarts for probe failures.
### In scope
* OM HA (`replicas > 1`): `startupProbe` via `ozone admin om roles`,
checking that the local hostname appears in the roles output
* SCM HA: `startupProbe` via `ozone admin scm roles` (same idea; hostname
match follows SCM roles format)
* **OM HA bootstrap wiring (required for roles startup to work):**
* `podManagementPolicy: Parallel` on the OM StatefulSet — avoids
OrderedReady stalling at only `om-0` while startup waits for peers
* `publishNotReadyAddresses: true` on the OM headless Service — peer DNS
must exist before startup passes; default headless only publishes Ready pods,
and roles startup keeps pods NotReady until the probe succeeds
* Probe timings from kind testing: `timeoutSeconds` must be larger than the
sample value of `5`, because `ozone admin * roles` often blocks while peers /
DNS / Ratis are not ready yet
### Out of scope (moved to
[[HDDS-15874](https://issues.apache.org/jira/browse/HDDS-15874)])
* `readinessProbe`
### Why readiness is not in this PR
The Jira issue and [[reviewer
sample](https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2631983830)])
included both `startupProbe` and `readinessProbe`. After kind testing,
readiness was split to
**[[HDDS-15874](https://issues.apache.org/jira/browse/HDDS-15874)]**; this PR
delivers startup only.
1. **Different problem, different tool**
The pain is *HA deployments take longer to initialize* → avoid premature
liveness restarts → **`startupProbe`**. A **`readinessProbe` does not fix
that**; it controls Service endpoint membership, not “do not kill me while
starting”.
2. **Historical concern for quorum OM may still applies**
[[HDDS-1668](https://issues.apache.org/jira/browse/HDDS-1668)] suggested
skipping readiness on the OM HA: Ready-gated Endpoints can fight peer DNS
needed at OM HA startup (`OzoneManager.loadOMHAConfigs`).
3. **Cost of roles-based readiness on kind**
Using `ozone admin * roles` as a **recurring** readiness probe has
ongoing cost (fork JVM/CLI + RPC). That is acceptable for a bounded startup
window; as a lifelong readiness check the cost is amplified.
### Follow-up
* **[[HDDS-15874](https://issues.apache.org/jira/browse/HDDS-15874)]**:
whether/how to add readiness (roles exec vs light `httpGet`, or document
deferral).
### Probe shape (illustrative; OM)
```yaml
startupProbe:
exec:
command:
- sh
- -c
- |
ozone admin om roles -id {{ .Values.clusterId }} 2>/dev/null \
| grep -q $(hostname) && exit 0 || exit 1
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 20 # sample value 5 is too low in practice
failureThreshold: 30 # multi-minute bootstrap window
```
SCM HA uses `ozone admin scm roles` with a hostname form that matches SCM
roles output (`$(hostname -f)` in testing).
Non-HA (`replicas == 1`) does not add a roles exec probe, to avoid assuming
multi-node Ratis roles output.
## What is the link to the Apache JIRA
* This issue (startup): https://issues.apache.org/jira/browse/HDDS-14349
* Readiness follow-up: https://issues.apache.org/jira/browse/HDDS-15874
* Parent: https://issues.apache.org/jira/browse/HDDS-14382
Review discussion:
https://github.com/apache/ozone-helm-charts/pull/20#discussion_r2631983830
## How was this patch tested?
### 1. kind — default install (OM HA)
On a fresh kind cluster:
```bash
helm upgrade --install ozone charts/ozone --wait --timeout 15m
```
Results:
* Release deployed; all pods Ready with no restarts
* During cold start, OM may show 1–2 Startup `timed out after 20s` events
while peers/roles are not ready; within `failureThreshold` the probe succeeds
and Ready is reached **without** repeated liveness Killing
* After Ready: `time ozone admin om roles -id cluster1` is about **2s**
(below `timeoutSeconds: 20`)
* Green CI run and manually verified leader transfer and basic ozone commands
### 2. kind — SCM HA (`scm.replicas=3`, fresh install)
On a **new** release/cluster (do not scale an existing `scm.replicas=1` in
place):
```bash
helm upgrade --install ozone charts/ozone \
--set scm.replicas=3 \
--wait --timeout 20m
```
Same pass criteria: Ready, no restart storm.
**Out of scope for this issue/PR:** `kubectl scale` or `helm upgrade --set
scm.replicas=3` migrating an existing **scm 1→3** deployment (known STS
revision / `addSCM` issues; track separately).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
