hani-fouladgar opened a new pull request, #11218:
URL: https://github.com/apache/ozone/pull/11218
## What changes were proposed in this pull request?
When an SCM is added to (or removed from) an SCM-HA ring, the OM only learns
the new membership from `ozone.scm.nodes.<serviceId>` /
`ozone.scm.address.<serviceId>.<nodeId>` at startup. The OM's SCM block and
container failover proxy providers build their node list once in their
constructor, so reaching a newly added SCM previously required restarting the
OM. This makes SCM scale-out/migration disruptive for the OM.
Please describe your PR in detail:
**Approach**
Make the OM's SCM proxy providers reloadable and drive the reload through
the existing reconfiguration mechanism, so no restart is needed.
- `SCMFailoverProxyProviderBase.changeConfig()` (new): reloads the node list
and addresses from the (already-updated) configuration.
- `loadConfigs()` now builds the new node list / proxy-info map in
temporaries and commits them only after the whole config parses successfully.
Adding an SCM requires two properties (the node list and the new node's
address); they may be applied in either order, and if the node list is updated
first the reload throws and leaves the previous state intact so the operator
can retry.
- Cached proxies for removed nodes, or nodes whose address changed, are
stopped (`RPC.stopProxy`) so the next call dials the fresh address.
- The current-proxy pointer is kept valid: it re-syncs its index to the
rebuilt list, falling back to the first node if the node it referenced was
removed.
- `HAUtils`: added overloads of `getScmBlockClient` /
`getScmContainerClient` that accept a caller-supplied proxy provider, so the OM
can keep the reference needed to reload it.
- `ScmClient.reloadScmNodes()` (new): delegates to changeConfig() on the
block and container providers; a no-op when providers are absent (e.g.
mock-constructed clients).
- `OzoneManager`: constructs and retains the block/container proxy
providers, passes them to `ScmClient`, and — only when an SCM service id is
configured (SCM HA) — registers `ozone.scm.nodes.<serviceId>` and, as a prefix,
`ozone.scm.address.<serviceId>.` as reconfigurable. The `reconfScmNodes`
callback rejects an empty node list and otherwise triggers `reloadScmNodes()`.
The address keys are registered as a prefix because a newly added node's key
does not exist at startup and cannot be registered by name in advance.
This mirrors the existing datanode-side SCM reconfiguration (HDDS-16.x,
`TestDatanodeSCMNodesReconfiguration`) and keeps service boundaries intact —
the OM only reloads its own client proxies.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16300
## How was this patch tested?
**Testing**
- `TestSCMFailoverProxyProviderChangeConfig` (unit): `changeConfig()` adds
a node, removes a node while keeping the current-proxy pointer valid, and fails
cleanly (leaving state intact) when a referenced node's address is missing.
- `TestScmClient` (unit): `reloadScmNodes()` delegates to both providers;
no-op when providers are null.
- `TestOmSCMNodesReconfiguration` (integration, SCM-HA
`MiniOzoneCluster`): the SCM node list and per-node address prefix are
reconfigurable on a running OM; an empty node list is rejected; a valid
reconfigure drives the proxy reload end-to-end.
**Manual Testing**
1) Manually shrink the SCM node list 3 -> 2
```
bash-5.1$ sed -i 's/scm1,scm2,scm3/scm1,scm2/'
/etc/hadoop/ozone-site.xmlp/ozone-site.xml
```
2) Get the value of `ozone.scm.nodes.scmservice` from OM (in memory)
```
bash-5.1$ curl -s http://localhost:9874/conf | tr -d '\n' | grep -oE
'<name>ozone\.scm\.(address|nodes)\.scmservice</name>\s*<value>[^<]+</value>'
<name>ozone.scm.nodes.scmservice</name><value>scm1,scm2,scm3</value>
```
3) Trigger `reconfig` the OM to get the new value from the config
```
bash-5.1$ ozone admin reconfig --service OM --address om1:9862 start
OM: Started reconfiguration task on node [om1:9862].
```
4) Check the `reconfig` status
```
bash-5.1$ ozone admin reconfig --service OM --address om1:9862 status 2>&1 |
grep -vE "TracingUtil|Sampling"
OM: Reconfiguring status for node [om1:9862]: started at Thu Sep 03 20:53:38
UTC 2026 and finished at Thu Sep 03 20:53:38 UTC 2026.
SUCCESS: Changed property ozone.scm.nodes.scmservice
From: "scm1,scm2,scm3"
To: "scm1,scm2"
```
5) Get the value of `ozone.scm.nodes.scmservice` from OM (in memory)
```
bash-5.1$ curl -s http://localhost:9874/conf | tr -d '\n' | grep -oE
'<name>ozone\.scm\.(address|nodes)\.scmservice</name>\s*<value>[^<]+</value>'
<name>ozone.scm.nodes.scmservice</name><value>scm1,scm2</value>
```
6) Add `scm3` back
```
bash-5.1$ sed -i 's/scm1,scm2/scm1,scm2,scm3/' /etc/hadoop/ozone-site.xml
```
7) Get the value of `ozone.scm.nodes.scmservice` from OM (in memory)
```
bash-5.1$ curl -s http://localhost:9874/conf | tr -d '\n' | grep -oE
'<name>ozone\.scm\.(address|nodes)\.scmservice</name>\s*<value>[^<]+</value>'
<name>ozone.scm.nodes.scmservice</name><value>scm1,scm2</value>
```
8) Trigger `reconfig` the OM to get the new value from the config
```
bash-5.1$ ozone admin reconfig --service OM --address om1:9862 start
OM: Started reconfiguration task on node [om1:9862].
```
9) Check the `reconfig` status
```
bash-5.1$ ozone admin reconfig --service OM --address om1:9862 status 2>&1 |
grep -vE "TracingUtil|Sampling"
OM: Reconfiguring status for node [om1:9862]: started at Thu Sep 03 21:01:59
UTC 2026 and finished at Thu Sep 03 21:01:59 UTC 2026.
SUCCESS: Changed property ozone.scm.nodes.scmservice
From: "scm1,scm2"
To: "scm1,scm2,scm3"
```
10) Get the value of `ozone.scm.nodes.scmservice` from OM (in memory)
curl -s http://localhost:9874/conf | tr -d '\n' | grep -oE
'<name>ozone\.scm\.(address|nodes)\.scmservice</name>\s*<value>[^<]+</value>'
<name>ozone.scm.nodes.scmservice</name><value>scm1,scm2,scm3</value>
bash-5.1$
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
