[
https://issues.apache.org/jira/browse/HDDS-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-17762:
------------------------------------
Target Version/s: 2.2.3
> DataNode fails to register with a dynamically-added SCM on a Kerberos-enabled
> cluste
> ------------------------------------------------------------------------------------
>
> Key: HDDS-17762
> URL: https://issues.apache.org/jira/browse/HDDS-17762
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode, SCM HA
> Environment: Kerberos ({{ozone.security.enabled=true}},
> {{hadoop.security.authentication=kerberos}}). Does not reproduce with
> security disabled.
> Reporter: Mohammadhani Fouladgar
> Assignee: Mohammadhani Fouladgar
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> HDDS-13890 lets a running DataNode pick up a newly added SCM by reconfiguring
> {{ozone.scm.nodes.<serviceId>}} (via {{ozone admin reconfig
> --service=DATANODE}}) instead of requiring a restart. On a secure cluster
> this does not work: after the reconfig, the DataNode never registers with the
> newly added SCM(s), which therefore stay in SCM safemode ("No Datanodes
> registered with SCM yet"). Connections to the pre-existing SCM(s) are
> unaffected. Restarting the DataNode resolves it (it then registers with all
> SCMs)
> Steps to reproduce
> 1. Secure (Kerberized) Ozone cluster with SCM HA, 1 SCM initially, DataNodes
> registered and out of safemode.
> 2. Add new SCM(s) and bootstrap them into the Ratis ring (they appear in
> {{ozone admin scm roles}}).
> 3. Deliver the updated {{ozone.scm.nodes.<svc>}} +
> {{ozone.scm.address.<svc>.<nodeId>}} to DataNodes and run {{ozone admin
> reconfig --service=DATANODE <dn> start}}. The reconfig reports success.
> 4. Observe the new SCM(s) stay in safemode; DataNode log shows repeated auth
> failures to the new SCM(s).
> Actual behavior — DataNode log, repeating against the new SCM only:
> WARN ...EndpointStateMachineTaskThread-<newscm>:9861... ipc.Client:
> Exception encountered while connecting to the server <newscm>/<ip>:9861
> javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos tgt)]
> at ...SaslRpcClient.saslConnect(...)
> at
> ...ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.getVersion(...)
> at ...container.common.states.endpoint.VersionEndpointTask.call(...)
> Connections to the original SCM succeed; only the dynamically-added endpoint
> fails.
> Expected behavior — the DataNode authenticates to the newly added SCM using
> its service (keytab) identity and registers, exactly as it does for SCMs
> present at startup; the new SCM leaves safemode without a DataNode restart.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]