[ 
https://issues.apache.org/jira/browse/HDDS-17762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Attila Doroszlai updated HDDS-17762:
------------------------------------
    Issue Type: Bug  (was: New Feature)

> DataNode fails to register with a dynamically-added SCM on a Kerberos-enabled 
> cluste
> ------------------------------------------------------------------------------------
>
>                 Key: HDDS-17762
>                 URL: https://issues.apache.org/jira/browse/HDDS-17762
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: Ozone Datanode, SCM HA
>         Environment: Kerberos ({{ozone.security.enabled=true}}, 
> {{hadoop.security.authentication=kerberos}}). Does not reproduce with 
> security disabled.
>            Reporter: Mohammadhani Fouladgar
>            Priority: Major
>              Labels: pull-request-available
>
> HDDS-13890 lets a running DataNode pick up a newly added SCM by reconfiguring 
> {{ozone.scm.nodes.<serviceId>}} (via {{ozone admin reconfig 
> --service=DATANODE}}) instead of requiring a restart. On a secure cluster 
> this does not work: after the reconfig, the DataNode never registers with the 
> newly added SCM(s), which therefore stay in SCM safemode ("No Datanodes 
> registered with SCM yet"). Connections to the pre-existing SCM(s) are 
> unaffected. Restarting the DataNode resolves it (it then registers with all 
> SCMs)
> Steps to reproduce
> 1. Secure (Kerberized) Ozone cluster with SCM HA, 1 SCM initially, DataNodes 
> registered and out of safemode.
> 2. Add new SCM(s) and bootstrap them into the Ratis ring (they appear in 
> {{ozone admin scm roles}}).
> 3. Deliver the updated {{ozone.scm.nodes.<svc>}} + 
> {{ozone.scm.address.<svc>.<nodeId>}} to DataNodes and run {{ozone admin 
> reconfig --service=DATANODE <dn> start}}. The reconfig reports success. 
> 4. Observe the new SCM(s) stay in safemode; DataNode log shows repeated auth 
> failures to the new SCM(s).
> Actual behavior — DataNode log, repeating against the new SCM only:
>  WARN  ...EndpointStateMachineTaskThread-<newscm>:9861... ipc.Client:
>      Exception encountered while connecting to the server <newscm>/<ip>:9861
>    javax.security.sasl.SaslException: GSS initiate failed
>      [Caused by GSSException: No valid credentials provided
>       (Mechanism level: Failed to find any Kerberos tgt)]
>        at ...SaslRpcClient.saslConnect(...)
>        at 
> ...ozone.protocolPB.StorageContainerDatanodeProtocolClientSideTranslatorPB.getVersion(...)
>        at ...container.common.states.endpoint.VersionEndpointTask.call(...)
> Connections to the original SCM succeed; only the dynamically-added endpoint 
> fails.
> Expected behavior — the DataNode authenticates to the newly added SCM using 
> its service (keytab) identity and registers, exactly as it does for SCMs 
> present at startup; the new SCM leaves safemode without a DataNode restart.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to