jojochuang opened a new pull request, #8531: URL: https://github.com/apache/ozone/pull/8531
## What changes were proposed in this pull request? Provide a one-liner summary of the changes in the PR **Title** field above. It should be in the form of `HDDS-1234. Short summary of the change`. Please describe your PR in detail: * Generated-by: Google Geimini Pro 2.5 (Preview) with the following prompt: ``` Read Ozone Distcp user doc https://ozone.apache.org/docs/edge/integration/distcp.html Create an updated user doc with the following instructions Be succinct, update and integrate with the existing text rather than overwrite the existing text Generate Markdown source output. Delegation Token Issues If a command fails due to being unable to retrieve a delegation token from the destination cluster (indicated by “OzoneToken” in the error output), add the following configuration to core-site.xml or ozone-site.xml: <property> <name>ozone.security.enabled</name> <value>true</value> </property> Cross-Realm Kerberos Affected Versions: For Ozone 1.x, issuing commands across clusters in different Kerberos realms may produce the following error: # hdfs dfs -ls ofs://ozone1707264383/ 24/02/07 18:47:36 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.io.IOException: DestHost:destPort ccycloud-1.weichiu-dst.root.comops.site:9862, LocalHost:localPort ccycloud-1.weichiu-src.local/10.140.99.144:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: om/[email protected], expecting: OM/ccycloud-1.weichiu-dst.local@REALM, while invoking $Proxy10.submitRequest over nodeId=om26,nodeAddress=ccycloud-1.weichiu-dst.local:9862 after 3 failover attempts. Trying to failover immediately. Cause: This occurs because the ozone.om.kerberos.principal property is not defined correctly. This issue is fixed in Ozone 2.0. Workaround: To resolve this, add the following property to ozone-site.xml: <property> <name>ozone.om.kerberos.principal.pattern</name> <value>*</value> </property> Fix: This bug is addressed by HDDS-10328 in Ozone 2.0. Bidirectional Cross-Realm Trust Environment In environments with bidirectional cross-realm trust, the command may fail with a token renewal error such as: 24/02/08 00:35:00 ERROR tools.DistCp: Exception encountered java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1707350431298_0001 to YARN: Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: 10.140.99.144:8020, Ident: (token for systest: HDFS_DELEGATION_TOKEN [email protected], renewer=yarn, realUser=, issueDate=1707352474394, maxDate=1707957274394, sequenceNumber=44, masterKeyId=14) Solution: Add the following parameter to prevent the DistCp job from attempting to renew the remote Ozone delegation token: -Dmapreduce.job.hdfs-servers.token-renewal.exclude=ozone1707264383 Example: If running the command on the destination cluster, use the following syntax: hadoop distcp \ -Dmapreduce.job.hdfs-servers.token-renewal.exclude=ccycloud-1.weichiu-src.root.comops.site \ -Ddfs.checksum.combine.mode=COMPOSITE_CRC \ -Dozone.client.checksum.type=CRC32C \ hdfs://ccycloud-1.weichiu-src.root.comops.site:8020/tmp/ \ ofs://ozone1707264383/tmp/dest ``` ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-11967 ## How was this patch tested? User doc only -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
