For example,when I type this command:
bin/hadoop distcp hdfs://hacluster:8020/mr-history/done/2013/02/18/000000/
hdfs://hacluster-slave:8020/mr-history/done/2013/02/18/000000/
hdfs://hacluster-slave is the destination HDFS cluster in HA mode. when it's
not configured for resourcemanager, distcp will fail with the following error:
13/02/25 12:00:27 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to run job : Unable to map logical nameservice URI
'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a
failover proxy provider configured.
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:384)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1212)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:148)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:113)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:368)
And RM's log has this:
2013-02-25 12:00:26,857 ERROR
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1235)
PriviledgedActionException as:mapred/had...@hadoop.com (auth:KERBEROS)
cause:java.io.IOException: Unable to map logical nameservice URI
'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a
failover proxy provider configured.
2013-02-25 12:00:26,857 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:290)
RMAppManager submit application exception
java.io.IOException: Unable to map logical nameservice URI
'hdfs://hacluster-slave' to a NameNode. Local configuration does not have a
failover proxy provider configured.
at
org.apache.hadoop.hdfs.DFSClient$Renewer.getNNProxy(DFSClient.java:875)
at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:840)
at org.apache.hadoop.security.token.Token.renew(Token.java:372)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:368)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$1.run(DelegationTokenRenewer.java:1)
... at
org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:280)
...
After I put remote cluster hacluster-slave's configraion into RM's
hdfs-site.xml file and restart RM, then distcp works.
My question is ,shall RM support reading configuration from client side when
doing delegation renew? Then no need to configure remote cluster configurations
and restart RM.
Thank you!
