[ https://issues.apache.org/jira/browse/HDFS-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231194#comment-13231194 ]
Hudson commented on HDFS-3062: ------------------------------ Integrated in Hadoop-Mapreduce-trunk #1021 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1021/]) HDFS-3062. Fix bug which prevented MR job submission from creating delegation tokens on an HA cluster. Contributed by Mingjie Lai. (Revision 1301285) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301285 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java > Fail to submit mapred job on a secured-HA-HDFS: logic URI cannot be picked up > by job submission. > ------------------------------------------------------------------------------------------------ > > Key: HDFS-3062 > URL: https://issues.apache.org/jira/browse/HDFS-3062 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, security > Affects Versions: 0.24.0 > Reporter: Mingjie Lai > Assignee: Mingjie Lai > Priority: Critical > Fix For: 0.24.0, 0.23.3 > > Attachments: HDFS-3062-trunk-2.patch, HDFS-3062-trunk.patch > > > When testing the combination of NN HA + security + yarn, I found that the > mapred job submission cannot pick up the logic URI of a nameservice. > I have logic URI configured in core-site.xml > {code} > <property> > <name>fs.defaultFS</name> > <value>hdfs://ns1</value> > </property> > {code} > HDFS client can work with the HA deployment/configs: > {code} > [root@nn1 hadoop]# hdfs dfs -ls / > Found 6 items > drwxr-xr-x - hbase hadoop 0 2012-03-07 20:42 /hbase > drwxrwxrwx - yarn hadoop 0 2012-03-07 20:42 /logs > drwxr-xr-x - mapred hadoop 0 2012-03-07 20:42 /mapred > drwxr-xr-x - mapred hadoop 0 2012-03-07 20:42 /mr-history > drwxrwxrwt - hdfs hadoop 0 2012-03-07 21:57 /tmp > drwxr-xr-x - hdfs hadoop 0 2012-03-07 20:42 /user > {code} > but cannot submit a mapred job with security turned on > {code} > [root@nn1 hadoop]# /usr/lib/hadoop/bin/yarn --config ./conf jar > share/hadoop/mapreduce/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar > randomwriter out > Running 0 maps. > Job started: Wed Mar 07 23:28:23 UTC 2012 > java.lang.IllegalArgumentException: java.net.UnknownHostException: ns1 > at > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:431) > at > org.apache.hadoop.security.SecurityUtil.buildDTServiceName(SecurityUtil.java:312) > at > org.apache.hadoop.fs.FileSystem.getCanonicalServiceName(FileSystem.java:217) > at > org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119) > at > org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:97) > at > org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137) > at > org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:411) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:326) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221) > at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) > .... > {code}0.24 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira