[ https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123232#comment-14123232 ]
Haohui Mai edited comment on HDFS-6776 at 9/5/14 5:47 PM: ---------------------------------------------------------- I've made it very clear that I'm opposed to put fragile hacks into {{WebHdfsFileSystem}}, though I'm okay if it's done at the application level (e.g. distcp). Unless this is addressed, I cannot give my +1. If you are not familiar with the distcp code, I'll take a look and see whether I can post a patch for it. was (Author: wheat9): I've made it very clear that I'm opposed to put fragile hacks into {{WebHdfsFileSystem}}, though I'm okay if it's done at the application level (e.g. distcp). Unless this is addressed, I cannot give my +1. If you don't want to take a look at the distcp code, I'll take a look and see whether I can post a patch for it. > distcp from insecure cluster (source) to secure cluster (destination) doesn't > work via webhdfs > ---------------------------------------------------------------------------------------------- > > Key: HDFS-6776 > URL: https://issues.apache.org/jira/browse/HDFS-6776 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 2.3.0, 2.5.0 > Reporter: Yongjun Zhang > Assignee: Yongjun Zhang > Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch, > HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch, > HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch, > HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch, > HDFS-6776.009.patch, HDFS-6776.010.patch, HDFS-6776.011.patch, > dummy-token-proxy.js > > > Issuing distcp command at the secure cluster side, trying to copy stuff from > insecure cluster to secure cluster, and see the following problem: > {code} > hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://<insure-cluster>:<port>/tmp > hdfs://<sure-cluster>:8020/tmp/tmptgt > 14/07/30 20:06:19 INFO tools.DistCp: Input Options: > DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, > ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', > copyStrategy='uniformsize', sourceFileListing=null, > sourcePaths=[webhdfs://<insecure-cluster>:<port>/tmp], > targetPath=hdfs://<secure-cluster>:8020/tmp/tmptgt, targetPathExists=true} > 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at > <secure-clister>:8032 > 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property > 'ssl.client.truststore.location' has not been set, no TrustStore will be > loaded > 14/07/30 20:06:20 WARN security.UserGroupInformation: > PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) > cause:java.io.IOException: Failed to get the token for hadoopuser, > user=hadoopuser > 14/07/30 20:06:20 WARN security.UserGroupInformation: > PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) > cause:java.io.IOException: Failed to get the token for hadoopuser, > user=hadoopuser > 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered > java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:781) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:796) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at > org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) > at > org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) > Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Failed > to get the token for hadoopuser, user=hadoopuser > at > org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:334) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:84) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:570) > ... 30 more > [hadoopuser@yjc5u-1 ~]$ > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)