[jira] [Updated] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage
[ https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-7035: Attachment: HDFS-7035.003.patch Update patch to address failures. > Refactor DataStorage and BlockSlicePoolStorage > --- > > Key: HDFS-7035 > URL: https://issues.apache.org/jira/browse/HDFS-7035 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.5.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, > HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch, > HDFS-7035.003.patch, HDFS-7035.003.patch > > > {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. > This jira extracts the common part of these two classes to simplify the logic > for both. > This is the ground work for handling partial failures during hot swapping > volumes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7035) Refactor DataStorage and BlockSlicePoolStorage
[ https://issues.apache.org/jira/browse/HDFS-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132694#comment-14132694 ] Hadoop QA commented on HDFS-7035: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668558/HDFS-7035.003.patch against trunk revision 98588cf. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.TestBPOfferService org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestDataTransferKeepalive {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8016//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8016//console This message is automatically generated. > Refactor DataStorage and BlockSlicePoolStorage > --- > > Key: HDFS-7035 > URL: https://issues.apache.org/jira/browse/HDFS-7035 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.5.0 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Attachments: HDFS-7035.000.combo.patch, HDFS-7035.000.patch, > HDFS-7035.001.combo.patch, HDFS-7035.001.patch, HDFS-7035.002.patch, > HDFS-7035.003.patch, HDFS-7035.003.patch > > > {{DataStorage}} and {{BlockPoolSliceStorage}} share many similar code path. > This jira extracts the common part of these two classes to simplify the logic > for both. > This is the ground work for handling partial failures during hot swapping > volumes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client
[ https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132875#comment-14132875 ] Allen Wittenauer commented on HDFS-6994: bq. Boost is already a configure option if the compiler support the features required in libhdfs3. This shouldn't be an option at all. If the compiler doesn't support the C++ bits needed, then the compile should fail. Boost shouldn't be referenced at all, anywhere, in any part of Hadoop. This is a library that has a large history of support issues and we have enough problems in Hadoop on the support side as it is. > libhdfs3 - A native C/C++ HDFS client > - > > Key: HDFS-6994 > URL: https://issues.apache.org/jira/browse/HDFS-6994 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs-client >Reporter: Zhanwei Wang > Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch > > > Hi All > I just got the permission to open source libhdfs3, which is a native C/C++ > HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol. > libhdfs3 provide the libhdfs style C interface and a C++ interface. Support > both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos > authentication. > libhdfs3 is currently used by HAWQ of Pivotal > I'd like to integrate libhdfs3 into HDFS source code to benefit others. > You can find libhdfs3 code from github > https://github.com/PivotalRD/libhdfs3 > http://pivotalrd.github.io/libhdfs3/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132981#comment-14132981 ] Gera Shegalov commented on HDFS-6888: - Thanks, [~airbots] for updating the patch. +1 (non-binding) Two nits: debugCmdSet can be declared as a general Set {code} private Set debugCmdSet = new HashSet(); {code} Its initialization can be more brief: {code} debugCmdSet.addAll(Arrays.asList(conf.getTrimmedStrings( DFSConfigKeys.DFS_AUDIT_LOG_DEBUG_CMDLIST))); {code} > Remove audit logging of getFIleInfo() > - > > Key: HDFS-6888 > URL: https://issues.apache.org/jira/browse/HDFS-6888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.5.0 >Reporter: Kihwal Lee >Assignee: Chen He > Labels: log > Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch > > > The audit logging of getFileInfo() was added in HDFS-3733. Since this is a > one of the most called method, users have noticed that audit log is now > filled with this. Since we now have HTTP request logging, this seems > unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7059) HAadmin transtionToActive with forceActive option can show confusing message.
[ https://issues.apache.org/jira/browse/HDFS-7059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133014#comment-14133014 ] Rushabh S Shah commented on HDFS-7059: -- Ran multiple times the failed test. Passing all the times on my local setup. Might be some intermittent test failures. > HAadmin transtionToActive with forceActive option can show confusing message. > - > > Key: HDFS-7059 > URL: https://issues.apache.org/jira/browse/HDFS-7059 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-7059.patch > > > Ran into this confusing message on our local HA setup. > One of the namenode was down and the other was in standby mode. > The namenode was not able to come out of safe mode so we did > transitionToActive with forceActive switch enabled. > Due to change in HDFS-2949, it will try connecting to all the namenode to > see whether they are active or not. > But since the other namenode is down it will try connect to that namenode for > 'ipc.client.connect.max.retries' number of times. > Every time it is not able to connect, it will log a message : > INFO ipc.Client: Retrying connect to server: . Already tried 0 > time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, > sleepTime=1000 MILLISECONDS) > Since in our configuration, the number of retries is 50, it will show this > message 50 times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6799) The invalidate method in SimulatedFSDataset.java failed to remove (invalidate) blocks from the file system.
[ https://issues.apache.org/jira/browse/HDFS-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Megasthenis Asteris updated HDFS-6799: -- Attachment: (was: HDFS-6799.patch) > The invalidate method in SimulatedFSDataset.java failed to remove > (invalidate) blocks from the file system. > --- > > Key: HDFS-6799 > URL: https://issues.apache.org/jira/browse/HDFS-6799 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, test >Affects Versions: 2.4.1 >Reporter: Megasthenis Asteris >Assignee: Megasthenis Asteris >Priority: Minor > > The invalidate(String bpid, Block[] invalidBlks) method in > SimulatedFSDataset.java should remove all invalidBlks from the simulated file > system. It currently fails to do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6799) The invalidate method in SimulatedFSDataset.java failed to remove (invalidate) blocks from the file system.
[ https://issues.apache.org/jira/browse/HDFS-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Megasthenis Asteris updated HDFS-6799: -- Attachment: HDFS-6799.patch > The invalidate method in SimulatedFSDataset.java failed to remove > (invalidate) blocks from the file system. > --- > > Key: HDFS-6799 > URL: https://issues.apache.org/jira/browse/HDFS-6799 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, test >Affects Versions: 2.4.1 >Reporter: Megasthenis Asteris >Assignee: Megasthenis Asteris >Priority: Minor > Attachments: HDFS-6799.patch > > > The invalidate(String bpid, Block[] invalidBlks) method in > SimulatedFSDataset.java should remove all invalidBlks from the simulated file > system. It currently fails to do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6799) The invalidate method in SimulatedFSDataset.java failed to remove (invalidate) blocks from the file system.
[ https://issues.apache.org/jira/browse/HDFS-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133053#comment-14133053 ] Megasthenis Asteris commented on HDFS-6799: --- The patch has been updated. > The invalidate method in SimulatedFSDataset.java failed to remove > (invalidate) blocks from the file system. > --- > > Key: HDFS-6799 > URL: https://issues.apache.org/jira/browse/HDFS-6799 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, test >Affects Versions: 2.4.1 >Reporter: Megasthenis Asteris >Assignee: Megasthenis Asteris >Priority: Minor > Attachments: HDFS-6799.patch > > > The invalidate(String bpid, Block[] invalidBlks) method in > SimulatedFSDataset.java should remove all invalidBlks from the simulated file > system. It currently fails to do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6799) The invalidate method in SimulatedFSDataset.java failed to remove (invalidate) blocks from the file system.
[ https://issues.apache.org/jira/browse/HDFS-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133062#comment-14133062 ] Hadoop QA commented on HDFS-6799: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668628/HDFS-6799.patch against trunk revision 98588cf. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8017//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8017//console This message is automatically generated. > The invalidate method in SimulatedFSDataset.java failed to remove > (invalidate) blocks from the file system. > --- > > Key: HDFS-6799 > URL: https://issues.apache.org/jira/browse/HDFS-6799 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, test >Affects Versions: 2.4.1 >Reporter: Megasthenis Asteris >Assignee: Megasthenis Asteris >Priority: Minor > Attachments: HDFS-6799.patch > > > The invalidate(String bpid, Block[] invalidBlks) method in > SimulatedFSDataset.java should remove all invalidBlks from the simulated file > system. It currently fails to do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6799) The invalidate method in SimulatedFSDataset.java failed to remove (invalidate) blocks from the file system.
[ https://issues.apache.org/jira/browse/HDFS-6799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133066#comment-14133066 ] Benoy Antony commented on HDFS-6799: +1. I intend to commit this on monday if there are no further comments. > The invalidate method in SimulatedFSDataset.java failed to remove > (invalidate) blocks from the file system. > --- > > Key: HDFS-6799 > URL: https://issues.apache.org/jira/browse/HDFS-6799 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, test >Affects Versions: 2.4.1 >Reporter: Megasthenis Asteris >Assignee: Megasthenis Asteris >Priority: Minor > Attachments: HDFS-6799.patch > > > The invalidate(String bpid, Block[] invalidBlks) method in > SimulatedFSDataset.java should remove all invalidBlks from the simulated file > system. It currently fails to do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7037) distcp via hftp fails in a similar way as webhdfs reported by HDFS-6776
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-7037: Description: Issuing "distcp hftp:// hdfs://" gave the following failure exception: {code} 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing remote token: java.io.IOException: Error when dealing remote token: Internal Server Error at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:248) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) 14/09/13 22:07:40 WARN security.UserGroupInformation: PriviledgedActionException as:hadoopu...@ent.cloudera.com (auth:KERBEROS) cause:java.io.IOException: Unable to obtain remote token 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered java.io.IOException: Unable to obtain remote token at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:248) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) Caused by: java.io.IOException: Error when dealing remote token: Internal Server Error at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationToke
[jira] [Updated] (HDFS-7037) distcp via hftp fails in a similar way as webhdfs reported by HDFS-6776
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-7037: Description: Issuing "distcp hftp:// hdfs://" gave the following failure exception: {quote} 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing remote token: java.io.IOException: Error when dealing remote token: Internal Server Error at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:248) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) 14/09/13 22:07:40 WARN security.UserGroupInformation: PriviledgedActionException as:hadoopu...@ent.cloudera.com (auth:KERBEROS) cause:java.io.IOException: Unable to obtain remote token 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered java.io.IOException: Unable to obtain remote token at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:248) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) Caused by: java.io.IOException: Error when dealing remote token: Internal Server Error at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTok
[jira] [Updated] (HDFS-7037) distcp via hftp fails in a similar way as webhdfs reported by HDFS-6776
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-7037: Description: Issuing "distcp hftp:// hdfs://" gave the following failure exception: {code} 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing remote token: java.io.IOException: Error when dealing remote token: Internal Server Error at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:248) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) 14/09/13 22:07:40 WARN security.UserGroupInformation: PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) cause:java.io.IOException: Unable to obtain remote token 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered java.io.IOException: Unable to obtain remote token at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:248) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) Caused by: java.io.IOException: Error when dealing remote token: Internal Server Error at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.
[jira] [Updated] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-7037: Attachment: HDFS-7037.001.patch > Using distcp to copy data from insecure to secure cluster via hftp doesn't > work (branch-2 only) > > > Key: HDFS-7037 > URL: https://issues.apache.org/jira/browse/HDFS-7037 > Project: Hadoop HDFS > Issue Type: Bug > Components: security, tools >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-7037.001.patch > > > This is a branch-2 only issue since hftp is only supported there. > Issuing "distcp hftp:// hdfs://" gave the > following failure exception: > {code} > 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing > remote token: > java.io.IOException: Error when dealing remote token: Internal Server Error > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at > org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) > at > org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) > 14/09/13 22:07:40 WARN security.UserGroupInformation: > PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) > cause:java.io.IOException: Unable to obtain remote token > 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered > java.io.IOException: Unable to obtain remote token > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at > org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.
[jira] [Updated] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-7037: Description: This is a branch-2 only issue since hftp is only supported there. Issuing "distcp hftp:// hdfs://" gave the following failure exception: {code} 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing remote token: java.io.IOException: Error when dealing remote token: Internal Server Error at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:248) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) 14/09/13 22:07:40 WARN security.UserGroupInformation: PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) cause:java.io.IOException: Unable to obtain remote token 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered java.io.IOException: Unable to obtain remote token at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) at org.apache.hadoop.fs.Globber.glob(Globber.java:248) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) Caused by: java.io.IOException: Error when dealing remote token: Internal Server Error at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) at org.apache.hadoop.hdfs.
[jira] [Updated] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-7037: Affects Version/s: 2.6.0 Status: Patch Available (was: Open) Submitted patch 001. Thanks for reviewing. > Using distcp to copy data from insecure to secure cluster via hftp doesn't > work (branch-2 only) > > > Key: HDFS-7037 > URL: https://issues.apache.org/jira/browse/HDFS-7037 > Project: Hadoop HDFS > Issue Type: Bug > Components: security, tools >Affects Versions: 2.6.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-7037.001.patch > > > This is a branch-2 only issue since hftp is only supported there. > Issuing "distcp hftp:// hdfs://" gave the > following failure exception: > {code} > 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing > remote token: > java.io.IOException: Error when dealing remote token: Internal Server Error > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at > org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) > at > org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) > 14/09/13 22:07:40 WARN security.UserGroupInformation: > PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) > cause:java.io.IOException: Unable to obtain remote token > 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered > java.io.IOException: Unable to obtain remote token > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133106#comment-14133106 ] Yongjun Zhang commented on HDFS-7037: - I referenced HDFS-3905 when working on the patch of this jira. Thanks [~daryn] for the work in HDFS-3905. > Using distcp to copy data from insecure to secure cluster via hftp doesn't > work (branch-2 only) > > > Key: HDFS-7037 > URL: https://issues.apache.org/jira/browse/HDFS-7037 > Project: Hadoop HDFS > Issue Type: Bug > Components: security, tools >Affects Versions: 2.6.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-7037.001.patch > > > This is a branch-2 only issue since hftp is only supported there. > Issuing "distcp hftp:// hdfs://" gave the > following failure exception: > {code} > 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing > remote token: > java.io.IOException: Error when dealing remote token: Internal Server Error > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at > org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) > at > org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) > 14/09/13 22:07:40 WARN security.UserGroupInformation: > PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) > cause:java.io.IOException: Unable to obtain remote token > 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered > java.io.IOException: Unable to obtain remote token > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.glo
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133108#comment-14133108 ] Hadoop QA commented on HDFS-7037: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12668640/HDFS-7037.001.patch against trunk revision 14e2639. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8018//console This message is automatically generated. > Using distcp to copy data from insecure to secure cluster via hftp doesn't > work (branch-2 only) > > > Key: HDFS-7037 > URL: https://issues.apache.org/jira/browse/HDFS-7037 > Project: Hadoop HDFS > Issue Type: Bug > Components: security, tools >Affects Versions: 2.6.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-7037.001.patch > > > This is a branch-2 only issue since hftp is only supported there. > Issuing "distcp hftp:// hdfs://" gave the > following failure exception: > {code} > 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing > remote token: > java.io.IOException: Error when dealing remote token: Internal Server Error > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at > org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) > at > org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) > 14/09/13 22:07:40 WARN security.UserGroupInformation: > PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) > cause:java.io.IOException: Unable to obtain remote token > 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered > java.io.IOException: Unable to obtain remote token > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSyste
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133109#comment-14133109 ] Yongjun Zhang commented on HDFS-7037: - Since it's branch-2 only, it's expected that the patch won't apply at trunk. > Using distcp to copy data from insecure to secure cluster via hftp doesn't > work (branch-2 only) > > > Key: HDFS-7037 > URL: https://issues.apache.org/jira/browse/HDFS-7037 > Project: Hadoop HDFS > Issue Type: Bug > Components: security, tools >Affects Versions: 2.6.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-7037.001.patch > > > This is a branch-2 only issue since hftp is only supported there. > Issuing "distcp hftp:// hdfs://" gave the > following failure exception: > {code} > 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing > remote token: > java.io.IOException: Error when dealing remote token: Internal Server Error > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at > org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) > at > org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) > 14/09/13 22:07:40 WARN security.UserGroupInformation: > PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS) > cause:java.io.IOException: Unable to obtain remote token > 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered > java.io.IOException: Unable to obtain remote token > at > org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at > org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at > org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at > org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)