[ https://issues.apache.org/jira/browse/HDFS-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118411#comment-13118411 ]
Rajit Saha commented on HDFS-2392: ---------------------------------- A few warnings can be useful for debugging >From NN ======= 2011-09-30 18:58:11,920 WARN org.mortbay.log: /data/user/hadoopqa/input1/part-00000: java.io.IOException: Security enabled but user not authenticated by filter But the same user can do distcp over hdfs:// successfully > Dist with hftp is failing again > ------------------------------- > > Key: HDFS-2392 > URL: https://issues.apache.org/jira/browse/HDFS-2392 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node > Affects Versions: 0.20.205.0 > Reporter: Rajit Saha > Assignee: Daryn Sharp > Priority: Critical > > $ hadoop distcp hftp://<NN Hostname>:50070/user/hadoopqa/input1/part-00000 > /user/hadoopqa/out3 > 11/09/30 18:57:59 INFO tools.DistCp: srcPaths=[hftp://<NN > Hostname>:50070/user/hadoopqa/input1/part-00000] > 11/09/30 18:57:59 INFO tools.DistCp: destPath=/user/hadoopqa/out3 > 11/09/30 18:58:00 INFO security.TokenCache: Got dt for > hftp://<NN Hostname>:50070/user/hadoopqa/input1/part-00000;uri=<NN > IP>:50470;t.service=<NN IP>:50470 > 11/09/30 18:58:00 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 24 > for hadoopqa on <NN IP>:8020 > 11/09/30 18:58:00 INFO security.TokenCache: Got dt for > /user/hadoopqa/out3;uri=<NN IP>:8020;t.service=<NN IP>:8020 > 11/09/30 18:58:00 INFO tools.DistCp: /user/hadoopqa/out3 does not exist. > 11/09/30 18:58:00 INFO tools.DistCp: sourcePathsCount=1 > 11/09/30 18:58:00 INFO tools.DistCp: filesToCopyCount=1 > 11/09/30 18:58:00 INFO tools.DistCp: bytesToCopyCount=1.0g > 11/09/30 18:58:01 INFO mapred.JobClient: Running job: job_201109300819_0007 > 11/09/30 18:58:02 INFO mapred.JobClient: map 0% reduce 0% > 11/09/30 18:58:25 INFO mapred.JobClient: Task Id : > attempt_201109300819_0007_m_000000_0, Status : FAILED > java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 > at > org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > 11/09/30 18:58:41 INFO mapred.JobClient: Task Id : > attempt_201109300819_0007_m_000000_1, Status : FAILED > java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 > at > org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > 11/09/30 18:58:56 INFO mapred.JobClient: Task Id : > attempt_201109300819_0007_m_000000_2, Status : FAILED > java.io.IOException: Copied: 0 Skipped: 0 Failed: 1 > at > org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > 11/09/30 18:59:14 INFO mapred.JobClient: Job complete: job_201109300819_0007 > 11/09/30 18:59:14 INFO mapred.JobClient: Counters: 6 > 11/09/30 18:59:14 INFO mapred.JobClient: Job Counters > 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=62380 > 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all reduces > waiting after reserving slots (ms)=0 > 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all maps > waiting after reserving slots (ms)=0 > 11/09/30 18:59:14 INFO mapred.JobClient: Launched map tasks=4 > 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 > 11/09/30 18:59:14 INFO mapred.JobClient: Failed map tasks=1 > 11/09/30 18:59:14 INFO mapred.JobClient: Job Failed: # of failed Map Tasks > exceeded allowed limit. FailedCount: 1. > LastFailedTask: task_201109300819_0007_m_000000 > With failures, global counters are inaccurate; consider running with -i > Copy failed: java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1257) > at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:881) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:908) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira