[jira] [Updated] (HDFS-2392) Dist with hftp is failing again

2011-10-05 Thread Matt Foley (Updated) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley updated HDFS-2392:
-

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

 Dist with hftp is failing again
 ---

 Key: HDFS-2392
 URL: https://issues.apache.org/jira/browse/HDFS-2392
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 0.20.205.0
Reporter: Rajit Saha
Assignee: Daryn Sharp
Priority: Critical
 Fix For: 0.20.205.0

 Attachments: HDFS-2392.patch


 $ hadoop distcp hftp://NN Hostname:50070/user/hadoopqa/input1/part-0 
 /user/hadoopqa/out3
 11/09/30 18:57:59 INFO tools.DistCp: srcPaths=[hftp://NN 
 Hostname:50070/user/hadoopqa/input1/part-0]
 11/09/30 18:57:59 INFO tools.DistCp: destPath=/user/hadoopqa/out3
 11/09/30 18:58:00 INFO security.TokenCache: Got dt for
 hftp://NN Hostname:50070/user/hadoopqa/input1/part-0;uri=NN 
 IP:50470;t.service=NN IP:50470
 11/09/30 18:58:00 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 24 
 for hadoopqa on NN IP:8020
 11/09/30 18:58:00 INFO security.TokenCache: Got dt for
 /user/hadoopqa/out3;uri=NN IP:8020;t.service=NN IP:8020
 11/09/30 18:58:00 INFO tools.DistCp: /user/hadoopqa/out3 does not exist.
 11/09/30 18:58:00 INFO tools.DistCp: sourcePathsCount=1
 11/09/30 18:58:00 INFO tools.DistCp: filesToCopyCount=1
 11/09/30 18:58:00 INFO tools.DistCp: bytesToCopyCount=1.0g
 11/09/30 18:58:01 INFO mapred.JobClient: Running job: job_201109300819_0007
 11/09/30 18:58:02 INFO mapred.JobClient:  map 0% reduce 0%
 11/09/30 18:58:25 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_0, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:58:41 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_1, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:58:56 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_2, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:59:14 INFO mapred.JobClient: Job complete: job_201109300819_0007
 11/09/30 18:59:14 INFO mapred.JobClient: Counters: 6
 11/09/30 18:59:14 INFO mapred.JobClient:   Job Counters 
 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=62380
 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all reduces 
 waiting after reserving slots (ms)=0
 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all maps 
 waiting after reserving slots (ms)=0
 11/09/30 18:59:14 INFO mapred.JobClient: Launched map tasks=4
 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
 11/09/30 18:59:14 INFO mapred.JobClient: Failed map tasks=1
 11/09/30 18:59:14 INFO mapred.JobClient: Job Failed:

[jira] [Updated] (HDFS-2392) Dist with hftp is failing again

2011-09-30 Thread Daryn Sharp (Updated) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-2392:
--

Attachment: HDFS-2392.patch

Fix to call setDelegationToken with a token located in the cache.

Move token initialization into a new method.  Update initialize method to call 
the token init.

Using aforementioned change to write unit tests to verify that the token is 
set, and that the correct token is set.

 Dist with hftp is failing again
 ---

 Key: HDFS-2392
 URL: https://issues.apache.org/jira/browse/HDFS-2392
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 0.20.205.0
Reporter: Rajit Saha
Assignee: Daryn Sharp
Priority: Critical
 Attachments: HDFS-2392.patch


 $ hadoop distcp hftp://NN Hostname:50070/user/hadoopqa/input1/part-0 
 /user/hadoopqa/out3
 11/09/30 18:57:59 INFO tools.DistCp: srcPaths=[hftp://NN 
 Hostname:50070/user/hadoopqa/input1/part-0]
 11/09/30 18:57:59 INFO tools.DistCp: destPath=/user/hadoopqa/out3
 11/09/30 18:58:00 INFO security.TokenCache: Got dt for
 hftp://NN Hostname:50070/user/hadoopqa/input1/part-0;uri=NN 
 IP:50470;t.service=NN IP:50470
 11/09/30 18:58:00 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 24 
 for hadoopqa on NN IP:8020
 11/09/30 18:58:00 INFO security.TokenCache: Got dt for
 /user/hadoopqa/out3;uri=NN IP:8020;t.service=NN IP:8020
 11/09/30 18:58:00 INFO tools.DistCp: /user/hadoopqa/out3 does not exist.
 11/09/30 18:58:00 INFO tools.DistCp: sourcePathsCount=1
 11/09/30 18:58:00 INFO tools.DistCp: filesToCopyCount=1
 11/09/30 18:58:00 INFO tools.DistCp: bytesToCopyCount=1.0g
 11/09/30 18:58:01 INFO mapred.JobClient: Running job: job_201109300819_0007
 11/09/30 18:58:02 INFO mapred.JobClient:  map 0% reduce 0%
 11/09/30 18:58:25 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_0, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:58:41 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_1, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:58:56 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_2, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:59:14 INFO mapred.JobClient: Job complete: job_201109300819_0007
 11/09/30 18:59:14 INFO mapred.JobClient: Counters: 6
 11/09/30 18:59:14 INFO mapred.JobClient:   Job Counters 
 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=62380
 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all reduces 
 waiting after reserving slots (ms)=0
 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all maps 
 waiting after reserving slots (ms)=0
 11/09/30 18:59:14 INFO mapred.JobClient: Launched map

[jira] [Updated] (HDFS-2392) Dist with hftp is failing again

2011-09-30 Thread Daryn Sharp (Updated) (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-2392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-2392:
--

Fix Version/s: 0.20.205.0
   Status: Patch Available  (was: Open)

 Dist with hftp is failing again
 ---

 Key: HDFS-2392
 URL: https://issues.apache.org/jira/browse/HDFS-2392
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: name-node
Affects Versions: 0.20.205.0
Reporter: Rajit Saha
Assignee: Daryn Sharp
Priority: Critical
 Fix For: 0.20.205.0

 Attachments: HDFS-2392.patch


 $ hadoop distcp hftp://NN Hostname:50070/user/hadoopqa/input1/part-0 
 /user/hadoopqa/out3
 11/09/30 18:57:59 INFO tools.DistCp: srcPaths=[hftp://NN 
 Hostname:50070/user/hadoopqa/input1/part-0]
 11/09/30 18:57:59 INFO tools.DistCp: destPath=/user/hadoopqa/out3
 11/09/30 18:58:00 INFO security.TokenCache: Got dt for
 hftp://NN Hostname:50070/user/hadoopqa/input1/part-0;uri=NN 
 IP:50470;t.service=NN IP:50470
 11/09/30 18:58:00 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 24 
 for hadoopqa on NN IP:8020
 11/09/30 18:58:00 INFO security.TokenCache: Got dt for
 /user/hadoopqa/out3;uri=NN IP:8020;t.service=NN IP:8020
 11/09/30 18:58:00 INFO tools.DistCp: /user/hadoopqa/out3 does not exist.
 11/09/30 18:58:00 INFO tools.DistCp: sourcePathsCount=1
 11/09/30 18:58:00 INFO tools.DistCp: filesToCopyCount=1
 11/09/30 18:58:00 INFO tools.DistCp: bytesToCopyCount=1.0g
 11/09/30 18:58:01 INFO mapred.JobClient: Running job: job_201109300819_0007
 11/09/30 18:58:02 INFO mapred.JobClient:  map 0% reduce 0%
 11/09/30 18:58:25 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_0, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:58:41 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_1, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:58:56 INFO mapred.JobClient: Task Id : 
 attempt_201109300819_0007_m_00_2, Status : FAILED
 java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
 at 
 org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
 at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:396)
 at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
 at org.apache.hadoop.mapred.Child.main(Child.java:249)
 11/09/30 18:59:14 INFO mapred.JobClient: Job complete: job_201109300819_0007
 11/09/30 18:59:14 INFO mapred.JobClient: Counters: 6
 11/09/30 18:59:14 INFO mapred.JobClient:   Job Counters 
 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=62380
 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all reduces 
 waiting after reserving slots (ms)=0
 11/09/30 18:59:14 INFO mapred.JobClient: Total time spent by all maps 
 waiting after reserving slots (ms)=0
 11/09/30 18:59:14 INFO mapred.JobClient: Launched map tasks=4
 11/09/30 18:59:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
 11/09/30 18:59:14 INFO mapred.JobClient: Failed map tasks=1
 11/09/30 18:59:14 INFO mapred.JobClient: Job Failed: # of failed Map Tasks