[
https://issues.apache.org/jira/browse/HADOOP-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12545256
]
Oscar Stiffelman commented on HADOOP-2129:
------------------------------------------
I experienced the same problem. Adding some logging revealed that the source
paths no longer had the "hdfs://host:port" portion of the URI when they were
used in the copy() method of the mapper. This caused the source file-system to
be incorrectly identified (it was set to the default file system, which was
what it found in the config file).
> distcp between two clusters does not work if it is run on the target cluster
> ----------------------------------------------------------------------------
>
> Key: HADOOP-2129
> URL: https://issues.apache.org/jira/browse/HADOOP-2129
> Project: Hadoop
> Issue Type: Bug
> Components: util
> Affects Versions: 0.16.0
> Environment: Nightly build:
> http://hadoopqa.yst.corp.yahoo.com:8080/hudson/job/Hadoop-LinuxTest/718/
> With patches for HADOOP-2033 and HADOOP-2048.
> Reporter: Murtaza A. Basrai
> Assignee: Chris Douglas
> Priority: Critical
>
> I am trying to copy a directory (~100k files, ~500GB) between two clusters A
> and B (~70 nodes), using a command like:
> hadoop distcp -log /logdir hdfs://namenode-of-A:8600/srcdir
> hdfs://namenode-of-B:8600/targetdir
> I tried 4 ways of doing it:
> 1) Copy from A to B, by running distcp on A
> 2) Copy from A to B, by running distcp on B
> 3) Copy from B to A, by running distcp on B
> 4) Copy from B to A, by running distcp on A
> Invocations 1 and 3 succeeded, but 2 and 4 failed.
> I got a lot of errors of the type below:
> 07/10/30 20:52:11 INFO mapred.JobClient: Running job: job_200710180049_0115
> 07/10/30 20:52:12 INFO mapred.JobClient: map 0% reduce 0%
> 07/10/30 20:54:41 INFO mapred.JobClient: map 1% reduce 0%
> 07/10/30 20:56:52 INFO mapred.JobClient: map 2% reduce 0%
> 07/10/30 20:57:41 INFO mapred.JobClient: Task Id :
> task_200710180049_0115_m_000184_0, Status : FAILED
> java.io.IOException: Some copies could not complete. See log for details.
> at
> org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:407)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)
> followed by the job failing:
> 07/10/30 22:07:41 INFO mapred.JobClient: map 99% reduce 100%
> Copy failed: java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:688)
> at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:481)
> at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:555)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:67)
> at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:566)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
