The error doesn't mean the file not existed in the HDFS, but it means local
disk. If you read the error stack trace:
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:581)
It indicates the error happened on Local file system.
If you try to copy data from an existing cluster to a new setup cluster, then
It most likely some misconfiguration in your new cluster causing this.
When you use the distcp command, hadoop will generate a lot of MR jobs to copy
the data in Mapper tasks. (No reducers in this case). You should check the JT
to find out which task node gave this error, and check the log to identify the
root cause, especially check the local.dir or log.dir settings in the new
cluster.
Yong
Date: Fri, 6 Sep 2013 15:36:48 +0800
Subject: distcp failed "Copy failed: ENOENT: No such file or directory"
From: justlo...@gmail.com
To: user@hadoop.apache.org
# sudo -u hdfs hadoop distcp hdfs://192.168.10.22:9000/alex
hdfs://192.168.10.220/benchmarks
13/09/06 15:34:11 INFO tools.DistCp: srcPaths=[hdfs://192.168.10.22:9000/alex]
13/09/06 15:34:11 INFO tools.DistCp: destPath=hdfs://192.168.10.220/benchmarks
13/09/06 15:34:12 WARN conf.Configuration: session.id is deprecated. Instead,
use dfs.metrics.session-id
13/09/06 15:34:12 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
13/09/06 15:34:12 WARN conf.Configuration: slave.host.name is deprecated.
Instead, use dfs.datanode.hostname
With failures, global counters are inaccurate; consider running with -i
Copy failed: ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:581)
at
org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:427)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:579)
at org.apache.hadoop.tools.DistCp.setup(DistCp.java:1038)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:666)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
the src and dest dir is all exist
# sudo -u hdfs hadoop fs -ls /alex
Found 3 items
drwxr-xr-x - root supergroup 0 2013-08-16 10:14 /alex/mydump
drwxr-xr-x - root supergroup 0 2013-08-28 10:48 /alex/tdump
drwxr-xr-x - root supergroup 0 2013-08-28 10:54 /alex/tdump2
# sudo -u hdfs hadoop fs -ls /benchmarks
Found 1 items
drwxr-xr-x - hdfs hdfs 0 2013-08-23 17:02 /benchmarks/TestDFSIO
