Hi,
Apologies for cross-posting.
We're in the process of migrating data from an Apache Hadoop 0.20.203.0
cluster to a 0.22.0 cluster using distcp with a hftp source and hdfs dest as
described in the manual. During the copy a handful of the following
cryptic IOExceptions occured and the job finally failed completely. Almost
20GB of data was not copied at all.
11/12/18 22:50:53 INFO mapreduce.Job: Task Id :
attempt_201112181848_0007_m_000097_0, Status : FAILED
java.io.IOException: Copied: 251 Skipped: 0 Failed: 16
at
org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:694)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)
The files being copied do not contain any fancy filenames and are all map or
sequence files with an occasional _SUCCESS file here and there. The cryptic
stack trace is similar to some CDH issue [1] which is still open. Any
thoughts?
[1]: https://issues.cloudera.org/browse/DISTRO-347
Thanks
