[
https://issues.apache.org/jira/browse/MAPREDUCE-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12893929#action_12893929
]
Ramya R commented on MAPREDUCE-1986:
I have the following setup on my cluster:
property
namehadoop.job.history.location/name
value/mapred/history1/value
description If job tracker is static the history files are stored in this
single well known place. If No
value is set here, by default,
it is in the local file system at ${hadoop.log.dir}/history.
/description/property
property
namemapred.job.tracker.history.completed.location/name
value/mapred/history2/done/value
description The completed job history files are stored at this single well
known location. If nothing is
specified, the files are stored at
${hadoop.job.history.location}/done.
/description
/property
i.e. both hadoop.job.history.location and
mapred.job.tracker.history.completed.location are configured to HDFS path.
The jobtracker successfully comes up without any error. However when the job
completes and it retires, I get the following exception:
{noformat}
ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE
canonical subfolder.
java.io.FileNotFoundException: File
/mapred/history1/job_201007290838_0001_1280392750589_rramya_Sleep+job does not
exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:207)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1118)
at
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1094)
at
org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1084)
at
org.apache.hadoop.mapred.JobHistory$JobHistoryFilesManager$1.run(JobHistory.java:300)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{noformat}
However, /mapred/history1 is created and the history files are very much
present on HDFS.
Thus the history files remain forever in hadoop.job.history.location and are
never moved to mapred.job.tracker.history.completed.location. The jobhistory is
never viewable thereafter.
History files fail to move to DONE folder when hadoop.job.history.location is
configured to a HDFS path
---
Key: MAPREDUCE-1986
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1986
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.2
Reporter: Ramya R
When hadoop.job.history.location is configured to a HDFS path and when the
job retires, the history files are never moved to
mapred.job.tracker.history.completed.location and they remain forever in
hadoop.job.history.location . The jobhistory is never viewable thereafter.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
