[ https://issues.apache.org/jira/browse/HADOOP-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557397#action_12557397 ]
Arun C Murthy commented on HADOOP-1876: --------------------------------------- bq. Can this patch make JobHistory log obsolete? Or at least is that intended? I hate to see same information logged at different places in different forms using different code paths. This patch doesn't do that, but definitely that is the direction I'd go too... +1. Should we broaden the scope HADOOP-2178 to re-work JobHistory to use Writables rather than the custom format? Or is that a new jira? bq. Other than being in text format (which has its pros and cons), job history log is event based [...] Yes, moving to Writable wouldn't hurt the _job analysis_ part since, as you point out, it's event-based - we just need to use Writable.readFields rather than the custom text-parsing... anyone sees other issues? > Persisting completed jobs status > -------------------------------- > > Key: HADOOP-1876 > URL: https://issues.apache.org/jira/browse/HADOOP-1876 > Project: Hadoop > Issue Type: Improvement > Components: mapred > Environment: all > Reporter: Alejandro Abdelnur > Assignee: Alejandro Abdelnur > Priority: Critical > Fix For: 0.16.0 > > Attachments: patch1876.txt, patch1876.txt > > > Currently the JobTracker keeps information about completed jobs in memory. > This information is flushed from the cache when it has outlived > (#RETIRE_JOB_INTERVAL) or because the limit of completed jobs in memory has > been reach (#MAX_COMPLETE_USER_JOBS_IN_MEMORY). > Also, if the JobTracker is restarted (due to being recycled or due to a > crash) information about completed jobs is lost. > If any of the above scenarios happens before the job information is queried > by a hadoop client (normally the job submitter or a monitoring component) > there is no way to obtain such information. > A way to avoid this is the JobTracker to persist in DFS the completed jobs > information upon job completion. This would be done at the time the job is > moved to the completed jobs queue. Then when querying the JobTracker for > information about a completed job, if it is not found in the memory queue, a > lookup in DFS would be done to retrieve the completed job information. > A directory in DFS (under mapred/system) would be used to persist completed > job information, for each completed job there would be a directory with the > job ID, within that directory all the information about the job: status, > jobprofile, counters and completion events. > A configuration property will indicate for how log persisted job information > should be kept in DFS. After such period it will be cleaned up automatically. > This improvement would not introduce API changes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.