[
https://issues.apache.org/jira/browse/MAPREDUCE-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252498#comment-13252498
]
Robert Joseph Evans commented on MAPREDUCE-3972:
------------------------------------------------
bq. Or put in a timelimt of some kind.
I think we need a time limit of some sort, because there are some issues that
are temporary and can be fixed with a retry. There are others like permission
problems in HDFS, that will never be fixed without manual intervention. We
need a way to still work even if there are issues. I would say that we
initialize the HistoryFileManager with the cleanup time window. If we see a
job that is not in done when we go to clean up we skip over it, if that job is
something that is still within that window. If it is older then the window we
remove it and go on. Similarly when scanning to add in new entries we directly
try to remove files that are older then the window.
bq. JobListCache - it should be possible to use a SkipListMap - and remove all
the synchronization.
I will look at this and see which races we can declare safe to make this happen.
bq. JobListCache.add() could be renamed to JobListCache.addIfAbsent
Great idea will do.
bq. Can the scanAndMove thread be added back ?
It is in there, but it was combined with the clean thread pool in JobHistory.
So now both of them execute as part of that thread pool.
bq. CachedHistoryStorage could check HistoryInfoState in some operations -
possible races with background deletion. Again, this is very unlikely. This
could potentially avoid the call from HistoryFileManager back to
CachedHistoryStorage (deadlock comment in code).
I will look into this too.
> Locking and exception issues in JobHistory Server.
> --------------------------------------------------
>
> Key: MAPREDUCE-3972
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3972
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Attachments: MR-3972.txt, MR-3972.txt, MR-3972.txt, MR-3972.txt
>
>
> The JobHistory server's locking is inconsistent and wrong in some cases.
> This is not super critical because the issues would only show up if a job is
> being cleaned up or moved from intermediate done to done, at the same time it
> is being parsed into a CompletedJob. However the locking is slowing down the
> server in some cases, and is a ticking time bomb that needs to be addressed.
> As part of this too we need to be sure that the Cleaner and Intermediate to
> Done migration threads handle exceptions properly. Now it appears that the
> exception is logged, and the thread just shuts down. This means that the
> history server could still be up and running for weeks and never remove old
> jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
