[
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670942#comment-13670942
]
Karthik Kambatla commented on MAPREDUCE-5268:
---------------------------------------------
bq. Actually I'm a bit surprised to see a templated interface for the private
class.
I initially had a custom/ un-templated implementation, but added the template
to limit the changes to JobListCache. Let me get rid of it.
bq. I was a bit puzzled to see the synchronized methods instead of the
AtomicInteger. What was the reasoning behind that change?
Agree using an AtomicInteger would be faster. However, my thinking was that
given we are abstracting it out, someone working on JobListCache in the future
might be oblivious to the potential race of mapSize not changing along with the
collection. But, I suppose we could explicitly comment that behavior.
Let me update the patch with the two changes.
> Improve history server startup performance
> ------------------------------------------
>
> Key: MAPREDUCE-5268
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
> Affects Versions: 0.23.7, 2.0.4-alpha
> Reporter: Jason Lowe
> Assignee: Karthik Kambatla
> Attachments: mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch
>
>
> The history server can easily take many minutes to startup when there are a
> significant number of jobs to scan in the done directory. However the
> scanning of files is not the bottleneck, rather it's the heavy use of
> ConcurrentSkipListMap.size in HistoryFileManager.
> ConcurrentSkipListMap.size is a very expensive operation, especially on maps
> with many entries, as it has to scan every entry to compute the size. We
> should avoid calling this method or at least minimize its use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
