[
https://issues.apache.org/jira/browse/MAPREDUCE-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307309#comment-14307309
]
Hudson commented on MAPREDUCE-6059:
-----------------------------------
SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #92 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/92/])
MAPREDUCE-6059. Speed up history server startup time (Siqi Li via aw) (aw: rev
fd57ab2002f97dcc83d455a5e0c770c8efde77a4)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
> Speed up history server startup time
> ------------------------------------
>
> Key: MAPREDUCE-6059
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6059
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.4.0
> Reporter: Siqi Li
> Assignee: Siqi Li
> Fix For: 3.0.0
>
> Attachments: YARN-2366.v1.patch
>
>
> When history server starts up, It scans every history directories and put all
> history files into a cache, whereas this cache only stores 20K recent history
> files. Therefore, it is wasting a large portion of time loading old history
> files into the cache, and the startup time will keep increasing if we don't
> trim the number of history files. For example, when history server starts up
> with 2.5M history files in HDFS, it took ~5 minutes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
