[ https://issues.apache.org/jira/browse/MAPREDUCE-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307397#comment-14307397 ]
Hudson commented on MAPREDUCE-6059: ----------------------------------- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2046 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2046/]) MAPREDUCE-6059. Speed up history server startup time (Siqi Li via aw) (aw: rev fd57ab2002f97dcc83d455a5e0c770c8efde77a4) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java * hadoop-mapreduce-project/CHANGES.txt > Speed up history server startup time > ------------------------------------ > > Key: MAPREDUCE-6059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6059 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Affects Versions: 2.4.0 > Reporter: Siqi Li > Assignee: Siqi Li > Fix For: 3.0.0 > > Attachments: YARN-2366.v1.patch > > > When history server starts up, It scans every history directories and put all > history files into a cache, whereas this cache only stores 20K recent history > files. Therefore, it is wasting a large portion of time loading old history > files into the cache, and the startup time will keep increasing if we don't > trim the number of history files. For example, when history server starts up > with 2.5M history files in HDFS, it took ~5 minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)