[ https://issues.apache.org/jira/browse/MAPREDUCE-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-6059: ---------------------------------- Fix Version/s: (was: 3.0.0) 2.7.0 Hadoop Flags: Reviewed Thanks Siqi and Allen! I committed this to branch-2 as well. > Speed up history server startup time > ------------------------------------ > > Key: MAPREDUCE-6059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6059 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Affects Versions: 2.4.0 > Reporter: Siqi Li > Assignee: Siqi Li > Fix For: 2.7.0 > > Attachments: YARN-2366.v1.patch > > > When history server starts up, It scans every history directories and put all > history files into a cache, whereas this cache only stores 20K recent history > files. Therefore, it is wasting a large portion of time loading old history > files into the cache, and the startup time will keep increasing if we don't > trim the number of history files. For example, when history server starts up > with 2.5M history files in HDFS, it took ~5 minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)