[ https://issues.apache.org/jira/browse/YARN-5008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266859#comment-15266859 ]
Karthik Kambatla commented on YARN-5008: ---------------------------------------- By the way, this also brings up another issue with our current RMStateStore implementations. We don't need to retain all the information for completed applications; for instance, the ASC is moot but contributes significantly to the statestore-footprint. This affects all stores, and the ZK store in particular. > LeveldbRMStateStore database can grow substantially leading to long recovery > times > ---------------------------------------------------------------------------------- > > Key: YARN-5008 > URL: https://issues.apache.org/jira/browse/YARN-5008 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager > Affects Versions: 2.7.0 > Reporter: Jason Lowe > Assignee: Jason Lowe > Fix For: 2.7.4 > > Attachments: YARN-5008.001.patch > > > On large clusters with high application churn the background compaction in > leveldb may not be able to keep up with the write rate. This can lead to > large leveldb databases that take many minutes to recover despite not having > very much real data in the database to load. Most the time is spent > traversing tables full of keys that have been deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org