[ https://issues.apache.org/jira/browse/MAPREDUCE-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Kanter updated MAPREDUCE-6970: ------------------------------------- Issue Type: Improvement (was: Bug) > archive-logs tool should throttle container requests > ---------------------------------------------------- > > Key: MAPREDUCE-6970 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6970 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Affects Versions: 2.8.0, 3.0.0-alpha1 > Reporter: Robert Kanter > > The {{mapred archive-logs}} command currently has no way to throttle the > number of requested containers. For example, we recently saw a busy cluster > where the tool hadn't been run for a while and there were about 20,000 apps > to process. This meant that the tool tried to request 20,000 containers and > got a ton of GC and then OOM trying to handle that. > This problem can be mitigated by setting {{-maxEligibleApps}} to a more > reasonable value, but doing so would require running the tool multiple times; > plus, the default value is {{-1}} (all). > We should add a way to throttle the max number of concurrently running > containers that the tool manages. Something like {{-concurrency <n>}} where > it would only allow up to {{n}} containers at a time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org