[ https://issues.apache.org/jira/browse/YARN-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16057718#comment-16057718 ]
Daryn Sharp commented on YARN-6679: ----------------------------------- [~jlowe] or [~nroberts] may be able to comment on the allocation throughput. I just reduced overhead found by a profiler. SLS may not be exercising the RM in the same manner as in a real-world setting. If you look at {{ResourcePBImpl}} it has to: # instantiate a builder – wasted object # builder and its parent class have unneeded instance variables – wasted memory # call setters for memory and vcores, each updates a bit field, assigns instance variable, marks parent builder dirty – unnecessary computational overhead By comparison, a simple object with 2 longs is clearly a win. Even if you aren't stressing the scheduler to its maximum, you should see fewer gc/min due to slower heap growth. I don't have the profile available but the cost of excessive Resource instantiations is still a non-trivial percent of the loop. > Reduce Resource instance overhead via non-PBImpl > ------------------------------------------------ > > Key: YARN-6679 > URL: https://issues.apache.org/jira/browse/YARN-6679 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager > Affects Versions: 2.8.0 > Reporter: Daryn Sharp > Assignee: Daryn Sharp > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: YARN-6679.2.branch-2.patch, YARN-6679.2.trunk.patch, > YARN-6679.3.branch-2.patch, YARN-6679.3.trunk.patch, > YARN-6679.branch-2.patch, YARN-6679.trunk.patch > > > Creating and using transient PB-based Resource instances during scheduling is > very expensive. The overhead can be transparently reduced by internally > using lightweight non-PB based instances. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org