[ https://issues.apache.org/jira/browse/YARN-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated YARN-5889: ----------------------------- Attachment: YARN-5889.0001.suggested.patchnotes Thanks [~sunilg] for providing this work. I am concerned about the way in which {{LeafQueue#getComputedUserLimit}} and {{LeafQueue#getComputedActiveUserLimit}} are deciding when to recompute user limits. - In the case of {{getComputedActiveUserLimit}}, if number of active users has increased or decreased, all active users in {{preComputedActiveUserLimit}} are invalided, and not just the one that was activated/deactivated. This requires recalculation for other users when it is not necessary. - In the case of {{getComputedUserLimit}}, comparing a cached {{lastUserCount}} with {{users.size()}} could miss occasions when recomputation would be necessary. My understanding of [~jlowe]'s [comment (above)|https://issues.apache.org/jira/browse/YARN-5889?focusedCommentId=15745552&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15745552] is that the {{LeafQueue}} would increment a change counter which would be cached in each {{User}}'s object and then would be used to determine when recalculation of user limit should occur. Rather than document each occurrence, I have uploaded a patch with my suggestions. {{YARN-5889.0001.suggested.patchnotes}} is the diff between {{YARN-5889.0001.patch}} and my suggestions. I haven't tested all of the scenarios yet (I'll complete that soon), but I think this covers my concerns about the current patch, and I think it is more efficient. Please have a look and let me know your thoughts. > Improve user-limit calculation in capacity scheduler > ---------------------------------------------------- > > Key: YARN-5889 > URL: https://issues.apache.org/jira/browse/YARN-5889 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler > Reporter: Sunil G > Assignee: Sunil G > Attachments: YARN-5889.0001.patch, > YARN-5889.0001.suggested.patchnotes, YARN-5889.v0.patch, YARN-5889.v1.patch, > YARN-5889.v2.patch > > > Currently user-limit is computed during every heartbeat allocation cycle with > a write lock. To improve performance, this tickets is focussing on moving > user-limit calculation out of heartbeat allocation flow. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org