[ https://issues.apache.org/jira/browse/YARN-7262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225516#comment-16225516 ]
Daniel Templeton commented on YARN-7262: ---------------------------------------- For the record, +1 > Add a hierarchy into the ZKRMStateStore for delegation token znodes to > prevent jute buffer overflow > --------------------------------------------------------------------------------------------------- > > Key: YARN-7262 > URL: https://issues.apache.org/jira/browse/YARN-7262 > Project: Hadoop YARN > Issue Type: Improvement > Affects Versions: 2.6.0 > Reporter: Robert Kanter > Assignee: Robert Kanter > Fix For: 2.9.0, 3.0.0 > > Attachments: YARN-7262.001.patch, YARN-7262.002.patch, > YARN-7262.003.patch, YARN-7262.003.patch > > > We've seen users who are running into a problem where the RM is storing so > many delegation tokens in the {{ZKRMStateStore}} that the _listing_ of those > znodes is higher than the jute buffer. This is fine during operations, but > becomes a problem on a fail over because the RM will try to read in all of > the token znodes (i.e. call {{getChildren}} on the parent znode). This is > particularly bad because everything appears to be okay, but then if a > failover occurs you end up with no active RMs. > There was a similar problem with the Yarn application data that was fixed in > YARN-2962 by adding a (configurable) hierarchy of znodes so the RM could pull > subchildren without overflowing the jute buffer (though it's off by default). > We should add a hierarchy similar to that of YARN-2962, but for the > delegation token znodes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org