[ https://issues.apache.org/jira/browse/YARN-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803148#comment-13803148 ]
Bikas Saha commented on YARN-1307: ---------------------------------- It is fine to store a batch of delegation token in the same znode. In fact that may be better than creating a huge number of znode. A setData operation is easy to do on a znode. So the stateStore can keep the current (upto 500kb) batch in memory. When a new delegation token arrives then the store can add it to the batch and overwrite the znode data with the new batch. Keep doing it till the batch size is ~500kb. When the data size exceeds the size of a batch then a new znode is created. There is no need to encode the sequence number in the name. It can be made part of the znode data. The sequence number znode stores the sequence number of the tokens. We encoded the sequence number in the name for HDFS so that we dont have to go to the data nodes to write the data. this is not necessary in ZK since it supports a setData operation that overwrites the previous data. We can simply name the znode something like RMDelegationTokenSequenceNumber. > Rethink znode structure for RM HA > --------------------------------- > > Key: YARN-1307 > URL: https://issues.apache.org/jira/browse/YARN-1307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager > Reporter: Tsuyoshi OZAWA > Assignee: Tsuyoshi OZAWA > Attachments: YARN-1307.1.patch, YARN-1307.2.patch > > > Rethink for znode structure for RM HA is proposed in some JIRAs(YARN-659, > YARN-1222). The motivation of this JIRA is quoted from Bikas' comment in > YARN-1222: > {quote} > We should move to creating a node hierarchy for apps such that all znodes for > an app are stored under an app znode instead of the app root znode. This will > help in removeApplication and also in scaling better on ZK. The earlier code > was written this way to ensure create/delete happens under a root znode for > fencing. But given that we have moved to multi-operations globally, this isnt > required anymore. > {quote} -- This message was sent by Atlassian JIRA (v6.1#6144)