Fengnan Li created HDFS-15383:
---------------------------------
Summary: RBF: Disable watch in ZKDelegationSecretManager for
performance
Key: HDFS-15383
URL: https://issues.apache.org/jira/browse/HDFS-15383
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Fengnan Li
Assignee: Fengnan Li
Based on the current design for delegation token in secure Router, the total
number of watches for tokens is the product of number of routers and number of
tokens, this is due to ZKDelegationTokenManager is using PathChildrenCache from
curator, which automatically sets the watch and ZK will push the sync
information to each router. There are some evaluations about the number of
watches in Zookeeper has negative performance impact to Zookeeper server.
In our practice when the number of watches exceeds 1.2 Million in a single ZK
server there will be significant ZK performance degradation. Thus this ticket
is to rewrite ZKDelegationTokenManagerImpl.java to explicitly disable the
PathChildrenCache and have Routers sync periodically from Zookeeper. This has
been working fine at the scale of 10 Routers with 2 million tokens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
