[ https://issues.apache.org/jira/browse/HDFS-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16307398#comment-16307398 ]
Yiqun Lin commented on HDFS-12934: ---------------------------------- Hi [~elgoiri], following are my comments: bq. I've checked MountTableResolver and it might make sense to use a similar TreeMap for the map in RouterQuotaLocalCache. It wo... Using TreeMap will benefit for function {{getChildrenPath}} I think. I don't this can simplify the recursive search in function {{RouterQuotaLocalCache#getQuotaUsage}}. Am I missing something? In additionally, use the read/write lock to make this thread-safe to use. bq. In Router, I would rename getChildrenPaths(String path) into getQuotaUsage(String path) and getQuotaUsageCache() getQuotaUsage().. I didn't fully understanding on this. Do you mean following changes: {noformat} getChildrenPaths -->getQuotaUsage getQuotaUsageCache-->getQuotaUsage {noformat} The first change looks strange. I only renamed getQuotaUsageCache to getQuotaUsage. {quote} Typo NamaService in RouterRpcServer and loc.getNameserviceId() should be extracted. Update TestMountTable to include the new field for the quota. {quote} Addressed. {quote} In TestRouterQuota, I think we shouldn't use the wait for breaking the quota but actually create successfully the first 3 and then fail in the next one. {quote} Since quota cache service is a periodical service, we should ensure the mount tables are loaded and their quota are updated. So I use {{GenericTestUtils#waitFor}} for waiting this instead of just sleeping fixed time. This is a simulated real scene. I found test of setQuota/getQuota can be simplified. Waiting operation is not needed. I have updated on this. bq. Do you mind opening the JIRA for the doc and the UI? Had filed JIRA HDFS-12972 and HDFS-12973 for tracking this. Attach the updated patch. > RBF: Federation supports global quota > ------------------------------------- > > Key: HDFS-12934 > URL: https://issues.apache.org/jira/browse/HDFS-12934 > Project: Hadoop HDFS > Issue Type: Sub-task > Affects Versions: 3.0.0 > Reporter: Yiqun Lin > Assignee: Yiqun Lin > Labels: RBF > Attachments: HDFS-12934.001.patch, HDFS-12934.002.patch, > HDFS-12934.003.patch, HDFS-12934.004.patch, RBF support global quota.pdf > > > Now federation doesn't support set the global quota for each folder. > Currently the quota will be applied for each subcluster under the specified > folder via RPC call. > It will be very useful for users that federation can support setting global > quota and exposing the command of this. > In a federated environment, a folder can be spread across multiple > subclusters. For this reason, we plan to solve this by following way: > # Set global quota across each subcluster. We don't allow each subcluster can > exceed maximun quota value. > # We need to construct one <Path, QuotaUsage> cache map for storing the sum > quota usage of these subclusters under federation folder. Every time we want > to do WRITE operation under specified folder, we will get its quota usage > from cache and verify its quota. If quota exceeded, throw exception, > otherwise update its quota usage in cache when finishing operations. > The quota will be set to mount table and as a new field in mount table. The > set/unset command will be like: > {noformat} > hdfs dfsrouteradmin -setQuota -ns <nsQuota> -ss <ssQuota> <mount table> > hdfs dfsrouteradmin -clrQuota <mount table> > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org