[ https://issues.apache.org/jira/browse/HDFS-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923449#comment-16923449 ]
He Xiaoqiao commented on HDFS-14633: ------------------------------------ hi [~xyao],[~LiJinglun]. Sorry for late response, just check TestQuota#testClrQuotaOnRoot failed in branch-2 cause it doesn't support traditional binary prefix, And I try to dig commit source, and meet the following commit info. {code:java} commit a524608d1e0a276fd822e9574f0a4a6d51298b13 Author: Xiaoyu Yao <x...@apache.org> Date: Fri Aug 30 16:46:04 2019 -0700 HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. (cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1) {code} I am confused that patch v007 source code is very different with commit for branch-2. is there anything I missed? or commit message is not match with code. Another way I do not trace back to `62d71fbac3789c7d484bc76ced9ec7fa6ff94de1` which is mark as cherrypick source. where is it?:) After check carefully, I think we could fix failed unittest `TestQuota#testClrQuotaOnRoot` with cherrypick HDFS-12009. Any suggestions? Thanks. > The StorageType quota and consume in QuotaFeature is not handled for rename > --------------------------------------------------------------------------- > > Key: HDFS-14633 > URL: https://issues.apache.org/jira/browse/HDFS-14633 > Project: Hadoop HDFS > Issue Type: Improvement > Reporter: Jinglun > Assignee: Jinglun > Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3 > > Attachments: HDFS-14633-testcases-explanation, HDFS-14633.002.patch, > HDFS-14633.003.patch, HDFS-14633.004.patch, HDFS-14633.005.patch, > HDFS-14633.006.patch, HDFS-14633.007.patch > > > The NameNode manages the global state of the cluster. We should always take > NameNode's records as the sole criterion because no matter what inconsistent > is the NameNode should finally make everything right based on it's records. > Let's call it rule NSC(NameNode is the Sole Criterion). That means when we do > all quota related rpcs, we do the quota check according to NameNode's records > regardless of any inconsistent situation, such as the replicas doesn't match > the storage policy of the file, or the replicas count doesn't match the > file's set replica. > The work SPS deals with the wrongly palced replicas. There is a thought > about putting off the consume update of the DirectoryQuota until all replicas > are re-placed by SPS. I can't agree that because if we do so we will abandon > letting the NameNode's records to be the sole criterion. The block > replication is a good example of the rule NSC. When we count the consume of a > file(CONTIGUOUS), we multiply the replication factor with the file's length, > no matter the blocks are under replicated or excessed. We should do the same > thing for the storage type quota. > Another concern is the change will let setStoragePolicy throw > QuotaByStorageTypeExceededException which it doesn't before. I don't think > it's a big problem since the setStoragePolicy already throws IOException. Or > we can wrap the QuotaByStorageTypeExceededException in an IOException, but I > won't recommend that because it's ugly. > To make storage type consume follow the rule NSC, we need change > rename(moving a file with storage policy inherited from it's parent) and > setStoragePolicy. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org