[
https://issues.apache.org/jira/browse/HDFS-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880323#comment-16880323
]
Jinglun commented on HDFS-14633:
--------------------------------
Hi [~ayushtkn], thanks for your explanation of backward compatibility, I agree
it will be a problem.:)
I read the code about setStoragePolicy's permission check, seems everyone with
FsAction.WRITE to the INode could change the policy. The super user priviledge
is not checked. So not only the admin but normal users can change it.
About rename, supposing I have a file /dir-1/A with storage policy HOT
inherited from it's parent. /dir-2 is a directory with storage policy COLD.
After I rename /dir-1/A to /dir-2/A, the consume in /dir-2 quota feature should
be ARCHIVE, but it's not. The quota usage recorded in /dir-2 doesn't match the
count result of /dir-2. Also the ARCHIVE quota in /dir-2 is not checked(It
checks the SSD quota). I upload a test case to explain it.
About setStoragePolicy, it's not handled at all. I tried to fix
setStoragePolicy when it's called on a file in HDFS-14568, and it's not enough.
Direcotry should also be handled.
The addBlock and delete rpcs handle the quota well, so I remove them from the
jira description. Sorry for the mistake, my bad.
Upload HDFS-14633-testcases-explanation to explain:
1. setStoragePolicy could be called by anyone with a write permission.
2. rename causes inconsistent in the count result and the usage recored in
quota feature.
Looking forward to your comments. Thanks~
> The StorageType quota and consume in QuotaFeature is not handled when rename
> and setStoragePolicy etc.
> -------------------------------------------------------------------------------------------------------
>
> Key: HDFS-14633
> URL: https://issues.apache.org/jira/browse/HDFS-14633
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Assignee: Jinglun
> Priority: Major
>
> The NameNode manages the global state of the cluster. We should always take
> NameNode's records as the sole criterion because no matter what inconsistent
> is the NameNode should finally make everything right based on it's records.
> Let's call it rule NSC(NameNode is the Sole Criterion). That means when we do
> all quota related rpcs, we do the quota check according to NameNode's records
> regardless of any inconsistent situation, such as the replicas doesn't match
> the storage policy of the file, or the replicas count doesn't match the
> file's set replica.
> The work SPS deals with the wrongly palced replicas. There is a thought
> about putting off the consume update of the DirectoryQuota until all replicas
> are re-placed by SPS. I can't agree that because if we do so we will abandon
> letting the NameNode's records to be the sole criterion. The block
> replication is a good example of the rule NSC. When we count the consume of a
> file(CONTIGUOUS), we multiply the replication factor with the file's length,
> no matter the blocks are under replicated or excessed. We should do the same
> thing for the storage type quota.
> Another concern is the change will let setStoragePolicy throw
> QuotaByStorageTypeExceededException which it doesn't before. I don't think
> it's a big problem since the setStoragePolicy already throws IOException. Or
> we can wrap the QuotaByStorageTypeExceededException in an IOException, but I
> won't recommend that because it's ugly.
> To make storage type consume follow the rule NSC, we need change
> rename(moving a file with storage policy inherited from it's parent) and
> setStoragePolicy.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
