[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323850#comment-14323850 ] Hudson commented on HDFS-7584: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #97 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/97/]) HDFS-7584. Update CHANGES.txt (arp: rev 9e33c9944cbcb96f9aab74eafce20fe50fe7c9e8) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319873#comment-14319873 ] Hudson commented on HDFS-7584: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #103 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/103/]) HDFS-7584. Update CHANGES.txt (arp: rev 9e33c9944cbcb96f9aab74eafce20fe50fe7c9e8) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319907#comment-14319907 ] Hudson commented on HDFS-7584: -- FAILURE: Integrated in Hadoop-Yarn-trunk #837 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/837/]) HDFS-7584. Update CHANGES.txt (arp: rev 9e33c9944cbcb96f9aab74eafce20fe50fe7c9e8) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320244#comment-14320244 ] Hudson commented on HDFS-7584: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #104 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/104/]) HDFS-7584. Update CHANGES.txt (arp: rev 9e33c9944cbcb96f9aab74eafce20fe50fe7c9e8) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320149#comment-14320149 ] Hudson commented on HDFS-7584: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2035 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2035/]) HDFS-7584. Update CHANGES.txt (arp: rev 9e33c9944cbcb96f9aab74eafce20fe50fe7c9e8) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320289#comment-14320289 ] Hudson commented on HDFS-7584: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2054 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2054/]) HDFS-7584. Update CHANGES.txt (arp: rev 9e33c9944cbcb96f9aab74eafce20fe50fe7c9e8) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318785#comment-14318785 ] Hudson commented on HDFS-7584: -- FAILURE: Integrated in Hadoop-trunk-Commit #7092 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7092/]) HDFS-7584. Update CHANGES.txt (arp: rev 9e33c9944cbcb96f9aab74eafce20fe50fe7c9e8) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318794#comment-14318794 ] Xiaoyu Yao commented on HDFS-7584: -- Thanks all for the review and feedbacks. I will update the release note for this feature under HDFS-7584. Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318795#comment-14318795 ] Xiaoyu Yao commented on HDFS-7584: -- Thanks all for the review and feedbacks. I will update the release note for this feature under HDFS-7584. Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 2.7.0 Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch, HDFS-7584.9c.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14297130#comment-14297130 ] Xiaoyu Yao commented on HDFS-7584: -- org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer needs the new binary editsStored in V8, I will upload a separate editsStored file. org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream needs to bump up the number of editOps from 49 to 50. I will post a new patch that fixes core tests and javac/findbugs shortly. Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch, HDFS-7584.8.patch, HDFS-7584.9.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14297157#comment-14297157
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695300/editsStored
against trunk revision fe2188a.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9372//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch,
HDFS-7584.8.patch, HDFS-7584.9.1.patch, HDFS-7584.9.patch, editsStored,
editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296627#comment-14296627
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695195/HDFS-7584.9.patch
against trunk revision 7882bc0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1193 javac
compiler warnings (more than the trunk's current 1191 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9370//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9370//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9370//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9370//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch,
HDFS-7584.8.patch, HDFS-7584.9.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14297658#comment-14297658
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695305/HDFS-7584.9a.patch
against trunk revision 57b8950.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDecommission
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9373//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9373//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9373//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9373//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch,
HDFS-7584.8.patch, HDFS-7584.9.1.patch, HDFS-7584.9.patch,
HDFS-7584.9a.patch, editsStored, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298027#comment-14298027
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695373/HDFS-7584.9b.patch
against trunk revision ad55083.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9375//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9375//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9375//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9375//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch,
HDFS-7584.8.patch, HDFS-7584.9.1.patch, HDFS-7584.9.patch,
HDFS-7584.9a.patch, HDFS-7584.9b.patch, editsStored, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298294#comment-14298294
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695445/HDFS-7584.9c.patch
against trunk revision f2c9109.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
org.apache.hadoop.hdfs.server.datanode.TestBlockScanner
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9378//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9378//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9378//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9378//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch,
HDFS-7584.8.patch, HDFS-7584.9.patch, HDFS-7584.9a.patch, HDFS-7584.9b.patch,
HDFS-7584.9c.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295897#comment-14295897
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695090/HDFS-7584.7.patch
against trunk revision caf7298.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9361//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch,
editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295864#comment-14295864
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695086/HDFS-7584.6.patch
against trunk revision caf7298.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9360//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296408#comment-14296408
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695191/HDFS-7584.8.patch
against trunk revision 5a0051f.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9369//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, HDFS-7584.6.patch, HDFS-7584.7.patch,
HDFS-7584.8.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14295671#comment-14295671
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12695054/HDFS-7584.5.patch
against trunk revision 9850e15.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9359//console
This message is automatically generated.
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, HDFS-7584.5.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292891#comment-14292891
]
Arpit Agarwal commented on HDFS-7584:
-
[~xyao], considering the size of the patch do you think we should split it into
at least two smaller patches. I can think of at least one natural split:
# Part 1 is the the API, protocol and tool changes.
# Part 2 is the NameNode implementation.
In any case I started looking at the NN changes. Some initial feedback below, I
will look at the rest later this week:
# I did not understand the todo in INode.java:464. If it is something that
would be broken and is not too hard to fix perhaps we should include it in the
same checkin? This is perhaps another argument for splitting into two patches
at a higher level.
# QuotaCounts: It has four telescoping constructors, all private. It is a
little confusing. e.g. the default constructor can be replaced with
initializers.
# QuotaCounts: {{typeSpaces}} and {{typeCounts}} are used interchangeably. We
should probably name them consistently.
# NameNodeLayoutVersion: description of the new layout is too terse, probably
unintentional.
# Could you please add a short comment for {{ONE_NAMESPACE}}? I realize it was
even more confusing before, thanks for adding the static initializer.
# {{INode.getQuotaCounts}} - don't need local variable {{qc}}.
# Nitpick, optional: {{EnumCounters.allLessOrEqual}} and {{.anyGreatOrEqual}} -
can we use foreach loop?
# DFSAdmin.java: _The space quota is set onto storage type_ should be _The
storage type specific quota is set when ..._
# Unintentional whitespace changes in Quota.java?
Enable Quota Support for Storage Types
--
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293051#comment-14293051 ] Xiaoyu Yao commented on HDFS-7584: -- Thanks Arpit again for the feedback. I've fixed your comments 1,2,5. bq. Just an observation, and you don't really need to fix it. I realize you are trying to avoid a new RPC call by modifying the existing ClientProtocol.setQuota call. But it does create a confusing difference between that and DFSClient.setQuota which has two overloads and the matching overload of DFSClient.setQuota behaves differently (throws exception on null). Perhaps it is better to add a new ClientProtocol.setQuota RPC call. Either is fine though. The decision to reuse existing setQuota RPC call compatibly instead of adding a new one is based on the feedbacks from design review. The API that throws exception on null storage type is the new API DFSClient#setQuotaByStorageType with a different signature. We keep the original DFSClient#setQuota(dsQuota, nsQuota) as-is so that it won't break existing clients. With a single setQuota RPC message, I think it should be fine. bq. Do we need the new config key DFS_QUOTA_BY_STORAGETYPE_ENABLED_KEY? The administrator can already choose to avoid configuring per-type quotas so I am not sure the new configuration is useful. Adding the key allow us to completely disable the feature. Without the key, the admin can accidentally configure and enable this feature. I can remove it if this is not needed. Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293018#comment-14293018 ] Xiaoyu Yao commented on HDFS-7584: -- Thanks Arpit for the review and feedback. I will provide a new patch with the update soon. bq. I can think of at least one natural split: Part 1 is the the API, protocol and tool changes. Part 2 is the NameNode implementation. That sounds good to me. I will try splitting it up after finishing the first round of review. bq. QuotaCounts: It has four telescoping constructors, all private. It is a little confusing. Can we simplify the constructors? e.g. the default constructor can be replaced with initializers. Agree. That's something I plan to refactor as well. I tried the build pattern, which could help to main this class when we need add more counters in future that need to be initialized. bq. I did not understand the todo in INode.java:464. If it is something that would be broken and is not too hard to fix perhaps we should include it in the same checkin? This is perhaps another argument for splitting into two patches at a higher level. This is for hadoop fs -count -q command where the majority of the change will not be under hadoop-hdfs-project. It also needs to move the StorageType.java from hadoop-hdfs to hadoop-common, which I prefer to change after the initial check in. bq. QuotaCounts: typeSpaces and typeCounts are used interchangeably. We should probably name them consistently. The parameter in the methods use typeSpaces to be consistent with the other parameters (namespace, diskspace). The class member variable is named typeCounts. bq. Nitpick, optional: EnumCounters.allLessOrEqual and .anyGreatOrEqual - can we use foreach loop? Agree that foreach will make it syntactically neat. Enable Quota Support for Storage Types -- Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292745#comment-14292745
]
Arpit Agarwal commented on HDFS-7584:
-
Few more comments to wrap up feedback on API and protocol.
# Continuing from previous, you also don't need the null check in
{{PBHelper.convertStorageType}}.
# DFSClient.java:3064 - Fix formatting.
# Just an observation, and you don't really need to fix it. I realize you are
trying to avoid a new RPC call by modifying the existing
{{ClientProtocol.setQuota}} call. But it does create a confusing difference
between that and {{DFSClient.setQuota}} which has two overloads and the
matching overload of {{DFSClient.setQuota}} behaves differently (throws
exception on null). Perhaps it is better to add a new
{{ClientProtocol.setQuota}} RPC call. Either is fine though.
# Do we need the new config key {{DFS_QUOTA_BY_STORAGETYPE_ENABLED_KEY}}? The
administrator can already choose to avoid configuring per-type quotas so I am
not sure the new configuration is useful.
# {{DistributedFileSystem.setQuotaByStorageType}} - the Javadoc _and one or
more [Storage Type, space Quota] pairs_ does not match the signature.
Looking at the NN changes next.
Enable Quota Support for Storage Types (SSD)
-
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14292398#comment-14292398
]
Arpit Agarwal commented on HDFS-7584:
-
Hi [~xyao], thanks for picking up this non-trivial task. Here is some initial
feedback on the patch:
# StorageType: You don't need a new constructor parameter for
{{supportQuotaByStorageType}}. You can just use {{!isTransient}}. We can add
the flexibility in the future if needed.
# {{StorageType.isStorageTypeChosen}} can be removed. It looks out of place in
its public interface.
# {{StorageType.getQuotableTypes}} is oddly named. Perhaps
{{typesSupportingQuota}} is better?
# You don't need the {{StorageTypeProto.NONE}} or the default value for it in
{{SetQuotaRequestProto}}. Since it is an optional field we can check for its
presence with {{req.hasStorageType}} in {{ClientNamenodeProtocolPB.setQuota}}.
# Similarly {{ClientNamenodeProtocolTranslatorPB.setQuota}} should not invoke
{{setStorageType}} if the {{type}} parameter is null.
# The loop in {{StorageType.parseStorageType(String)}} can be replaced with
{{StorageType.valueOf(s.toUpperCase())}} right?
# The loop in {{StorageType.parseStorageType(int)}} is not necessary. You can
just use {{StorageType.values()[i]}} instead. {{StorageType.values[]}} can be
cached in a static context to avoid allocating it repeatedly. The 'return null'
fallback will be lost and we will bubble up an exception instead. I checked the
callers and think that will be fine.
More feedback to follow.
Enable Quota Support for Storage Types (SSD)
-
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290903#comment-14290903 ] Zhe Zhang commented on HDFS-7584: - [~xyao] Will the legacy quota become deprecated after this change? If so we should mention it in the documentation. Otherwise we should also add some guideline on how to set both legacy and type-aware quotas. The above example, where the supposedly overall quora is smaller than aggregate type quotas, isn't so easy to understand. Otherwise the patch looks good to me. Thanks again for the good work! Enable Quota Support for Storage Types (SSD) - Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch, HDFS-7584.4.patch, editsStored Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[
https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14290736#comment-14290736
]
Hadoop QA commented on HDFS-7584:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12694377/HDFS-7584.4.patch
against trunk revision 8f26d5a.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9323//console
This message is automatically generated.
Enable Quota Support for Storage Types (SSD)
-
Key: HDFS-7584
URL: https://issues.apache.org/jira/browse/HDFS-7584
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf,
HDFS-7584.0.patch, HDFS-7584.1.patch, HDFS-7584.2.patch, HDFS-7584.3.patch,
HDFS-7584.4.patch, editsStored
Phase II of the Heterogeneous storage features have completed by HDFS-6584.
This JIRA is opened to enable Quota support of different storage types in
terms of storage space usage. This is more important for certain storage
types such as SSD as it is precious and more performant.
As described in the design doc of HDFS-5682, we plan to add new
quotaByStorageType command and new name node RPC protocol for it. The quota
by storage type feature is applied to HDFS directory level similar to
traditional HDFS space quota.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288467#comment-14288467 ] Zhe Zhang commented on HDFS-7584: - bq. If you have 1GB SSD quota dn 2GB DISK quota, which will limit your maximum overall space usage at 3GB of space quota. You don't need to set legacy quota in this case. This makes sense. But what happens if the user did configure a legacy quota, say 2GB in this case? Will it be enforced together with the 1GB SSD + 2GB DISK quotas? bq. The exchange rate actually could be a guideline for admin setup quotas of different storage types. If automatic correlation for quota of different storage types is desired, it should be explicitly set though policies. Sounds good to me. We can revisit it as a follow up. Enable Quota Support for Storage Types (SSD) - Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288474#comment-14288474 ] Xiaoyu Yao commented on HDFS-7584: -- bq. But what happens if the user did configure a legacy quota, say 2GB in this case? Will it be enforced together with the 1GB SSD + 2GB DISK quotas? Good question. Yes. they will be enforced together. When both are configured, whatever usage exceeds first will throw a QuotaExceeded exception. Enable Quota Support for Storage Types (SSD) - Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch, HDFS-7584.1.patch Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288118#comment-14288118 ] Xiaoyu Yao commented on HDFS-7584: -- bq. But then how should we coordinate type-based quotas and the traditional space quota? Is the latter the summation of quotas from all types? If I set my SSD quota as 1GB and DISK quota as 2GB, do I need to separately set my overall space quota to be 3GB? We propose not to correlate storage type quota with legacy space quota. Legacy space quota lacks fine-grain control of storage usage and is left as-is for backward compatibility. If you have 1GB SSD quota dn 2GB DISK quota, which will limit your maximum overall space usage at 3GB of space quota. You don't need to set legacy quota in this case. bq. This is where it would be helpful to exchange type quotas. Revising the example a little bit, let's assume I have 3 blocks of SSD quota and no DISK quota, and the cluster has no SSD storage and plenty of DISK storage. When trying to create a 1 block file with replication factor of 3, under ONE_SSD policy, the user would expect a success because he has 3 blocks of quota in the most expensive type. I understand setting exchange rates needs more discussion. Can we at least allow more expensive quotas to be used on less expensive storages? The example above looks like a configuration that needs admin to either add more SSD storage or increase the DISK quota. By default, quota should be simple and strict. The exchange rate actually could be a guideline for admin setup quotas of different storage types. If automatic correlation for quota of different storage types is desired, it should be explicitly set though policies. Enable Quota Support for Storage Types (SSD) - Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288001#comment-14288001 ] Xiaoyu Yao commented on HDFS-7584: -- Attach a preliminary patch to demo the proposal. I will open separate JIRAs for # Add option to display quota by storage type info in hadoop shell command hadoop fs -count -q, this requires StorageType.java being moved from hadoop-hdfs to hadoop-common. # Update the editsStored for TestOfflineEditsViewer for new SetQuotaByStorageType op. # Adding more unit tests for quota by storage type, more specifically on snapshot and checkpoint. Enable Quota Support for Storage Types (SSD) - Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287990#comment-14287990 ] Zhe Zhang commented on HDFS-7584: - bq. the cumulative disk space quota in the spec means traditional space quota not quota of DISK type Thanks for clarifying. Then section 5.1 is a pure intended-usage based policy, and makes more sense now. But then how should we coordinate type-based quotas and the traditional space quota? Is the latter the summation of quotas from all types? If I set my SSD quota as 1GB and DISK quota as 2GB, do I need to separately set my overall space quota to be 3GB? bq. If the fallback still can't be satisfied due to DISK quota unavailable, the user will get QuotaByStorageTypeExceeded exception. This is where it would be helpful to exchange type quotas. Revising the example a little bit, let's assume I have 3 blocks of SSD quota and no DISK quota, and the cluster has no SSD storage and plenty of DISK storage. When trying to create a 1 block file with replication factor of 3, under ONE_SSD policy, the user would expect a success because he has 3 blocks of quota in the most expensive type. I understand setting exchange rates needs more discussion. Can we at least allow more expensive quotas to be used on less expensive storages? Enable Quota Support for Storage Types (SSD) - Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286520#comment-14286520 ] Zhe Zhang commented on HDFS-7584: - This is a very useful feature and thanks [~xyao] for initiating the work. A few comments: # bq. For example, when SSD is not available but Quota of SSD is available, the write will fallback to DISK and the storage usage deducted from both SSD and the cumulative disk space quota of the directory even no SSD space is being consumed. This is an interesting scenario and is worth more discussion. It is a conservative and safe policy to deduct from both SSD and DISK quotas. However it doesn't fully comply with the principle of _quota based on intended usage_, which might make it appear counter-intuitive to users (e.g. why am I double charged?). As an extreme example, what if the user doesn't have any DISK quota? # How about calculating quota truly based on intended usage? The charged quota might be different than the usage, but so is the case with existing quota logic. What are other disadvantages? # If we do want to charge by actual usage (5.2), maybe we should allow different quota currencies to be exchanged? Something like 1GB of SSD = 2GB of DISK = 4GB of ARCHIVAL. Or at least allow a user with _only_ 1GB SSD quota to use 1GB DISK space. Enable Quota Support for Storage Types (SSD) - Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7584) Enable Quota Support for Storage Types (SSD)
[ https://issues.apache.org/jira/browse/HDFS-7584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14287001#comment-14287001 ] Xiaoyu Yao commented on HDFS-7584: -- Thanks [~zhz] for the feedback. bq. 1. This is an interesting scenario and is worth more discussion. It is a conservative and safe policy to deduct from both SSD and DISK quotas. However it doesn't fully comply with the principle of quota based on intended usage, which might make it appear counter-intuitive to users (e.g. why am I double charged?). As an extreme example, what if the user doesn't have any DISK quota? Agree. the cumulative disk space quota in the spec means traditional space quota not quota of DISK type. I will update it to avoid confusion. We are actually calculating quota based on intended usage by not deducting DISK quotas due to policy fallback. For example, we have a directory \ssd1 with ONE_SSD policy enabled. The directory currently has 2 blocks of SSD Quota and 3 blocks of DISK Quota remaining. If we want to create a file of 1 block with a replication factor of 3 under \ssd1 but the actual available SSD storage is 0, the creation fallback to DISK. After that, the remaining SSD and DISK Quota should be 1 block and 1 block instead of 2 block and 0 block, respectively. If the fallback still can't be satisfied due to DISK quota unavailable, the user will get QuotaByStorageTypeExceeded exception. bq. 2. How about calculating quota truly based on intended usage? The charged quota might be different than the usage, but so is the case with existing quota logic. What are other disadvantages? Quota calculation based on intended usage saves the replication monitor from updating traditional space/namespace quota usage for under/over replicated blocks. For quota by storage type, it can similarly save the Mover from updating quota when the blocks are moved across storage tiers to meet the policy requirement. bq. 3. If we do want to charge by actual usage (5.2), maybe we should allow different quota currencies to be exchanged? Something like 1GB of SSD = 2GB of DISK = 4GB of ARCHIVAL. Or at least allow a user with only 1GB SSD quota to use 1GB DISK space. As mentioned above, we prefer charging by intended usage for its simplicity and consistency. Correlation quota of different storage types looks interesting, it may requires additional tuning to get appropriate currency rates between storage types for different user scenarios. Enable Quota Support for Storage Types (SSD) - Key: HDFS-7584 URL: https://issues.apache.org/jira/browse/HDFS-7584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-7584 Quota by Storage Type - 01202015.pdf, HDFS-7584.0.patch Phase II of the Heterogeneous storage features have completed by HDFS-6584. This JIRA is opened to enable Quota support of different storage types in terms of storage space usage. This is more important for certain storage types such as SSD as it is precious and more performant. As described in the design doc of HDFS-5682, we plan to add new quotaByStorageType command and new name node RPC protocol for it. The quota by storage type feature is applied to HDFS directory level similar to traditional HDFS space quota. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
