vtutrinov commented on code in PR #5721:
URL: https://github.com/apache/ozone/pull/5721#discussion_r1417203747
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/StorageLocationReportMXBean.java:
##########
@@ -33,6 +33,8 @@ public interface StorageLocationReportMXBean {
long getRemaining();
Review Comment:
@xichen01
I've pushed a new commit with the proposed approach and the build is green,
as such as the manual test from the jira ticket. But I'm concerned about the
datanode storage metrics (output of the **ozone admin datanode usageinfo**
command) and recon's reports - from the user's perspective it will seen contr
ordinal:
* I've written a 200KiB key to the EC-bucket with rs-6-3-1024k replication
(the cluster has 10 datanodes with 2Gb storages mounted to /data)
* datanodes report before the key was written:
```
Usage Information (1 Datanodes)
UUID : 8ac06e73-a434-4fd3-9be5-bc271b8f6bf7
IP Address : 172.23.0.15
Hostname : ozone-datanode10-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : 9e3bf76a-8c98-4fa3-b034-c68973d7560b
IP Address : 172.23.0.10
Hostname : ozone-datanode3-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : a5df4b7d-92b3-407a-b56d-aaeb924f0cd6
IP Address : 172.23.0.7
Hostname : ozone-datanode2-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 27365376 B (26.10 MB)
Total Used % : 1.27%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2120118272 B (1.97 GB)
Remaining % : 98.73%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : e5a630fc-ce77-4211-89dc-15ab0ca67fde
IP Address : 172.23.0.6
Hostname : ozone-datanode8-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : 2d00f163-7f22-4395-b82b-e8353c74698b
IP Address : 172.23.0.5
Hostname : ozone-datanode7-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : 3a22dc09-acc5-4270-8007-135056031ed9
IP Address : 172.23.0.12
Hostname : ozone-datanode9-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : 4478a423-0416-429e-b41c-5e58b8ec7ed5
IP Address : 172.23.0.16
Hostname : ozone-datanode5-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : 5ffda4af-e5dd-4b11-a945-7fef0b53356b
IP Address : 172.23.0.17
Hostname : ozone-datanode1-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : 6fe5b830-5031-4a4c-a8fa-ae909c55d187
IP Address : 172.23.0.3
Hostname : ozone-datanode6-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : 7ce23e7c-41bf-4c5e-aea7-63af06049ad0
IP Address : 172.23.0.19
Hostname : ozone-datanode4-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31571968 B (30.11 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115911680 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
```
* datanode report after the key was written:
```
Usage Information (1 Datanodes)
UUID : 8ac06e73-a434-4fd3-9be5-bc271b8f6bf7
IP Address : 172.23.0.15
Hostname : ozone-datanode10-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1252982784 B (1.17 GB)
Total Used % : 58.35%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 894500864 B (853.06 MB)
Remaining % : 41.65%
Container(s) : 1
Usage Information (1 Datanodes)
UUID : 9e3bf76a-8c98-4fa3-b034-c68973d7560b
IP Address : 172.23.0.10
Hostname : ozone-datanode3-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1252982784 B (1.17 GB)
Total Used % : 58.35%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 894500864 B (853.06 MB)
Remaining % : 41.65%
Container(s) : 1
Usage Information (1 Datanodes)
UUID : a5df4b7d-92b3-407a-b56d-aaeb924f0cd6
IP Address : 172.23.0.7
Hostname : ozone-datanode2-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1248776192 B (1.16 GB)
Total Used % : 58.15%
Ozone Used : 204800 B (200 KB)
Ozone Used % : 0.01%
Remaining : 898707456 B (857.07 MB)
Remaining % : 41.85%
Container(s) : 1
Usage Information (1 Datanodes)
UUID : e5a630fc-ce77-4211-89dc-15ab0ca67fde
IP Address : 172.23.0.6
Hostname : ozone-datanode8-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 31596544 B (30.13 MB)
Total Used % : 1.47%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 2115887104 B (1.97 GB)
Remaining % : 98.53%
Container(s) : 0
Usage Information (1 Datanodes)
UUID : 2d00f163-7f22-4395-b82b-e8353c74698b
IP Address : 172.23.0.5
Hostname : ozone-datanode7-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1252982784 B (1.17 GB)
Total Used % : 58.35%
Ozone Used : 204800 B (200 KB)
Ozone Used % : 0.01%
Remaining : 894500864 B (853.06 MB)
Remaining % : 41.65%
Container(s) : 1
Usage Information (1 Datanodes)
UUID : 3a22dc09-acc5-4270-8007-135056031ed9
IP Address : 172.23.0.12
Hostname : ozone-datanode9-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1252982784 B (1.17 GB)
Total Used % : 58.35%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 894500864 B (853.06 MB)
Remaining % : 41.65%
Container(s) : 1
Usage Information (1 Datanodes)
UUID : 4478a423-0416-429e-b41c-5e58b8ec7ed5
IP Address : 172.23.0.16
Hostname : ozone-datanode5-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1252982784 B (1.17 GB)
Total Used % : 58.35%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 894500864 B (853.06 MB)
Remaining % : 41.65%
Container(s) : 1
Usage Information (1 Datanodes)
UUID : 5ffda4af-e5dd-4b11-a945-7fef0b53356b
IP Address : 172.23.0.17
Hostname : ozone-datanode1-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1252982784 B (1.17 GB)
Total Used % : 58.35%
Ozone Used : 204800 B (200 KB)
Ozone Used % : 0.01%
Remaining : 894500864 B (853.06 MB)
Remaining % : 41.65%
Container(s) : 1
Usage Information (1 Datanodes)
UUID : 6fe5b830-5031-4a4c-a8fa-ae909c55d187
IP Address : 172.23.0.3
Hostname : ozone-datanode6-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1252982784 B (1.17 GB)
Total Used % : 58.35%
Ozone Used : 204800 B (200 KB)
Ozone Used % : 0.01%
Remaining : 894500864 B (853.06 MB)
Remaining % : 41.65%
Container(s) : 1
Usage Information (1 Datanodes)
UUID : 7ce23e7c-41bf-4c5e-aea7-63af06049ad0
IP Address : 172.23.0.19
Hostname : ozone-datanode4-1.ozone_default
Capacity : 2147483648 B (2 GB)
Total Used : 1252982784 B (1.17 GB)
Total Used % : 58.35%
Ozone Used : 0 B (0 B)
Ozone Used % : 0.00%
Remaining : 894500864 B (853.06 MB)
Remaining % : 41.65%
Container(s) : 1
```
WHAT ??? 200KiB written key has changed the remaining storage size of 9
datanodes from 1.97GiB to 853MB, looks too strange, doesn't it?
I would report the committed bytes as a separate report data field as the
remaining storage size will be more obvious for the user and the cluster admins
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
