[
https://issues.apache.org/jira/browse/HDDS-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700980#comment-17700980
]
Tsz-wo Sze commented on HDDS-8128:
----------------------------------
We can explain the 7.5 GB OM rocks db usage as below.
MultipartKeyInfo has a list of PartKeyInfo (partKeyInfoList) while each
PartKeyInfo has a KeyInfo and the size of each KeyInfo is ~1.5 KB.
{code}
message MultipartKeyInfo {
required string uploadID = 1;
...
repeated PartKeyInfo partKeyInfoList = 5;
...
}
message PartKeyInfo {
required string partName = 1;
required uint32 partNumber = 2;
required KeyInfo partKeyInfo = 3; // the size of a KeyInfo is ~1.5 KB
}
{code}
Multipart upload repeatedly put the same key with a MultipartKeyInfo object for
each part.
# When uploading the first part, partKeyInfoList has 1 element.
# When uploading the second part, partKeyInfoList has 2 elements.
# When uploading the third part, partKeyInfoList has 3 elements.
...
1000. When uploading the 1000th part, partKeyInfoList has 1000 elements.
Then, the number of PartKeyInfo objects uploaded is
- (1 + 1000)*1000/2 = 500,500 and
- the size is 1.5 KB * 500,500 = 733.15 MB.
Since we have uploaded 10 times in the test, the size become 7.16 GB.
> OM rocksdb uses a lot of space
> ------------------------------
>
> Key: HDDS-8128
> URL: https://issues.apache.org/jira/browse/HDDS-8128
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Tsz-wo Sze
> Assignee: Tsz-wo Sze
> Priority: Blocker
>
> In a multipart upload test, the key "testKey" had 1000-parts with 8KB each.
> The same key was uploaded 10 times sequentially (i.e. it overwrote the
> previous upload) in a newly formatted cluster. The replication was 3, so the
> total raw size of the key is ~ 24 MB. After the test has completed, OM rocks
> db uses ~ 7.5 GB.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
