[
https://issues.apache.org/jira/browse/HDDS-12691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939990#comment-17939990
]
Peter Lee commented on HDDS-12691:
----------------------------------
Sure, thanks.
> Calculation of committed space in Datanode seems incorrect
> ----------------------------------------------------------
>
> Key: HDDS-12691
> URL: https://issues.apache.org/jira/browse/HDDS-12691
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Siddhant Sangwan
> Assignee: Peter Lee
> Priority: Major
>
> The following method is called on every writeChunk request, after writing
> bytes to the disk to update stats:
> {code}
> public void incrWriteBytes(long bytes) {
> long unused = getMaxSize() - getBytesUsed();
> this.writeBytes.addAndGet(bytes);
> /*
> Increase the cached Used Space in VolumeInfo as it
> maybe not updated, DU or DedicatedDiskSpaceUsage runs
> periodically to update the Used Space in VolumeInfo.
> */
> this.getVolume().incrementUsedSpace(bytes);
> // only if container size < max size
> if (committedSpace && unused > 0) {
> //with this write, container size might breach max size
> long decrement = Math.min(bytes, unused);
> this.getVolume().incCommittedBytes(0 - decrement);
> }
> }
> {code}
> For a container, 5GB (default) is committed when the container is created.
> Then every writeChunk is supposed to increment usedSpace and decrement
> committedSpace. Here, the logic of long decrement = Math.min(bytes, unused);
> seems incorrect to me.
> Consider that the latest writeChunk wrote 256MB into a close-to-full
> container. Assume that now, unused = getMaxSize() - getBytesUsed() = 100MB.
> The current logic would decrement the committedSpace by 100MB, whereas it
> should have been decremented by 256MB.
> Moreover, what about the case where the because of the latest chunk, the used
> size of the container exceeds the max size (used > max, for example I've seen
> 13GB containers in actual clusters since the max is just a soft limit),
> shouldn't we still decrement the committed space by getMaxSize() -
> (getBytesUsed() - bytes)? That's the remaining amount of committed space that
> this container was taking up in the volume before the "bytes" sized chunk was
> written to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
