[
https://issues.apache.org/jira/browse/HDDS-12691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17938260#comment-17938260
]
Sumit Agrawal commented on HDDS-12691:
--------------------------------------
[~siddhant] decrement = Math.min(bytes, unused); is used to avoid going -ve
value of committed for the container.
There are possibility for container crossing the committed space, and under
boundary condition, there is chance it can cross volume capacity assigned, and
may cause write failure, but pre-closer for container is triggered when
reaching 90% of capacity to SCM.
Need re-look solution like,
# closing container hard way if availability of disk space is less than <
hdds.datanode.volume.min.free.space
# appropriate configuration of hdds.datanode.volume.min.free.space based on
volume size like in in 100s of TB.
> Calculation of committed space in Datanode seems incorrect
> ----------------------------------------------------------
>
> Key: HDDS-12691
> URL: https://issues.apache.org/jira/browse/HDDS-12691
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Siddhant Sangwan
> Priority: Major
>
> The following method is called on every writeChunk request, after writing
> bytes to the disk to update stats:
> {code}
> public void incrWriteBytes(long bytes) {
> long unused = getMaxSize() - getBytesUsed();
> this.writeBytes.addAndGet(bytes);
> /*
> Increase the cached Used Space in VolumeInfo as it
> maybe not updated, DU or DedicatedDiskSpaceUsage runs
> periodically to update the Used Space in VolumeInfo.
> */
> this.getVolume().incrementUsedSpace(bytes);
> // only if container size < max size
> if (committedSpace && unused > 0) {
> //with this write, container size might breach max size
> long decrement = Math.min(bytes, unused);
> this.getVolume().incCommittedBytes(0 - decrement);
> }
> }
> {code}
> For a container, 5GB (default) is committed when the container is created.
> Then every writeChunk is supposed to increment usedSpace and decrement
> committedSpace. Here, the logic of long decrement = Math.min(bytes, unused);
> seems incorrect to me.
> Consider that the latest writeChunk wrote 256MB into a close-to-full
> container. Assume that now, unused = getMaxSize() - getBytesUsed() = 100MB.
> The current logic would decrement the committedSpace by 100MB, whereas it
> should have been decremented by 256MB.
> Moreover, what about the case where the because of the latest chunk, the used
> size of the container exceeds the max size (used > max, for example I've seen
> 13GB containers in actual clusters since the max is just a soft limit),
> shouldn't we still decrement the committed space by getMaxSize() -
> (getBytesUsed() - bytes)? That's the remaining amount of committed space that
> this container was taking up in the volume before the "bytes" sized chunk was
> written to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
