[
https://issues.apache.org/jira/browse/HDDS-12658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddhant Sangwan resolved HDDS-12658.
-------------------------------------
Resolution: Duplicate
> Verify datanode space availability when selecting pipeline to write to
> ----------------------------------------------------------------------
>
> Key: HDDS-12658
> URL: https://issues.apache.org/jira/browse/HDDS-12658
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM
> Reporter: Siddhant Sangwan
> Assignee: Siddhant Sangwan
> Priority: Major
>
> When writing in Ozone, SCM selects a container to write to. We have a
> PipelineChoosePolicy interface that's used to select which pipeline the write
> will go to. The default implementation picks a pipeline randomly. However it
> doesn't check whether the Datanodes in this pipeline have enough disk space
> available. If a full pipeline is handed over to the client, it can cause
> writes to fail and reduce write performance.
> We do have a check later on in that code path that checks whether the chosen
> container has sufficient space:
> {code}
> containerInfo.getUsedBytes() + size <= this.containerSize
> {code}
> But this in itself is not sufficient.
> I propose adding some common logic to only pick a pipeline if all the
> Datanodes in that pipeline have sufficient available space. All
> implementations of PipelineChoosePolicy should use it. I need to check how
> this area is handling parallel writes and other synchronization/race details.
> Note that we also have another implementation of the PipelineChoosePolicy,
> CapacityPipelineChoosePolicy:
> {code}
> Pipeline choose policy that randomly choose pipeline with relatively lower
> utilization.
> {code}
> I think all policies including this one will benefit from having the proposed
> check.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
