[
https://issues.apache.org/jira/browse/HDDS-12810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Lee updated HDDS-12810:
-----------------------------
Description:
If VolumeChoosingPolicy#chooseVolume is not synchronized and caller required
space reserved before chooseVolume returns, there is no easy way to guarantee
chosen volume will have enough space.
VolumeChoosingPolicy#chooseVolume is used for create container and container
import. Make VolumeChoosingPolicy#chooseVolume synchronized or part of
chooseVolume synchronized will add some latency, which is negligible for
container import, may noticeable for create container. We can compare the
create container latency metrics with and without the synchronization.
VolumeChoosingPolicy#chooseVolume, its Java Doc says "The implementations of
this interface must be thread-safe.", regarding the space full check and space
reservation, it can be done as an atomic operation in the chooseVolume
internally, so that there will no over allocation of space due to concurrent
container creation and container import.
Also need to add test to check if committedBytes is checked in container
creation unit tests and container import, container replication unit tests. If
not, could you add the committedBytes checks? Besides, we also need the unit
tests to verify that if the container creation fails, container import fails,
or container replication fails, the resevered committedBytes are released.
was:
KeyValueHandler and ContainerImporter have its owner VolumeChoosingPolicy
instances. As the VolumeChoosingPolicy is decided by same
"hdds.datanode.volume.choosing.policy" property, so KeyValueHandler and
ContainerImporter can use the same VolumeChoosingPolicy instance.
The choosing policy is should initialized in {{DatanodeStateMachine and pass
the instance to both KeyValueHandler and ContainerImporter.}}
> Make chooseVolume synchronized and changes around committing space.
> -------------------------------------------------------------------
>
> Key: HDDS-12810
> URL: https://issues.apache.org/jira/browse/HDDS-12810
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Peter Lee
> Assignee: Peter Lee
> Priority: Major
> Fix For: 2.1.0
>
>
> If VolumeChoosingPolicy#chooseVolume is not synchronized and caller required
> space reserved before chooseVolume returns, there is no easy way to guarantee
> chosen volume will have enough space.
> VolumeChoosingPolicy#chooseVolume is used for create container and container
> import. Make VolumeChoosingPolicy#chooseVolume synchronized or part of
> chooseVolume synchronized will add some latency, which is negligible for
> container import, may noticeable for create container. We can compare the
> create container latency metrics with and without the synchronization.
>
> VolumeChoosingPolicy#chooseVolume, its Java Doc says "The implementations of
> this interface must be thread-safe.", regarding the space full check and
> space reservation, it can be done as an atomic operation in the chooseVolume
> internally, so that there will no over allocation of space due to concurrent
> container creation and container import.
>
> Also need to add test to check if committedBytes is checked in container
> creation unit tests and container import, container replication unit tests.
> If not, could you add the committedBytes checks? Besides, we also need the
> unit tests to verify that if the container creation fails, container import
> fails, or container replication fails, the resevered committedBytes are
> released.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
