[
https://issues.apache.org/jira/browse/HDDS-4961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17313550#comment-17313550
]
Sammi Chen commented on HDDS-4961:
----------------------------------
Hey [~weichiu], Delete in Ozone is a lazy way. When a key is request to
delete, OM will mark the key as deleted and put the blocks related into a
pending delete CF. OM will schedule tasks in background, which sends block
delete command to SCM. And SCM just do the same as OM. When SCM send delete
block command to DN, DN also just record the command and do it lazily. So
eventually, the blocks will be deleted, just not that fast.
> Disk space leak after deleting Ozone files
> ------------------------------------------
>
> Key: HDDS-4961
> URL: https://issues.apache.org/jira/browse/HDDS-4961
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> I'm finding cases where deleting Ozone files does not reclaim disk space,
> even if the corresponding containers are empty and in CLOSED state.
> In some cases, blocks are removed from container, resulting in empty and
> closed container. However, the container metadata is not removed, and SCM
> still records the container. No way to remove them. In this case, each
> dangling container occupies ~140mb of space. Overtime, tens or even hundreds
> of GB is wasted on each DN.
> Because of HDDS-2534, there is no way to manually delete the containers.
> Steps to reproduce:
> {code}
> ozone sh volume create o3://ozone1/vol1
> ozone sh bucket create o3://ozone1/vol1/bucket1
> # Generated data.
> export EXAMPLES_PATH=
> yarn jar
> /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce//hadoop-mapreduce-examples.jar
> teragen -Dmapreduce.job.maps=10 -DmDmapreduce.map.memory.mb=4096
> -Dmapreduce.reduce.memory.mb=4096 1000000000
> o3fs://bucket1.vol1.ozone1/teragen11
> Delete skip trash:
> hdfs dfs -rm -r -skipTrash "o3fs://bucket1.vol1.ozone1/*”
> (run "ozone admin container close ${containerid} to force closing the
> containers)
> # check disk usage:
> du -h
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
> # the container directory is not removed
> ...
> 142M
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/34-dn-container.db
> 4.0K
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata/db.checkpoints
> 142M
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/metadata
> 4.0K
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34/chunks
> 142M
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/34
> 13G
> /hadoop-ozone/datanode/data/hdds/8a1c8d3a-3a36-4852-a7b6-234ee48b7fb7/current/containerDir0/
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
