[
https://issues.apache.org/jira/browse/HDDS-15187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-15187:
-----------------------------------
Description:
h1. Symptoms
Files marked for deletion are never marked reclaimable because when a snapshot
is deleted, its contents are appended to the next Active snapshot. Because
there is always an Active snapshot remaining when a snapshot is deleted, the
files marked for deletion will be kept with the current Active snapshots
perpetually. We can see this mechanism at work in the Ozone Manager role logs
(/var/log/hadoop-ozone/ozone-om.log by default) where log messages specify
“entries will be moved to next active snapshot”:
{noformat}
2026-04-14 22:21:02,311 INFO
[SnapshotDeletingService#0]-org.apache.hadoop.ozone.om.service.SnapshotDeletingService:
Started Snapshot Deletion Processing for snapshot :
/snaptest1/snapbucket1/s20260414-212019.561
2026-04-14 22:21:02,312 INFO
[SnapshotDeletingService#0]-org.apache.hadoop.ozone.om.service.SnapshotDeletingService:
Snapshot: /snaptest1/snapbucket1/s20260414-212019.561 entries will be moved to
next active snapshot: /snaptest1/snapbucket1/s20260414-214018.921
{noformat}
h1. Proposed solution
The Ozone Snapshot GC project (HDDS-12558) refactored Ozone Snapshot
implementation. HDDS-11411 added a configuration property
ozone.snapshot.deep.cleaning.enabled which moves deleted keys from AOS or
deleted snapshot to the next snapshot. But this flag is false by default.
We need to enable this flag by default to prevent orphaned files accumulating
in the cluster.
h1. Workaround
Set ozone.snapshot.deep.cleaning.enabled=true in the Ozone Manager's
ozone-site.xml.
was:
h1. Symptoms
Files marked for deletion are never marked reclaimable because when a snapshot
is deleted, its contents are appended to the next Active snapshot. Because
there is always an Active snapshot remaining when a snapshot is deleted, the
files marked for deletion will be kept with the current Active snapshots
perpetually. We can see this mechanism at work in the Ozone Manager role logs
(/var/log/hadoop-ozone/ozone-om.log by default) where log messages specify
“entries will be moved to next active snapshot”:
2026-04-14 22:21:02,311 INFO
[SnapshotDeletingService#0]-org.apache.hadoop.ozone.om.service.SnapshotDeletingService:
Started Snapshot Deletion Processing for snapshot :
/snaptest1/snapbucket1/s20260414-212019.561
2026-04-14 22:21:02,312 INFO
[SnapshotDeletingService#0]-org.apache.hadoop.ozone.om.service.SnapshotDeletingService:
Snapshot: /snaptest1/snapbucket1/s20260414-212019.561 entries will be moved to
next active snapshot: /snaptest1/snapbucket1/s20260414-214018.921
h1. Proposed solution
The Ozone Snapshot GC project (HDDS-12558) refactored Ozone Snapshot
implementation. HDDS-11411 added a configuration property
ozone.snapshot.deep.cleaning.enabled which moves deleted keys from AOS or
deleted snapshot to the next snapshot. But this flag is false by default.
We need to enable this flag by default to prevent orphaned files accumulating
in the cluster.
h1. Workaround
Set ozone.snapshot.deep.cleaning.enabled=true in the Ozone Manager's
ozone-site.xml.
> Enable ozone.snapshot.deep.cleaning.enabled by default
> ------------------------------------------------------
>
> Key: HDDS-15187
> URL: https://issues.apache.org/jira/browse/HDDS-15187
> Project: Apache Ozone
> Issue Type: Task
> Components: Snapshot
> Affects Versions: 2.2.0
> Reporter: Wei-Chiu Chuang
> Priority: Blocker
>
> h1. Symptoms
> Files marked for deletion are never marked reclaimable because when a
> snapshot is deleted, its contents are appended to the next Active snapshot.
> Because there is always an Active snapshot remaining when a snapshot is
> deleted, the files marked for deletion will be kept with the current Active
> snapshots perpetually. We can see this mechanism at work in the Ozone Manager
> role logs (/var/log/hadoop-ozone/ozone-om.log by default) where log messages
> specify “entries will be moved to next active snapshot”:
> {noformat}
> 2026-04-14 22:21:02,311 INFO
> [SnapshotDeletingService#0]-org.apache.hadoop.ozone.om.service.SnapshotDeletingService:
> Started Snapshot Deletion Processing for snapshot :
> /snaptest1/snapbucket1/s20260414-212019.561
> 2026-04-14 22:21:02,312 INFO
> [SnapshotDeletingService#0]-org.apache.hadoop.ozone.om.service.SnapshotDeletingService:
> Snapshot: /snaptest1/snapbucket1/s20260414-212019.561 entries will be moved
> to next active snapshot: /snaptest1/snapbucket1/s20260414-214018.921
> {noformat}
> h1. Proposed solution
> The Ozone Snapshot GC project (HDDS-12558) refactored Ozone Snapshot
> implementation. HDDS-11411 added a configuration property
> ozone.snapshot.deep.cleaning.enabled which moves deleted keys from AOS or
> deleted snapshot to the next snapshot. But this flag is false by default.
> We need to enable this flag by default to prevent orphaned files accumulating
> in the cluster.
> h1. Workaround
> Set ozone.snapshot.deep.cleaning.enabled=true in the Ozone Manager's
> ozone-site.xml.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
