[
https://issues.apache.org/jira/browse/HDDS-15862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yashaswini G A updated HDDS-15862:
----------------------------------
Description:
During snapshot defrag boundary QE testing (~10k snapshots on a single bucket),
OM snapshot defrag fails to drain the needsDefrag backlog. Approximately half
of snapshot checkpoint yaml entries remain stuck at needsDefrag=true.
OM logs error stack
{code:java}
2026-07-13 03:43:16,516 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Defragmenting snapshot:
/vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441
(ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:16,516 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading SnapshotId: '082ca862-7220-4e26-98b2-190486862fbe'
2026-07-13 03:43:16,517 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,659 INFO
SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointManager:
Created checkpoint in rocksDB at
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590
in 68 milliseconds
2026-07-13 03:43:16,659 INFO
SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointUtils:
Waited for 0 milliseconds for checkpoint directory
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590
availability.
2026-07-13 03:43:16,660 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,893 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,117 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Performing incremental defragmentation for snapshot:
/vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441
(ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:17,348 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading SnapshotId: 'eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda'
2026-07-13 03:43:17,349 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,420 ERROR
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Exception while defragmenting snapshot: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda
org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Corruption:
RocksDatabase[/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590]:
Failed to Failed to ingest external files
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/differSstFiles/rdbDiffer/1072.sst
of fileTable
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:475)
at
org.apache.hadoop.hdds.utils.db.RDBSstFileLoader.load(RDBSstFileLoader.java:39)
at
org.apache.hadoop.hdds.utils.db.RDBTable.loadFromFile(RDBTable.java:317)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.performIncrementalDefragmentation(SnapshotDefragService.java:426)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.checkAndDefragSnapshot(SnapshotDefragService.java:637)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.triggerSnapshotDefragOnce(SnapshotDefragService.java:692)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService$SnapshotDefragTask.call(SnapshotDefragService.java:540)
at
org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:155)
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.rocksdb.RocksDBException: An external sst file with version 2
have global seqno property with value ^A, while largest seqno in the file is 0
at org.rocksdb.RocksDB.ingestExternalFile(Native Method)
at org.rocksdb.RocksDB.ingestExternalFile(RocksDB.java:4060)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:469)
... 14 more{code}
*Steps to reproduce*
* Create ~10k snapshots on a single bucket (boundary test workload with
periodic random deletes and snapdiff every 5k).
* Wait for OM defrag cycles (ozone.om.snapshot.defrag.interval default 10 min).
* Check needsDefrag counts on OM leader
* Observe that thousands of snapshots remain with needsDefrag=true, OM logs
show "External file version not found" during defrag.
was:
During snapshot defrag boundary QE testing (~10k snapshots on a single bucket),
OM snapshot defrag fails to drain the needsDefrag backlog. Approximately half
of snapshot checkpoint yaml entries remain stuck at needsDefrag=true.
OM logs error stack
{code:java}
2026-07-13 03:43:16,516 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Defragmenting snapshot:
/vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441
(ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:16,516 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading SnapshotId: '082ca862-7220-4e26-98b2-190486862fbe'
2026-07-13 03:43:16,517 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,659 INFO
SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointManager:
Created checkpoint in rocksDB at
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590
in 68 milliseconds
2026-07-13 03:43:16,659 INFO
SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointUtils:
Waited for 0 milliseconds for checkpoint directory
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590
availability.
2026-07-13 03:43:16,660 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,893 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,117 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Performing incremental defragmentation for snapshot:
/vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441
(ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:17,348 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading SnapshotId: 'eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda'
2026-07-13 03:43:17,349 INFO
SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,420 ERROR
SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Exception while defragmenting snapshot: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda
org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Corruption:
RocksDatabase[/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590]:
Failed to Failed to ingest external files
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/differSstFiles/rdbDiffer/1072.sst
of fileTable
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:475)
at
org.apache.hadoop.hdds.utils.db.RDBSstFileLoader.load(RDBSstFileLoader.java:39)
at
org.apache.hadoop.hdds.utils.db.RDBTable.loadFromFile(RDBTable.java:317)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.performIncrementalDefragmentation(SnapshotDefragService.java:426)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.checkAndDefragSnapshot(SnapshotDefragService.java:637)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.triggerSnapshotDefragOnce(SnapshotDefragService.java:692)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService$SnapshotDefragTask.call(SnapshotDefragService.java:540)
at
org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:155)
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.rocksdb.RocksDBException: An external sst file with version 2
have global seqno property with value ^A, while largest seqno in the file is 0
at org.rocksdb.RocksDB.ingestExternalFile(Native Method)
at org.rocksdb.RocksDB.ingestExternalFile(RocksDB.java:4060)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:469)
... 14 more{code}
> Snapshot incremental defrag fails at scale with - External file version not
> found (~5k snapshots stuck with needsDefrag=true)
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-15862
> URL: https://issues.apache.org/jira/browse/HDDS-15862
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Yashaswini G A
> Priority: Major
>
> During snapshot defrag boundary QE testing (~10k snapshots on a single
> bucket), OM snapshot defrag fails to drain the needsDefrag backlog.
> Approximately half of snapshot checkpoint yaml entries remain stuck at
> needsDefrag=true.
> OM logs error stack
> {code:java}
> 2026-07-13 03:43:16,516 INFO
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
> Defragmenting snapshot:
> /vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441
> (ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
> 2026-07-13 03:43:16,516 INFO
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
> Loading SnapshotId: '082ca862-7220-4e26-98b2-190486862fbe'
> 2026-07-13 03:43:16,517 INFO
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
> OmKeyInfo.getCodec ignorePipeline = true
> 2026-07-13 03:43:16,659 INFO
> SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointManager:
> Created checkpoint in rocksDB at
> /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590
> in 68 milliseconds
> 2026-07-13 03:43:16,659 INFO
> SnapshotDefragService#0-org.apache.hadoop.hdds.utils.db.RDBCheckpointUtils:
> Waited for 0 milliseconds for checkpoint directory
> /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590
> availability.
> 2026-07-13 03:43:16,660 INFO
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
> OmKeyInfo.getCodec ignorePipeline = true
> 2026-07-13 03:43:16,893 INFO
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
> OmKeyInfo.getCodec ignorePipeline = true
> 2026-07-13 03:43:17,117 INFO
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
> Performing incremental defragmentation for snapshot:
> /vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441
> (ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
> 2026-07-13 03:43:17,348 INFO
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
> Loading SnapshotId: 'eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda'
> 2026-07-13 03:43:17,349 INFO
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
> OmKeyInfo.getCodec ignorePipeline = true
> 2026-07-13 03:43:17,420 ERROR
> SnapshotDefragService#0-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
> Exception while defragmenting snapshot: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda
> org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Corruption:
> RocksDatabase[/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590]:
> Failed to Failed to ingest external files
> /var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/differSstFiles/rdbDiffer/1072.sst
> of fileTable
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:475)
> at
> org.apache.hadoop.hdds.utils.db.RDBSstFileLoader.load(RDBSstFileLoader.java:39)
> at
> org.apache.hadoop.hdds.utils.db.RDBTable.loadFromFile(RDBTable.java:317)
> at
> org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.performIncrementalDefragmentation(SnapshotDefragService.java:426)
> at
> org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.checkAndDefragSnapshot(SnapshotDefragService.java:637)
> at
> org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.triggerSnapshotDefragOnce(SnapshotDefragService.java:692)
> at
> org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService$SnapshotDefragTask.call(SnapshotDefragService.java:540)
> at
> org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:155)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: org.rocksdb.RocksDBException: An external sst file with version 2
> have global seqno property with value ^A, while largest seqno in the file is > 0
> at org.rocksdb.RocksDB.ingestExternalFile(Native Method)
> at org.rocksdb.RocksDB.ingestExternalFile(RocksDB.java:4060)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:469)
> ... 14 more{code}
>
> *Steps to reproduce*
> * Create ~10k snapshots on a single bucket (boundary test workload with
> periodic random deletes and snapdiff every 5k).
> * Wait for OM defrag cycles (ozone.om.snapshot.defrag.interval default 10
> min).
> * Check needsDefrag counts on OM leader
> * Observe that thousands of snapshots remain with needsDefrag=true, OM logs
> show "External file version not found" during defrag.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
