Yashaswini G A created HDDS-15862:
-------------------------------------
Summary: Snapshot incremental defrag fails at scale with -
External file version not found (~5k snapshots stuck with needsDefrag=true)
Key: HDDS-15862
URL: https://issues.apache.org/jira/browse/HDDS-15862
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Manager
Reporter: Yashaswini G A
During snapshot defrag boundary QE testing (~10k snapshots on a single bucket),
OM snapshot defrag fails to drain the needsDefrag backlog. Approximately half
of snapshot checkpoint yaml entries remain stuck at needsDefrag=true.
OM logs error stack
2026-07-13 03:43:16,516 INFO
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Defragmenting snapshot:
/vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441
(ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:16,516 INFO
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading SnapshotId: '082ca862-7220-4e26-98b2-190486862fbe'
2026-07-13 03:43:16,517 INFO
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,659 INFO
[SnapshotDefragService#0]-org.apache.hadoop.hdds.utils.db.RDBCheckpointManager:
Created checkpoint in rocksDB at
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590
in 68 milliseconds
2026-07-13 03:43:16,659 INFO
[SnapshotDefragService#0]-org.apache.hadoop.hdds.utils.db.RDBCheckpointUtils:
Waited for 0 milliseconds for checkpoint directory
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590
availability.
2026-07-13 03:43:16,660 INFO
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:16,893 INFO
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,117 INFO
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Performing incremental defragmentation for snapshot:
/vol-defrag-boundary-1781889720/buck-defrag-boundary-1781889720/snap-002441
(ID: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda)
2026-07-13 03:43:17,348 INFO
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.SnapshotCache:
Loading SnapshotId: 'eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda'
2026-07-13 03:43:17,349 INFO
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.helpers.OmKeyInfo:
OmKeyInfo.getCodec ignorePipeline = true
2026-07-13 03:43:17,420 ERROR
[SnapshotDefragService#0]-org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService:
Exception while defragmenting snapshot: eae48f14-9c6b-4c54-b0c3-7f6dfcbf3eda
org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Corruption:
RocksDatabase[/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/om.db-082ca862-7220-4e26-98b2-190486862fbe-4_checkpoint_1783939396590]:
Failed to Failed to ingest external files
/var/lib/hadoop-ozone/om/data889052/db.snapshots/checkpointState/tmp_defrag/differSstFiles/rdbDiffer/1072.sst
of fileTable
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:475)
at
org.apache.hadoop.hdds.utils.db.RDBSstFileLoader.load(RDBSstFileLoader.java:39)
at
org.apache.hadoop.hdds.utils.db.RDBTable.loadFromFile(RDBTable.java:317)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.performIncrementalDefragmentation(SnapshotDefragService.java:426)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.checkAndDefragSnapshot(SnapshotDefragService.java:637)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService.triggerSnapshotDefragOnce(SnapshotDefragService.java:692)
at
org.apache.hadoop.ozone.om.snapshot.defrag.SnapshotDefragService$SnapshotDefragTask.call(SnapshotDefragService.java:540)
at
org.apache.hadoop.hdds.utils.BackgroundService$PeriodicalTask.lambda$run$0(BackgroundService.java:155)
at
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.rocksdb.RocksDBException: An external sst file with version 2
have global seqno property with value ^A, while largest seqno in the file is 0
at org.rocksdb.RocksDB.ingestExternalFile(Native Method)
at org.rocksdb.RocksDB.ingestExternalFile(RocksDB.java:4060)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.ingestExternalFile(RocksDatabase.java:469)
... 14 more
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
