[
https://issues.apache.org/jira/browse/HDDS-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-13173:
------------------------------
Status: Patch Available (was: Open)
> [snapshot] OM start failed with exception java.io.IOException
> -------------------------------------------------------------
>
> Key: HDDS-13173
> URL: https://issues.apache.org/jira/browse/HDDS-13173
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Jyotirmoy Sinha
> Priority: Major
> Labels: ozone-snapshot, pull-request-available
>
> Steps :
> * Execute snapshot load workloads
> * Workload executed for 3+ days
> * Workload details -
> ** Volume - 100
> ** Buckets - 400 (4 per volume)
> *** Ratis-fso
> *** Ratis-obs
> *** Ec-fso
> *** Ec-obs
> ** Keys -
> *** 500-1000 per snapshot
> *** Size = 10kb
> * Configs -
> ** ozone.compaction.service.enabled = True
> ** ozone.om.compaction.service.run.interval = 1m
> ** ozone.om.snapshot.compaction.dag.prune.daemon.run.interval = 5m
> ** ozone.scm.block.size = 1KB
>
> OM Error stacktrace -
> {code:java}
> 2025-06-04 05:09:28,028 INFO
> [main]-org.apache.hadoop.hdds.utils.NativeLibraryLoader: Loading Library:
> ozone_rocksdb_tools
> 2025-06-04 05:09:28,312 INFO
> [main]-org.apache.ozone.rocksdiff.RocksDBCheckpointDiffer: Shutting down
> CompactionDagPruningService.
> 2025-06-04 05:09:28,313 ERROR
> [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with
> exception
> java.io.IOException: Failed to create RDBStore from
> /var/lib/hadoop-ozone/om/data/om.db
> at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:186)
> at
> org.apache.hadoop.hdds.utils.db.DBStoreBuilder.build(DBStoreBuilder.java:236)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:603)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.loadDB(OmMetadataManagerImpl.java:564)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.start(OmMetadataManagerImpl.java:554)
> at
> org.apache.hadoop.ozone.om.OmMetadataManagerImpl.<init>(OmMetadataManagerImpl.java:338)
> at
> org.apache.hadoop.ozone.om.OzoneManager.instantiateServices(OzoneManager.java:909)
> at
> org.apache.hadoop.ozone.om.OzoneManager.<init>(OzoneManager.java:685)
> at
> org.apache.hadoop.ozone.om.OzoneManager.createOm(OzoneManager.java:874)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.start(OzoneManagerStarter.java:189)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.startOm(OzoneManagerStarter.java:86)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.call(OzoneManagerStarter.java:74)
> at org.apache.hadoop.hdds.cli.GenericCli.call(GenericCli.java:38)
> at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
> at picocli.CommandLine.access$1300(CommandLine.java:145)
> at
> picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
> at
> picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
> at picocli.CommandLine.execute(CommandLine.java:2078)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:103)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:94)
> at
> org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58)
> Caused by: org.apache.hadoop.hdds.utils.db.RocksDatabaseException:
> IOError(NoSpace): class org.apache.hadoop.hdds.utils.db.RocksDatabase: Failed
> to open /var/lib/hadoop-ozone/om/data/om.db
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.toRocksDatabaseException(RocksDatabase.java:93)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:170)
> at org.apache.hadoop.hdds.utils.db.RDBStore.<init>(RDBStore.java:118)
> ... 22 more
> Caused by: org.rocksdb.RocksDBException: While appending to file:
> /var/lib/hadoop-ozone/om/data/om.db/062092.dbtmp: No space left on device
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:307)
> at
> org.apache.hadoop.hdds.utils.db.managed.ManagedRocksDB.open(ManagedRocksDB.java:84)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.open(RocksDatabase.java:156)
> ... 23 more
> 2025-06-04 05:09:28,318 INFO
> [shutdown-hook-0]-org.apache.hadoop.ozone.om.OzoneManagerStarter:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down OzoneManager at
> ccycloud-6.quasar-vckttz.root.comops.site/10.140.147.64
> ************************************************************/ {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
