[
https://issues.apache.org/jira/browse/HDDS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730366#comment-17730366
]
Dave Teng commented on HDDS-8786:
---------------------------------
Two other exceptions during snapshot creation:
{code:java}
2023-06-08 02:01:14,769 [OMDoubleBufferFlushThread] ERROR
org.apache.hadoop.hdds.utils.db.RDBCheckpointManager: Unable to create RocksDB
Snapsho
t.
java.io.IOException: RocksDatabase[/var/lib/hadoop-ozone/om/data/om.db]: Failed
to flush; status : IOError; message : While open a file for appe
nding: /var/lib/hadoop-ozone/om/data/om.db/050410.sst: Too many open files
at
org.apache.hadoop.hdds.utils.HddsServerUtil.toIOException(HddsServerUtil.java:588)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.toIOException(RocksDatabase.java:88)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.flush(RocksDatabase.java:502)
at
org.apache.hadoop.hdds.utils.db.RDBCheckpointManager.createCheckpoint(RDBCheckpointManager.java:81)
at
org.apache.hadoop.hdds.utils.db.RDBStore.getSnapshot(RDBStore.java:325)
at
org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:391)
at
org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
at
org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:414)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:237)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:412)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:333)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:312)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:277)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.rocksdb.RocksDBException: While open a file for appending:
/var/lib/hadoop-ozone/om/data/om.db/050410.sst: Too many open files
at org.rocksdb.RocksDB.flush(Native Method)
at org.rocksdb.RocksDB.flush(RocksDB.java:3785)
at org.rocksdb.RocksDB.flush(RocksDB.java:3763)
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.flush(RocksDatabase.java:498)
... 12 more {code}
> Encounter error of OzoneManagerDoubleBuffer Terminating during snapshot
> creation
> --------------------------------------------------------------------------------
>
> Key: HDDS-8786
> URL: https://issues.apache.org/jira/browse/HDDS-8786
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Dave Teng
> Priority: Major
> Labels: ozone-snapshot
>
> When creating snapshot, encountered following error in OM:
> {code:java}
> 2023-06-08 02:01:14,771 [OMDoubleBufferFlushThread] ERROR
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with
> exit statu
> s 1: During flush to DB encountered error in OMDoubleBuffer flush thread
> OMDoubleBufferFlushThread when handling OMRequest: cmdType: CreateSnaps
> hot
> traceID: ""
> success: true
> status: OK
> CreateSnapshotResponse {
> snapshotInfo {
> snapshotID: "3ce30204-da71-41ad-9e78-607c688e8c5c"
> name: "snap5670"
> volumeName: "vol2"
> bucketName: "bucket2"
> snapshotStatus: SNAPSHOT_ACTIVE
> creationTime: 1686189673557
> deletionTime: 18446744073709551615
> pathPreviousSnapshotID: "180bd9f4-a83e-4bfb-97b3-a19d0eef0450"
> globalPreviousSnapshotID: "180bd9f4-a83e-4bfb-97b3-a19d0eef0450"
> snapshotPath: "vol2/bucket2"
> checkpointDir: "-3ce30204-da71-41ad-9e78-607c688e8c5c"
> dbTxSequenceNumber: 3269138
> }
> }
> java.io.IOException: Rocks Database is closed
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.assertClose(RocksDatabase.java:442)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:827)
> at
> org.apache.hadoop.hdds.utils.db.RDBTable.iterator(RDBTable.java:194)
> at
> org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:361)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager.deleteKeysFromDelKeyTableInSnapshotScope(OmSnapshotManager.java:573)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:394)
> at
> org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
> at
> org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:414)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:237)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:412)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:333)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:312)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:277)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
