[
https://issues.apache.org/jira/browse/HDDS-13062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-13062:
------------------------------
Summary: Upon OM startup, verify that ozone.metadata.dirs and
ozone.om.db.dirs to be in the same mount (was: Verify that ozone.metadata.dirs
and ozone.om.db.dirs to be in the same mount upon OM startup)
> Upon OM startup, verify that ozone.metadata.dirs and ozone.om.db.dirs to be
> in the same mount
> ---------------------------------------------------------------------------------------------
>
> Key: HDDS-13062
> URL: https://issues.apache.org/jira/browse/HDDS-13062
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
>
> h2. Problem
> When {{ozone.metadata.dirs}} and {{ozone.om.db.dirs}} are set to different
> mounts, the following exception would occur:
> {code}
> 2025-05-03 20:49:12,016 ERROR
> [pool-30-thread-1]-org.apache.hadoop.ozone.om.OzoneManager: Failed to move
> downloaded DB checkpoint
> /hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate to
> metadata directory /hadoop-ozone/datanode02/om/data/om.db. Exception: {}.
> Resetting to original DB.
> java.nio.file.FileSystemException:
> /hadoop-ozone/datanode02/om/data/om.db/000079.sst ->
> /hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate/000079.sst:
> Invalid cross-device link
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
> at java.nio.file.Files.createLink(Files.java:1086)
> at
> org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.linkFiles(OmSnapshotUtils.java:169)
> at
> org.apache.hadoop.ozone.om.OzoneManager.moveCheckpointFiles(OzoneManager.java:3880)
> at
> org.apache.hadoop.ozone.om.OzoneManager.replaceOMDBWithCheckpoint(OzoneManager.java:3860)
> at
> org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3734)
> at
> org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3669)
> at
> org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3646)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$5(OzoneManagerStateMachine.java:505)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> 2025-05-03 20:49:12,016 ERROR
> [pool-30-thread-1]-org.apache.hadoop.ozone.om.OzoneManager: Failed to install
> Snapshot from om1546339340 as OM failed to replace DB with downloaded
> checkpoint. Reloading old OM state.
> java.nio.file.FileSystemException:
> /hadoop-ozone/datanode02/om/data/om.db/000079.sst ->
> /hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate/000079.sst:
> Invalid cross-device link
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
> at java.nio.file.Files.createLink(Files.java:1086)
> at
> org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.linkFiles(OmSnapshotUtils.java:169)
> at
> org.apache.hadoop.ozone.om.OzoneManager.moveCheckpointFiles(OzoneManager.java:3880)
> at
> org.apache.hadoop.ozone.om.OzoneManager.replaceOMDBWithCheckpoint(OzoneManager.java:3860)
> at
> org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3734)
> at
> org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3669)
> at
> org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3646)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$5(OzoneManagerStateMachine.java:505)
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> At this point, the follower OM won't ever succeed bootstrapping and will lag
> behind the leader indefinitely.
> h2. Solution
> 1. Upon OM startup, verify that {{ozone.metadata.dirs}} and
> {{ozone.om.db.dirs}} are in the same mount
> 2. If possible, also check that the partition mount supports hardlinks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
