[
https://issues.apache.org/jira/browse/HDDS-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kodali Bhavya Sree updated HDDS-14653:
--------------------------------------
Description:
*Actual:* Ratis pipeline is getting allocated, but not moving to the open state
as creating directory operation is failing on Data node. It then times out and
tries again after 5 minutes controlled by the default configs
*Expected:* Ratis pipeline should move to OPEN from ALLOCATED as soon as
possible.
Scenario: The storage configurations were updated by one of the test modules
from /var/lib/hadoop-ozone/datanode/ratis/data to
/var/lib/hadoop-ozone/datanode/ratis/data148766 in order to reset the ozone
data and metadata. After updating the configs, the ozone stayed in safemode as
it couldn't open the pipeline with error as AccessDeniedException
{code:java}
2026-02-15 02:19:13,746 WARN
[7921c309-3218-4c72-bd8e-cc8a03ff2270-groupManagement]-org.apache.ratis.util.FileUtils:
Failed to Files.createDirectories
/var/lib/hadoop-ozone/datanode/ratis/data148766/3c1e6c09-c270-4421-bdf0-f99e6e215a3c:
java.nio.file.AccessDeniedException:
/var/lib/hadoop-ozone/datanode/ratis/data148766 2026-02-15 02:19:13,746 WARN
[7921c309-3218-4c72-bd8e-cc8a03ff2270-groupManagement]-org.apache.ratis.server.RaftServer$Division:
Failed to initialize a new directory
/var/lib/hadoop-ozone/datanode/ratis/data148766/3c1e6c09-c270-4421-bdf0-f99e6e215a3c
java.nio.file.AccessDeniedException:
/var/lib/hadoop-ozone/datanode/ratis/data148766 at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
at java.base/java.nio.file.Files.createDirectory(Files.java:700) at
java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) at
java.base/java.nio.file.Files.createDirectories(Files.java:793) at
org.apache.ratis.util.FileUtils.lambda$createDirectories$8(FileUtils.java:188)
at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38) at
org.apache.ratis.util.FileUtils.createDirectories(FileUtils.java:187) at
org.apache.ratis.util.FileUtils.createDirectories(FileUtils.java:183) at
org.apache.ratis.server.storage.RaftStorageDirectoryImpl.analyzeStorage(RaftStorageDirectoryImpl.java:139)
at
org.apache.ratis.server.storage.RaftStorageImpl.initialize(RaftStorageImpl.java:58)
at
org.apache.ratis.server.storage.StorageImplUtils$Op.format(StorageImplUtils.java:148)
at
org.apache.ratis.server.storage.StorageImplUtils$Op.run(StorageImplUtils.java:127)
at
org.apache.ratis.server.storage.StorageImplUtils.initRaftStorage(StorageImplUtils.java:100)
at
org.apache.ratis.server.impl.ServerState.lambda$new$2(ServerState.java:119) at
org.apache.ratis.util.MemoizedCheckedSupplier.get(MemoizedCheckedSupplier.java:68)
at org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:136)
at org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:402)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$groupAddAsync$13(RaftServerProxy.java:519)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635){code}
was:
*Actual:* Ratis pipeline is getting allocated, but not moving to the open state
as creating directory operation is failing on Data node. It then times out and
tries again after 5 minutes controlled by the default configs
*Expected:* Ratis pipeline should move to OPEN from ALLOCATED as soon as
possible.
Scenario: The storage configurations were updated by one of the test modules
from /var/lib/hadoop-ozone/datanode/ratis/data to
/var/lib/hadoop-ozone/datanode/ratis/data148766 in order to reset the ozone
data and metadata. After updating the configs, the ozone stayed in safemode as
it couldn't open the pipeline with error as AccessDeniedException
{code:java}
2026-02-15 02:19:13,746 WARN
[7921c309-3218-4c72-bd8e-cc8a03ff2270-groupManagement]-org.apache.ratis.util.FileUtils:
Failed to Files.createDirectories
/var/lib/hadoop-ozone/datanode/ratis/data148766/3c1e6c09-c270-4421-bdf0-f99e6e215a3c:
java.nio.file.AccessDeniedException:
/var/lib/hadoop-ozone/datanode/ratis/data148766 2026-02-15 02:19:13,746 WARN
[7921c309-3218-4c72-bd8e-cc8a03ff2270-groupManagement]-org.apache.ratis.server.RaftServer$Division:
Failed to initialize a new directory
/var/lib/hadoop-ozone/datanode/ratis/data148766/3c1e6c09-c270-4421-bdf0-f99e6e215a3c
java.nio.file.AccessDeniedException:
/var/lib/hadoop-ozone/datanode/ratis/data148766 at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
at java.base/java.nio.file.Files.createDirectory(Files.java:700) at
java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) at
java.base/java.nio.file.Files.createDirectories(Files.java:793) at
org.apache.ratis.util.FileUtils.lambda$createDirectories$8(FileUtils.java:188)
at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38) at
org.apache.ratis.util.FileUtils.createDirectories(FileUtils.java:187) at
org.apache.ratis.util.FileUtils.createDirectories(FileUtils.java:183) at
org.apache.ratis.server.storage.RaftStorageDirectoryImpl.analyzeStorage(RaftStorageDirectoryImpl.java:139)
at
org.apache.ratis.server.storage.RaftStorageImpl.initialize(RaftStorageImpl.java:58)
at
org.apache.ratis.server.storage.StorageImplUtils$Op.format(StorageImplUtils.java:148)
at
org.apache.ratis.server.storage.StorageImplUtils$Op.run(StorageImplUtils.java:127)
at
org.apache.ratis.server.storage.StorageImplUtils.initRaftStorage(StorageImplUtils.java:100)
at
org.apache.ratis.server.impl.ServerState.lambda$new$2(ServerState.java:119) at
org.apache.ratis.util.MemoizedCheckedSupplier.get(MemoizedCheckedSupplier.java:68)
at org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:136)
at org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:402)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$groupAddAsync$13(RaftServerProxy.java:519)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635){code}
{\{ }}
> Ratis pipeline is getting allocated, but not moving to open state as creating
> directory operation is failing on a DataNode
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-14653
> URL: https://issues.apache.org/jira/browse/HDDS-14653
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode
> Affects Versions: 2.0.0
> Reporter: Kodali Bhavya Sree
> Priority: Critical
>
> *Actual:* Ratis pipeline is getting allocated, but not moving to the open
> state as creating directory operation is failing on Data node. It then times
> out and tries again after 5 minutes controlled by the default configs
> *Expected:* Ratis pipeline should move to OPEN from ALLOCATED as soon as
> possible.
> Scenario: The storage configurations were updated by one of the test modules
> from /var/lib/hadoop-ozone/datanode/ratis/data to
> /var/lib/hadoop-ozone/datanode/ratis/data148766 in order to reset the ozone
> data and metadata. After updating the configs, the ozone stayed in safemode
> as it couldn't open the pipeline with error as AccessDeniedException
> {code:java}
> 2026-02-15 02:19:13,746 WARN
> [7921c309-3218-4c72-bd8e-cc8a03ff2270-groupManagement]-org.apache.ratis.util.FileUtils:
> Failed to Files.createDirectories
> /var/lib/hadoop-ozone/datanode/ratis/data148766/3c1e6c09-c270-4421-bdf0-f99e6e215a3c:
> java.nio.file.AccessDeniedException:
> /var/lib/hadoop-ozone/datanode/ratis/data148766 2026-02-15 02:19:13,746 WARN
> [7921c309-3218-4c72-bd8e-cc8a03ff2270-groupManagement]-org.apache.ratis.server.RaftServer$Division:
> Failed to initialize a new directory
> /var/lib/hadoop-ozone/datanode/ratis/data148766/3c1e6c09-c270-4421-bdf0-f99e6e215a3c
> java.nio.file.AccessDeniedException:
> /var/lib/hadoop-ozone/datanode/ratis/data148766 at
> java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
> at
> java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
> at
> java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397)
> at java.base/java.nio.file.Files.createDirectory(Files.java:700) at
> java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) at
> java.base/java.nio.file.Files.createDirectories(Files.java:793) at
> org.apache.ratis.util.FileUtils.lambda$createDirectories$8(FileUtils.java:188)
> at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38) at
> org.apache.ratis.util.FileUtils.createDirectories(FileUtils.java:187) at
> org.apache.ratis.util.FileUtils.createDirectories(FileUtils.java:183) at
> org.apache.ratis.server.storage.RaftStorageDirectoryImpl.analyzeStorage(RaftStorageDirectoryImpl.java:139)
> at
> org.apache.ratis.server.storage.RaftStorageImpl.initialize(RaftStorageImpl.java:58)
> at
> org.apache.ratis.server.storage.StorageImplUtils$Op.format(StorageImplUtils.java:148)
> at
> org.apache.ratis.server.storage.StorageImplUtils$Op.run(StorageImplUtils.java:127)
> at
> org.apache.ratis.server.storage.StorageImplUtils.initRaftStorage(StorageImplUtils.java:100)
> at
> org.apache.ratis.server.impl.ServerState.lambda$new$2(ServerState.java:119)
> at
> org.apache.ratis.util.MemoizedCheckedSupplier.get(MemoizedCheckedSupplier.java:68)
> at
> org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:136) at
> org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:402)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$groupAddAsync$13(RaftServerProxy.java:519)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
