[ https://issues.apache.org/jira/browse/HDFS-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979430#comment-16979430 ]
Stephen O'Donnell commented on HDFS-14993: ------------------------------------------ If I chown and chmod a directory so it is not writeable, then I get this stack on startup: {code} 2019-11-21 15:52:44,383 INFO datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT] 2019-11-21 15:52:44,529 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2019-11-21 15:52:44,647 INFO checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/tmp/hadoop-sodonnell/dfs/data 2019-11-21 15:52:44,654 INFO checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/tmp/hadoop-sodonnell/dfs/data2 2019-11-21 15:52:44,680 WARN checker.StorageLocationChecker: Exception checking StorageLocation [DISK]file:/tmp/hadoop-sodonnell/dfs/data ExitCodeException exitCode=1: chmod: Unable to change file mode on /private/tmp/hadoop-sodonnell/dfs/data: Operation not permitted at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008) at org.apache.hadoop.util.Shell.run(Shell.java:901) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213) at org.apache.hadoop.util.Shell.execCommand(Shell.java:1307) at org.apache.hadoop.util.Shell.execCommand(Shell.java:1289) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:867) at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:550) at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:531) at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:553) at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:234) at org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:141) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:239) at org.apache.hadoop.hdfs.server.datanode.StorageLocation.check(StorageLocation.java:52) at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker$1.call(ThrottledAsyncChecker.java:142) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-11-21 15:52:44,682 ERROR datanode.DataNode: Exception in secureMain org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 1, volumes configured: 2, volumes failed: 1, volume failures tolerated: 0 at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:233) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2836) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2749) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2793) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2937) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2961) 2019-11-21 15:52:44,683 INFO util.ExitUtil: Exiting with status 1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Too many failed volumes - current valid volumes: 1, volumes configured: 2, volumes failed: 1, volume failures tolerated: 0 {code} So it looks like a disk check is scheduled already by org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2836) which finds its way into DiskChecker: {code} private static void checkDirInternal(LocalFileSystem localFS, Path dir, FsPermission expected) throws DiskErrorException, IOException { mkdirsWithExistsAndPermissionCheck(localFS, dir, expected); checkAccessByFileMethods(localFS.pathToFile(dir)); } {code} The call checkAccessByFileMethods then runs a read and write test. Rather than moving the call to checkDiskError() after the addBlockPool() call, I wonder if we need it there at all? > checkDiskError doesn't work during datanode startup > --------------------------------------------------- > > Key: HDFS-14993 > URL: https://issues.apache.org/jira/browse/HDFS-14993 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Reporter: Yang Yun > Assignee: Yang Yun > Priority: Major > Attachments: HDFS-14993.patch, HDFS-14993.patch > > > the function checkDiskError() is called before addBlockPool, but list > bpSlices is empty this time. So the function check() in FsVolumeImpl.java > does nothing. > @Override > public VolumeCheckResult check(VolumeCheckContext ignored) > throws DiskErrorException { > // TODO:FEDERATION valid synchronization > for (BlockPoolSlice s : bpSlices.values()) { > s.checkDirs(); > } > return VolumeCheckResult.HEALTHY; > } -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org