[jira] [Commented] (HDFS-16316) Improve DirectoryScanner: add regular file check related block
[
https://issues.apache.org/jira/browse/HDFS-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17781202#comment-17781202
]
姚凡 commented on HDFS-16316:
---
!image-2023-10-31-11-24-39-387.png!
When a meta file that does not exist is transferred, the blk is incorrectly set
to corrupt. You need to check whether the file exists.
> Improve DirectoryScanner: add regular file check related block
> --
>
> Key: HDFS-16316
> URL: https://issues.apache.org/jira/browse/HDFS-16316
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Something unusual happened in the online environment.
> The DataNode is configured with 11 disks (${dfs.datanode.data.dir}). It is
> normal for 10 disks to calculate the used capacity, and the calculated value
> for the other 1 disk is much larger, which is very strange.
> This is about the live view on the NameNode:
> !screenshot-1.png!
> This is about the live view on the DataNode:
> !screenshot-2.png!
> We can look at the view on linux:
> !screenshot-3.png!
> There is a big gap here, regarding'/mnt/dfs/11/data'. This situation should
> be prohibited from happening.
> I found that there are some abnormal block files.
> There are wrong blk_.meta in some subdir directories, causing abnormal
> computing space.
> Here are some abnormal block files:
> !screenshot-4.png!
> Such files should not be used as normal blocks. They should be actively
> identified and filtered, which is good for cluster stability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16316) Improve DirectoryScanner: add regular file check related block
[
https://issues.apache.org/jira/browse/HDFS-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17492356#comment-17492356
]
JiangHua Zhu commented on HDFS-16316:
-
Sorry I'm late. Thanks to [~aajisaka] and [~weichiu] for your attention.
Yes, this happens occasionally, I've been monitoring it for a long time and
still can't find a specific cause.
But I think this situation may be related to the linux environment, when the
normal data flow works, there is no exception. (I will continue to monitor this
situation)
When this happens, we can do some work, such as doing more standardized checks
on Blocks. Because for Block, it should be a standard file.
> Improve DirectoryScanner: add regular file check related block
> --
>
> Key: HDFS-16316
> URL: https://issues.apache.org/jira/browse/HDFS-16316
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Something unusual happened in the online environment.
> The DataNode is configured with 11 disks (${dfs.datanode.data.dir}). It is
> normal for 10 disks to calculate the used capacity, and the calculated value
> for the other 1 disk is much larger, which is very strange.
> This is about the live view on the NameNode:
> !screenshot-1.png!
> This is about the live view on the DataNode:
> !screenshot-2.png!
> We can look at the view on linux:
> !screenshot-3.png!
> There is a big gap here, regarding'/mnt/dfs/11/data'. This situation should
> be prohibited from happening.
> I found that there are some abnormal block files.
> There are wrong blk_.meta in some subdir directories, causing abnormal
> computing space.
> Here are some abnormal block files:
> !screenshot-4.png!
> Such files should not be used as normal blocks. They should be actively
> identified and filtered, which is good for cluster stability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16316) Improve DirectoryScanner: add regular file check related block
[
https://issues.apache.org/jira/browse/HDFS-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17488633#comment-17488633
]
Wei-Chiu Chuang commented on HDFS-16316:
Interesting. The file type 'b' is a block device file.
I've never seen anything like this before, but it's just me. I think it's a
good idea to add a sanity check.
> Improve DirectoryScanner: add regular file check related block
> --
>
> Key: HDFS-16316
> URL: https://issues.apache.org/jira/browse/HDFS-16316
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Something unusual happened in the online environment.
> The DataNode is configured with 11 disks (${dfs.datanode.data.dir}). It is
> normal for 10 disks to calculate the used capacity, and the calculated value
> for the other 1 disk is much larger, which is very strange.
> This is about the live view on the NameNode:
> !screenshot-1.png!
> This is about the live view on the DataNode:
> !screenshot-2.png!
> We can look at the view on linux:
> !screenshot-3.png!
> There is a big gap here, regarding'/mnt/dfs/11/data'. This situation should
> be prohibited from happening.
> I found that there are some abnormal block files.
> There are wrong blk_.meta in some subdir directories, causing abnormal
> computing space.
> Here are some abnormal block files:
> !screenshot-4.png!
> Such files should not be used as normal blocks. They should be actively
> identified and filtered, which is good for cluster stability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16316) Improve DirectoryScanner: add regular file check related block
[
https://issues.apache.org/jira/browse/HDFS-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477006#comment-17477006
]
Akira Ajisaka commented on HDFS-16316:
--
Do you know why did the abnormal block appear?
> Improve DirectoryScanner: add regular file check related block
> --
>
> Key: HDFS-16316
> URL: https://issues.apache.org/jira/browse/HDFS-16316
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Major
> Labels: pull-request-available
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Something unusual happened in the online environment.
> The DataNode is configured with 11 disks (${dfs.datanode.data.dir}). It is
> normal for 10 disks to calculate the used capacity, and the calculated value
> for the other 1 disk is much larger, which is very strange.
> This is about the live view on the NameNode:
> !screenshot-1.png!
> This is about the live view on the DataNode:
> !screenshot-2.png!
> We can look at the view on linux:
> !screenshot-3.png!
> There is a big gap here, regarding'/mnt/dfs/11/data'. This situation should
> be prohibited from happening.
> I found that there are some abnormal block files.
> There are wrong blk_.meta in some subdir directories, causing abnormal
> computing space.
> Here are some abnormal block files:
> !screenshot-4.png!
> Such files should not be used as normal blocks. They should be actively
> identified and filtered, which is good for cluster stability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
