[jira] [Comment Edited] (HDFS-16320) Datanode retrieve slownode information from NameNode
[ https://issues.apache.org/jira/browse/HDFS-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17443276#comment-17443276 ] Janus Chow edited comment on HDFS-16320 at 11/14/21, 9:41 AM: -- [~hexiaoqiao] Thank you for the review. The issue we met is we have clients writing to the slownode and it took a very long time to finish writing for a normal file. After we checked the metrics, we found we can avoid the pipeline creating on the slownodes with "dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled" set to true. It will work fine for new clients, but for clients already using the slownode as pipeline, they have to suffer the slownode. (Maybe the slownode is reported by this pipeline.) Since when clients are writing data, it will only be clients and datanodes communicating, so even NameNode has the information that the datanode in the pipeline is slow, clients can not do too much to avoid it. Our proposal would be, to let Datanodes get the information from heartbeats reports, then during the writing, datanodes can report it to clients, then clients can choose to rebuild the pipeline to improve the writing performance. was (Author: symious): [~hexiaoqiao] Thank you for the review. The issue we met is we have clients writing to the slownode and it took a very long time to finish writing for a normal file. After we checked the metrics, we found we can avoid the pipeline creating on the slownodes with "dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled" set to true. It will work fine for new clients, but for clients already using the slownode as pipeline, they have to suffer the slownode. (Maybe the slownode is reported by this pipeline.) Since when clients are writing data, it will only be clients and datanodes communicating, so even NameNode has the information that the datanode in the pipeline is slow, clients can do too much to avoid it. Our proposal would be, to let Datanodes get the information from heartbeats reports, then during the writing, datanodes can report it to clients, then clients can choose to rebuild the pipeline to improve the writing performance. > Datanode retrieve slownode information from NameNode > > > Key: HDFS-16320 > URL: https://issues.apache.org/jira/browse/HDFS-16320 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The current information of slownode is reported by reportingNode, and stored > in NameNode. > This ticket is to let the slownode retrieve the information from NameNode, so > that it can do other performance improvement actions based on this > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16320) Datanode retrieve slownode information from NameNode
[ https://issues.apache.org/jira/browse/HDFS-16320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17443351#comment-17443351 ] Janus Chow edited comment on HDFS-16320 at 11/14/21, 2:13 PM: -- [~hexiaoqiao] {quote}I mean that DataNode has the total information to decide if he is SLOW based on response time or throughput rather than based on command from NameNode. Furthermore there is possible to false positive at NameNode side. {quote} In my opinion, the slownode information of NameNode is kind of union choices from DataNodes. All the slownodes are reported by other DataNodes(calculated by statistics), and NameNode does the summary and chooses the top reported DataNodes. Till this part, the data of "slownode" should be much confident. {quote}I am not against the idea but we should have more proper way to solve this problem. {quote} I tried to find other ways to find out the slownode, especially from DataNode themselves. But after I checked the implementation of "OutlierDetector.java" and "DataNodePeerMetrics.java", I think the current calculation is very good to spot the slownode. {quote}client and DataNode/Pipeline communication could estimate if there are slow nodes and which one is slow {quote} Since in a pipeline, the client only talks to the first DataNode, it could be difficult to track the slowness for the slowness between the three DataNodes. I think that's why the slownode is only calculated on the penultimate node and the last node. Another thing is in this ticket, it's kind of a slowness statement. Until now, the DataNode only shows the state of slowness tagged by each NameNode in the metrics. It's a kind of real-time status updated by heartbeat. was (Author: symious): {quote}I mean that DataNode has the total information to decide if he is SLOW based on response time or throughput rather than based on command from NameNode. Furthermore there is possible to false positive at NameNode side. {quote} In my opinion, the slownode information of NameNode is kind of union choices from DataNodes. All the slownodes are reported by other DataNodes(calculated by statistics), and NameNode does the summary and chooses the top reported DataNodes. Till this part, the data of "slownode" should be much confident. {quote}I am not against the idea but we should have more proper way to solve this problem. {quote} I tried to find other ways to find out the slownode, especially from DataNode themselves. But after I checked the implementation of "OutlierDetector.java" and "DataNodePeerMetrics.java", I think the current calculation is very good to spot the slownode. {quote}client and DataNode/Pipeline communication could estimate if there are slow nodes and which one is slow {quote} Since in a pipeline, the client only talks to the first DataNode, it could be difficult to track the slowness for the slowness between the three DataNodes. I think that's why the slownode is only calculated on the penultimate node and the last node. Another thing is in this ticket, it's kind of a slowness statement. Until now, the DataNode only shows the state of slowness tagged by each NameNode in the metrics. It's a kind of real-time status updated by heartbeat. > Datanode retrieve slownode information from NameNode > > > Key: HDFS-16320 > URL: https://issues.apache.org/jira/browse/HDFS-16320 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The current information of slownode is reported by reportingNode, and stored > in NameNode. > This ticket is to let the slownode retrieve the information from NameNode, so > that it can do other performance improvement actions based on this > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org