[jira] [Commented] (HDFS-7786) Handle slow writers for DFSStripedOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354898#comment-16354898 ] liaoyuxiangqin commented on HDFS-7786: -- [~libo-intel] I take a think on this issue, and want to process as follows: 1.select the slow streamers when one block group write end through comprehensive consideration slow threshold and time consuming Median. 2.use slow streams target datanodes as excluded nodes for next block group allocate, so that avoid write data to slow datanodes. Thanks. Can you give me more advice! > Handle slow writers for DFSStripedOutputStream > -- > > Key: HDFS-7786 > URL: https://issues.apache.org/jira/browse/HDFS-7786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Fix For: HDFS-7285 > > > The streamers in DFSStripedOutputStream may have different write speed. We > need to consider and handle the situation when one or more streamers begin to > write slowly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7786) Handle slow writers for DFSStripedOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724894#comment-14724894 ] Keisuke Ogiwara commented on HDFS-7786: --- Hi Li Bo, I have two questions. 1. Slow streamer is caused by network delay, right? 2. I think we could set the rule like bellow: If actual retry count becomes more than the retry count which we will add to config file, we set the streamer's status as failed(StripedDataStreamer#isFailed)? Does it make sense to you? Thank you. > Handle slow writers for DFSStripedOutputStream > -- > > Key: HDFS-7786 > URL: https://issues.apache.org/jira/browse/HDFS-7786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Keisuke Ogiwara > Fix For: HDFS-7285 > > > The streamers in DFSStripedOutputStream may have different write speed. We > need to consider and handle the situation when one or more streamers begin to > write slowly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7786) Handle slow writers for DFSStripedOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516029#comment-14516029 ] Keisuke Ogiwara commented on HDFS-7786: --- Hi Li Bo, I want to work on this jira. Could you assign to me? Thank you very much. > Handle slow writers for DFSStripedOutputStream > -- > > Key: HDFS-7786 > URL: https://issues.apache.org/jira/browse/HDFS-7786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > Fix For: HDFS-7285 > > > The streamers in DFSStripedOutputStream may have different write speed. We > need to consider and handle the situation when one or more streamers begin to > write slowly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7786) Handle slow writers for DFSStripedOutputStream
[ https://issues.apache.org/jira/browse/HDFS-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516067#comment-14516067 ] Li Bo commented on HDFS-7786: - Sure.Thank you for working on this. > Handle slow writers for DFSStripedOutputStream > -- > > Key: HDFS-7786 > URL: https://issues.apache.org/jira/browse/HDFS-7786 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Li Bo >Assignee: Li Bo > Fix For: HDFS-7285 > > > The streamers in DFSStripedOutputStream may have different write speed. We > need to consider and handle the situation when one or more streamers begin to > write slowly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
