[jira] [Commented] (HDFS-6016) Update datanode replacement policy to make writes more robust
[ https://issues.apache.org/jira/browse/HDFS-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963516#comment-13963516 ] Tsz Wo Nicholas Sze commented on HDFS-6016: --- > ... Making it also add additional DN when r == 2 and the current number of > DNs is 1. When creating a file with r == 2, the user actually cares more about the performance. (Otherwise, one should use r == 3.) Also, if datanode replacement is preferred, one could set the policy to ALWAYS. So I suggest don't make this change. > Update datanode replacement policy to make writes more robust > - > > Key: HDFS-6016 > URL: https://issues.apache.org/jira/browse/HDFS-6016 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-6016.patch, HDFS-6016.patch > > > As discussed in HDFS-5924, writers that are down to only one node due to node > failures can suffer if a DN does not restart in time. We do not worry about > writes that began with single replica. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6016) Update datanode replacement policy to make writes more robust
[ https://issues.apache.org/jira/browse/HDFS-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915151#comment-13915151 ] Brandon Li commented on HDFS-6016: -- The patch looks good. Some nitpicks: * also need to update hdfs-default.xml for the property description of dfs.client.block.write.replace-datanode-on-failure.policy * a couple typos in the comments of getMinimumNumberOfReplicasAllowed() > Update datanode replacement policy to make writes more robust > - > > Key: HDFS-6016 > URL: https://issues.apache.org/jira/browse/HDFS-6016 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-6016.patch, HDFS-6016.patch > > > As discussed in HDFS-5924, writers that are down to only one node due to node > failures can suffer if a DN does not restart in time. We do not worry about > writes that began with single replica. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-6016) Update datanode replacement policy to make writes more robust
[ https://issues.apache.org/jira/browse/HDFS-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912973#comment-13912973 ] Kihwal Lee commented on HDFS-6016: -- The test failure was not caused by this patch. > Update datanode replacement policy to make writes more robust > - > > Key: HDFS-6016 > URL: https://issues.apache.org/jira/browse/HDFS-6016 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-6016.patch, HDFS-6016.patch > > > As discussed in HDFS-5924, writers that are down to only one node due to node > failures can suffer if a DN does not restart in time. We do not worry about > writes that began with single replica. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-6016) Update datanode replacement policy to make writes more robust
[
https://issues.apache.org/jira/browse/HDFS-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912560#comment-13912560
]
Hadoop QA commented on HDFS-6016:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631119/HDFS-6016.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6242//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6242//console
This message is automatically generated.
> Update datanode replacement policy to make writes more robust
> -
>
> Key: HDFS-6016
> URL: https://issues.apache.org/jira/browse/HDFS-6016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ha, hdfs-client, namenode
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-6016.patch, HDFS-6016.patch
>
>
> As discussed in HDFS-5924, writers that are down to only one node due to node
> failures can suffer if a DN does not restart in time. We do not worry about
> writes that began with single replica.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-6016) Update datanode replacement policy to make writes more robust
[ https://issues.apache.org/jira/browse/HDFS-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912158#comment-13912158 ] Kihwal Lee commented on HDFS-6016: -- One test failure is likely unrelated, but the rest may be due to this change. I will investigate further on the cause of the failures. > Update datanode replacement policy to make writes more robust > - > > Key: HDFS-6016 > URL: https://issues.apache.org/jira/browse/HDFS-6016 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-6016.patch > > > As discussed in HDFS-5924, writers that are down to only one node due to node > failures can suffer if a DN does not restart in time. We do not worry about > writes that began with single replica. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-6016) Update datanode replacement policy to make writes more robust
[
https://issues.apache.org/jira/browse/HDFS-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912051#comment-13912051
]
Hadoop QA commented on HDFS-6016:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630994/HDFS-6016.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestFileAppend4
org.apache.hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6232//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6232//console
This message is automatically generated.
> Update datanode replacement policy to make writes more robust
> -
>
> Key: HDFS-6016
> URL: https://issues.apache.org/jira/browse/HDFS-6016
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ha, hdfs-client, namenode
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Attachments: HDFS-6016.patch
>
>
> As discussed in HDFS-5924, writers that are down to only one node due to node
> failures can suffer if a DN does not restart in time. We do not worry about
> writes that began with single replica.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
