[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14248733#comment-14248733 ] Nathan Roberts commented on HDFS-2825: -- [~tlipcon] - It might help. I'm a little concerned about the additional round trip and how we might enable it globally for an entire cluster. We'll be running some experiments with a global config for the block placement policy, and then go from there. > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243637#comment-14243637 ] Todd Lipcon commented on HDFS-2825: --- [~nroberts] - have you seen HDFS-3702? It's related and might be useful for this scenario. > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232257#comment-14232257 ] Nathan Roberts commented on HDFS-2825: -- Any chance folks have altered their view on this and would accept a config option in the default placement policy? I'm happy to file the jira and put up the patch. It's very straightforward and makes sense for a lot of cases. imho, it actually makes more sense as the default than node-local (except for obvious things like Hbase). Today with node-local-first we wind up with some very hot nodes that dramatically slow down subsequent jobs, all because there is an entire copy of a large file on a single node of a rack. > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194857#comment-13194857 ] Todd Lipcon commented on HDFS-2825: --- Yes, for create/append > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194479#comment-13194479 ] Arun C Murthy commented on HDFS-2825: - bq. Maybe we should open a new JIRA to do this as a client-side config, where the client would pass a new flag when asking the NN for a pipeline? It should just be an option for the create call? > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194436#comment-13194436 ] Todd Lipcon commented on HDFS-2825: --- Maybe we should open a new JIRA to do this as a client-side config, where the client would pass a new flag when asking the NN for a pipeline? > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194306#comment-13194306 ] Milind Bhandarkar commented on HDFS-2825: - Actually, there is a valid use case for turning off local datanode preferrence (not only for testing), especially with Yarn being used to deploy long running services. One such service is a kafka-like daemon which is listening to event producers, and loads batches into HDFS. This would fill up the local datanode. > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192142#comment-13192142 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Mapreduce-trunk #968 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/968/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192130#comment-13192130 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Mapreduce-0.23-Build #170 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/170/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192111#comment-13192111 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Hdfs-0.23-Build #148 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/148/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192089#comment-13192089 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Hdfs-trunk #935 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/935/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191685#comment-13191685 ] Todd Lipcon commented on HDFS-2825: --- Allen - I agree it would be useful to set on the writer side. But that's not quite the original patch, which operates DFS-wide. > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191663#comment-13191663 ] Allen Wittenauer commented on HDFS-2825: This would have actually been useful for ETL jobs. Oh well. Maybe I'll throw the original patch into our build anyway. > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191627#comment-13191627 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1591 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1591/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191558#comment-13191558 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Mapreduce-0.23-Commit #422 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/422/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191559#comment-13191559 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Common-0.23-Commit #407 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/407/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191551#comment-13191551 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Hdfs-trunk-Commit #1646 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1646/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191545#comment-13191545 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Common-trunk-Commit #1573 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1573/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235025 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191538#comment-13191538 ] Hudson commented on HDFS-2825: -- Integrated in Hadoop-Hdfs-0.23-Commit #397 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/397/]) HDFS-2825. Add test hook to turn off the writer preferring its local DN. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235024 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.24.0, 0.23.1 > > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191523#comment-13191523 ] Arun C Murthy commented on HDFS-2825: - +1 for the setter. > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add test hook to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191467#comment-13191467 ] Hadoop QA commented on HDFS-2825: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511547/hdfs-2825.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1796//console This message is automatically generated. > Add test hook to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt, hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. > _[per discussion below, this was changed to not add a config, but only to add > a hook for testing]_ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
