[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191390#comment-13191390 ] Eli Collins commented on HDFS-2825: --- bq. If this is for testing only, then one could just do it by setting BlockPlacementPolicyDefault#preferLocal to false from the test without any new configuration. Seems reasonable. Can always add the config later if we want to enable it on a real cluster. > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191375#comment-13191375 ] Suresh Srinivas commented on HDFS-2825: --- bq. It is not meant for general purpose use by users bq. Description: In some network designs, within-rack bandwidth is never constrained so this doesn't give much of an advantage. It would also be really useful to disable this for MiniDFSCluster tests, since currently if you start a multi-DN cluster and write with replication level 1, all of the replicas go to the same DN. Since the description says "It would also be useful", I assumed this is a generic capability that is being added. If this is for testing only, then one could just do it by setting BlockPlacementPolicyDefault#preferLocal to false from the test without any new configuration. > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191337#comment-13191337 ] Eli Collins commented on HDFS-2825: --- Agree with Todd, writing a new block placement policy for this case doesn't make sense. This options is in the same spirit as eg considerLoad, something that you'd like to configure but we don't expect to become the default. > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191324#comment-13191324 ] Todd Lipcon commented on HDFS-2825: --- (do you have another way of writing the test in HDFS-2826? it's impossible to get replicas spread evenly around a minicluster without this patch, as far as I know) > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191321#comment-13191321 ] Todd Lipcon commented on HDFS-2825: --- This still reuses 99% of the code from the existing policy -- it places the first replica rack-local and the other two on a separate (but same) rack. It still takes into account load as well as disk space on all the nodes. The primary purpose is for unit tests - see HDFS-2826 for example. It is not meant for general purpose use by users, though as noted in the description there is potentially a use case. Adding an entirely new block placement implementation for this purpose is overkill, since it will duplicate all of the code from the existing one. > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191305#comment-13191305 ] Suresh Srinivas commented on HDFS-2825: --- Why are we adding another configuration to change the default behavior? If this is not required, using a different block placement policy, which is already pluggable is the right way to go. > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191296#comment-13191296 ] Arun C Murthy commented on HDFS-2825: - I'm a little leery of adding more such arcane configs, wouldn't it be better to add a notion of a block-placement policy? Maybe HDFS-1094, maybe some other jira along these lines? > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191292#comment-13191292 ] Eli Collins commented on HDFS-2825: --- +1 lgtm > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2825) Add config option to turn off the writer preferring its local DN
[ https://issues.apache.org/jira/browse/HDFS-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190871#comment-13190871 ] Hadoop QA commented on HDFS-2825: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12511471/hdfs-2825.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1794//console This message is automatically generated. > Add config option to turn off the writer preferring its local DN > > > Key: HDFS-2825 > URL: https://issues.apache.org/jira/browse/HDFS-2825 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: hdfs-2825.txt > > > Currently, the default block placement policy always places the first replica > in the pipeline on the local node if there is a valid DN running there. In > some network designs, within-rack bandwidth is never constrained so this > doesn't give much of an advantage. It would also be really useful to disable > this for MiniDFSCluster tests, since currently if you start a multi-DN > cluster and write with replication level 1, all of the replicas go to the > same DN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
