[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044749#comment-14044749 ] Hudson commented on HDFS-6595: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1813 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1813/]) HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer, datanode >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 2.5.0 > > Attachments: HDFS-6595.patch, HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044705#comment-14044705 ] Hudson commented on HDFS-6595: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1786 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1786/]) HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer, datanode >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 2.5.0 > > Attachments: HDFS-6595.patch, HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044568#comment-14044568 ] Hudson commented on HDFS-6595: -- FAILURE: Integrated in Hadoop-Yarn-trunk #595 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/595/]) HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer, datanode >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 2.5.0 > > Attachments: HDFS-6595.patch, HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043953#comment-14043953 ] Hudson commented on HDFS-6595: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5779 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5779/]) HDFS-6595. Allow the maximum threads for balancing on datanodes to be configurable. Contributed by Benoy Antony (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605565) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer, datanode >Reporter: Benoy Antony >Assignee: Benoy Antony >Priority: Minor > Fix For: 2.5.0 > > Attachments: HDFS-6595.patch, HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[
https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043087#comment-14043087
]
Hadoop QA commented on HDFS-6595:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652334/HDFS-6595.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7228//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7228//console
This message is automatically generated.
> Configure the maximum threads allowed for balancing on datanodes
>
>
> Key: HDFS-6595
> URL: https://issues.apache.org/jira/browse/HDFS-6595
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-6595.patch, HDFS-6595.patch
>
>
> Currently datanode allows a max of 5 threads to be used for balancing.
> In some cases, , it may make sense to use a different number of threads to
> the purpose of moving.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042570#comment-14042570 ] Tsz Wo Nicholas Sze commented on HDFS-6595: --- Benoy, thanks for working on this. Some comments on the patch: - Balancer.MAX_NUM_CONCURRENT_MOVES should be moved to DFSConfigKeys, i.e. DFSConfigKeys.DFS_DATANODE_BALANCE_MAX_NUM_CONCURRENT_MOVES_DEFAULT. - We should also change Balancer to read the new conf and use it to replace MAX_NUM_CONCURRENT_MOVES. > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042547#comment-14042547 ] Benoy Antony commented on HDFS-6595: Thanks for the suggestion, [~jnp]. I'll try that approach. > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042498#comment-14042498 ] Jitendra Nath Pandey commented on HDFS-6595: Its a good idea to add a unit test. I am not clear why we need Balancer to allow more than 5 concurrent moves to test this. Just set the max number of threads to a smaller number (e.g. 3) and see the exception? > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042408#comment-14042408 ] Benoy Antony commented on HDFS-6595: Thanks @jnp. I have tested this manually by increasing the number of concurrent moves on the balancer side. I can add a unit test to the same, but I would need to change the _Balancer_ code also so that it allows to more than 5 concurrent moves per datanode. > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041783#comment-14041783 ] Jitendra Nath Pandey commented on HDFS-6595: The patch looks good to me. Since there are no unit tests included, please mention the testing done on the patch. > Configure the maximum threads allowed for balancing on datanodes > > > Key: HDFS-6595 > URL: https://issues.apache.org/jira/browse/HDFS-6595 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HDFS-6595.patch > > > Currently datanode allows a max of 5 threads to be used for balancing. > In some cases, , it may make sense to use a different number of threads to > the purpose of moving. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6595) Configure the maximum threads allowed for balancing on datanodes
[
https://issues.apache.org/jira/browse/HDFS-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041716#comment-14041716
]
Hadoop QA commented on HDFS-6595:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12652111/HDFS-6595.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7220//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7220//console
This message is automatically generated.
> Configure the maximum threads allowed for balancing on datanodes
>
>
> Key: HDFS-6595
> URL: https://issues.apache.org/jira/browse/HDFS-6595
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Attachments: HDFS-6595.patch
>
>
> Currently datanode allows a max of 5 threads to be used for balancing.
> In some cases, , it may make sense to use a different number of threads to
> the purpose of moving.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
