[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427126#comment-13427126
]
Harsh J commented on HDFS-3475:
---
Thanks for following up Adam. Yes, and also since this is a multiplier config,
growth of blocks will induce higher memory, so this must only be done if
greatly needed and must be carefully monitored.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.1.0-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426186#comment-13426186
]
Adam Muise commented on HDFS-3475:
--
Note on this value: while we tested at 100, this may be too high for a cluster
under even moderate workload. The cluster in question had very powerful nodes
and dual-bonded 10Gb interfaces. We also had to increase the DataNode memory in
the range of 4-6Gb. You may choose to start the value at 10 and go up based on
your available memory for the Datanode, I/O, and network capacity.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.1.0-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403847#comment-13403847
]
Hudson commented on HDFS-3475:
--
Integrated in Hadoop-Hdfs-trunk #1091 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1091/])
HDFS-3475. Make the replication monitor multipliers configurable.
Contributed by Harsh J Chouraria (Revision 1355089)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355089
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.0.1-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403894#comment-13403894
]
Harsh J commented on HDFS-3475:
---
The value for the multiplier we used was 100. This gave us a good boost, but
was tried on a cluster that had a very good network gear.
Yes I do feel the default of 2 is too low today. However, it is low to prevent
unnecessary replication in face of long-taking DN restarts and for that reason
it does well not to consume too much network. Hence I had left in the defaults.
It has to be raised only for fast networks, otherwise the reads/writes may get
choked. So I do not think we need to raise it, given the variance of users who
use Hadoop.
Let me know if you feel otherwise Eli.
P.s. The commit HDFS build failed cause of HDFS-RAID project issues, not HDFS.
No alarm, hence.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.0.1-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403920#comment-13403920
]
Hudson commented on HDFS-3475:
--
Integrated in Hadoop-Mapreduce-trunk #1124 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1124/])
HDFS-3475. Make the replication monitor multipliers configurable.
Contributed by Harsh J Chouraria (Revision 1355089)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355089
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.0.1-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13404241#comment-13404241
]
Eli Collins commented on HDFS-3475:
---
Thanks Harsh, makes sense.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.0.1-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403176#comment-13403176
]
Hadoop QA commented on HDFS-3475:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533831/HDFS-3475.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2717//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2717//console
This message is automatically generated.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403288#comment-13403288
]
Eli Collins commented on HDFS-3475:
---
+1 Looks good. Style nit going forward: no need to use this.foo when
referencing members in the constructor, understand you're doing so to be
consistent with the surrounding code (which doesn't need to do so either)
Per offline conversation Harsh tested this on a real cluster with higher
multiplier and verfied that re-replication proceeded in kind. Harsh, seems
like the current defaults (which you've preserved here) should be revisited?
They seem a little low to me, if you concur please file a separate jira for
that (to raise them).
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403324#comment-13403324
]
Hudson commented on HDFS-3475:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2471 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2471/])
HDFS-3475. Make the replication monitor multipliers configurable.
Contributed by Harsh J Chouraria (Revision 1355089)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355089
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.0.1-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403332#comment-13403332
]
Hudson commented on HDFS-3475:
--
Integrated in Hadoop-Common-trunk-Commit #2403 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2403/])
HDFS-3475. Make the replication monitor multipliers configurable.
Contributed by Harsh J Chouraria (Revision 1355089)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355089
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.0.1-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403449#comment-13403449
]
Hudson commented on HDFS-3475:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2422 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2422/])
HDFS-3475. Make the replication monitor multipliers configurable.
Contributed by Harsh J Chouraria (Revision 1355089)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355089
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Fix For: 2.0.1-alpha
Attachments: HDFS-3475.patch, HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13400599#comment-13400599
]
Aaron T. Myers commented on HDFS-3475:
--
One small comment: I think you should add some info to the hdfs-default.xml
description for {{dfs.namenode.invalidate.work.pct.per.iteration}} saying
that the value should be between 0-100, or whatever's appropriate. For that
matter, since this is a brand new config, you might want to change it to be in
the range 0 - 1.0, which I think is a more common way in the Hadoop code base
to represent percentages.
Other than that the patch looks good. +1 pending a fix for the above and an
explanation of the two test failures.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Attachments: HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13399617#comment-13399617
]
Hadoop QA commented on HDFS-3475:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533056/HDFS-3475.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 13 warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2685//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2685//console
This message is automatically generated.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Attachments: HDFS-3475.patch, HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398829#comment-13398829
]
Hadoop QA commented on HDFS-3475:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12532920/HDFS-3475.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 13 warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2680//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2680//console
This message is automatically generated.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Attachments: HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398872#comment-13398872
]
Harsh J commented on HDFS-3475:
---
The javadocs warnings are not related to this patch. The tests warn I've
explained in my previous comment.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Attachments: HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13398912#comment-13398912
]
Harsh J commented on HDFS-3475:
---
Btw I finally played around with the block repl multiplier on a local cluster
and I could notice increase in usage when a DN went down and the blocks began
moving about, with a higher multiplier value. The logs grepped and counted too
showed higher asking DN to replicate count per DN heartbeat at NN.
This should be cautiously increased though, cause even if your network is
awesome, your disks cumulatively need to keep up with it anyway. And too much
of payload per DN command transfer-back is bad as well. This is why the
defaults, I've stuck them as-is and added warnings in documented props.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
Attachments: HDFS-3475.patch
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3475) Make the replication monitor multipliers configurable
[
https://issues.apache.org/jira/browse/HDFS-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13285943#comment-13285943
]
Aaron T. Myers commented on HDFS-3475:
--
I'm +1 to making these configurable.
Make the replication monitor multipliers configurable
-
Key: HDFS-3475
URL: https://issues.apache.org/jira/browse/HDFS-3475
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Trivial
BlockManager currently hardcodes the following two constants:
{code}
private static final int INVALIDATE_WORK_PCT_PER_ITERATION = 32;
private static final int REPLICATION_WORK_MULTIPLIER_PER_ITERATION = 2;
{code}
These are used to throttle/limit the amount of deletion and
replication-to-other-DN work done per heartbeat interval of a live DN.
Not many have had reasons to want these changed so far but there have been a
few requests I've faced over the past year from a variety of clusters I've
helped maintain. I think with the improvements in disks and network thats
already started to be rolled out in production environments out there,
changing these may start making sense to some.
Lets at least make it advanced-configurable with proper docs that warn
adequately, with the defaults being what they are today. With hardcodes, it
comes down to a recompile for admins, which is not something they may like.
Please let me know your thoughts.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
