[jira] [Commented] (HDFS-8862) Improve BlockManager#excessReplicateMap

2015-08-05 Thread Hadoop QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659513#comment-14659513
 ] 

Hadoop QA commented on HDFS-8862:
-

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  15m 19s | Findbugs (version ) appears to 
be broken on trunk. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear 
to include any new or modified tests.  Please justify why no new tests are 
needed for this patch. Also please list what manual steps were performed to 
verify this patch. |
| {color:green}+1{color} | javac |   7m 36s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 46s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m  4s | The applied patch generated  
171 new checkstyle issues (total was 0, now 171). |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 25s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 31s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   2m 28s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native |   3m  3s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 159m 37s | Tests passed in hadoop-hdfs. 
|
| | | 201m 15s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12748980/HDFS-8862.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f59612e |
| checkstyle |  
https://builds.apache.org/job/PreCommit-HDFS-Build/11915/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
 |
| hadoop-hdfs test log | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11915/artifact/patchprocess/testrun_hadoop-hdfs.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11915/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/11915/console |


This message was automatically generated.

> Improve BlockManager#excessReplicateMap
> ---
>
> Key: HDFS-8862
> URL: https://issues.apache.org/jira/browse/HDFS-8862
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8862.001.patch
>
>
> Per [~cmccabe]'s comments in HDFS-8792, this JIRA is to discuss improving 
> {{BlockManager#excessReplicateMap}}.
> That's right HashMap don't ever shrink when elements are removed,  but 
> TreeMap entry needs to store more (memory) references (left,  right, parent) 
> than HashMap entry (only one reference next),  even when there is element 
> removing and cause some entry empty, the empty HashMap entry is just a 
> {{null}} reference (4 bytes),  so they are close at this point.  On the other 
> hand, the key of {{excessReplicateMap}} is datanode uuid, so the entries 
> number is almost fixed, so HashMap memory is good than TreeMap memory in this 
> case.   I think the most important is the search/insert/remove performance, 
> HashMap is absolutely better than TreeMap.  Because we don't need to sort,  
> we should use HashMap instead of TreeMap



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-8862) Improve BlockManager#excessReplicateMap

2015-08-17 Thread Colin Patrick McCabe (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700352#comment-14700352
 ] 

Colin Patrick McCabe commented on HDFS-8862:


I guess the set of datanodes is not going to shrink that much over the life of 
the cluster.  So the fact that this data structure can't shrink should be OK.  
We may want to look into whether that {{LightWeightLinkedSet}} can 
shrink... but that is outside the scope of this JIRA.  +1.

> Improve BlockManager#excessReplicateMap
> ---
>
> Key: HDFS-8862
> URL: https://issues.apache.org/jira/browse/HDFS-8862
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8862.001.patch
>
>
> Per [~cmccabe]'s comments in HDFS-8792, this JIRA is to discuss improving 
> {{BlockManager#excessReplicateMap}}.
> That's right HashMap don't ever shrink when elements are removed,  but 
> TreeMap entry needs to store more (memory) references (left,  right, parent) 
> than HashMap entry (only one reference next),  even when there is element 
> removing and cause some entry empty, the empty HashMap entry is just a 
> {{null}} reference (4 bytes),  so they are close at this point.  On the other 
> hand, the key of {{excessReplicateMap}} is datanode uuid, so the entries 
> number is almost fixed, so HashMap memory is good than TreeMap memory in this 
> case.   I think the most important is the search/insert/remove performance, 
> HashMap is absolutely better than TreeMap.  Because we don't need to sort,  
> we should use HashMap instead of TreeMap



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-8862) Improve BlockManager#excessReplicateMap

2015-08-17 Thread Yi Liu (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700560#comment-14700560
 ] 

Yi Liu commented on HDFS-8862:
--

Thanks [~cmccabe] for the review. Will commit the patch later.

{{LightWeightLinkedSet}} can shrink..



> Improve BlockManager#excessReplicateMap
> ---
>
> Key: HDFS-8862
> URL: https://issues.apache.org/jira/browse/HDFS-8862
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-8862.001.patch
>
>
> Per [~cmccabe]'s comments in HDFS-8792, this JIRA is to discuss improving 
> {{BlockManager#excessReplicateMap}}.
> That's right HashMap don't ever shrink when elements are removed,  but 
> TreeMap entry needs to store more (memory) references (left,  right, parent) 
> than HashMap entry (only one reference next),  even when there is element 
> removing and cause some entry empty, the empty HashMap entry is just a 
> {{null}} reference (4 bytes),  so they are close at this point.  On the other 
> hand, the key of {{excessReplicateMap}} is datanode uuid, so the entries 
> number is almost fixed, so HashMap memory is good than TreeMap memory in this 
> case.   I think the most important is the search/insert/remove performance, 
> HashMap is absolutely better than TreeMap.  Because we don't need to sort,  
> we should use HashMap instead of TreeMap



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)