[
https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122898#comment-15122898
]
Colin Patrick McCabe commented on HDFS-9260:
--------------------------------------------
Thanks, [~sfriberg].
bq. 7. The reason for duplicating it is basically to avoid that the NN
allocates 4 LinkedLists as part of each block that is being reported in an IBR.
Potentially one could change the fullBR to not rely on lists and simply
add/remove as it finds entries. Two issues that needs to be thought about for
this, how should logging be handled since some counting is done as part of
number of handled blocks, and, is it better to have multiple loops with smaller
code footprint than expanding the already large one with even more code to
handle each case directly. I agree with you that it is bad with the two code
paths, but I think it the reduction in allocation for IBRs could be worth it.
Yeah, this seems like something that should be done in a follow-on JIRA.
{code}
+ public static final String
DFS_NAMENODE_STORAGEINFO_EFFICIENCY_INTERVAL_MS_KEY
+ = "dfs.namenode.storageinfo.efficiency.interval.ms";
{code}
I appreciate that this now has time units associated with it, but I feel that
"defragmenter" should be somewhere in the name. This is basically a
configuration key for the defragmenter.
> Improve performance and GC friendliness of startup and FBRs
> -----------------------------------------------------------
>
> Key: HDFS-9260
> URL: https://issues.apache.org/jira/browse/HDFS-9260
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode, performance
> Affects Versions: 2.7.1
> Reporter: Staffan Friberg
> Assignee: Staffan Friberg
> Attachments: FBR processing.png, HDFS Block and Replica Management
> 20151013.pdf, HDFS-7435.001.patch, HDFS-7435.002.patch, HDFS-7435.003.patch,
> HDFS-7435.004.patch, HDFS-7435.005.patch, HDFS-7435.006.patch,
> HDFS-7435.007.patch, HDFS-9260.008.patch, HDFS-9260.009.patch,
> HDFS-9260.010.patch, HDFS-9260.011.patch, HDFS-9260.012.patch,
> HDFS-9260.013.patch, HDFS-9260.014.patch, HDFS-9260.015.patch,
> HDFS-9260.016.patch, HDFSBenchmarks.zip, HDFSBenchmarks2.zip
>
>
> This patch changes the datastructures used for BlockInfos and Replicas to
> keep them sorted. This allows faster and more GC friendly handling of full
> block reports.
> Would like to hear peoples feedback on this change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
