[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223757#comment-14223757
]
Daryn Sharp commented on HDFS-7435:
-----------------------------------
While I agree it's a questionably nice to have feature - see below - decoding
the block report into a contiguous or non-contiguous array is an implementation
detail that can be changed in another jira w/o updating the layout version.
This patch simply undoes the damage caused by moving from Writables to PBs. I
changed it internally from long[] -> dynamically growing ArrayList<Long> -> PB
-> dynamically growing ArrayList<Long> -> long[]; to simply fixed long[] -> PB
-> fixed long[]. Further changes to segment the list in ways not previously
done are feature creep.
If 20-30MB is going to cause a promotion failure in a namenode servicing a
hundred millions of blocks - it's already game over. 2.x easily generates over
1GB garbage/sec at a mere ~20k ops/sec. That's an average of 54MB/request.
This is why I'm submitting GC friendly changes.
> PB encoding of block reports is very inefficient
> ------------------------------------------------
>
> Key: HDFS-7435
> URL: https://issues.apache.org/jira/browse/HDFS-7435
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HDFS-7435.patch
>
>
> Block reports are encoded as a PB repeating long. Repeating fields use an
> {{ArrayList}} with default capacity of 10. A block report containing tens or
> hundreds of thousand of longs (3 for each replica) is extremely expensive
> since the {{ArrayList}} must realloc many times. Also, decoding repeating
> fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
