[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-7435:
------------------------------
Attachment: HDFS-7435.patch
Thanks for the review, Charles. I think I've addressed your comments. I
mildly disagree will overly verbose message for UnsupportedOperationExceptions
since the JDK rarely uses message and the class & method is in the stack trace.
The ReplicaState, although a varint64, is actually just a byte on the wire.
Using a varint64 just allows for future use of the bits.
I’ll note that I must backpedal a bit on some of my claims. The
ByteString.Output creates new overflow buffers with larger capacity. That
means substring returns bounded instances, not that big a deal, unless we
assume the internal implementation of PBs. Even then, it returns bounded
instances when it shouldn’t. Multiple buffers are required before it might
return a non-bounded buffer. Therefor I optimized the single small buffer case
to not substring.
I think the whole fragmented over-the-write buffer implementation is over
engineered. The average encoded block is ~15 bytes. The majority of cluster
aren’t likely to exceed one buffer (I reduced from 1MB to 64K) per storage, and
if they do, multiple little allocations is more work for GC. Removing the
intermediate long[] conversions is the real problem. But I digress.
I added comprehensive tests for client auto-detection of NN support; encoding &
decoding as buffers and longs.
> PB encoding of block reports is very inefficient
> ------------------------------------------------
>
> Key: HDFS-7435
> URL: https://issues.apache.org/jira/browse/HDFS-7435
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HDFS-7435.000.patch, HDFS-7435.001.patch,
> HDFS-7435.002.patch, HDFS-7435.patch, HDFS-7435.patch, HDFS-7435.patch
>
>
> Block reports are encoded as a PB repeating long. Repeating fields use an
> {{ArrayList}} with default capacity of 10. A block report containing tens or
> hundreds of thousand of longs (3 for each replica) is extremely expensive
> since the {{ArrayList}} must realloc many times. Also, decoding repeating
> fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
