[ https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daryn Sharp updated HDFS-7435: ------------------------------ Attachment: HDFS-7435.patch Thanks for the review, Charles. I think I've addressed your comments. I mildly disagree will overly verbose message for UnsupportedOperationExceptions since the JDK rarely uses message and the class & method is in the stack trace. The ReplicaState, although a varint64, is actually just a byte on the wire. Using a varint64 just allows for future use of the bits. I’ll note that I must backpedal a bit on some of my claims. The ByteString.Output creates new overflow buffers with larger capacity. That means substring returns bounded instances, not that big a deal, unless we assume the internal implementation of PBs. Even then, it returns bounded instances when it shouldn’t. Multiple buffers are required before it might return a non-bounded buffer. Therefor I optimized the single small buffer case to not substring. I think the whole fragmented over-the-write buffer implementation is over engineered. The average encoded block is ~15 bytes. The majority of cluster aren’t likely to exceed one buffer (I reduced from 1MB to 64K) per storage, and if they do, multiple little allocations is more work for GC. Removing the intermediate long[] conversions is the real problem. But I digress. I added comprehensive tests for client auto-detection of NN support; encoding & decoding as buffers and longs. > PB encoding of block reports is very inefficient > ------------------------------------------------ > > Key: HDFS-7435 > URL: https://issues.apache.org/jira/browse/HDFS-7435 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode > Affects Versions: 2.0.0-alpha, 3.0.0 > Reporter: Daryn Sharp > Assignee: Daryn Sharp > Priority: Critical > Attachments: HDFS-7435.000.patch, HDFS-7435.001.patch, > HDFS-7435.002.patch, HDFS-7435.patch, HDFS-7435.patch, HDFS-7435.patch > > > Block reports are encoded as a PB repeating long. Repeating fields use an > {{ArrayList}} with default capacity of 10. A block report containing tens or > hundreds of thousand of longs (3 for each replica) is extremely expensive > since the {{ArrayList}} must realloc many times. Also, decoding repeating > fields will box the primitive longs which must then be unboxed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)