[
https://issues.apache.org/jira/browse/HDFS-7435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335128#comment-14335128
]
Daryn Sharp commented on HDFS-7435:
-----------------------------------
Do I have the luxury of bumping the DN's minimum NN version? That would
greatly simplify the implementation. It's easy for the NN to use the presence
of the protobuf fields to determine if old/new. However the prior patches
illustrate it's not so easy for the DN to auto-detect.
I believe the standard upgrade procedure is upgrade the NN, then rolling
upgrade the DNs. Per above, upgraded NN supports old/new reports from DNs.
The only scenario in which a "problem" can occur is the cluster is fully or
partially upgraded, and the NN is downgraded. The new DNs won't be able to
communicate with the old NN, hence why I'd like to bump the minimum version so
the DN doesn't continue to send block reports that appear to be empty to the
old NN. I'd argue that if the NN is downgraded, there's going to be downtime,
so you might as well rollback the DNs too.
Thoughts?
> PB encoding of block reports is very inefficient
> ------------------------------------------------
>
> Key: HDFS-7435
> URL: https://issues.apache.org/jira/browse/HDFS-7435
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
> Affects Versions: 2.0.0-alpha, 3.0.0
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HDFS-7435.000.patch, HDFS-7435.001.patch,
> HDFS-7435.002.patch, HDFS-7435.patch
>
>
> Block reports are encoded as a PB repeating long. Repeating fields use an
> {{ArrayList}} with default capacity of 10. A block report containing tens or
> hundreds of thousand of longs (3 for each replica) is extremely expensive
> since the {{ArrayList}} must realloc many times. Also, decoding repeating
> fields will box the primitive longs which must then be unboxed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
