[
https://issues.apache.org/jira/browse/HDFS-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736017#comment-14736017
]
Tsz Wo Nicholas Sze commented on HDFS-9011:
-------------------------------------------
Another possible solution is to accumulate partial reports in NN. It seems
fine since DN supposes to send all its partial reports at once. NN can store
the partial report in the block report lease temporarily.
The lease expiry time for partial reports can be very short, say 3 minutes.
When NN receives a partial report, it stores it in the lease and renews the
lease. When NN receives the last partial report, it processes the full report.
When the lease expires, NN removes the accumulated partial reports and reject
future partial reports with the same ID.
> Support splitting BlockReport of a storage into multiple RPC
> ------------------------------------------------------------
>
> Key: HDFS-9011
> URL: https://issues.apache.org/jira/browse/HDFS-9011
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jing Zhao
> Assignee: Jing Zhao
> Attachments: HDFS-9011.000.patch, HDFS-9011.001.patch,
> HDFS-9011.002.patch
>
>
> Currently if a DataNode has too many blocks (more than 1m by default), it
> sends multiple RPC to the NameNode for the block report, each RPC contains
> report for a single storage. However, in practice we've seen sometimes even a
> single storage can contains large amount of blocks and the report even
> exceeds the max RPC data length. It may be helpful to support sending
> multiple RPC for the block report of a storage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
