[
https://issues.apache.org/jira/browse/HADOOP-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893060#comment-16893060
]
Erik Krogen commented on HADOOP-16452:
--------------------------------------
{quote}
Breaking the BRs into multiple messages is a possible (and probably better)
solution
{quote}
I agree, and I see you already linked HDFS-11313 with a proposal to do this.
{quote}
The block report processing logic releases the NN lock every 4 milliseconds.
(BlockManager.BlockReportProcessingThread#processQueue)
{quote}
This is only for IBRs. A single volume FBR is still processed under a lock
without release; HDFS-14657 proposes to fix this. But it is an issue for now.
{quote}
Additionally, when you get into this sort of situation, the only real solution
is to increase the limit, so it probably makes sense to bump this to 128MB by
default. If the cluster is running with this many blocks, then the NN heap is
probably big enough to accommodate the larger report size anyway.
{quote}
+1
> Increase ipc.maximum.data.length default from 64MB to 128MB
> -----------------------------------------------------------
>
> Key: HADOOP-16452
> URL: https://issues.apache.org/jira/browse/HADOOP-16452
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 2.6.0
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> Reason for bumping the default:
> Denser DataNodes are common. It is not uncommon to find a DataNode with > 7
> million blocks these days.
> With such a high number of blocks, the block report message can exceed the
> 64mb limit (defined by ipc.maximum.data.length). The block reports are
> rejected, causing missing blocks in HDFS. We had to double this configuration
> value in order to work around the issue.
> We are seeing an increasing number of these cases. I think it's time to
> revisit some of these default values as the hardware evolves.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
