[
https://issues.apache.org/jira/browse/HADOOP-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16891175#comment-16891175
]
Wei-Chiu Chuang commented on HADOOP-16452:
------------------------------------------
Thanks Anu.
I am not so worried about block reports blocking NN lock. The block report
processing logic releases the NN lock every 4 milliseconds.
(BlockManager.BlockReportProcessingThread#processQueue)
Breaking the BRs into multiple messages is a possible (and probably better)
solution -- Serializing/deserializing a large protobuf message is bad as it
creates lots of GCs.
We already break down BRs per volume. Unfortunately we still see 7-8 million
blocks per volume. Small files problems yes. But this is such a common issue
that we can't simply say "go back fix your app"
> Increase ipc.maximum.data.length default from 64MB to 128MB
> -----------------------------------------------------------
>
> Key: HADOOP-16452
> URL: https://issues.apache.org/jira/browse/HADOOP-16452
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 2.6.0
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> Reason for bumping the default:
> Denser DataNodes are common. It is not uncommon to find a DataNode with > 7
> million blocks these days.
> With such a high number of blocks, the block report message can exceed the
> 64mb limit (defined by ipc.maximum.data.length). The block reports are
> rejected, causing missing blocks in HDFS. We had to double this configuration
> value in order to work around the issue.
> We are seeing an increasing number of these cases. I think it's time to
> revisit some of these default values as the hardware evolves.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
