[
https://issues.apache.org/jira/browse/HADOOP-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12501313
]
dhruba borthakur commented on HADOOP-1448:
------------------------------------------
In the short term, I like Doug's proposal #1. The current code already supports
a configuration setting named dfs.replication.max. It is set to 512.
For installations that are seeing this problem, maybe they can change
dfs.replication.max to a smaller number, say 10 - 20 or so. An application that
was successfully able to set a replication factor of 512 with the current
configuration will now experience an exception when dfs.replication.max is set
to 20.
> Setting the replication factor of a file too high causes namenode cpu overload
> ------------------------------------------------------------------------------
>
> Key: HADOOP-1448
> URL: https://issues.apache.org/jira/browse/HADOOP-1448
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: dhruba borthakur
>
> The replication factor of a file in set to 300 (on a 800 node cluster). Then
> all mappers try to open this file. For every open call that the namenode
> receives from each of these 800 clients, it sorts all the replicas of the
> block(s) based on the distance from the client. This causes CPU usage
> overload on the namenode.
> One proposal is to make the namenode return a non-sorted list of datanodes to
> the client. Information about each replica also contains the rack on which
> that replica resides. The client can look at the replicas to determine if
> there is a copy on the local node. If not, then it can find out if there is a
> replica on the local rack. If not then it can choose a replica at random.
> This proposal is scalable because the sorting and selection of replicas is
> done by the client rather than the Namenode.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
