[
https://issues.apache.org/jira/browse/HDFS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867207#action_12867207
]
Dmytro Molkov commented on HDFS-599:
------------------------------------
@Hairong
1. With the Balancer we were treating it more like a client as far as this
patch is concerned since it doesn't keep the cluster alive. The problem we were
trying to solve is communication of crucial parts of the cluster. DataNodes,
NameNode and the BackupNode. However if you feel strongly that Balancer has to
have the same kind of priority in the system I can modify that part.
2. The reason all Protocols are available on all the ports is to serve the
usecase when for administrative reasons you want to run clients commands on the
service port, while the client port is closed (firewalled). The modification
here that would make sense is to limit the datanodes to use the service port
only, so that you do not have "misconfigured" datanodes in your cluster.
3. This is a RPC server change that can be done as a different set of JIRAs
(Common + HDFS) since currently the only way to do it is to modify the
configuration on the fly before passing it to creation of Server and I do not
want to go down that path.
@Eli the way we address the original problem with this patch is creating a
different queue for the DataNode heartbeats. This way if you have thousands of
clients hammering the NameNode while it is being slow your heartbeat RPC will
not get lost in the heap of client requests which is what was happening before:
if the namenode is slow, there is a huge backlog and it starts dropping DN
requests, which will not happen now, and having a completely separate set of
handlers the datanode request will get process soon after arrival even if there
was a thousand of client request on the client port already.
> Improve Namenode robustness by prioritizing datanode heartbeats over client
> requests
> ------------------------------------------------------------------------------------
>
> Key: HDFS-599
> URL: https://issues.apache.org/jira/browse/HDFS-599
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-599.patch
>
>
> The namenode processes RPC requests from clients that are reading/writing to
> files as well as heartbeats/block reports from datanodes.
> Sometime, because of various reasons (Java GC runs, inconsistent performance
> of NFS filer that stores HDFS transacttion logs, etc), the namenode
> encounters transient slowness. For example, if the device that stores the
> HDFS transaction logs becomes sluggish, the Namenode's ability to process
> RPCs slows down to a certain extent. During this time, the RPCs from clients
> as well as the RPCs from datanodes suffer in similar fashion. If the
> underlying problem becomes worse, the NN's ability to process a heartbeat
> from a DN is severly impacted, thus causing the NN to declare that the DN is
> dead. Then the NN starts replicating blocks that used to reside on the
> now-declared-dead datanode. This adds extra load to the NN. Then the
> now-declared-datanode finally re-establishes contact with the NN, and sends a
> block report. The block report processing on the NN is another heavyweight
> activity, thus casing more load to the already overloaded namenode.
> My proposal is tha the NN should try its best to continue processing RPCs
> from datanodes and give lesser priority to serving client requests. The
> Datanode RPCs are integral to the consistency and performance of the Hadoop
> file system, and it is better to protect it at all costs. This will ensure
> that NN recovers from the hiccup much faster than what it does now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
