[
https://issues.apache.org/jira/browse/HDFS-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867600#action_12867600
]
Dmytro Molkov commented on HDFS-599:
------------------------------------
Hairong, thanks for your comments.
Not starting RPC client server until we are out of safemode is the second patch
that we have been running internally for a while now and I will port it to
trunk as soon as this jira makes it in. I felt like adding both parts in one
jira will be too huge.
DFSAdmin in that case does have to run on the service port.
The clean separation makes sense, but I do not think we can fully make that
separation available in the case of this JIRA.
The way to effectively administratively solve the problem of two ports is to
firewall service ports from external clients + do not include information about
this port in the mapreduce configuration. This way only HDFS cluster will have
the information in the configuration and only the datanodes will be accessing
it. This is the way we were operating internally at FB.
This of course doesn't help solve the problem of malicious clients still
accessing the service port by hacking the values in the code (since it should
not be available in the configuration).
However removing the ClientProtocol from the service port will effectively make
it impossible for administrator to perform any client operations like LS, or
even getting out of safemode (which is still in ClientProtocol) if we postpone
the start of the client port until we are out of safemode.
So essentially I feel like this problem can partly be solved by administrative
measures and the value that we get from keeping the Client protocol and others
available on the service port still outweigh the problem of malicious clients
that might get in on that port.
> Improve Namenode robustness by prioritizing datanode heartbeats over client
> requests
> ------------------------------------------------------------------------------------
>
> Key: HDFS-599
> URL: https://issues.apache.org/jira/browse/HDFS-599
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: HDFS-599.patch
>
>
> The namenode processes RPC requests from clients that are reading/writing to
> files as well as heartbeats/block reports from datanodes.
> Sometime, because of various reasons (Java GC runs, inconsistent performance
> of NFS filer that stores HDFS transacttion logs, etc), the namenode
> encounters transient slowness. For example, if the device that stores the
> HDFS transaction logs becomes sluggish, the Namenode's ability to process
> RPCs slows down to a certain extent. During this time, the RPCs from clients
> as well as the RPCs from datanodes suffer in similar fashion. If the
> underlying problem becomes worse, the NN's ability to process a heartbeat
> from a DN is severly impacted, thus causing the NN to declare that the DN is
> dead. Then the NN starts replicating blocks that used to reside on the
> now-declared-dead datanode. This adds extra load to the NN. Then the
> now-declared-datanode finally re-establishes contact with the NN, and sends a
> block report. The block report processing on the NN is another heavyweight
> activity, thus casing more load to the already overloaded namenode.
> My proposal is tha the NN should try its best to continue processing RPCs
> from datanodes and give lesser priority to serving client requests. The
> Datanode RPCs are integral to the consistency and performance of the Hadoop
> file system, and it is better to protect it at all costs. This will ensure
> that NN recovers from the hiccup much faster than what it does now.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
