[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377660#comment-16377660
]
Erik Krogen commented on HDFS-13183:
------------------------------------
[~hexiaoqiao], thanks for raising this issue! I was actually just about to do
the same today.
Regarding the existing patch, we should not remove {{checkOperation}}
altogether, but rather it should be {{OperationCategory.UNCHECKED}}. We should
have this feature be opt-in, and also only enabled if HA is enabled, i.e.:
{code:java}
if (balancerReadsFromStandby && haEnabled && haContext != null &&
haContext.getState().getServiceState() == HAServiceState.ACTIVE) {
...
}
{code}
Additionally the exception that should be thrown to purposefully trigger
failover for a client is currently a {{StandbyException}}, not a generic
{{IOException}}. However, this violates the description of a
{{StandbyException}}:
{code:java}
/**
* Thrown by a remote server when it is up, but is not the active server in a
* set of servers in which only a subset may be active.
*/
{code}
I think the right approach here will depend on how exactly we tackle HDFS-12976
which is closely related.
[~ajayydv], as a member of the team pushing HDFS-12943, we think there is still
value in enabling the balancer to read from the existing SbNN. A few notes:
* The work in HDFS-12975 is primarily necessary because reads from SbNN should
not get blocked by e.g. checkpointing. If the balancer is temporarily (few
minutes) delayed by such operations, there is no negative impact to the cluster.
* The work in HDFS-13150 is primarily necessary to decrease lag between the
ANN and SbNN. Since the balancer will mostly read blocks that have not been
recently modified (probabilistically), the lag time is not critical for
balancer.
* The overall read-from-standby feature is a much larger feature, and would
require setting up additional admin machines (for the ObserverNode). The
balancer can benefit from the existing standby.
We think the Balancer is a good point to provide intermediate functionality
between the current state of affairs and the full read-from-standby feature due
to its high performance impact, narrow scope, and low consistency requirements.
> Standby NameNode process getBlocks request to reduce Active load
> ----------------------------------------------------------------
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
> Affects Versions: 2.7.5, 3.1.0, 2.9.1, 2.8.4, 3.0.2
> Reporter: He Xiaoqiao
> Assignee: He Xiaoqiao
> Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
