[
https://issues.apache.org/jira/browse/HDFS-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xudong Cao updated HDFS-14963:
------------------------------
Description:
In multi-NameNodes scenery, hdfs client always begins a rpc call from the 1st
namenode, simply polls, and finally determines the current Active namenode.
This brings at least two problems：
# Extra failover consumption, especially in the case of frequent startup of
new client processes.
# Unnecessary log printing, suppose there are 3 NNs and the 3rd is ANN, and
then a client starts rpc with the 1st NN, it will be silent when failover from
the 1st NN to the 2nd NN, but when failover from the 2nd NN to the 3rd NN, it
prints some unnecessary logs, in some scenarios, these logs will be very
numerous:
{code:java}
2019-11-07 11:35:41,577 INFO retry.RetryInvocationHandler:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category READ is not supported in state standby. Visit
https://s.apache.org/sbnn-error
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2052)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1459)
...{code}
We can introduce a solution for this problem: in client side, for every hdfs
cluster, caching its current Active NameNode index in a separate cache file, so:
# When a client starts, it reads the current Active NameNode index from the
corresponding cache file based on the target hdfs uri, and then directly make
an rpc call toward the right ANN.
# After each time client failovers, it need to write the latest Active
NameNode index to the corresponding cache file based on the target hdfs uri.
Suppose there are hdfs://ns1 and hdfs://ns2, and the client own cache file
directory is /tmp, then:
# the ns1 cluster related cache file is /tmp/ns1
# the ns2 cluster related cache file is /tmp/ns2
was:
In multi-NameNodes scenery, hdfs client always begins a rpc call from the 1st
namenode, simply polls, and finally determines the current Active namenode.
This brings at least two problems：
# Extra failover consumption, especially in the case of frequent startup of
new client processes.
# Unnecessary log printing, suppose there are 3 NNs and the 3rd is ANN, and
then a client starts rpc with the 1st NN, it will be silent when failover from
the 1st NN to the 2nd NN, but when failover from the 2nd NN to the 3rd NN, it
prints some unnecessary logs, in some scenarios, these logs will be very
numerous:
{code:java}
2019-11-07 11:35:41,577 INFO retry.RetryInvocationHandler:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
Operation category READ is not supported in state standby. Visit
https://s.apache.org/sbnn-error
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2052)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1459)
...{code}
We can introduce a solution for this problem: in client side, for every hdfs
cluster, caching its current Active NameNode index in a separate cache file, so:
# When a client starts, it reads the current Active NameNode index from the
corresponding cache file based on the target hdfs uri, and then directly make
an rpc call toward the right ANN.
# After each time client failovers, it need to write the latest Active
NameNode index to the corresponding cache file based on the target hdfs uri.
Suppose there are hdfs://ns1 and hdfs://ns2, and the client own cache file
directory is /tmp, then:
# the ns1 cluster related cache file is /tmp/ns1
# the ns2 cluster related cache file is /tmp/ns2
> Add DFS Client caching active namenode mechanism.
> -------------------------------------------------
>
> Key: HDFS-14963
> URL: https://issues.apache.org/jira/browse/HDFS-14963
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 3.1.3
> Reporter: Xudong Cao
> Assignee: Xudong Cao
> Priority: Minor
>
> In multi-NameNodes scenery, hdfs client always begins a rpc call from the 1st
> namenode, simply polls, and finally determines the current Active namenode.
> This brings at least two problems：
> # Extra failover consumption, especially in the case of frequent startup of
> new client processes.
> # Unnecessary log printing, suppose there are 3 NNs and the 3rd is ANN, and
> then a client starts rpc with the 1st NN, it will be silent when failover
> from the 1st NN to the 2nd NN, but when failover from the 2nd NN to the 3rd
> NN, it prints some unnecessary logs, in some scenarios, these logs will be
> very numerous:
> {code:java}
> 2019-11-07 11:35:41,577 INFO retry.RetryInvocationHandler:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category READ is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> at
> org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:98)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2052)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1459)
> ...{code}
> We can introduce a solution for this problem: in client side, for every hdfs
> cluster, caching its current Active NameNode index in a separate cache file,
> so:
> # When a client starts, it reads the current Active NameNode index from the
> corresponding cache file based on the target hdfs uri, and then directly make
> an rpc call toward the right ANN.
> # After each time client failovers, it need to write the latest Active
> NameNode index to the corresponding cache file based on the target hdfs uri.
> Suppose there are hdfs://ns1 and hdfs://ns2, and the client own cache file
> directory is /tmp, then:
> # the ns1 cluster related cache file is /tmp/ns1
> # the ns2 cluster related cache file is /tmp/ns2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
