[
https://issues.apache.org/jira/browse/HADOOP-7472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079042#comment-13079042
]
Daryn Sharp commented on HADOOP-7472:
-------------------------------------
+1 Minor concern about round-robin dns causing connection attempts forever.
Just so everyone understands the caveats:
# A process that is running when the IP changes will reconnect and continue to
use its tokens since it knows how it original contacted the service -- new
behavior
# New processes using a token cache (ie. mapreduce) will be unable to find
their tokens since the token is keyed on IP -- same as before
# Job tracker will be unable to renew tokens when the ip changes -- same as
before.
What it means is this change is useful in cases such as the NN IP changes.
Existing mappers will reconnect & complete, but mappers spawned after the IP
change won't find their tokens.
> RPC client should deal with the IP address changes
> --------------------------------------------------
>
> Key: HADOOP-7472
> URL: https://issues.apache.org/jira/browse/HADOOP-7472
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.20.205.0
> Reporter: Kihwal Lee
> Assignee: Kihwal Lee
> Priority: Minor
> Fix For: 0.20.205.0
>
> Attachments: addr_change_dfs-1.patch.txt,
> addr_change_dfs-2.patch.txt, addr_change_dfs-3.patch.txt,
> addr_change_dfs.patch.txt, addr_change_dfs_0_20s-1.patch.txt,
> addr_change_dfs_0_20s.patch.txt, addr_change_dfs_trunk-1.patch.txt,
> addr_change_dfs_trunk-2.patch.txt, addr_change_dfs_trunk.patch.txt
>
>
> The current RPC client implementation and the client-side callers assume that
> the hostname-address mappings of servers never change. The resolved address
> is stored in an immutable InetSocketAddress object above/outside RPC, and the
> reconnect logic in the RPC Connection implementation also trusts the resolved
> address that was passed down.
> If the NN suffers a failure that requires migration, it may be started on a
> different node with a different IP address. In this case, even if the
> name-address mapping is updated in DNS, the cluster is stuck trying old
> address until the whole cluster is restarted.
> The RPC client-side should detect this situation and exit or try to recover.
> Updating ConnectionId within the Client implementation may get the system
> work for the moment, there always is a risk of the cached address:port become
> connectable again unintentionally. The real solution will be notifying upper
> layer of the address change so that they can re-resolve and retry or
> re-architecture the system as discussed in HDFS-34.
> For 0.20 lines, some type of compromise may be acceptable. For example, raise
> a custom exception for some well-defined high-impact upper layer to do
> re-resolve/retry, while other will have to restart. For TRUNK, the HA work
> will most likely determine what needs to be done. So this Jira won't cover
> the solutions for TRUNK.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
