[
https://issues.apache.org/jira/browse/HADOOP-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116992#comment-17116992
]
Dhiraj Hegde commented on HADOOP-17052:
---------------------------------------
[~hemanthboyina] Here is a stack trace of read failing due to the exception
thrown:
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at
org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3363)
at
org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:817)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:710)
at
org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:356)
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:666)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:875)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:927)
...
FATAL [MemStoreFlusher.0] regionserver.HRegionServer - ABORTING region server
> NetUtils.connect() throws an exception the prevents any retries when hostname
> resolution fails
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-17052
> URL: https://issues.apache.org/jira/browse/HADOOP-17052
> Project: Hadoop Common
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.10.0, 2.9.2, 3.2.1, 3.1.3
> Reporter: Dhiraj Hegde
> Assignee: Dhiraj Hegde
> Priority: Major
>
> Hadoop components are increasingly being deployed on VMs and containers. One
> aspect of this environment is that DNS is dynamic. Hostname records get
> modified (or deleted/recreated) as a container in Kubernetes (or even VM) is
> being created/recreated. In such dynamic environments, the initial DNS
> resolution request might return resolution failure briefly as DNS client
> doesn't always get the latest records. This has been observed in Kubernetes
> in particular. In such cases NetUtils.connect() appears to throw
> java.nio.channels.UnresolvedAddressException. In much of Hadoop code (like
> DFSInputStream and DFSOutputStream), the code is designed to retry
> IOException. However, since UnresolvedAddressException is not child of
> IOException, no retry happens and the code aborts immediately. It is much
> better if NetUtils.connect() throws java.net.UnknownHostException as that is
> derived from IOException and the code will treat this as a retry-able error.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
