[
https://issues.apache.org/jira/browse/HADOOP-20027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-20027:
------------------------------------
Labels: ipv6 pull-request-available (was: ipv6)
> DatanodeID builds unbracketed IPv6 host:port strings
> ----------------------------------------------------
>
> Key: HADOOP-20027
> URL: https://issues.apache.org/jira/browse/HADOOP-20027
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: ipv6, pull-request-available
>
> {{DatanodeID}} builds its address strings by plain concatenation:
> {code:java}
> this.xferAddr = ipAddr + ":" + xferPort; // setIpAndXferPort
> return ipAddr + ":" + ipcPort; // getIpcAddr
> return ipAddr + ":" + infoPort; // getInfoAddr
> return ipAddr + ":" + infoSecurePort; // getInfoSecureAddr
> {code}
> [DatanodeID.java:138, 204, 211,
> 218|https://github.com/apache/hadoop/blob/9ede2e95c6fb577d7ae8664833d5e151cbfc47c9/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java#L138]
> For a DataNode with an IPv6 address this gives an ambiguous string such as
> {{2001:db8:0:0:0:0:0:1:9866}} instead of {{[2001:db8:0:0:0:0:0:1]:9866}}.
> {{NetUtils.createSocketAddr}} parses it as the authority of a URI.
> {{java.net.URI}} returns a null host for the unbracketed form, so
> {{createSocketAddr}} throws {{IllegalArgumentException: Does not contain a
> valid host:port authority}}.
> {{getXferAddr}} is what the client uses to reach DataNodes: {{DataStreamer}}
> for writes, {{DFSInputStream}} for reads, {{DFSUtilClient}} and
> {{ClientContext}}. On an IPv6-only cluster that uses IP addresses
> ({{dfs.client.use.datanode.hostname=false}}, the default), clients cannot
> connect to any DataNode. The xfer address is cached in the {{DatanodeID}} and
> sent to clients in every block location, so one bad string affects every read
> and write. The info addresses have the same problem in the web UI and in
> WebHDFS redirects.
> {{getXferAddrWithHostname}} and {{getIpcAddrWithHostname}} (lines 225 and
> 232) use the same pattern. They are only wrong if the host name is itself an
> IPv6 literal, but they should use the same helper.
> *Proposed fix:* build all of these strings with
> {{NetUtils.getHostPortString(host, port)}}, which brackets IPv6 literals
> since HADOOP-19958. IPv4 and host name output is unchanged. Add a unit test
> that round-trips each address through {{NetUtils.createSocketAddr}} for IPv4,
> IPv6 and host names.
> HADOOP-17543 reports an HDFS put failure on an IPv6 cluster with no stack
> trace; this is a likely cause.
> POC:
> https://github.com/smengcl/hadoop/commit/8aeae47d67401180a954f7859ae423acf641477c
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
