Siyao Meng created HDFS-18000:
---------------------------------
Summary: HDFS builds NameNode URIs and host:port strings from IPv6
addresses without brackets
Key: HDFS-18000
URL: https://issues.apache.org/jira/browse/HDFS-18000
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Siyao Meng
Assignee: Siyao Meng
Several HDFS helpers build a URI or {{host:port}} string by concatenating
{{InetSocketAddress.getHostName()}} and the port. For an IPv6 address with no
reverse DNS name, {{getHostName()}} returns the literal without brackets, for
example {{2001:db8:0:0:0:0:0:10}}. The result looks like
{{hdfs://2001:db8:0:0:0:0:0:10:8021}}. {{java.net.URI}} parses this without an
error but returns a null host, and {{DistributedFileSystem.initialize}},
{{NetUtils.createSocketAddr}} and {{URI.toURL()}} then reject it.
*1. DFSUtilClient.getNNUri*
([DFSUtilClient.java:920|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java#L920])
{code:java}
return URI.create(HdfsConstants.HDFS_URI_SCHEME + "://"
+ namenode.getHostName() + portString);
{code}
{{NameNodeRpcServer}} builds {{clientRpcAddress}} from
{{rpcAddr.getHostName()}} and passes it to {{NameNode.setRpcServerAddress}},
which writes {{getNNUri()}} into {{fs.defaultFS}} of the NameNode's own
configuration
([NameNodeRpcServer.java:512|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java#L512],
[NameNode.java:784|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L784]).
With {{fs.defaultFS=hdfs://[2001:db8::10]:8020}} and no PTR record, it becomes
{{hdfs://2001:db8:0:0:0:0:0:10}}. When the NameNode becomes active with
{{fs.trash.interval > 0}}, {{startTrashEmptier}} initializes a
{{DistributedFileSystem}} from this URI
([NameNode.java:1078|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L1078]),
which throws "Incomplete HDFS URI, no host", and {{startActiveServices}} shuts
the NameNode down. {{fsck -move}} reads the same value through
{{DFSUtilClient.getNNAddress(conf)}}.
*2. DFSUtil.substituteForWildcardAddress*
([DFSUtil.java:1101|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java#L1101])
{code:java}
return defaultHost + ":" + sockAddr.getPort();
{code}
When the NameNode HTTP address is a wildcard ({{[::]:9870}}, or the default
{{0.0.0.0:9870}}), this substitutes {{defaultHost}}. {{RemoteNameNodeInfo}} and
{{StandbyCheckpointer}} pass {{getHostName()}} of the other NameNode's service
address and call {{toURL()}} on {{http://2001:db8:0:0:0:0:0:11:9870}}, which
throws {{MalformedURLException}}. {{EditLogTailer}} reports "Remote NameNodes
not correctly configured!", so the NameNode cannot start standby services.
{{DFSck}}, {{DFSAdmin -fetchImage}}, {{DFSZKFailoverController}} and
{{BackupNode}} call {{getInfoServer}} the same way.
*3. GetConf -nnRpcAddresses*
([GetConf.java:248|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java#L248])
{code:java}
tool.printOut(rpc.getHostName()+":"+rpc.getPort());
{code}
This prints {{2001:db8:0:0:0:0:0:10:8020}}. {{refresh-namenodes.sh}} runs
{{dfsadmin -fs hdfs://$namenode -refreshNodes}} with each line, which fails in
{{DistributedFileSystem.initialize}} for the same reason.
All three need {{getHostName()}} to return the literal: no PTR record, or
{{hadoop.security.token.service.use_ip=false}}, where the resolver keeps the
literal as the host name. With a PTR record the host name is used and these
paths work. Tests on {{::1}} usually get {{localhost}} from {{/etc/hosts}},
which hides the problem. {{NameNode.initializeGenericKeys}} is not affected: it
copies the configured value, so a bracketed address stays valid.
Verified by reading the code paths above on trunk, and with small JDK 21
programs: {{getHostName()}} for {{2001:db8::10}} with no PTR record returns
{{2001:db8:0:0:0:0:0:10}}; {{URI.create("hdfs://2001:db8:0:0:0:0:0:10")}}
succeeds with a null host;
{{URI.create("http://2001:db8:0:0:0:0:0:10:9870").toURL()}} throws
{{MalformedURLException}}. Not run on a NameNode.
*Proposed fix:* build these strings with {{NetUtils.getHostPortString(host,
port)}}, which brackets IPv6 literals since HADOOP-19958, or with the
multi-argument {{URI}} constructor. Keep omitting the default port in
{{getNNUri}}. Add IPv6 cases to {{TestDFSUtil}} and {{TestGetConf}}.
POC:
https://github.com/smengcl/hadoop/commit/c246cc6908b262e93a4231a04576b752a28cbe8c
and the GetConf part of
https://github.com/smengcl/hadoop/commit/7db88e0273b4e8672fab8d99b12314d78b6b18ff
(its {{initializeGenericKeys}} change is not needed)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]