Siyao Meng created HDFS-18002:
---------------------------------
Summary: RBF: Router builds unbracketed IPv6 host:port strings for
heartbeats and the State Store
Key: HDFS-18002
URL: https://issues.apache.org/jira/browse/HDFS-18002
Project: Hadoop HDFS
Issue Type: Bug
Components: rbf
Reporter: Siyao Meng
Assignee: Siyao Meng
The Router builds {{host:port}} strings by concatenation in two places. For an
IPv6 host the result has no brackets, for example
{{2001:db8:0:0:0:0:0:10:8020}}, and {{NetUtils.createSocketAddr}} rejects it
with "Does not contain a valid host:port authority".
*1. NamenodeHeartbeatService with name resolution*
([NamenodeHeartbeatService.java:206|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java#L206])
{code:java}
rpcAddress = resolvedHost + ":"
+ NetUtils.getPortFromHostPortString(rpcAddress);
serviceAddress = resolvedHost + ":"
+ NetUtils.getPortFromHostPortString(serviceAddress);
{code}
With
{{dfs.federation.router.monitor.namenode.nameservice.resolution-enabled.<ns>=true}},
the Router resolves the configured NameNode name to its addresses and creates
one heartbeat service per address with {{resolvedHost = isa.getHostName()}}
([Router.java:577|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java#L577]).
For an AAAA record with no PTR record that is the IPv6 literal.
{{lifelineAddress}} and {{webAddress}} are built the same way. {{serviceInit}}
then creates an {{NNHAServiceTarget}} from {{serviceAddress}}, whose
{{createSocketAddr}} call throws. The heartbeat service is a child service of
the Router, so the Router fails to start. {{getRpcAddress()}} has the same
pattern in its fallback
([NamenodeHeartbeatService.java:299|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java#L299]),
reached when a nameservice is monitored without an nnId and lists one
NameNode. Without resolution the addresses are the configured, bracketed
strings, and these paths work.
*2. Router admin addresses in the State Store*
([StateStoreUtils.java:137|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreUtils.java#L137],
[StateStoreUtils.java:152|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreUtils.java#L152])
{code:java}
return hostName + ":" + address.getPort();
...
return inet.getHostAddress() + ":" + address.getPort();
{code}
{{RouterHeartbeatService}} stores each Router's admin address with
{{getHostPortString}} (default) or {{getIpPortString}}
({{dfs.federation.router.heartbeat.with.ip.enable=true}}). {{getIpPortString}}
always gives an unbracketed IPv6 string; {{getHostPortString}} does when there
is no PTR record. {{getHostPortString}} also only replaces the IPv4 wildcard
{{0.0.0.0}}, so {{dfs.federation.router.admin-address=[::]:8111}} is stored as
{{0:0:0:0:0:0:0:0:8111}}. With
{{dfs.federation.router.mount-table.cache.update=true}} and two or more
Routers, {{MountTableRefresherService}} passes these addresses to
{{createSocketAddr}}
([MountTableRefresherService.java:170|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/MountTableRefresherService.java#L170]).
The {{IllegalArgumentException}} reaches the caller wrapped in Guava's
{{UncheckedExecutionException}}, which the {{catch (ExecutionException)}} at
line 248 does not catch, so add, update and remove mount table calls return an
error after the entry has been written, and the other Routers are not refreshed.
Verified by reading the code, and on JDK 21: {{getHostAddress()}} and the
no-PTR {{getHostName()}} return unbracketed literals, and a copy of the
{{createSocketAddr}} parsing rejects them. The Guava wrapping is from the
documented {{LoadingCache.get}} contract. Not run on a Router or with real DNS.
*Proposed fix:* use {{NetUtils.getHostPortString(host, port)}} for the four
heartbeat addresses, the {{getRpcAddress()}} fallback, and both
{{StateStoreUtils}} helpers. In {{getHostPortString}}, treat any
{{isAnyLocalAddress()}} address as the wildcard. Optionally catch
{{RuntimeException}} per Router in {{MountTableRefresherService}} so one bad
record cannot fail the admin call. Add unit tests with IPv6 hosts.
POC:
https://github.com/smengcl/hadoop/commit/daab9184f1b0818161b3c53a50753aaaefb9f568
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]