Siyao Meng created HDFS-17999:
---------------------------------
Summary: WebHDFS NameNode rejects IPv6 excludedatanodes entries
with NumberFormatException
Key: HDFS-17999
URL: https://issues.apache.org/jira/browse/HDFS-17999
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Reporter: Siyao Meng
Assignee: Siyao Meng
{{NamenodeWebHdfsMethods.chooseDatanode}} splits each {{excludedatanodes}}
entry at the first colon, so an IPv6 entry fails with {{NumberFormatException}}
and the request returns HTTP 400.
[NamenodeWebHdfsMethods.java:298|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L298]
{code:java}
int idx = host.indexOf(':');
...
excludeNode = bm.getDatanodeManager().getDatanodeByXferAddr(
host.substring(0, idx), Integer.parseInt(host.substring(idx + 1)));
{code}
The WebHDFS client fills this parameter itself. When a DataNode advertises an
IPv6 literal as its host name, the NameNode builds the redirect with the
multi-argument {{URI}} constructor
([NamenodeWebHdfsMethods.java:465|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L465]),
which brackets it: {{http://[2001:db8::1]:9864/webhdfs/v1/...}}. If the client
then fails to connect to or read from that DataNode, it adds
{{[2001:db8::1]:9864}} to {{excludedatanodes}}
([WebHdfsFileSystem.java:744|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L744]
uses {{URL.getHost()}}, which keeps the brackets;
[WebHdfsFileSystem.java:2652|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L2652]
uses {{getAuthority()}}). On the retry the NameNode computes {{idx = 5}} and
calls {{Integer.parseInt("db8::1]:9864")}}:
{code}
java.lang.NumberFormatException: For input string: "db8::1]:9864"
{code}
{{ExceptionHandler}} maps this to 400 Bad Request. The client does not retry a
{{RemoteException}}, so the read or write fails instead of going to another
DataNode. Unbracketed entries ({{2001:db8:0:0:0:0:0:1:9866}}) and bare literals
fail the same way. The client sends the parameter on retries, which happen when
HA failover is configured or {{dfs.http.client.retry.policy.enabled}} is true.
A REST caller that passes an IPv6 {{excludedatanodes}} value hits it directly.
{{ExcludeDatanodesParam}} does not validate the value.
Verified with a standalone JDK 21 program: {{URI}} and {{URL}} produce the
bracketed authority above, and the same parsing logic throws the exception
shown. The 400 mapping and the retry behaviour were traced by reading
{{ExceptionHandler}}, {{NamenodeWebHdfsMethods.doAsExternalCall}} and
{{RetryPolicies.FailoverOnNetworkExceptionRetry}}. Not run on an IPv6
MiniDFSCluster.
Stripping the brackets is not enough for an entry to match. {{Host2NodesMap}}
is keyed by {{DatanodeID.ipAddr}}, which {{DatanodeManager}} sets from
{{InetAddress.getHostAddress()}} (expanded form, such as
{{2001:db8:0:0:0:0:0:1}}), so the literal also has to be normalized.
*Proposed fix:* parse each entry bracket-aware. {{[addr]:port}} is an address
and port. An entry with more than one colon and no brackets is a bare IPv6
literal with no port. Anything else is {{host:port}} or {{host}}. Normalize IP
literals with {{InetAddresses.forString(addr).getHostAddress()}} before the
lookup. Catch {{IllegalArgumentException}} per entry, log it and skip the
entry, so one malformed entry cannot fail the request. Add a case to
{{TestWebHdfsDataLocality}} that passes a bracketed IPv6 entry to
{{chooseDatanode}}.
Possibly related, found by reading only: the client sends the DataNode's HTTP
port, while {{getDatanodeByXferAddr}} compares the transfer port. That may make
client-generated exclusions ineffective on IPv4 as well. Not verified.
POC (the NameNode part needs the normalization above; the {{WebHdfsFileSystem}}
change is not needed):
https://github.com/smengcl/hadoop/commit/6f3f5e696bbe8da605d5fc6de96cb6c15bcc66bf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]