Siyao Meng created HADOOP-20025:
-----------------------------------
Summary: Block pool ID generated on an IPv6 NameNode contains
colons
Key: HADOOP-20025
URL: https://issues.apache.org/jira/browse/HADOOP-20025
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Siyao Meng
Assignee: Siyao Meng
{{NNStorage.newBlockPoolID}} puts the NameNode's IP address into the block pool
ID:
{code:java}
return "BP-" + rand + "-"+ ip + "-" + Time.now();
{code}
[NNStorage.java:1042|https://github.com/apache/hadoop/blob/9ede2e95c6fb577d7ae8664833d5e151cbfc47c9/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java#L1042]
When {{DNS.getDefaultIP}} returns an IPv6 address, the ID contains colons, for
example {{BP-1234-fd00:dead:beef:0:0:0:0:10-1760000000000}}. The ID is used as
a directory name in every DataNode volume and inside local file names. On the
IPv6 docker-compose test cluster, DataNode storage initialization failed with
{{URISyntaxException: Relative path in absolute URI}} when
{{ChecksumFileSystem}} built a {{.crc}} sidecar path from such a name. Colons
in directory names also cannot exist on Windows.
*Proposed fix:* when the IP is IPv6, replace {{:}} with {{_}} and drop any
scope ID before building the ID. IPv4 block pool IDs are unchanged, and
existing block pool IDs are not touched, because the ID is generated only when
a namespace is formatted.
The general problem of colons in local file names under {{ChecksumFileSystem}}
is HADOOP-17845.
POC:
https://github.com/smengcl/hadoop/commit/a2fbda753b05be8f20ba263451a6ecd68e966605
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
