Teo Klestrup Röijezon created HDFS-16577:
--------------------------------------------
Summary: Let administrator override connection details when
registering datanodes
Key: HDFS-16577
URL: https://issues.apache.org/jira/browse/HDFS-16577
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 3.2.2
Reporter: Teo Klestrup Röijezon
Currently (as of 3.2.2, but reading through the release notes this doesn't seem
to have changed since then) DataNodes use the same properties for deciding
which port to bind each service to, as for deciding which ports are included in
the `DatanodeRegistration` sent to the NameNode. Further, NameNodes overwrite
the DataNode's IP address with the incoming address during registration.
Both of these prevent external users from connecting to DataNodes that are
hosted behind some sort of NAT (such as Kubernetes).
I have created a spike branch
([https://github.com/stackabletech/hadoop/tree/spike/override-datanode-id,]
based on v3.2.2) that I have confirmed solves this problem for us. There's
clearly some work to be done integrating this properly (such as using the
regular Hadoop config system and falling back to the old behaviour if no
override is configured). I'd be happy to take that on to the best of my ability
(with the caveats that I'm not super familiar with the Hadoop codebase, and
that my Java is quite rusty at this point) if the overall direction seems
acceptable.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
