Hello,
Tl;dr question: How to make the NodeManager give the "yarn.nodemanager.hostname" value to the Yarn ResourceManager during registration for IPC communication. Explanations: I have trouble making a Yarn cluster works in a Kubernetes deployment where every components run as a Docker container. When the NodeManager registers with the Yarn ResourceManager, it give its hostname for NodeId. Then the ResourceManager uses this NodeId to communicate with the node. The problem is that the hostname comes from system calls and not from the Yarn's configuration files. The ResourceManager is unable to talk to the NodeManager because the container hostname is invalid in the DNS context. Moreover, Docker disallows a dynamic change of a container's hostname. My Yarn NodeManager's configuration looks like this: * <property> * <name>yarn.resourcemanager.hostname</name> * <value>yarnrm-int-svc.default.svc.cluster.docker</value> * </property> * * <property> * <name>yarn.nodemanager.hostname</name> * <value>100-66-10-4.default.pod.cluster.docker</value> * </property> * <property> * <name>yarn.nodemanager.address</name> * <value>100-66-10-4.default.pod.cluster.docker:8111</value> * </property> * <property> * <name>yarn.nodemanager.webapp.address</name> * <value>100-66-10-4.default.pod.cluster.docker:8042</value> * </property> * <property> * <name>yarn.nodemanager.localizer.address</name> * <value>100-66-10-4.default.pod.cluster.docker:8040</value> * </property> I tried to simply use IP but it didn't work either. For now I had to fork the Kubernetes's DNS plugin to please Yarn but its not a proper solution. It's difficult to understand why the NodeId not configurable in this setup. If so, why do the "yarn.nodemanager.hostname" variable exists in the first place ?. Anyway, if someone knows a solution to this problem it would be very helpful. Thanks Geoffroy
