Replying to my own thread here. While we got a good handle on the IP based hadoop cluster by using the settings mentioned above, we are now upgrading the Cloudera 5.1.0 packages and Yarn.
So far most everything seemed to work well, except that for some reason Yarn insists on making use of DNS, which is causing issues in our environment. We've set the following things in our yarn-site.xml (customized for each host through automation): The issue is that I still get java.net.NoRouteToHostException while running jobs, and the exception is showing the fqdn of the hosts an not the IP addresses which I want Yarn to use. When looking at http://1.2.3.100:8042/conf I see that most values provided in the xml are honored, but not all: It seems to be a bug that yarn.nodemanager.localizer.address is set 'programatically' and not from yarn-site.xml. I have even added it as -Dyarn.nodemanager.localizer.address=1.2.3.100:8040 to YARN_OPTS when launching each individual node manager without any change shown in http://1.2.3.100:8042/conf. My guess is that we need to be able to explicitaly what we want yarn.nodemanager.localizer.address to be, but maybe also a way to tell the ressourcemanager to not be 'smart' about the nodes fqdn, like 'hadoop.security.token.service.use_ip' is supposed to do. -- View this message in context: http://hadoop.6.n7.nabble.com/IP-based-hadoop-cluster-tp70191p73503.html Sent from the common-user mailing list archive at Nabble.com.
