Hello Hadoop users,

I have a question about having a different Hadoop Home directory on every slave 
machine. First, some background: right now, to get around an issue where we do 
not have a dedicated hadoop user, I am specifying an SSH config file in 
SSH_OPTS of hadoop-env.sh.  This config file has a host, user, and key so SSH 
knows what key to use and what user to connect as for each slave.  In our case, 
every slave is likely to have a different user.  This, unfortunately, is not 
something I can change, but the SSH Config file works and allows the server 
admin to SSH passphraseless to the home folder of the individual users at each 
client.

The issue I'm running into is the fact that each client, since it has a unique 
user, also has a unique hadoop_home directory.  For example, UserB's 
hadoop_home directory is likely to be /home/UserB/hadoop-0.18.2 whereas the 
server's home directory is /home/UserA/hadoop-0.18.2.  This causes a problem 
because when you run bin/start-all.sh, it tries to start each server by SSHing 
(which works successfully) and then changing directory to the same directory as 
the server's hadoop home.  The specific SSH command that the server is running 
to start each node (as defined in slaves.sh) is "ssh -q -F /home/userA/ 
hadoop-0.18.2/bin/../sshconfig hostname "cd 
/home/userA/hadoop-0.18.2/bin/../conf start datanode"." 

The ultimate question that arises out of this is: Is there a way I can get it 
to read the Hadoop_Home from the slave node and then try to start the datanode? 
 Right now it's using Hadoop_Home of the server, and assuming it is the same as 
the slaves, which it probably never will be.  In the above example, changing 
userA to userB would work, but the users will vary.  It seems strange to me 
that there's nothing (that I can find) to account for varying Hadoop_Home 
locations on the nodes.

I'd be happy to hear any suggestions that anybody has.  Unfortunately, the 
option of having a dedicated Hadoop user or a consistent Hadoop_Home are both 
out of my control.

Thank you for any feedback,
Alyssa Hargraves

Reply via email to