I solved how to set different data directories on the same NFS $HOME
directories shared by all nodes in the cluster.
The HADOOP_OPTS value set in conf/hadoop-env.sh is the used to set custom
properties. All the values set to this variable will be set to JVM.
Just like we set java.net.preferIPv4Stack=true before to prefer use IPv4. It
actually did this:
hadoop xxxx -D java.net.preferIPv4Stack=true
or more low level
java xxxx -Djava.net.preferIPv4Stack=true
(Notice here the grammar of the -D option is different in hadoop with in
JVM)
Therefore, we can also set other properties just as you like.
You can find detailed explanations in "Hadoop The Definitive Guide" chapter
5 "Developing a MapReduce Application" section 1 "The Configuration API".
So for my problem, I set the HADOOP_OPTS as:
HADOOP_OPTS="-Dhostname=$HOSTNAME -Dhome=$HOME
-Djava.net.preferIPv4Stack=true"
So that I can use ${hostname} in the core-site.xml to get the variable of
system environment $HOSTNAME.
Then I can get different directories in the shared $HOME directory for
different nodes.
Second, after I set like this, I find I can't use hadoop.tmp.dir value
anymore, or else there will be an java.lang.IllegalArgumentException: Wrong
FS: exception.So I can only through set dfs.name.dir and dfs.data.dir in
conf/hdfs.xml to solve this problem.I.e., strictly according to the Hadoop
"cluster set" said.
Best Regards
Jiamin Lu
