Hi Sungho, Here is a "recipe" for how to run multiple nodes on a single server, posted to this list on Sept. 15: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201009.mbox/%3c8a898c33-dc4e-418c-adc0-5689d434b...@yahoo-inc.com%3E
For v22 and later, the world has been split into three parts; where there was formerly HADOOP_HOME, there is now HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, and HADOOP_MAPRED_HOME, and in the default configuration each of them has its own "conf/" subdirectory. However, it is acceptable to pile all the contents of the three conf directories into a single conf directory somewhere else (the only name conflict is "configuration.xsl" which can be shared), set an environment variable $HADOOP_CONF_DIR to point to it, and pass that value in with the "--config" option whenever you launch processes with bin/hadoop or bin/hdfs. Now, the above recipe assumes you want multiple nodes from ONE cluster running on a single server. I suggest you start with that and get it working, so you understand the hdfs-site.xml file and how it is used. You seem to be asking to run multiple CLUSTERS on a single server. I believe the same mechanism will work (pointing different "node" invocations at different config directories), but you will need to make several more changes in the $HADOOP_CONF_DIR/hdfs-site.xml files, to create different namenode configurations as well as the different datanode configurations addressed in the recipe. Please look at the documentation for which parameters to change. A couple comments: - You probably can't run two namenodes simultaneously in the same server, unless it has a huge amount of memory and you don't care about performance. But you can have two different configurations stored, and run them at different times. - If the ONLY difference in the two clusters is the number of datanodes, you actually don't have to have different namenode configurations. You can just configure 10 datanodes, and then sometimes run only 5 of them (clearing storage in between test runs, of course, so it doesn't look like you lost half your stored blocks!). This is because namenodes have no configuration for which or how many datanodes to expect; namenodes simply accept registration from any datanode that initiates communication with it. - Your statement "I can control number of datanode by change conf and restart" is therefore not entirely correct. Each datanode launched has to be pointed at its own config, but there is no place in the config to define how many datanodes to launch. (This is partly because running multiple nodes on a single server is not considered normal for a production environment, even though it is useful for a test environment.) You may be thinking of the "slaves" file, which is used by some launch scripts, but that is a tool to assist users in launching clusters, not part of namenode configuration, and is also not really oriented toward launching multiple nodes in a single server, if you read the scripts. If you want launch scripts to help you locally launch different numbers of nodes with different configs, you'll have to write them yourself, but they're really easy. They just consist of multiple lines that look like $HADOOP_COMMON_HOME/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script $HADOOP_HDFS_HOME/bin/hdfs start <datanode|namenode> [args] with different values of $HADOOP_CONF_DIR for each line. The same lines with "stop" instead of "start" will give you a well-behaved kill script. As always you have to start and stop each node with appropriate userId so they have read/write and i/o access permissions. Hope this helps, --Matt On Mar 1, 2011, at 4:19 AM, Sungho Jeon wrote: Hi, I'm graduate student and my major is computer science, data mining. Is that possible that install multiple hadoop in one node? I mean, I want to install several hadoop that have different conf. Specifically, one hadoop has 5 datanode and other hadoop has 10 datanode. Of course I can control number of datanode by change conf and restart. But, without changing conf, install multiple hadoop in one node is possible? Thanks