I agree, we will go the standard route. Like you suggested we will go step by step to the full cluster deployment. After the first node configuration we will use clonezilla to replicate it and then setup them one by one..
On the workernodes I was thinking to run ubuntu server, namenode will run ubuntu desktop. I am interested how should I configure the environment that I will able to remotely monitor, analyse and configure the cluster. I will run jobs outsite the local network via ssh to the namenode, however in this situation I will not be abble to access the web interface of the job and tasktracker. So I am wondering how to analyze them and how did you configure your environment to be as practical as possible. For monitoring the cluster I saw that ganglia is one of the option, but in this stage of testing probably job-history files will be enough.. On 23 September 2011 17:09, GOEKE, MATTHEW (AG/1000) < matthew.go...@monsanto.com> wrote: > If you are starting from scratch with no prior Hadoop install experience I > would configure stand-alone, migrate to pseudo distributed and then to fully > distributed verifying functionality at each step by doing a simple word > count run. Also, if you don't mind using the CDH distribution then SCM / > their rpms will greatly simplify both the bin installs as well as the user > creation. > > Your VM route will most likely work but I can imagine the amount of hiccups > during migration from that to the real cluster will not make it worth your > time. > > Matt > > -----Original Message----- > From: Merto Mertek [mailto:masmer...@gmail.com] > Sent: Friday, September 23, 2011 10:00 AM > To: common-user@hadoop.apache.org > Subject: Environment consideration for a research on scheduling > > Hi, > in the first phase we are planning to establish a small cluster with few > commodity computer (each 1GB, 200GB,..). Cluster would run ubuntu server > 10.10 and a hadoop build from the branch 0.20.204 (i had some issues with > version 0.20.203 with missing > libraries< > http://hadoop-common.472056.n3.nabble.com/Development-enviroment-problems-eclipse-hadoop-0-20-203-td3186022.html#a3188567 > >). > Would you suggest any other version? > > In the second phase we are planning to analyse, test and modify some of > hadoop schedulers. > > Now I am interested what is the best way to deploy ubuntu and hadop to this > few machine. I was thinking to configure the system in the local VM and > then > converting it to each physical machine but probably this is not the best > option. If you know any other please share.. > > Thanks you! > This e-mail message may contain privileged and/or confidential information, > and is intended to be received only by persons entitled > to receive such information. If you have received this e-mail in error, > please notify the sender immediately. Please delete it and > all attachments from any servers, hard drives or any other media. Other use > of this e-mail by you is strictly prohibited. > > All e-mails and attachments sent and received are subject to monitoring, > reading and archival by Monsanto, including its > subsidiaries. The recipient of this e-mail is solely responsible for > checking for the presence of "Viruses" or other "Malware". > Monsanto, along with its subsidiaries, accepts no liability for any damage > caused by any such code transmitted by or accompanying > this e-mail or any attachment. > > > The information contained in this email may be subject to the export > control laws and regulations of the United States, potentially > including but not limited to the Export Administration Regulations (EAR) > and sanctions regulations issued by the U.S. Department of > Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this > information you are obligated to comply with all > applicable U.S. export laws and regulations. > >
