Configurations for Multiple Clusters

2012-03-26 Thread Eli Finkelshteyn

Hi,
I've recently run into a situation where I'll have access to my local 
single node cluster, and development cluster, and a production cluster. 
What's the best way to work with all three of these, easily switching 
between which I want to use? I found this project 
http://code.google.com/p/hadoopenv/wiki/README that seems like it's 
trying to address my issue, but it hasn't been updated since 2010. Is 
this still the preferred way of managing multiple clusters, or is there 
something better (preferably something easy to install, as I'll need to 
convince others at my company to use whatever solution I find).


Thanks!
Eli


Re: Configurations for Multiple Clusters

2012-03-26 Thread Harsh J
Eli,

Couple of things I've done (Not necessarily ideal):

* Use a git repo for configs with branches. Switch when needed.
* Point HADOOP_CONF_DIR to the right conf-dir location (or have
triggers that do these for you). Similar to above I guess.
* Alias hadoop multiple times to use different --config conf dir
opts, and use convenient aliased names.

Taking a brief look at hadoopenv sources (Last update seems to be mid
2011), I think it would work with the stable versions of Hadoop even
today. You may give it a whirl too if its approach attracts you.

On Mon, Mar 26, 2012 at 9:30 PM, Eli Finkelshteyn iefin...@gmail.com wrote:
 Hi,
 I've recently run into a situation where I'll have access to my local single
 node cluster, and development cluster, and a production cluster. What's the
 best way to work with all three of these, easily switching between which I
 want to use? I found this project
 http://code.google.com/p/hadoopenv/wiki/README that seems like it's trying
 to address my issue, but it hasn't been updated since 2010. Is this still
 the preferred way of managing multiple clusters, or is there something
 better (preferably something easy to install, as I'll need to convince
 others at my company to use whatever solution I find).

 Thanks!
 Eli



-- 
Harsh J