Re: Running hadoop jobs from a client and tuning (was Re: How does hadoop deal with hadoop-site.xml?)

2009-08-21 Thread stephen mulcahy
Hi Amogh, Thanks for your reply. Some comments below. Amogh Vasekar wrote: AFAIK, hadoop.tmp.dir : Used by NN and DN for directory listings and metadata ( don't have much info on this ) I've been running some test jobs against a local hadoop cluster from eclipse using the eclipse plugin.

Running hadoop jobs from a client and tuning (was Re: How does hadoop deal with hadoop-site.xml?)

2009-08-20 Thread stephen mulcahy
Hi folks, Sorry to cut across this discussion but I'm experiencing some similar confusion about where to change some parameters. In particular, I'm not entirely clear on how the following should be used - clarification welcome (I'm happy to pull some of this together on a blog once I get

Re: How does hadoop deal with hadoop-site.xml?

2009-08-19 Thread Aaron Kimball
Hi Inifok, This is a confusing aspect of Hadoop, I'm afraid. Settings are divided into two categories: per-job and per-node. Unfortunately, which are which, isn't documented. Some settings are applied to the node that is being used. So for example, if you set fs.default.name on a node to be

Re: How does hadoop deal with hadoop-site.xml?

2009-08-19 Thread yang song
Thank you, Aaron. I've benefited a lot. per-node means some settings associated with the node. e.g., fs.default.name, mapred.job.tracker, etc. per-job means some settings associated with the jobs which are submited from the node. e.g., mapred.reduce.tasks. That means, if I set per-job