Re: How to make hadoop use all nodes?

2013-09-23 Thread Antoine Vandecreme
Hi Omkar, (which has 40 containers slots.) for total cluster? Yes, it was just an hypotetical value though. Below are my real configurations. 1) yarn-site.xml - what is the resource memory configured for per node? 12288mb 2) yarn-site.xml - what is the minimum resource allocation for the

Re: How to make hadoop use all nodes?

2013-09-20 Thread Antoine Vandecreme
Hello Omkar, Thanks for your reply. Yes, all 4 points are corrects. However, my application is requesting let say 100 containers on my cluster which has 40 containers slots. So I expected to see all containers slots used but that is not the case. Just in case it matters, it is the only

How to make hadoop use all nodes?

2013-09-19 Thread Vandecreme, Antoine
Hi all, I am working with Hadoop 2.0.5 (I plan to migrate to 2.1.0 soon). When I am starting a Job, I notice that some nodes are not used or partially used. For example, if my nodes can hold 2 containers, I notice that some nodes are not running any or just 1 while others are running 2. All my

Re: How to make hadoop use all nodes?

2013-09-19 Thread Omkar Joshi
Hi, Let me clarify few things. 1) you are making container requests which are not explicitly looking for certain nodes. (No white listing). 2) All nodes are identical in terms of resources (memory/cores) and every container requires same amount of resources. 3) All nodes have capacity to run say