Hi, On Wed, Nov 24, 2010 at 10:23 PM, Grandl Robert <[email protected]> wrote: > Hi, > I am sorry bothering again about this subject, but still I am not very > convinced what Hadoop assumes a slot is. I understood it represent smth in > terms of CPU/Memory, so you have to allocate corresponding numbers of > map/reduce slots based on your configurations. > BUT, I cannot understand yet, if Hadoop make any mapping between the concept > of slot and physical resources itself, or are just some numbers and you can > go over only with this numbers.
The slot amount is the user's homework for now. > I looked on the code, but I am not able to figure out if Hadoop really did > some checking between number of slots and physical resources, or just is > limited by the 2 numbers(for maximum number of map slots and reduce slots) > and play with this numbers only. That means, the user should give more > interpretation of what a slot really may be: (Only one slot per core, one > slot per 512 MB, etc) when configure the number of map/reduce slots on his > machines. Yes, Hadoop does not dynamically detect any such thing yet. The setup is ignorant to a machine's hardware and blindly relies on the configurations passed at start up. I usually set M = No. of CPUs + 1, and R = Prime nearest No. of CPUs. But needs may vary depending on the nature of jobs it is going to perform; sometimes you may need lesser CPU but more Memory/Task, so configure based on your application knowledge. -- Harsh J www.harshj.com
