Hi,
Thanks for your help, your suggestion is very usefully.
I have another question that is whether the sum of maps and reduces equals
to the total number of cores.
regards!
2012-01-10
hao.wang
发件人： Harsh J
发送时间： 2012-01-10 16:44:07
收件人： common-user
抄送：
主题： Re: how to set mapred.tasktracker.map.tasks.maximum and
mapred.tasktracker.reduce.tasks.maximum
Hello Hao,
Am sorry if I confused you. By CPUs I meant the CPUs visible to your OS
(/proc/cpuinfo), so yes the total number of cores.
On 10-Jan-2012, at 12:39 PM, hao.wang wrote:
> Hi ,
>
> Thanks for your reply!
> According to your suggestion, Maybe I can't apply it to our hadoop cluster.
> Cus, each server in our hadoop cluster just contains 2 CPUs.
> So, I think maybe you mean the core # but not CPU # in each searver?
> I am looking for your reply.
>
> regards!
>
>
> 2012-01-10
>
>
>
> hao.wang
>
>
>
> 发件人： Harsh J
> 发送时间： 2012-01-10 11:33:38
> 收件人： common-user
> 抄送：
> 主题： Re: how to set mapred.tasktracker.map.tasks.maximum and
> mapred.tasktracker.reduce.tasks.maximum
>
> Hello again,
> Try a 4:3 ratio between maps and reduces, against a total # of available CPUs
> per node (minus one or two, for DN and HBase if you run those). Then tweak it
> as you go (more map-only loads or more map-reduce loads, that depends on your
> usage, and you can tweak the ratio accordingly over time -- changing those
> props do not need JobTracker restarts, just TaskTracker).
> On 10-Jan-2012, at 8:17 AM, hao.wang wrote:
>> Hi,
>> Thanks for your reply!
>> I had already read the pages before, can you give me sme more specific
>> suggestions about how to choose the values of
>> mapred.tasktracker.map.tasks.maximum and
>> mapred.tasktracker.reduce.tasks.maximum according to our cluster
>> configuration if possible?
>>
>> regards!
>>
>>
>> 2012-01-10
>>
>>
>>
>> hao.wang
>>
>>
>>
>> 发件人： Harsh J
>> 发送时间： 2012-01-09 23:19:21
>> 收件人： common-user
>> 抄送：
>> 主题： Re: how to set mapred.tasktracker.map.tasks.maximum and
>> mapred.tasktracker.reduce.tasks.maximum
>>
>> Hi,
>> Please read
>> http://hadoop.apache.org/common/docs/current/single_node_setup.html to learn
>> how to configure Hadoop using the various *-site.xml configuration files,
>> and then follow
>> http://hadoop.apache.org/common/docs/current/cluster_setup.html to achieve
>> optimal configs for your cluster.
>> On 09-Jan-2012, at 5:50 PM, hao.wang wrote:
>>> Hi ,all
>>> Our hadoop cluster has 22 nodes including one namenode, one jobtracker and
>>> 20 datanodes.
>>> Each node has 2 * 12 cores with 32G RAM
>>> Dose anyone tell me how to config following parameters:
>>> mapred.tasktracker.map.tasks.maximum
>>> mapred.tasktracker.reduce.tasks.maximum
>>>
>>> regards!
>>> 2012-01-09
>>>
>>>
>>>
>>> hao.wang
