Re: only one reducer running in a hadoop cluster

2009-02-09 Thread Nick Cen 
Thanks everyone. I find the solution for this one, in my main method, i call
the setNumReductTask() on JobConf with the value i want.

2009/2/9 Owen O'Malley 

>
> On Feb 7, 2009, at 11:52 PM, Nick Cen wrote:
>
>  Hi,
>>
>> I hava a hadoop cluster with 4 pc. And I wanna to integrate hadoop and
>> lucene together, so i copy some of the source code from nutch's Indexer
>> class, but when i run my job, i found that there is only 1 reducer running
>> on 1 pc, so the performance is not as far as expect.
>>
>
> Set mapred.reduce.tasks in your configuration to the number of reduces, you
> want your jobs to have by default. Typically this should be 0.99 *
> mapred.tasktracker.reduce.tasks.maximum * number of computers.
>



-- 
http://daily.appspot.com/food/

Re: only one reducer running in a hadoop cluster

2009-02-09 Thread Owen O'Malley 


On Feb 7, 2009, at 11:52 PM, Nick Cen wrote:


Hi,

I hava a hadoop cluster with 4 pc. And I wanna to integrate hadoop and
lucene together, so i copy some of the source code from nutch's  
Indexer
class, but when i run my job, i found that there is only 1 reducer  
running

on 1 pc, so the performance is not as far as expect.


Set mapred.reduce.tasks in your configuration to the number of  
reduces, you want your jobs to have by default. Typically this should  
be 0.99 * mapred.tasktracker.reduce.tasks.maximum * number of computers.

Re: only one reducer running in a hadoop cluster

2009-02-08 Thread Amareshwari Sriramadasu 

Nick Cen wrote:

Hi,

I hava a hadoop cluster with 4 pc. And I wanna to integrate hadoop and
lucene together, so i copy some of the source code from nutch's Indexer
class, but when i run my job, i found that there is only 1 reducer running
on 1 pc, so the performance is not as far as expect.

  

what is the configuration of mapred.tasktracker.reduce.tasks.maximum ?


-Amareshwari

only one reducer running in a hadoop cluster

2009-02-07 Thread Nick Cen 
Hi,

I hava a hadoop cluster with 4 pc. And I wanna to integrate hadoop and
lucene together, so i copy some of the source code from nutch's Indexer
class, but when i run my job, i found that there is only 1 reducer running
on 1 pc, so the performance is not as far as expect.

-- 
http://daily.appspot.com/food/