RE: How to balance reduce job

2013-05-07 Thread Tony Burton
) is a good rule of thumb to follow. Tony From: bejoy.had...@gmail.com [mailto:bejoy.had...@gmail.com] Sent: 17 April 2013 07:19 To: user@hadoop.apache.org Cc: Mohammad Tariq Subject: Re: How to balance reduce job Yes, That is a valid point. The partitioner might do non uniform distribution

Re: How to balance reduce job

2013-05-07 Thread shashwat shriparv
** ** ** ** ** ** ** ** *From:* bejoy.had...@gmail.com [mailto:bejoy.had...@gmail.com] *Sent:* 17 April 2013 07:19 *To:* user@hadoop.apache.org *Cc:* Mohammad Tariq *Subject:* Re: How to balance reduce job ** ** Yes, That is a valid point. The partitioner might do non uniform distribution

Re: Re: How to balance reduce job

2013-04-21 Thread Rahul Bhattacharjee
remote device, Please excuse typos -- *From: *rauljin liujin666...@sina.com *Date: *Wed, 17 Apr 2013 12:53:37 +0800 *To: *user@hadoop.apache.orguser@hadoop.apache.org *ReplyTo: *user@hadoop.apache.org *Subject: *How to balance reduce job 8 datanode in my hadoop

Re: How to balance reduce job

2013-04-17 Thread Ajay Srivastava
@hadoop.apache.orgmailto:user@hadoop.apache.org; Bejoy Ksbejoy.had...@gmail.commailto:bejoy.had...@gmail.com Subject: Re: How to balance reduce job Just to add to Bejoy's comments, it also depends on the data distribution. Is your data properly distributed across the HDFS? Warm Regards, Tariq https

Re: How to balance reduce job

2013-04-17 Thread bejoy . hadoop
Cc: Mohammad Tariqdonta...@gmail.com Subject: Re: How to balance reduce job Tariq probably meant distribution of keys from key, value pair emitted by mapper. Partitioner distributes these pairs to different reducers based on key. If data is such that keys are skewed then most of the records may

Re: Re: How to balance reduce job

2013-04-17 Thread rauljin
in the configuration? rauljin From: bejoy.hadoop Date: 2013-04-17 13:09 To: user; liujin666jin Subject: Re: How to balance reduce job Hi Rauljin Few things to check here. What is the number of reduce slots in each Task Tracker? What is the number of reduce tasks for your job? Based

How to balance reduce job

2013-04-16 Thread rauljin
8 datanode in my hadoop cluseter ,when running reduce job,there is only 2 datanode running the job . I want to use the 8 datanode to run the reduce job,so I can balance the I/O press. Any ideas? Thanks. rauljin

Re: How to balance reduce job

2013-04-16 Thread bejoy . hadoop
typos -Original Message- From: rauljin liujin666...@sina.com Date: Wed, 17 Apr 2013 12:53:37 To: user@hadoop.apache.orguser@hadoop.apache.org Reply-To: user@hadoop.apache.org Subject: How to balance reduce job 8 datanode in my hadoop cluseter ,when running reduce job,there is only 2

Re: How to balance reduce job

2013-04-16 Thread Mohammad Tariq
-- *From: * rauljin liujin666...@sina.com *Date: *Wed, 17 Apr 2013 12:53:37 +0800 *To: *user@hadoop.apache.orguser@hadoop.apache.org *ReplyTo: * user@hadoop.apache.org *Subject: *How to balance reduce job 8 datanode in my hadoop cluseter ,when running reduce job,there is only 2

Re: How to balance reduce job

2013-04-16 Thread bejoy . hadoop
Message- From: Mohammad Tariq donta...@gmail.com Date: Wed, 17 Apr 2013 10:46:27 To: user@hadoop.apache.orguser@hadoop.apache.org; Bejoy Ksbejoy.had...@gmail.com Subject: Re: How to balance reduce job Just to add to Bejoy's comments, it also depends on the data distribution. Is your data