) is a good rule of thumb to follow.
Tony
From: bejoy.had...@gmail.com [mailto:bejoy.had...@gmail.com]
Sent: 17 April 2013 07:19
To: user@hadoop.apache.org
Cc: Mohammad Tariq
Subject: Re: How to balance reduce job
Yes, That is a valid point.
The partitioner might do non uniform distribution
** **
** **
** **
** **
*From:* bejoy.had...@gmail.com [mailto:bejoy.had...@gmail.com]
*Sent:* 17 April 2013 07:19
*To:* user@hadoop.apache.org
*Cc:* Mohammad Tariq
*Subject:* Re: How to balance reduce job
** **
Yes, That is a valid point.
The partitioner might do non uniform distribution
remote device, Please excuse typos
--
*From: *rauljin liujin666...@sina.com
*Date: *Wed, 17 Apr 2013 12:53:37 +0800
*To: *user@hadoop.apache.orguser@hadoop.apache.org
*ReplyTo: *user@hadoop.apache.org
*Subject: *How to balance reduce job
8 datanode in my hadoop
@hadoop.apache.orgmailto:user@hadoop.apache.org;
Bejoy Ksbejoy.had...@gmail.commailto:bejoy.had...@gmail.com
Subject: Re: How to balance reduce job
Just to add to Bejoy's comments, it also depends on the data distribution. Is
your data properly distributed across the HDFS?
Warm Regards,
Tariq
https
Cc: Mohammad Tariqdonta...@gmail.com
Subject: Re: How to balance reduce job
Tariq probably meant distribution of keys from key, value pair emitted by
mapper.
Partitioner distributes these pairs to different reducers based on key. If data
is such that keys are skewed then most of the records may
in the configuration?
rauljin
From: bejoy.hadoop
Date: 2013-04-17 13:09
To: user; liujin666jin
Subject: Re: How to balance reduce job
Hi Rauljin
Few things to check here.
What is the number of reduce slots in each Task Tracker? What is the number of
reduce tasks for your job?
Based
8 datanode in my hadoop cluseter ,when running reduce job,there is only 2
datanode running the job .
I want to use the 8 datanode to run the reduce job,so I can balance the I/O
press.
Any ideas?
Thanks.
rauljin
typos
-Original Message-
From: rauljin liujin666...@sina.com
Date: Wed, 17 Apr 2013 12:53:37
To: user@hadoop.apache.orguser@hadoop.apache.org
Reply-To: user@hadoop.apache.org
Subject: How to balance reduce job
8 datanode in my hadoop cluseter ,when running reduce job,there is only 2
--
*From: * rauljin liujin666...@sina.com
*Date: *Wed, 17 Apr 2013 12:53:37 +0800
*To: *user@hadoop.apache.orguser@hadoop.apache.org
*ReplyTo: * user@hadoop.apache.org
*Subject: *How to balance reduce job
8 datanode in my hadoop cluseter ,when running reduce job,there is only 2
Message-
From: Mohammad Tariq donta...@gmail.com
Date: Wed, 17 Apr 2013 10:46:27
To: user@hadoop.apache.orguser@hadoop.apache.org; Bejoy
Ksbejoy.had...@gmail.com
Subject: Re: How to balance reduce job
Just to add to Bejoy's comments, it also depends on the data distribution.
Is your data
10 matches
Mail list logo