Thanks for the update ;O)
Regards,
Chris MacKenzie
http://www.chrismackenziephotography.co.uk/Expert in all aspects of
photography
telephone: 0131 332 6967 tel:0131 332 6967
email: stu...@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
hi all,
we are tuning yarn (or trying to) on our environment (shared fielsystem,
no hdfs) using terasort and one of the main issue we are seeing is that
an avg map task takes 15sec. some tuning guides and websites suggest
that ideally map tasks run between 40sec to 1 or 2 minutes.
It's my understanding that you don't get map tasks as such but containers.
My experience is with version 2 +
And if that's true containers are based on memory tuning in mapred-site.xml
Otherwise I'd love to learn more.
Sent from my iPhone
On 27 Aug 2014, at 12:14, Stijn De Weirdt
hi all,
someone PM'ed me suggesting i'd take a look in the input split setting,
and indeed, the splitsize is determining the number of tasks
stijn
On 08/27/2014 06:23 PM, Chris MacKenzie wrote:
It's my understanding that you don't get map tasks as such but containers.
My experience is
romedius.we...@student.uibk.ac.at
To: user@hadoop.apache.org
Sent: Wednesday, October 3, 2012 5:00 AM
Subject: Re: How to lower the total number of map tasks
Hi!
According to the article @YDN*
The on-node parallelism is controlled by the
mapred.tasktracker.map.tasks.maximum parameter.
[http
I am running Hadoop 1.0.3 in Pseudo distributed mode.
When I submit a map/reduce job to process a file of size about 16 GB, in
job.xml, I have the following
mapred.map.tasks =242
mapred.min.split.size =0
dfs.block.size = 67108864
I would like to reduce mapred.map.tasks to see if it
Those numbers make sense, considering 1 map task per block. 16 GB file /
64 MB block size = ~242 map tasks.
When you doubled dfs.block.size, how did you accomplish that? Typically,
the block size is selected at file write time, with a default value from
system configuration used if not
Hi
You need to alter the value of mapred.max.split size to a value larger than
your block size to have less number of map tasks than the default.
On Tue, Oct 2, 2012 at 10:04 PM, Shing Hing Man mat...@yahoo.com wrote:
I am running Hadoop 1.0.3 in Pseudo distributed mode.
When I submit a
Sorry for the typo, the property name is mapred.max.split.size
Also just for changing the number of map tasks you don't need to modify the
hdfs block size.
On Tue, Oct 2, 2012 at 10:31 PM, Bejoy Ks bejoy.had...@gmail.com wrote:
Hi
You need to alter the value of mapred.max.split size to a
: Bejoy Ks bejoy.had...@gmail.com
To: user@hadoop.apache.org; Shing Hing Man mat...@yahoo.com
Sent: Tuesday, October 2, 2012 6:03 PM
Subject: Re: How to lower the total number of map tasks
Sorry for the typo, the property name is mapred.max.split.size
Also just for changing the number of map
@hadoop.apache.orguser@hadoop.apache.org
Reply-To: user@hadoop.apache.org
Subject: Re: How to lower the total number of map tasks
I have tried
Configuration.setInt(mapred.max.split.size,134217728);
and setting mapred.max.split.size in mapred-site.xml. ( dfs.block.size is left
unchanged at 67108864
From: Bejoy KS bejoy.had...@gmail.com
To: user@hadoop.apache.org; Shing Hing Man mat...@yahoo.com
Sent: Tuesday, October 2, 2012 6:37 PM
Subject: Re: How to lower the total number of map tasks
Shing
This doesn't change the block size of existing files in hdfs, only new
I only have one big input file.
Shing
From: Bejoy KS bejoy.had...@gmail.com
To: user@hadoop.apache.org; Shing Hing Man mat...@yahoo.com
Sent: Tuesday, October 2, 2012 6:46 PM
Subject: Re: How to lower the total number of map tasks
Hi Shing
Is your input
13 matches
Mail list logo
