Hello All,
I have a few questions concerning the TaskTracker's JVM re-use that I couldn't
unearth some details about:
Is the configured amount of tasks for reuse a suggestion or will it actually
use it? For example, if I've configured it to use a JVM for 4 tasks, will a
TaskTracker that has 8
t restricted to block size if you tune the
right parameters/classes.
Matt
From: Adam Shook [mailto:ash...@clearedgeit.com]
Sent: Thursday, September 22, 2011 3:33 PM
To: mapreduce-user@hadoop.apache.org
Subject: RE: FairScheduler Local Task Restriction
Each box has 3 (seems weird) quad core H
11 at 1:21 PM, Adam Shook wrote:
> Okay, I put a Thread.sleep to test my theory and it will run all 128 at a
> time - they are just completing too quickly. I guess there is no other way
> to get around it, unless someone knows how to make the scheduler schedule
> faster...
>
>
&g
rg
Subject: RE: FairScheduler Local Task Restriction
Just to confirm your configuration, how many logical cores do these boxes
actually have (I am assuming dual quad core HT'ed)? Do you not have any reduce
slots allocated?
Matt
From: Adam Shook [mailto:ash...@clearedgeit.com]
Sent: Thursda
Okay, I put a Thread.sleep to test my theory and it will run all 128 at a time
- they are just completing too quickly. I guess there is no other way to get
around it, unless someone knows how to make the scheduler schedule faster...
-- Adam
From: Adam Shook [mailto:ash...@clearedgeit.com
er it is the
same 16 tasktrackers slots that are getting the map tasks? Long shot but it
could be that it is actually distributing across your cluster and there is some
other issue that is springing up. Also, how long does each of your map tasks
take?
Matt
From: Adam Shook [mailt
Hello All,
I have recently switched my small Hadoop dev cluster (v0.20.1) to use the
FairScheduler. I have a max of 128 map tasks available and recently noticed
that my jobs seem to use a maximum of 16 at any given time (the job I am
looking at in particular runs for about 15 minutes) - they a
Hi All,
I am writing a sequence file to HDFS from an application as a pre-process to a
MapReduce job. (It isn't being written from a MR job, just open, write, close)
The file is around 32 MBs in size. When the MapReduce job starts up, it starts
with 256 map tasks. I am writing SequenceFiles
