Did you place the fair scheduler on your classpath? Can you see its UI if
you go to http://:50030/scheduler ?
On Mon, Nov 30, 2009 at 4:04 PM, Mike Kendall wrote:
> startup sequence is fine. there is no log file generated, just the xml and
> jar.
>
> the jobtracker gives this on failure:
>
> 20
Hi Ives,
It's actually pretty easy to do this with the Fair Scheduler in trunk. You
can just set the mapred.fairscheduler.locality.delay config property to a
large number, say 1. This parameter controls how long the
scheduler waits before it is allowed to launch non-local tasks from a
If you want the code to happen on only one machine, why not run it in your
driver program that submits the MapReduce job?
You could also create a special input record that tells the mapper who gets
that record that it's the chosen one. However, note that that mapper may be run
multiple times du
Not in any nice way, as far as I know. You could shut down the TaskTrackers one
at a time, update their config files to add slots, and start them up again, but
you'd cause some tasks to fail this way, and you might also have the JobTracker
deciding that map outputs on a given TT can't be fetched
Hi Erik,
With four priority levels like this, you should just be able to use Hadoop's
priorities, because it has five of them (very high, high, normal, low and very
low). You can just use the default scheduler for this (i.e. don't enable either
the fair or the capacity scheduler). Or am I missi
Ideally, to evaluate whether you want to go for 10GbE NICs, you would profile
your target Hadoop workload and see whether it's communication-bound. Hadoop
jobs can definitely be communication-bound if you shuffle a lot of data between
map and reduce, but I've also seen a lot of clusters that are
How long are your tasks, and which version of Hadoop are you using? In older
versions (0.20.*), the fair scheduler doesn't launch multiple tasks per
heartbeat, so it performs poorly when your tasks are small (less than 5-10
seconds). You may be able to improve it a bit by setting
mapred.fairsch
Okay, great!
Unfortunately the task launching is still slow even if you have JVM reuse set
to -1 because of heartbeats (the slave node only updates its state with the
master every ~5 seconds).
Matei
On Aug 18, 2011, at 5:25 PM, Mck wrote:
>
>> How long are your tasks, and which version of Ha
Are you trying to use the capacity scheduler or the fair scheduler? Your
mapred-site.xml says to use the capacity scheduler but then points to a fair
scheduler allocation file. Take a look at
http://hadoop.apache.org/common/docs/r0.20.204.0/fair_scheduler.html for
setting up the fair scheduler
Hi Merto,
> a) is LATE scheduler a standalone scheduler or is integrated in
> fairscheduler? If is standalone where to find it and which hadoop version
> does it support?
LATE is actually part of the common scheduling code that all of the schedulers
in Hadoop use, implemented in the JobInProgres
The one in 0.20.2 doesn't support it. However, the Cloudera
Distribution of Hadoop has backported preemption (and the other fair
scheduler features in 0.21), so you could try that if you want
preemption on a 0.20 cluster.
Matei
On 8/29/2010 10:37 PM, xiujin yang wrote:
> Hadoop Version: 0.20.2
>
Do you mean profiling the data path in MapReduce? I think the general consensus
is that a decent amount of time is spent in deserialization and in data copies
in the HDFS stack, although of course there is work to improve this. For
example, take a look at https://issues.apache.org/jira/browse/HD
Preemption is only available in Hadoop 0.20+ or in distributions of Hadoop that
have applied that patch, such as Cloudera's distribution. If you are running
one of these, check out
http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html for
information on how to enable preemption.
Sorry, I meant 0.21+, not 0.20+ for the Apache releases.
Matei
On May 31, 2011, at 4:05 PM, Matei Zaharia wrote:
> Preemption is only available in Hadoop 0.20+ or in distributions of Hadoop
> that have applied that patch, such as Cloudera's distribution. If you are
> runnin
Usually the number of speculatively executed tasks is equal to the number of
"killed" tasks in the UI (as opposed to "failed"). When Hadoop runs a
speculative task, it ends up killing either the original or the speculative
task, depending on which one finishes first.
I don't think OOM errors wo
15 matches
Mail list logo
