For the record, I figured this out.
The default task scheduler just assigns tasks to trackers on a first come
first served basis as trackers' heartbeats are received; although this might
tend to favour data-local and then rack-local trackers at a large enough
scale.
I switched over to using FairS
I found from Googling around that I should probably be seeing messages like
"Choosing data-local task" and "Choosing rack-local task" - from
JobInProgress::addRunningTaskToTIP(). (e.g. here:
http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-dev/201012.mbox/%3C2120373776.44711293532894724.Ja
Harsh, thanks for the clarification.
But my mappers always seem to run elsewhere. Here's an example with 2
splits, both on rack1node1, but the 2 mappers get started on other nodes.
Could the "choosing a non-local task" message be significant?
I have actually read through the JobTracker source, bu
Ian,
The reducer needn't run on the same TaskTracker as it does not work on
local data. The mapper 21, however, should've run on
node:/rack1/rack1node1.local.
On Tue, Jun 7, 2011 at 5:09 AM, Ian Halperin wrote:
> Hi,
>
> I might be misunderstanding how scheduling is supposed to work, or I mi
Hi,
I might be misunderstanding how scheduling is supposed to work, or I might
have something misconfigured, but my Map/Reduce jobs don't seem to run where
my data is located.
I get a bunch of these messages:
INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201106062049_0001_
m_21 has sp
Hi,
On Sun, Nov 7, 2010 at 12:40 AM, nitin reddy wrote:
> where in the source code i mean the class of map reduce the jobs are
> mapped from the job scheduler to the task scheduler so that i could
> plugin my trust concept to the tasks scheduler and increase the
The task-scheduler in play is inv
I am working on my cloud computing project, introducing the concept of
trust for a node using some data mining techniques. I want to know
where in the source code i mean the class of map reduce the jobs are
mapped from the job scheduler to the task scheduler so that i could
plugin my trust concept
