FairScheduler locality delay may put heavy pressure on Jobtracker
-----------------------------------------------------------------
Key: MAPREDUCE-1764
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1764
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Scott Chen
Assignee: Scott Chen
FairScheduler locality delay feature holds the scheduling of jobs until it gets
good locality.
This greatly improves the locality of the tasks. Reduce the cost of traffic.
We have observed the following problem on FairScheduler locality delay:
We have some machines have older data and some newly added machines do not have
important data.
When these machines send heartbeat, JT scans tasks to find jobs has the right
locality.
Often time, these machines will scan all of the tasks of all the jobs and do
not get any tasks.
Scanning all the tasks on the JT is very costly. This makes JT very slow.
And these machines often time do not get scheduled. This hurts the cluster
utilization.
Any ideas?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
