Add support for scheduling jobs based on memory requirements to the
Fairscheduler
---------------------------------------------------------------------------------
Key: HADOOP-5567
URL: https://issues.apache.org/jira/browse/HADOOP-5567
Project: Hadoop Core
Issue Type: New Feature
Components: contrib/fair-share
Reporter: Hemanth Yamijala
In HADOOP-3759, we added the ability for users to specify jobs requesting for a
certain amount of virtual memory. For e.g. users can say that their jobs
require 2GB of memory to run. In HADOOP-4035, functionality was added to the
capacity scheduler to schedule jobs based on this specified amount. This JIRA
is to add similar support to the Fairshare scheduler.
The basic use case is that there are jobs that require a certain known amount
of virtual memory, usually more than the JVM's heap size. This happens
specifically for streaming jobs that can launch several processes from the
child. Without being aware of these requirements, if tasks are scheduled on
nodes just based on available slots, they have a potential of affecting the
other processes running on the node, or if memory protection features are
enabled (HADOOP-3581), they could result in the task being killed by the
tasktracker.
The scheduler must take into account the requested amount of memory by the job,
the amount of memory that can be committed to by a tracker, and schedule based
on these inputs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
