Need a job control utility to submit and monitor a group of jobs which have DAG
dependency
------------------------------------------------------------------------------------------
Key: HADOOP-322
URL: http://issues.apache.org/jira/browse/HADOOP-322
Project: Hadoop
Type: New Feature
Reporter: Runping Qi
Assigned to: Runping Qi
In my applications, some jobs depend on the outputs of other jobs. Therefore,
job dependency forms a DAG. A job is ready to run if and only if it does not
have any dependency or all the jobs it depends are finished successfully. To
help schedule and monitor a group of jobs like that, I am thinking of
implementing a utility that:
- accept jobs with dependency specification
- monitor job status
- submit jobs when they are ready
With such a utility, the application can construct its jobs, specify their
dependency and then hand the jobs to the utility class. The utility takes care
of the details of job submission.
I'll post my design skech for comments/suggestion.
Eventually, I'll submit a patch for the utility.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
