[ http://issues.apache.org/jira/browse/HADOOP-322?page=comments#action_12428200 ] Doug Cutting commented on HADOOP-322: -------------------------------------
This looks good. One minor nit is that the sleep in the unit test should probably be smaller. If I change the sleep to 1000ms, then the entire test only takes 37 seconds, 20 seconds longer than the 60 second sleep you have. I run unit tests a lot and am sensitive to them running slowly. Also, this should be in a mapred subpackage, since it is mapred-specific stuff. I'd suggest org.apache.hadoop.mapred.jobcontrol. Thanks for your patience & persistence! > Need a job control utility to submit and monitor a group of jobs which have > DAG dependency > ------------------------------------------------------------------------------------------ > > Key: HADOOP-322 > URL: http://issues.apache.org/jira/browse/HADOOP-322 > Project: Hadoop > Issue Type: New Feature > Reporter: Runping Qi > Assigned To: Runping Qi > Attachments: job_control_patch.txt > > > In my applications, some jobs depend on the outputs of other jobs. Therefore, > job dependency forms a DAG. A job is ready to run if and only if it does not > have any dependency or all the jobs it depends are finished successfully. To > help schedule and monitor a group of jobs like that, I am thinking of > implementing a utility that: > - accept jobs with dependency specification > - monitor job status > - submit jobs when they are ready > With such a utility, the application can construct its jobs, specify their > dependency and then hand the jobs to the utility class. The utility takes > care of the details of job submission. > I'll post my design skech for comments/suggestion. > Eventually, I'll submit a patch for the utility. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
