Hey Brian, On Jun 6, 2012, at 5:40 PM, Brian Foster wrote:
> Okay... totally got behind on this thread... the purpose of this is that > regardless of whether we are talking about CAS-PGE running in the resource > manager or just some other generic resource manager job, typically any job > requires some set of files to exist before it runs, a temp directory to work > in, and temp directory cleanup... currently CAS-PGE (or any other job) has to > implement this logic (this really should be controlled at a higher level -- > this will also avoid directory collisions across jobs as well)... now if > CAS-PGE needs a file from the filemgr that is something CAS-PGE should be > responsible for. So in relation to the emails below, pge-config.xml is the > file that needs to exist on or be visible to the machine before CAS-PGE is > run (CAS-PGE really shouldn't have to stage that file -- it makes for a hacky > implementation in CAS-PGE anyway). Sure, gotcha. > > I invision such a change to the resource manager would include being able to > specify a XML file with a list of need files for the job to run, and at > runtime the resource manager would stage those files to the temp working > directory it created for the job and then clean them up after job execution. > Something like: > > <reqInput class="file.staging.class"> > <file src="/path/to/pge-config.xml" > dest="path/relative/to/temp/working/dir/pge-config.xml"/> > </reqInput> What about making a specialized extension or orthogonal implementation of the Job interface in Resource manager. In other words, a "WorkflowTaskJob" and "WorkflowTaskJobInput" 2.0-type of thing? > > you could imagine that later you could even extend it to support zip packages > which it could stage and unzip: > <reqInput class="file.staging.class"> > <file src="/path/to/package.zip" > dest="path/relative/to/temp/working/dir/package" > postCopyHandler="unzip.logic.class"/> > </reqInput> > +1. > This would be ideal for cloud computing since you could then package up your > JDK, binaries, etc and the resource manager would make sure they were > installed on the machine before executing its job. Yap, agreed. Let me know if you think WorkflowTaskJob and WorkflowTaskJobInput seem like a good place to put stuff like this. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
