Am 23.05.2012 um 09:58 schrieb Beat Rubischon: > Hi William! > > On 23.05.12 09:30, William Hay wrote: >> On 23 May 2012 08:12, Beat Rubischon <[email protected]> wrote: >>> Basically a good idea. One big issue is the filesystem. There is no file >>> staging from the client to the cluster, the Grid Engine needs a shared >>> filesystem. Even it has "Grid" in its name :-) >> While grid engine doesn't provide file staging it does provide hooks >> so you can write your own. > > There are hooks, but it's tricky to get files from a client outside :-) > Usually they are used to stage from the queuemaster to the exec node and > back. > > Some time ago I played with them and finally failed. There is no way to > move the output files completely as the staging process by itself is > still writing its STDOUT and STDERR to those files. And you need the > output to be able to debug the sometimes complex procedure... I finally > managed the whole file staging inside of the job script and used the > post script for a cleanup in case of a qdel.
It's also not fixed what input and output files need to be transferred, i.e. is it just a single file at all? I set up a transfer queue in SGE6, and use the job context where the user has to specify which files or directories should be transferred to the remote cluster. Some applications like TurboMole or ADF need a complete directory for their files, others like Gaussian might need a checkpoint file in addition of the input file. The starter method for the remote queue took care of it, and also of the transfer back (by hostbased ssh authentication for scp or rsync, depending on the type of transfer; i.e. from the headnode of cluster A to the headnode of cluster B). After the job on the remote cluster finished, there is also nothing written any more to stdout/-err. If someone is interested, I can post the stuff. -- Reuti > Beat > > -- > \|/ Beat Rubischon <[email protected]> > ( 0-0 ) http://www.0x1b.ch/~beat/ > oOO--(_)--OOo--------------------------------------------------- > Meine Erlebnisse, Gedanken und Traeume: http://www.0x1b.ch/blog/ > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
