Steve White wrote:
Jan,
I agree with your assessment that the need to adjust the memory
use per
process is a general one in cluster job submission, and that it is
in
some way implemented by any underlying job management system, and
that
these extensions ought not to be PBS-specific.
I also looked at your "messy solution". (The code looks very
professional,
really.) It won't do for my purposes, because I need to present a
minimal,
easily understood solution.
Let me explain my situation:
None of the compute resources is under my control. I can point
out
problems to admins, that is all.
I have been assigned two jobs.
I and our users are familiar with doing conventional cluster job
submission. One job was to bring them into the grid fold, showing
them the
advantages
of globusrun-ws. If it can be shown to be really a cross-platform
solution, giving them the ability to (almost) effortlessly switch
between grid clusters, the effort will be a success.
My other job is to write a report on practical MPI job submission
over
the grid.
We have come a long way, but still have to deal with a couple of
practical
details. At this point, it looks like both of them will end up as
work-arounds to incomplete implementation of a job submission
interface
in Globus.
If with a future release of Globus, these issues can be dealt
with, grid
job submission will look very attractive to real researchers.
Hi,
Based on my experience with Globus, you might be following a wrong
route
(the route to disenchantment). I view Globus more as a middleware
that
has to be adapted (as in: "wrapped around" or "slightly modified")
according to your users' needs and which plays an important role
behind
the scenes, but it probably should not be exposed directly to users
as a
drop-in replacement for their familiar job submission tools.
There is a reason for that more important than the limitations you
have
discovered so far: Globus doesn't ship with command-line job
management
commands on par with those of TORQUE/Maui, Condor or SGE. If you
let
users submit jobs with globus-job-submit, the next thing they are
going
to ask you is "how can I see what jobs I have submitted", "how
can I
cancel the job or resubmit it elsewhere", "is my job running or
not",
"why is my job not running", "when is my job going to start", etc.
You need something in front of Globus to make your users' life
bearable.
Some projects lean toward application-specific web portals (I think
that's AstroGrid's approach). In our project, we have deployed a
largely
application-agnostic frontend based on Condor-G, but even so there
was
some customization and some user training required. The Condor-G
approach might be relevant for you because it covers the scenario
of
making a transparent transition from a local batch system to a
Grid -
the Condor tools for submitting jobs and status querying are pretty
much
the same regardless of whether your job goes to a machine from a
local
pool (equivalent to an SGE or PBS-managed cluster) or to a pool of
Globus hosts. (In fact, Condor can submit to GT2 [gLite], GT4,
Unicore,
and some more Grid middlewares.)
The disadvantage of Condor is that it is a rather huge software
product
and trying to understand all of it can be daunting. Still, I
suppose you
could get the Grid submission piece of it running in a couple of
hours
if you wish to give it a try (by following our tutorials and asking
questions where necessary).
Regards,
Jan Ploski
