[EMAIL PROTECTED] schrieb am 05/05/2008 09:46:52 PM: > On May 5, 2008, at 11:12 AM, Jan Ploski wrote: > > > I think it's not really a good case for metascheduling because it > > doesn't > > make sense to separate the (typically very short) setup stage from the > > actual (long) computation - there are simply no advantages in doing > > that. > > If you use a metascheduler, there is risk that your setup job becomes > > separated in the queue from the computation job by other users' jobs, > > unnecessarily delaying the whole execution. > > Hi Jan, > > Even after reflection, I do not really agree. This is really just > one step along a workflow. Setup jobs can generally be done via > fork job manager submissions, or submissions to fast-turnaround > queues that in principle always complete faster than the scheduling > for the bulk-execution MPI or multiple slow-serial jobs. Workflow or > metascheduling tools can simply be used to ensure the order of > execution of these steps. > > One way or the other, you are always pursuing a workflow of data > production, analysis and/or processing steps with such jobs, so this > is just a special-case variant. > > Of course, everyone is welcome to their opinion (and preferred way of > doing it). With such a simple use case, you could just build jobs > with a lock file in common space that lets the first job to start do > the setup, and others ignore it if the book-keeping lock file made by > the first job is in place. Of course, that would be the very > definition of a race condition, so you would want to be careful -- > and it would also depend on having access to shared-access common > space between jobs, not always guaranteed for job execution sandbox > space. > > That's why I would prefer to think of this as a workflow step.
Hi Alan, I agree that it can be considered a workflow step, and your point about 'fork' jobs is well made. My main gripe is that if you are not using a workflow engine yet and view jobs as logical independent units of computation, then it is a poor excuse to introduce a workflow engine just for this use case - just not easy enough. On the other hand, if you already have a workflow engine (and it's likely you are going to need one anyway), then it fits. Basically, I see two ways of argumentation in favor of workflow engines: 1) you *must* use them due to simple functionality gaps in the basic middleware 2) you *should* use them to gain additional benefits (say, job coordination across clusters, application-specific fault tolerance) You are also right about the difficulties of correct (and especially - portable) interprocess synchronization within Grid clusters. That's why I believe such functionality should be offered by the middleware (and it would seem rather simple to implement in GRAM), not by middleware users, in the long term. One could argue that MPI can already do it - you could put your initialization stuff into your MPI program. However, often users of MPI programs are not their authors and not really familiar with MPI, so again it would seem an overkill for something that simple. Regards, Jan Ploski
