On Thu, 2009-07-16 at 10:51 -0700, Jeff Haferman wrote:
> We have a SGE array task that we wish to run with elements 1-7.
> Each task generates output and takes roughly 20 seconds to 4 minutes
> of CPU time. We're doing them on a machine with about 144 8-core nodes,
> and we've divvied the
Have each node record results locally, and then merge pair-wise until
a single node is left with the final results? If you can do merges
that way while reducing the size of the result set, then that's
probably going to be the most scalable way to generate overall
results.
On Thu, Jul 16, 2009 at
We have a SGE array task that we wish to run with elements 1-7.
Each task generates output and takes roughly 20 seconds to 4 minutes
of CPU time. We're doing them on a machine with about 144 8-core nodes,
and we've divvied the job up to do about 500 at a time.
So, we have 500 jobs at a t
