We have a SGE array task that we wish to run with elements 1-70000. Each task generates output and takes roughly 20 seconds to 4 minutes of CPU time. We're doing them on a machine with about 144 8-core nodes, and we've divvied the job up to do about 500 at a time.
So, we have 500 jobs at a time writing to the same ZFS partition. What is the best way to collect the results of the task? Currently we are having each task write to STDOUT and then are combining the results. This nails our ZFS partition to the wall and kills performance for other users of the system. We tried setting up a MySQL server to receive the results, but it couldn't take 1000 simultaneous inbound connections. Jeff _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
