it would be worth placing an upper bound on the # of concurrent
operations to see if this has an impact on the 128 client case; could be
that the servers just don't do that well when there are that many
concurrent operations. i note that the 32 server case is the only
alt-aio read case that *didn't* have poor performance at 128 clients, so
this seems like a reasonable approach.
rob
Robert Latham wrote:
Another round of those mpi-io-test runs on jazz have finally made it
through the queue:
http://www.mcs.anl.gov/~robl/pvfs2/jazz/20061012/
http://www.mcs.anl.gov/~robl/pvfs2/jazz/20061012-altio
Those directories have plots for write (including time to sync after
write), plots for read (reading from a hot VFS buffer cache) and plots
of the min/max/avg for each case (to see if wierd dips and spikes in
the graph are one-offs or consistent).
Some items of interest:
- Jazz is old. It's thread library is the old linuxthreads stuff.
- The reads are coming out of VFS cache, so I hypothesized there would
be not much difference in trove methods for reads. That hypothesis
was incorrect. I would have expected the different Trove methods to
matter more for writes, but that too was incorrect, probalby because
MPI_File_sync calls PVFS_sys_flush from one processor. The 128
server case is worth noting: see below
- Even though the thread library isn't ideal, AltAio gives better peak
performance and more consistent results
- Strangely, AltAio takes a bad turn when 128 clients read from 64 or 128
servers
- Even more strangely, AltAio for 128 servers does very poorly. This
could be an artifact of the way we implement MPI_File_sync: as the
number of clients increases, there is more data to flush to disk. I
don't know why AltAio would handle this worse than default, though.
Certainly, there is the same amount of data when 128 clients write
to 64 or 32 servers. why is 128 so bad?
- It's hard to see from the scale, but AltAio for writes reaches the
plateau faster than default.
- AltAio bandwitdhs are much more consistent than the default: the
candlesticks for the most part are much smaller in the altaoi case.
So, based on these numbers, I'd be tempted to make AltAio the default.
I just wish I could explain why performance drops off for this
benchmark with higher numbers of servers and clients.
==rob
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
