A couple of comments regarding issues raised by this thread. 1) In my opinion Netpipe is not such a great network benchmarking tool for HPC applications. It measures timings based on the completion of the send call on the transmitter not the completion of the receive. Thus, if there is a delay in copying the send buffer across the net, it will report a misleading timing compared with the wall-clock time. This is particularly problematic with multiple pairs of edge exchanges, which can oversubscribe most GigE switches. Here the netpipe timings can be off by orders of magnitude compared with the wall clock. The good thing about writing your own code is that you know what it has done (of course no one else knows, which can be a problem). But it seems many people are unaware of the timing issue in Netpipe.
2) Its worth distinguishing between ethernet and TCP/IP. With MPIGAMMA, the Intel Pro 1000 NIC has a latency of 12 microsecs including the switch and a duplex bandwidth of 220 MBytes/sec. With the Extreme Networks X450a-48t switch we can sustain 220MBytes/sec over 48 ports at once. This is not IB performance but it seems sufficient to scale a number of applications to the 100 cpu level, and perhaps beyond. Tony ------------------------------- Tony Ladd Professor, Chemical Engineering University of Florida PO Box 116005 Gainesville, FL 32611-6005 Tel: 352-392-6509 FAX: 352-392-9513 Email: tl...@che.ufl.edu Web: http://ladd.che.ufl.edu
