Louwtjie Burger writes: > Hi there. > > Not sure if this is the right place, maby it should be in storage ... > question: How would a rather large number of Opterons running Solaris > compare on a HPC benchmark when used with: > > (a) Fibre channel HBA's and switches, running IP over FC > (b) Gbit with regular TCP switches > (c) 10Gbit (ditto) > > OF course, price is not an option (only to such an extend that any other > interconnect like Infiniband cannot be used ;) >
How about Myrinet? It is certainly cheaper than a high density 10GbE switch and 10GbE nics. I assume IB would also be cheaper. Anyway, the advantages of Myrinet (and IB) over TCP/IP based solutions are low latency, and low overhead and high bandwidth. Myrinet 1/2rtt pingpong latency is roughly 2.5us with both the 2G and 10G lines, and the host overhead is nearly zero even when running at 18Gb/s (9Gb/s send + 9Gb/s recv simultaneously) because most of the protocol handling is done by the NIC, and the data can be DMA'ed directly to/from its source/destination with no copies. See http://www.myri.com/opteron for some sample Pallas benchmark data from the 10G products. Generally, HPC programs will respond to a combination of low latency, high bandwidth, and low overhead. TCP/IP going through the kernel doesn't give you all three, but interconnects like Myrinet (and IB) can and do. That is why the most efficient clusters in the top500 use a non-ethernet interconnect. Specifically, your HPC program could have a very coarse-grained parallelism, and you might do just fine with TCP/IP. It all depends on how your target application behaves. Drew PS: Disclaimer, I work for Myricom. But I do ethernet drivers and don't have much to do with the HPC side of the company... ;) _______________________________________________ networking-discuss mailing list [email protected]
