On Sep 5, 2012, at 4:12 PM, Ezra Kissel wrote: > On 9/5/2012 3:48 PM, Atchley, Scott wrote: >> On Sep 5, 2012, at 3:06 PM, Christoph Lameter wrote: >> >>> On Wed, 5 Sep 2012, Atchley, Scott wrote: >>> >>>>> AFAICT the network stack is useful up to 1Gbps and >>>>> after that more and more band-aid comes into play. >>>> >>>> Hmm, many 10G Ethernet NICs can reach line rate. I have not yet tested any >>>> 40G Ethernet NICs, but I hope that they will get close to line rate. If >>>> not, what is the point? ;-) >>> >>> Oh yes they can under restricted circumstances. Large packets, multiple >>> cores etc. With the band-aids…. >> >> With Myricom 10G NICs, for example, you just need one core and it can do >> line rate with 1500 byte MTU. Do you count the stateless offloads as >> band-aids? Or something else? >> >> I have not tested any 40G NICs yet, but I imagine that one core will not be >> enough. >> > Since you are using netperf, you might also considering experimenting > with the TCP_SENDFILE test. Using sendfile/splice calls can have a > significant impact for sockets-based apps. > > Using 40G NICs (Mellanox ConnectX-3 EN), I've seen our applications hit > 22Gb/s single core/stream while fully CPU bound. With sendfile/splice, > there is no issue saturating a 40G link with about 40-50% core > utilization. That being said, binding to the right core/node, message > size and memory alignment, interrupt handling, and proper host/NIC > tuning all have an impact on the performance. The state of > high-performance networking is certainly not plug-and-play.
Thanks for the tip. The app we want to test does not use sendfile() or splice(). I do bind to the "best" core (determined by testing all combinations on client and server). I have heard others within DOE reach ~16 Gb/s on a 40G Mellanox NIC. I'm glad to hear that you got to 22 Gb/s for a single stream. That is more reassuring. Scott-- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
