Hi Francesco, I confirm your benchmark using two systems with the same 100 Gbps Mellanox NICs but with an intervening Juniper QFX5200 switch (100 Gbps ports).
To reach ~25 Gbps with the largest message sizes required "jumbo frame" MTU. The default mtu=1500 allows only ~20 Gbps. I also tried two more doubling of zmsg size in the benchmark and these produce no significant increase in throughput. OTOH, pinning the receiver (local_thr) to a CPU gets it up to 33 Gbps. I note that iperf3 can achieve almost 40 Gbps (20 Gbps w MTU=1500). Multiple simultaneous iperf3 tests can, in aggregate, use 90-100 Gbps. In both the ZMQ and singular iperf3 tests, it seems that CPU is the bottleneck. For ZeroMQ the receiver's I/O thread is pegged at 100%. With iperf3 it's that of the client/sender. The other ends in both cases are at about 50%. The zguide suggests to use one I/O thread per GByte/s (faq says "Gbps") so I tried the naive thing and hacked the ZMQ remote_thr.cpp and local_thr.cpp so each use ten I/O threads. While I see all ten threads in "top -H", still only one thread uses any CPU and it remains pegged at 100% on the receiver (local_thr) and about 50% on the sender (remote_thr). I think now that I misinterpreted this advice and it's really relevant to the case of handling a very large number of connections. Any suggestions on how to let ZeroMQ get higher throughput at 100 Gbps? If so, I'll give them a try. Cheers, -Brett. Francesco <francesco.monto...@gmail.com> writes: > Hi all, > > I placed here: > http://zeromq.org/results:100gbe-tests-v432 > the results I collected using 2 Mellanox ConnectX-5 linked by 100Gbps > fiber cable. > > The results are not too much different from those at 10gpbs > (http://zeromq.org/results:10gbe-tests-v432 )... the difference in TCP > throughput is that > - even using 100kB-long messages we still cannot saturate the link > - latency is very much improved for messages > 10kB long > > Hopefully we will be able to improve performances in the future to > improve these benchmarks... > > Francesco > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
signature.asc
Description: PGP signature
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
