Hi, Dotan Barak wrote: > Marcel Heinz wrote: >> [low multicast throughput of ~250MB/s with own benchmark tool] > > 1) I know that ib_send_bw supports multicast as well, can you please > check that you can reproduce your problem > on this benchmark too?
Well, the last time I've checked this, ib_send_bw didn't support multicast, but this was some months ago. That multicast support seems a bit odd, since it doesn't create/join the multicast groups and there is still a 1:1 TCP connection used to establish the IB connection, so one cannot benchmark "real" multicast scenarios with more than one receiver. However, here are the results (I just used ipoib to let it create some multicast groups for me): | [EMAIL PROTECTED]:~$ ib_send_bw -c UD -g mhtest1 | ------------------------------------------------------------------ | Send BW Multicast Test | Connection type : UD | Max msg size in UD is 2048 changing to 2048 | Inline data is used up to 400 bytes message | local address: LID 0x01, QPN 0x4a0405, PSN 0x8667a7 | remote address: LID 0x03, QPN 0x4a0405, PSN 0x5d41b6 | Mtu : 2048 | ------------------------------------------------------------------ | #bytes #iterations BW peak[MB/sec] BW average[MB/sec] | 2048 1000 301.12 247.05 | ------------------------------------------------------------------ This is the same result as my own benchmark showed in that scenario. > 2) You should expect that multicast messages will be slower than > unicast because the HCA/switch treat them in different way > (message duplication need to be done if needed). Yes, but 250MB/s vs. 1100MB/s (UD unicast throughput) seems to be a bit too much of overhead, don't you think? Especially if I take into account that with my own benchmark, I can get ~950MB/s when I start another receiver on the same host as the sender. Note that both of the receivers, the local and the remote one, are seeing all packets at that rate, so the HCAs and the switch must be able to handle multicast packets with this throughput. The other strange thing is that multicast traffics slows down other traffic way more than the bandwith it consumes. Moreover, it seems like it limits any other connections to the same throughput than that of the multicast traffic, which looks suspicious to me. The same behavior can be reproduced with ib_send_bw, by starting an unicast and multicast run in parallel: | [EMAIL PROTECTED]:~$ ib_send_bw -c UD mhtest1 & ib_send_bw -c UD -g mhtest1\ | -p 18516 | ./ib_send_bw -c UD -g mhtest1 -p 18516 | [1] 4927 | ------------------------------------------------------------------ | Send BW Test | Connection type : UD | Max msg size in UD is 2048 changing to 2048 | ------------------------------------------------------------------ | Send BW Multicast Test | Connection type : UD | Max msg size in UD is 2048 changing to 2048 | Inline data is used up to 400 bytes message | Inline data is used up to 400 bytes message | local address: LID 0x01, QPN 0x530405, PSN 0xe98523 | local address: LID 0x01, QPN 0x530406, PSN 0x3b338e | remote address: LID 0x03, QPN 0x540405, PSN 0x5c53e2 | Mtu : 2048 | remote address: LID 0x03, QPN 0x540406, PSN 0xff883f | Mtu : 2048 | ------------------------------------------------------------------ | #bytes #iterations BW peak[MB/sec] BW average[MB/sec] | ------------------------------------------------------------------ | #bytes #iterations BW peak[MB/sec] BW average[MB/sec] | 2048 1000 692.41 270.26 | 2048 1000 246.00 244.68 | ------------------------------------------------------------------ | ------------------------------------------------------------------ Doing 2 unicast UD runs in parallel, I'm getting ~650MB/s average bandwith for each, which sounds reasonable. Also, when using bidircetional mode, I'm getting ~1900MB/s (amlost doubled) throughput for unicast, but still ~250MBs for multicast. Regards, Marcel _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
