Infiniband uses multicast internally. It's not something you have a choice with. You won't see it on the local interface any more than you'd see individual drives of a raid 5.
I believe it's one of the reasons the connection setup speeds are kept under the requisite 1.2usec limits etc. On Jun 10, 2016 4:16 AM, "Daniel Swarbrick" < daniel.swarbr...@profitbricks.com> wrote: On 10/06/16 02:33, Christian Balzer wrote: > > > This thread brings back memories of this one: > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-April/008792.html > > According to Robert IPoIB still uses IB multicast under the hood even when > from an IP perspective traffic would be unicast. I'd be interested to see some concrete proof of that. We run several IB fabrics here using Mellanox QDR HCAs, and run a mixture of SRP and IPoIB over them. We don't explicitly override the mcast rate, so it's safe to assume that this is the default SDR, 10 Gbps rate. Testing with iperf3, I've seen single flow IPoIB (CM) reach about 20 Gbps, and multiple flows top out at around a combined 25 Gbps. On the other hand, testing with ib_write_bw (RDMA, single "flow"), we usually get just under 30 Gbps. So there is a fair bit of overhead in IPoIB, but I'm skeptical that it uses mcast IB all the time. Nothing in the Linux IPoIB kernel modules stands out as looking like "use multicast for everything." > > The biggest issue pointed out in that mail and the immensely long > and complex thread he mentioned in it is that you can't change the speed > settings on the fly. > which means that if you're already in production it's unlikely that there > will ever be a time to entirely tear down your IB network... > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
