On Aug 30, 2013, at 1:38 PM, "Hefty, Sean" <sean.he...@intel.com> wrote:
>> Another strange issue: >> >> $ sudo LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c >> 172.17.0.2 >> ------------------------------------------------------------ >> Client connecting to 172.17.0.2, TCP port 5001 >> TCP window size: 128 KByte (default) > > Increasing the window size may improve the results. E.g. on my systems I go > from 17.7 Gbps at 128 KB to 24.3 Gbps for 512 KB. > >> ------------------------------------------------------------ >> [ 3] local 172.17.0.1 port 57926 connected with 172.17.0.2 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-10.0 sec 12.2 GBytes 10.4 Gbits/sec >> >> $ iperf -c 172.17.0.2 >> ------------------------------------------------------------ >> Client connecting to 172.17.0.2, TCP port 5001 >> TCP window size: 648 KByte (default) >> ------------------------------------------------------------ >> [ 3] local 172.17.0.1 port 58113 connected with 172.17.0.2 port 5001 >> [ ID] Interval Transfer Bandwidth >> [ 3] 0.0-10.0 sec 14.5 GBytes 12.5 Gbits/sec >> >> rsocket slower than IPoIB ? > > This is surprising to me - just getting 12.5 Gbps out of ipoib is surprising. > Does iperf use sendfile()? I have a pair of nodes connected by QDR via a switch. Using normal IPoIB, a single Netperf can reach 18.4 Gb/s if I bind to the same core that the IRQ handler is bound to. With four concurrent Netperfs, I can reach 23 Gb/s. This is in datagram mode. Connected mode is slower. I have not tried rsockets on these nodes. Scott > > My results with iperf (version 2.0.5) over ipoib (default configurations) > vary considerably based on the TCP window size. (Note that this is a 40 Gbps > link.) Results summarized: > > TCP window size: 27.9 KByte (default) > [ 3] 0.0-10.0 sec 12.8 GBytes 11.0 Gbits/sec > > TCP window size: 416 KByte (WARNING: requested 500 KByte) > [ 3] 0.0-10.0 sec 8.19 GBytes 7.03 Gbits/sec > > TCP window size: 250 KByte (WARNING: requested 125 KByte) > [ 3] 0.0-10.0 sec 4.99 GBytes 4.29 Gbits/sec > > I'm guessing that there are some settings I can change to increase the ipoib > performance on my systems. Using rspreload, I get: > > LD_PRELOAD=/usr/local/lib/rsocket/librspreload.so iperf -c 192.168.0.103 > TCP window size: 512 KByte (default) > [ 3] 0.0-10.0 sec 28.3 GBytes 24.3 Gbits/sec > > It seems that ipoib bandwidth should be close to rsockets, similar to what > you see. I also don't understand the effect that the TCP window size is > having on the results. The smallest window gives the best bandwidth for > ipoib?! > > - Sean > -- > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
