Generally, to get good performance with many NICs, you will need multiple streams. The problem is that the dispatch latencies and round trip times mean that without parallelization you won't get the best possible numbers. Additionally, with some products, there are hardware rings for parallelization, and getting full performance requires engaging multiple rings, which you can't do with a single stream. Try running 4-5 of those netcats in parallel and see what happens.
Also please include your file size when posting times. I was concerned that perhaps you were mixing bytes and bits. (Probably not, but it comes up often enough that I prefer to check.) On Sat, Jul 19, 2014 at 6:42 AM, Chris Ferebee via smartos-discuss < [email protected]> wrote: > > I'm trying to debug a network performance issue. > > I have two servers running SmartOS (20140613T024634Z and > 20140501T225642Z), one is a Supermicro dual Xeon E5649 (64 GB RAM) and the > other is a dual Xeon E5-2620v2 (128 GB RAM). Each has an Intel X520-DA1 > 10GbE card, and they are both connected to 10GbE ports on a NetGear > GS752TXS switch. > > The switch reports 10GbE links: > > 1/xg49 Enable 10G Full 10G Full Link Up > Enable 1518 20:0C:C8:46:C8:3E 49 49 > 1/xg50 Enable 10G Full 10G Full Link Up > Enable 1518 20:0C:C8:46:C8:3E 50 50 > > as do both hosts: > > [root@90-e2-ba-00-2a-e2 ~]# dladm show-phys > LINK MEDIA STATE SPEED DUPLEX DEVICE > igb0 Ethernet down 0 > half igb0 > igb1 Ethernet down 0 > half igb1 > ixgbe0 Ethernet up 10000 full > ixgbe0 > > [root@00-1b-21-bf-e1-b4 ~]# dladm show-phys > LINK MEDIA STATE SPEED DUPLEX DEVICE > igb0 Ethernet down 0 > half igb0 > ixgbe0 Ethernet up 10000 full > ixgbe0 > igb1 Ethernet down 0 > half igb1 > > Per dladm show-linkprop, maxbw is not set on either of the net0 vnic > interfaces. > > And yet, as measured via netcat, throughput is just below 1 Gbit/s: > > [root@90-e2-ba-00-2a-e2 ~]# time cat /zones/test/10gb | nc -v -v -n > 192.168.168.5 8888 > Connection to 192.168.168.5 8888 port [tcp/*] succeeded! > > real 1m34.662s > user 0m11.422s > sys 1m53.957s > > (In this test, 10gb is a test file that is warm in RAM and transfers via > dd to /dev/null at approx. 2.4 GByte/s.) > > What could be causing the slowdown, and how might I go about debugging > this? > > FTR, disk throughput, while not an issue here, appears to be perfectly > reasonable, approx. 900 MB/s read performance. > > Thanks for any pointers! > > Chris > > > > > ------------------------------------------- > smartos-discuss > Archives: https://www.listbox.com/member/archive/184463/=now > RSS Feed: > https://www.listbox.com/member/archive/rss/184463/22103350-51080293 > Modify Your Subscription: > https://www.listbox.com/member/?&; > Powered by Listbox: http://www.listbox.com > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
