Chris - out of interest, could you try it with iperf? Nick
On 20 July 2014 20:05, André Soileau via smartos-discuss < [email protected]> wrote: > Would you report your speeds simply changing the protocol between tcp and > udp? > > > On Sun, Jul 20, 2014 at 8:08 AM, Chris Ferebee via smartos-discuss < > [email protected]> wrote: > >> Allan, >> >> I increased the window size from 1 MB to 2 MB via >> >> # ndd -set /dev/tcp tcp_cwnd_max 2097152 >> >> as suggested for 10GbE on the page mentioned by Nick, < >> http://www.solarisinternals.com/wiki/index.php/Networks>, but while that >> (or one of the other suggestions) improved aggregate throughput when >> running multiple streams, it didn't do much for the 100 MByte/s barrier I'm >> seeing for a single stream over 10GbE. >> >> Thanks, >> Chris >> >> >> >> Am 20.07.2014 um 15:34 schrieb Allan McAleavy via smartos-discuss < >> [email protected]>: >> >> > What's your tcp Window size I have seen in the past when using say >> veritas vvr increasing this helps just a passing thought >> > >> > >> > On 20 Jul 2014 14:19, "Chris Ferebee via smartos-discuss" < >> [email protected]> wrote: >> > > >> > > Nick, >> > > >> > > running 12 streams in parallel, I do see one CPU maxed out (core 0 as >> shown below). If I understand things correctly, it might be worthwhile to >> bump up rx_queue_number and tx_queue_number based on that. >> > > >> > > OTOH, when running just one stream, the one busy core doesn't seem to >> drop below 30% idle. I will still try increasing the number of queues next >> time I can schedule a reboot. >> > > >> > > Thanks, >> > > Chris >> > > >> > > ===== cpu stats while running 12 netcat streams: >> > > >> > > [root@90-e2-ba-00-2a-e2 ~]# mpstat 10 40 >> > > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys >> wt idl >> > > 0 0 0 0 11671 9657 342 92 0 4263 0 0 0 100 >> 0 0 >> > > 1 294 0 4 199 34 3876 9 73 771 0 3 0 5 >> 0 95 >> > > 2 242 0 0 83 10 2384 0 26 576 0 2 0 2 >> 0 98 >> > > 3 820 0 0 292 248 7959 1 18 1972 0 6 0 7 >> 0 93 >> > > 4 267 0 0 24 3 2672 0 7 691 0 115 0 2 >> 0 98 >> > > 5 983 0 0 36 2 8969 1 9 1857 0 114 0 5 >> 0 95 >> > > 6 279 0 0 22 1 3800 13 17 2907 0 89029 3 14 >> 0 82 >> > > 7 548 0 0 31 1 7160 21 30 3124 0 84476 3 17 >> 0 80 >> > > 8 324 0 0 22 2 4290 15 20 3146 0 86657 3 14 >> 0 83 >> > > 9 476 0 0 30 2 5989 18 25 3520 0 102163 4 20 >> 0 75 >> > > 10 261 0 0 24 2 3570 14 18 2579 0 88500 4 14 >> 0 82 >> > > 11 480 0 0 42 4 6199 26 26 2838 0 81498 3 14 >> 0 83 >> > > 12 2 0 0 18113 18090 41 0 8 432 0 0 0 10 >> 0 90 >> > > 13 137 0 0 5 0 1258 0 3 322 0 1 0 1 >> 0 99 >> > > 14 706 0 0 43 28 7634 0 6 1743 0 44 0 5 >> 0 95 >> > > 15 120 0 0 20 2 1441 0 3 328 0 3 0 1 >> 0 99 >> > > 16 933 0 0 36 8 11714 0 9 2721 0 10 0 5 >> 0 95 >> > > 17 620 0 0 10049 10009 6046 0 12 1688 0 4 0 7 >> 0 93 >> > > 18 656 0 0 621 579 7786 29 31 2128 0 61547 3 13 >> 0 85 >> > > 19 433 0 0 10657 10626 5522 14 26 1595 0 47377 2 19 >> 0 80 >> > > 20 706 0 0 133 93 8344 33 29 2536 0 60154 2 12 >> 0 85 >> > > 21 486 0 0 15933 15904 5968 10 28 1611 0 38225 2 25 >> 0 74 >> > > 22 409 0 0 613 576 4730 26 28 3327 0 97758 4 17 >> 0 79 >> > > 23 179 0 0 22 2 2528 15 18 2827 0 96562 4 15 >> 0 81 >> > > >> > > >> > > Am 19.07.2014 um 23:16 schrieb Nick Perry via smartos-discuss < >> [email protected]>: >> > > >> > > > Hi Chris. >> > > > >> > > > How much improvement do you get with jumbo frames? >> > > > >> > > > Can you achieve significantly higher output if you try multiple >> streams in parallel? >> > > > >> > > > During the test are there any CPU cores with very low idle time? >> > > > >> > > > Depending on the answers to the above it might be interesting to >> see if there is any improvement by increasing rx_queue_number and >> tx_queue_number on the ixgbe driver. >> > > > >> > > > Regards, >> > > > >> > > > Nick >> > > > >> > > > >> > > > On 19 July 2014 14:42, Chris Ferebee via smartos-discuss < >> [email protected]> wrote: >> > > > >> > > > I'm trying to debug a network performance issue. >> > > > >> > > > I have two servers running SmartOS (20140613T024634Z and >> 20140501T225642Z), one is a Supermicro dual Xeon E5649 (64 GB RAM) and the >> other is a dual Xeon E5-2620v2 (128 GB RAM). Each has an Intel X520-DA1 >> 10GbE card, and they are both connected to 10GbE ports on a NetGear >> GS752TXS switch. >> > > > >> > > > The switch reports 10GbE links: >> > > > >> > > > 1/xg49 Enable 10G Full 10G Full >> Link Up Enable 1518 20:0C:C8:46:C8:3E 49 49 >> > > > 1/xg50 Enable 10G Full 10G Full >> Link Up Enable 1518 20:0C:C8:46:C8:3E 50 50 >> > > > >> > > > as do both hosts: >> > > > >> > > > [root@90-e2-ba-00-2a-e2 ~]# dladm show-phys >> > > > LINK MEDIA STATE SPEED DUPLEX >> DEVICE >> > > > igb0 Ethernet down 0 >> half igb0 >> > > > igb1 Ethernet down 0 >> half igb1 >> > > > ixgbe0 Ethernet up 10000 >> full ixgbe0 >> > > > >> > > > [root@00-1b-21-bf-e1-b4 ~]# dladm show-phys >> > > > LINK MEDIA STATE SPEED DUPLEX >> DEVICE >> > > > igb0 Ethernet down 0 >> half igb0 >> > > > ixgbe0 Ethernet up 10000 >> full ixgbe0 >> > > > igb1 Ethernet down 0 >> half igb1 >> > > > >> > > > Per dladm show-linkprop, maxbw is not set on either of the net0 >> vnic interfaces. >> > > > >> > > > And yet, as measured via netcat, throughput is just below 1 Gbit/s: >> > > > >> > > > [root@90-e2-ba-00-2a-e2 ~]# time cat /zones/test/10gb | nc -v -v >> -n 192.168.168.5 8888 >> > > > Connection to 192.168.168.5 8888 port [tcp/*] succeeded! >> > > > >> > > > real 1m34.662s >> > > > user 0m11.422s >> > > > sys 1m53.957s >> > > > >> > > > (In this test, 10gb is a test file that is warm in RAM and >> transfers via dd to /dev/null at approx. 2.4 GByte/s.) >> > > > >> > > > What could be causing the slowdown, and how might I go about >> debugging this? >> > > > >> > > > FTR, disk throughput, while not an issue here, appears to be >> perfectly reasonable, approx. 900 MB/s read performance. >> > > > >> > > > Thanks for any pointers! >> > > > >> > > > Chris >> > > > >> > > > >> > > > >> > > > >> > > > ------------------------------------------- >> > > > smartos-discuss >> > > > Archives: https://www.listbox.com/member/archive/184463/=now >> > > > RSS Feed: >> https://www.listbox.com/member/archive/rss/184463/22416839-083bd2e9 >> > > > Modify Your Subscription: https://www.listbox.com/member/?&; >> > > > Powered by Listbox: http://www.listbox.com >> > > > >> > > > smartos-discuss | Archives | Modify Your Subscription >> > > >> > > >> > > >> > > ------------------------------------------- >> > > smartos-discuss >> > > Archives: https://www.listbox.com/member/archive/184463/=now >> > > RSS Feed: >> https://www.listbox.com/member/archive/rss/184463/23685903-e846680a >> > > Modify Your Subscription: https://www.listbox.com/member/?&; >> > > Powered by Listbox: http://www.listbox.com >> > >> > smartos-discuss | Archives | Modify Your Subscription >> >> >> >> ------------------------------------------- >> smartos-discuss >> Archives: https://www.listbox.com/member/archive/184463/=now >> RSS Feed: >> https://www.listbox.com/member/archive/rss/184463/24535378-4ca6b74b >> >> Modify Your Subscription: https://www.listbox.com/member/?&; >> Powered by Listbox: http://www.listbox.com >> > > *smartos-discuss* | Archives > <https://www.listbox.com/member/archive/184463/=now> > <https://www.listbox.com/member/archive/rss/184463/22416839-083bd2e9> | > Modify > <https://www.listbox.com/member/?&;> > Your Subscription <http://www.listbox.com> > ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
