Ian,
Right now I'm fighting with my Finder/AFP/netatalk/getcwd() performance issues,
which are a landmine, so the 10GbE slowdowns are the least of my worries. But
here is what I did find out.
It helps to tune the following TCP stack parameters:
# ndd -set /dev/tcp tcp_recv_hiwat 400000
# ndd -set /dev/tcp tcp_xmit_hiwat 400000
# ndd -set /dev/tcp tcp_max_buf 2097152
# ndd -set /dev/tcp tcp_cwnd_max 16777216
Still, I can max out one CPU core at 100% by running a small number of threads
of netstat or iperf (doesn't really matter which) in parallel. The other cores
stay mostly idle.
After tuning the parameters as above, I was seeing about 3 Gbit/s throughput
over the X520 with several threads.
I think the bottleneck is in the ixgbe driver, because running the same tests
on localhost gives about 200 Gbit/s throughput, so the threads producing and
consuming the data are definitely not at fault.
Considering comments by Nick Perry and others I suspect it would be worth
trying to increase the ixgbe driver's rx_queue_number and tx_queue_number via
/kernel/drv/ixgbe.conf. AIR the max # of queues depends on your hardware
revision and is either 8 or 16 depending on the Intel part number, while the
default is 1. As I understand it, this would allow parallelizing the driver
load across multiple cores.
Thanks to
<http://dtrace.org/blogs/wesolows/2013/12/28/anonymous-tracing-on-smartos/>
it is actually quite easy to tune kernel parameters without rebuilding a new
platform image. (Keith's instructions seemed a bit scary to me, but they are
actually very clear and work exactly as advertised. One thing to note is that
your modifications will show up in /system/boot when the system is running -
they just override the original files that are still present in the platform
image.)
I intended to play with rx_queue_number and tx_queue_number via this mechanism,
but it will be a while before I can do that. Please let me know what you find
if you decide to try it.
Best,
Chris
>> I'm trying to debug a network performance issue.
>>
>> I have two servers running SmartOS (20140613T024634Z and 20140501T225642Z),
>> one is a Supermicro dual Xeon E5649 (64 GB RAM) and the other is a dual Xeon
>> E5-2620v2 (128 GB RAM). Each has an Intel X520-DA1 10GbE card, and they are
>> both connected to 10GbE ports on a NetGear GS752TXS switch.
>
> Did you reach a satisfactory conclusion with this?
>
> I'm experiencing the what looks like the same issue with two SmartOS systems
> using Intel X540 10GE cards. I didn't see the problem when I was testing
> these two boxes with SmartOS on one and Solaris 11 on the other, so looks to
> be a SmartOS/Illumos issue.
>
> I also have some Dell systems which show the same issue, because I hadn't
> seen the problem on my own systems, I'd put this down to the Dell modules.
> Now I think it is more general.
>
> --
> Ian.
-------------------------------------------
smartos-discuss
Archives: https://www.listbox.com/member/archive/184463/=now
RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00
Modify Your Subscription:
https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb
Powered by Listbox: http://www.listbox.com
