Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On 2012-08-05, Kapeatanakis Giannis wrote: > On 05/08/12 00:13, Mike Belopuhov wrote: >> to elaborate on this point a bit: please make sure you understand what >> you're testing! tcpbench and iperf both test how fast your tcp or udp >> server running in userland can receive and transmit information >> through the socket interface. this has nothing to do with router >> workloads for example. >>> Yes, we would like to be faster but to get more speed large changes are >>> needed since currently only one CPU is doing all the work in the kernel. >>> >>> -- >>> :wq Claudio > > Well if the OpenBSD is in the middle of two other machines like B in A-B-C > and you do iperf/tcpbench between A and C then it is a valid test for > router workloads, isn't it? Well it's a valid test if you're interested in seeing how it performs when routing a single or limited number of flows but it's not valid for standard internet traffic forwarding. "normal" internet traffic has lots of separate flows, lots of destinations and a mixture of packet sizes. And "normal" is, of course, different depending on the network. Neither iperf/tcpbench are good for emulating this. People pay tens of thousands of (insert favourite currency unit here) for test equipment for this.. Can't remember if it came up in this thread yet or not, but make sure that pool_debug is off if you're doing performance testing.
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On Sun, Aug 05, 2012 at 01:50:26PM +0300, Kapeatanakis Giannis wrote: > On 05/08/12 00:13, Mike Belopuhov wrote: > >to elaborate on this point a bit: please make sure you understand > >what you're testing! tcpbench and iperf both test how fast your > >tcp or udp server running in userland can receive and transmit > >information through the socket interface. this has nothing to do > >with router workloads for example. > >>Yes, we would like to be faster but to get more speed large changes are > >>needed since currently only one CPU is doing all the work in the kernel. > >> > >>-- > >>:wq Claudio > > Well if the OpenBSD is in the middle of two other machines like B in A-B-C > and you do iperf/tcpbench between A and C then it is a valid test > for router workloads, isn't it? > Depends. tcpbench and iperf do not emulate real traffic. In most cases they will run a very limited number of sessions/flows. In the end you need to know if your profiling run does measure the right things. At least the tcpbench workload (tcp session in one direction) produce a not so common network pattern. iperf has a similar issue but comes with more buttons. In short tcpbench and iperf can give you an indication how a router behaves in your A-B-C test. -- :wq Claudio
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On 05/08/12 00:13, Mike Belopuhov wrote: to elaborate on this point a bit: please make sure you understand what you're testing! tcpbench and iperf both test how fast your tcp or udp server running in userland can receive and transmit information through the socket interface. this has nothing to do with router workloads for example. Yes, we would like to be faster but to get more speed large changes are needed since currently only one CPU is doing all the work in the kernel. -- :wq Claudio Well if the OpenBSD is in the middle of two other machines like B in A-B-C and you do iperf/tcpbench between A and C then it is a valid test for router workloads, isn't it? Giannis
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On Sat, Aug 04, 2012 at 17:43 +0200, Claudio Jeker wrote: > On Tue, Jul 24, 2012 at 02:22:57PM +0200, Henrik Lund Kramshøj wrote: > > On 24/07/2012, at 14.16, Otto Moerbeek wrote: > > > > > On Tue, Jul 24, 2012 at 01:52:03PM +0200, Peter Hessler wrote: > > > > > >> On 2012 Jul 24 (Tue) at 12:40:00 +0200 (+0200), Henrik Lund Kramsh?j > > wrote: > > >> :Current tuning, apart from kernel defaults: > > >> :and also modified /etc/sysctl somewhat - most changes below are from > > >> :https://calomel.org/network_performance.html: > > >> > > >> Revert all changes recommended by this page, they hurt performance and > > >> do not help. > > >> > > >> calomel.org is only bad juju. > > >> > > >> > > >> -- > > >> In case of atomic attack, the federal ruling against prayer in schools > > >> will be temporarily canceled. > > > > > > Also, the OP didn't say if pf is active or not. By default pf is > > > enabled these days, switch off by setting pf=NO in rc.conf.local > > > > sorry, but tried both, some difference, some improvement, as shown below > > > > hlk@xpc02:hlk$ sudo pfctl -d > > pf disabled > > hlk@xpc02:hlk$ tcpbench -u 10.0.10.30 > > Elapsed:1000 Mbps:1943.217 Peak Mbps:1943.217 Tx PPS: > > 165015 > > Elapsed:2000 Mbps:1944.761 Peak Mbps:1944.761 Tx PPS: > > 165146 > > Elapsed:3000 Mbps:1942.852 Peak Mbps:1944.761 Tx PPS: > > 164984 > > Elapsed:4000 Mbps:1942.710 Peak Mbps:1944.761 Tx PPS: > > 164972 > > > > hlk@xpc02:hlk$ tcpbench -n 4 10.0.10.30 > > elapsed_ms bytes mbps bwidth > > 1000 114326840 914.615 28.58% > > 1000 86697552 693.580 21.67% > > 1000 7 889.431 27.79% > > 1000 87886360 703.091 21.97% > > Conn: 4 Mbps: 3200.717 Peak Mbps: 3200.717 Avg Mbps: 800.179 > > 2001 79059352 631.843 19.21% > > 2001 127067792 1016.542 30.87% > > 2001 135531352 1084.251 32.93% > > 2001 69944248 559.554 16.99% > > Conn: 4 Mbps: 3292.190 Peak Mbps: 3292.190 Avg Mbps: 823.048 > > 3003 101616720 812.122 25.16% > > 3003 98757024 789.267 24.45% > > 3003 103894144 830.323 25.72% > > 3003 99604280 796.038 24.66% > > Conn: 4 Mbps: 3227.750 Peak Mbps: 3292.190 Avg Mbps: 806.937 > > > > But still way below expectations :-( > > > > Then you should adjust your expectations. OpenBSD never said to do 10G > wirespeed or anything in that regard. Btw. depending on how you plan to > use the system it may not make sense to run tcpbench on the system under > test since that will skew the test results. > to elaborate on this point a bit: please make sure you understand what you're testing! tcpbench and iperf both test how fast your tcp or udp server running in userland can receive and transmit information through the socket interface. this has nothing to do with router workloads for example. > Yes, we would like to be faster but to get more speed large changes are > needed since currently only one CPU is doing all the work in the kernel. > > -- > :wq Claudio
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On Tue, Jul 24, 2012 at 02:22:57PM +0200, Henrik Lund Kramshøj wrote: > On 24/07/2012, at 14.16, Otto Moerbeek wrote: > > > On Tue, Jul 24, 2012 at 01:52:03PM +0200, Peter Hessler wrote: > > > >> On 2012 Jul 24 (Tue) at 12:40:00 +0200 (+0200), Henrik Lund Kramsh?j > wrote: > >> :Current tuning, apart from kernel defaults: > >> :and also modified /etc/sysctl somewhat - most changes below are from > >> :https://calomel.org/network_performance.html: > >> > >> Revert all changes recommended by this page, they hurt performance and > >> do not help. > >> > >> calomel.org is only bad juju. > >> > >> > >> -- > >> In case of atomic attack, the federal ruling against prayer in schools > >> will be temporarily canceled. > > > > Also, the OP didn't say if pf is active or not. By default pf is > > enabled these days, switch off by setting pf=NO in rc.conf.local > > sorry, but tried both, some difference, some improvement, as shown below > > hlk@xpc02:hlk$ sudo pfctl -d > pf disabled > hlk@xpc02:hlk$ tcpbench -u 10.0.10.30 > Elapsed:1000 Mbps:1943.217 Peak Mbps:1943.217 Tx PPS: 165015 > Elapsed:2000 Mbps:1944.761 Peak Mbps:1944.761 Tx PPS: 165146 > Elapsed:3000 Mbps:1942.852 Peak Mbps:1944.761 Tx PPS: 164984 > Elapsed:4000 Mbps:1942.710 Peak Mbps:1944.761 Tx PPS: 164972 > > hlk@xpc02:hlk$ tcpbench -n 4 10.0.10.30 > elapsed_ms bytes mbps bwidth > 1000 114326840 914.615 28.58% > 1000 86697552 693.580 21.67% > 1000 7 889.431 27.79% > 1000 87886360 703.091 21.97% > Conn: 4 Mbps: 3200.717 Peak Mbps: 3200.717 Avg Mbps: 800.179 > 2001 79059352 631.843 19.21% > 2001 127067792 1016.542 30.87% > 2001 135531352 1084.251 32.93% > 2001 69944248 559.554 16.99% > Conn: 4 Mbps: 3292.190 Peak Mbps: 3292.190 Avg Mbps: 823.048 > 3003 101616720 812.122 25.16% > 3003 98757024 789.267 24.45% > 3003 103894144 830.323 25.72% > 3003 99604280 796.038 24.66% > Conn: 4 Mbps: 3227.750 Peak Mbps: 3292.190 Avg Mbps: 806.937 > > But still way below expectations :-( > Then you should adjust your expectations. OpenBSD never said to do 10G wirespeed or anything in that regard. Btw. depending on how you plan to use the system it may not make sense to run tcpbench on the system under test since that will skew the test results. Yes, we would like to be faster but to get more speed large changes are needed since currently only one CPU is doing all the work in the kernel. -- :wq Claudio
10Gbit Intel 10GbE SFP+ (82599) performance
Hi I have bought two lab systems for testing 10Gbit and have some issues with 10Gbit performance. Note: I have the devices triple-booting FreeBSD, Linux and OpenBSD and can get about 9.4Gbits/sec with Linux iperf using iperf -t 60 I have read and updated kernel to latest snapshot OpenBSD 5.2 (GENERIC.MP) #362: Sun Jul 22 14:10:40 MDT 2012 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17169907712 (16374MB) avail mem = 16690479104 (15917MB) also tried GENERIC-nonMP - gave worse performance Current performance, a little better than 5.1 root@xpc02:root# tcpbench -u 10.0.10.30 Elapsed:1000 Mbps:1841.437 Peak Mbps:1841.437 Tx PPS: 156372 Elapsed:2000 Mbps:1841.901 Peak Mbps:1841.901 Tx PPS: 156411 Elapsed:3000 Mbps:1839.553 Peak Mbps:1841.901 Tx PPS: 156212 root@xpc03:root# tcpbench -n 4 -s elapsed_ms bytes mbps bwidth 999 92113072 737.642 27.54% 999 98731880 790.646 29.52% 999 72763448 582.690 21.75% 999 70892632 567.709 21.19% Conn: 4 Mbps: 2678.687 Peak Mbps: 2678.687 Avg Mbps: 669.672 2000 65977576 527.821 18.87% 1999 107496584 860.834 30.75% 1999 67310912 539.026 19.25% 1999 108839944 871.591 31.13% Conn: 4 Mbps: 2799.272 Peak Mbps: 2799.272 Avg Mbps: 699.818 OTOH it seems to be very stable, even running over longer periods :-) Better stable than fast :-) Basic specs Shuttle SX58H7 Pro Barebone, using SX58 chipset - has x8 PCI-E slot Dual port 10Gbit Intel 10GbE SFP+ (82599) cpu0: Intel(R) Core(TM) i7 CPU 960 @ 3.20GHz, 3200.46 MHz spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-10600 spdmem1 at iic0 addr 0x51: 4GB DDR3 SDRAM PC3-10600 spdmem2 at iic0 addr 0x52: 4GB DDR3 SDRAM PC3-10600 spdmem3 at iic0 addr 0x54: 4GB DDR3 SDRAM PC3-10600, BIOS overclocking memory a bit up to 1333MHz Current tuning, apart from kernel defaults: and also modified /etc/sysctl somewhat - most changes below are from https://calomel.org/network_performance.html: hlk@xpc02:hlk$ grep -v "^#" /etc/sysctl.conf net.inet6.icmp6.rediraccept=1 # 1=Accept IPv6 ICMP redirects (for hosts) kern.pool_debug=0 # 0=Disable pool corruption checks (faster) machdep.allowaperture=2 # See xf86(4) ddb.panic=0# do not enter ddb console on kernel panic, reboot if possible kern.bufcachepercent=90# Allow the kernel to use up to 90% of the RAM for cache (default 10%) machdep.allowaperture=2# Access the X Window System (if you use X on the system) net.inet.ip.forwarding=1 # Permit forwarding (routing) of packets through the firewall net.inet.ip.ifq.maxlen=512 # Maximum allowed input queue length (256*number of physical interfaces) net.inet.ip.mtudisc=0 # TCP MTU (Maximum Transmission Unit) discovery off since our mss is small enough net.inet.tcp.rfc3390=2 # RFC3390 increasing TCP's Initial Congestion Window to 14600 for SPDY net.inet.tcp.mssdflt=1440 # maximum segment size (1440 from scrub pf.conf match statement) net.inet.udp.recvspace=131072 # Increase UDP "receive" buffer size. Good for 200Mbit without packet drop. net.inet.udp.sendspace=131072 # Increase UDP "send" buffer size. Good for 200Mbit without packet drop. I have a more detailed pastebin at: http://pastebin.com/iNpk3DgV The devices are NOT intended for production and I will be happy to allow access to these systems, only requirement is that you have IPv6 (I am moving these systems around a bit, and NAT sucks) - email me your SSH public key and I will create an account. I have this week until saturday dedicated for working on this and other related projects, so contact me if you want to join. Skype: kramshoej, EFnet: hlkv6, freenode: hlkv6 Current cabling is using direct attach copper cable, I only have direct attach cables, but multiple switches, did not make any significant difference. I got a Force10 switch to report the following in testing, so the hardware looks fine: force10-2#sh interfaces Tengigabitethernet 0/0 | grep Mbits Input 35.00 Mbits/sec, 62616 packets/sec, 0.45% of line-rate Output 9869.00 Mbits/sec, 812987 packets/sec, 99.90% of line-rate The main steps for getting FreeBSD tuned were: kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 sysctl hw.intr_storm_threshold=9000 - note I see LOTS and lots of interrupts, so maybe some /boot/loader.conf ixgbe_load="YES" hw.ixgbe.txd=4096 hw.ixgbe.rxd=4096 - this made a huge difference Killing xhci from kernel, was using lots of interrupts, buggy driver? Best regards Henrik -- Henrik Lund Kramshøj, Follower of the Great Way of Unix h...@kramse.org h...@solidonetworks.com +45 2026 6000 cand.scient CISSP http://solidonetworks.com/ Network Security is a business enabler
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On 24/07/2012, at 14.16, Otto Moerbeek wrote: > On Tue, Jul 24, 2012 at 01:52:03PM +0200, Peter Hessler wrote: > >> On 2012 Jul 24 (Tue) at 12:40:00 +0200 (+0200), Henrik Lund Kramsh?j wrote: >> :Current tuning, apart from kernel defaults: >> :and also modified /etc/sysctl somewhat - most changes below are from >> :https://calomel.org/network_performance.html: >> >> Revert all changes recommended by this page, they hurt performance and >> do not help. >> >> calomel.org is only bad juju. >> >> >> -- >> In case of atomic attack, the federal ruling against prayer in schools >> will be temporarily canceled. > > Also, the OP didn't say if pf is active or not. By default pf is > enabled these days, switch off by setting pf=NO in rc.conf.local sorry, but tried both, some difference, some improvement, as shown below hlk@xpc02:hlk$ sudo pfctl -d pf disabled hlk@xpc02:hlk$ tcpbench -u 10.0.10.30 Elapsed:1000 Mbps:1943.217 Peak Mbps:1943.217 Tx PPS: 165015 Elapsed:2000 Mbps:1944.761 Peak Mbps:1944.761 Tx PPS: 165146 Elapsed:3000 Mbps:1942.852 Peak Mbps:1944.761 Tx PPS: 164984 Elapsed:4000 Mbps:1942.710 Peak Mbps:1944.761 Tx PPS: 164972 hlk@xpc02:hlk$ tcpbench -n 4 10.0.10.30 elapsed_ms bytes mbps bwidth 1000 114326840 914.615 28.58% 1000 86697552 693.580 21.67% 1000 7 889.431 27.79% 1000 87886360 703.091 21.97% Conn: 4 Mbps: 3200.717 Peak Mbps: 3200.717 Avg Mbps: 800.179 2001 79059352 631.843 19.21% 2001 127067792 1016.542 30.87% 2001 135531352 1084.251 32.93% 2001 69944248 559.554 16.99% Conn: 4 Mbps: 3292.190 Peak Mbps: 3292.190 Avg Mbps: 823.048 3003 101616720 812.122 25.16% 3003 98757024 789.267 24.45% 3003 103894144 830.323 25.72% 3003 99604280 796.038 24.66% Conn: 4 Mbps: 3227.750 Peak Mbps: 3292.190 Avg Mbps: 806.937 But still way below expectations :-( Best regards Henrik -- Henrik Lund Kramshøj, Follower of the Great Way of Unix h...@kramse.org h...@solidonetworks.com +45 2026 6000 cand.scient CISSP http://solidonetworks.com/ Network Security is a business enabler
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On Tue, Jul 24, 2012 at 02:22:57PM +0200, Henrik Lund Kramsh?j wrote: > > On 24/07/2012, at 14.16, Otto Moerbeek wrote: > > > On Tue, Jul 24, 2012 at 01:52:03PM +0200, Peter Hessler wrote: > > > >> On 2012 Jul 24 (Tue) at 12:40:00 +0200 (+0200), Henrik Lund Kramsh?j wrote: > >> :Current tuning, apart from kernel defaults: > >> :and also modified /etc/sysctl somewhat - most changes below are from > >> :https://calomel.org/network_performance.html: > >> > >> Revert all changes recommended by this page, they hurt performance and > >> do not help. > >> > >> calomel.org is only bad juju. > >> > >> > >> -- > >> In case of atomic attack, the federal ruling against prayer in schools > >> will be temporarily canceled. > > > > Also, the OP didn't say if pf is active or not. By default pf is > > enabled these days, switch off by setting pf=NO in rc.conf.local > > sorry, but tried both, some difference, some improvement, as shown below > > hlk@xpc02:hlk$ sudo pfctl -d > pf disabled > hlk@xpc02:hlk$ tcpbench -u 10.0.10.30 > Elapsed:1000 Mbps:1943.217 Peak Mbps:1943.217 Tx PPS: 165015 > Elapsed:2000 Mbps:1944.761 Peak Mbps:1944.761 Tx PPS: 165146 > Elapsed:3000 Mbps:1942.852 Peak Mbps:1944.761 Tx PPS: 164984 > Elapsed:4000 Mbps:1942.710 Peak Mbps:1944.761 Tx PPS: 164972 > > hlk@xpc02:hlk$ tcpbench -n 4 10.0.10.30 > elapsed_ms bytes mbps bwidth > 1000 114326840 914.615 28.58% > 1000 86697552 693.580 21.67% > 1000 7 889.431 27.79% > 1000 87886360 703.091 21.97% > Conn: 4 Mbps: 3200.717 Peak Mbps: 3200.717 Avg Mbps: 800.179 > 2001 79059352 631.843 19.21% > 2001 127067792 1016.542 30.87% > 2001 135531352 1084.251 32.93% > 2001 69944248 559.554 16.99% > Conn: 4 Mbps: 3292.190 Peak Mbps: 3292.190 Avg Mbps: 823.048 > 3003 101616720 812.122 25.16% > 3003 98757024 789.267 24.45% > 3003 103894144 830.323 25.72% > 3003 99604280 796.038 24.66% > Conn: 4 Mbps: 3227.750 Peak Mbps: 3292.190 Avg Mbps: 806.937 > > But still way below expectations :-( Ok, now that obvious things are out of the way, I thunk I have to leave this to the newtork gurus. BTW, did you try iperf on OpenBSD? -Otto
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On Tue, Jul 24, 2012 at 01:52:03PM +0200, Peter Hessler wrote: > On 2012 Jul 24 (Tue) at 12:40:00 +0200 (+0200), Henrik Lund Kramsh?j wrote: > :Current tuning, apart from kernel defaults: > :and also modified /etc/sysctl somewhat - most changes below are from > :https://calomel.org/network_performance.html: > > Revert all changes recommended by this page, they hurt performance and > do not help. > > calomel.org is only bad juju. > > > -- > In case of atomic attack, the federal ruling against prayer in schools > will be temporarily canceled. Also, the OP didn't say if pf is active or not. By default pf is enabled these days, switch off by setting pf=NO in rc.conf.local -Otto
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On 24/07/2012, at 13.52, Peter Hessler wrote: > On 2012 Jul 24 (Tue) at 12:40:00 +0200 (+0200), Henrik Lund Kramshøj wrote: > :Current tuning, apart from kernel defaults: > :and also modified /etc/sysctl somewhat - most changes below are from > :https://calomel.org/network_performance.html: > > Revert all changes recommended by this page, they hurt performance and > do not help. ok, I did of course test before applying and did not change much > > calomel.org is only bad juju. Understood, then lets pretend that part was never said - sorry for the interruption Best regards Henrik -- Henrik Lund Kramshøj, Follower of the Great Way of Unix h...@kramse.org h...@solidonetworks.com +45 2026 6000 cand.scient CISSP http://solidonetworks.com/ Network Security is a business enabler
Re: 10Gbit Intel 10GbE SFP+ (82599) performance
On 2012 Jul 24 (Tue) at 12:40:00 +0200 (+0200), Henrik Lund Kramshøj wrote: :Current tuning, apart from kernel defaults: :and also modified /etc/sysctl somewhat - most changes below are from :https://calomel.org/network_performance.html: Revert all changes recommended by this page, they hurt performance and do not help. calomel.org is only bad juju. -- In case of atomic attack, the federal ruling against prayer in schools will be temporarily canceled.
10Gbit Intel 10GbE SFP+ (82599) performance
Hi I have bought two lab systems for testing 10Gbit and have some issues with 10Gbit performance. Note: I have the devices triple-booting FreeBSD, Linux and OpenBSD and can get about 9.4Gbits/sec with Linux iperf using iperf -t 60 I have read and updated kernel to latest snapshot OpenBSD 5.2 (GENERIC.MP) #362: Sun Jul 22 14:10:40 MDT 2012 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17169907712 (16374MB) avail mem = 16690479104 (15917MB) also tried GENERIC-nonMP - gave worse performance Current performance, a little better than 5.1 root@xpc02:root# tcpbench -u 10.0.10.30 Elapsed:1000 Mbps:1841.437 Peak Mbps:1841.437 Tx PPS: 156372 Elapsed:2000 Mbps:1841.901 Peak Mbps:1841.901 Tx PPS: 156411 Elapsed:3000 Mbps:1839.553 Peak Mbps:1841.901 Tx PPS: 156212 root@xpc03:root# tcpbench -n 4 -s elapsed_ms bytes mbps bwidth 999 92113072 737.642 27.54% 999 98731880 790.646 29.52% 999 72763448 582.690 21.75% 999 70892632 567.709 21.19% Conn: 4 Mbps: 2678.687 Peak Mbps: 2678.687 Avg Mbps: 669.672 2000 65977576 527.821 18.87% 1999 107496584 860.834 30.75% 1999 67310912 539.026 19.25% 1999 108839944 871.591 31.13% Conn: 4 Mbps: 2799.272 Peak Mbps: 2799.272 Avg Mbps: 699.818 OTOH it seems to be very stable, even running over longer periods :-) Better stable than fast :-) Basic specs Shuttle SX58H7 Pro Barebone, using SX58 chipset - has x8 PCI-E slot Dual port 10Gbit Intel 10GbE SFP+ (82599) cpu0: Intel(R) Core(TM) i7 CPU 960 @ 3.20GHz, 3200.46 MHz spdmem0 at iic0 addr 0x50: 4GB DDR3 SDRAM PC3-10600 spdmem1 at iic0 addr 0x51: 4GB DDR3 SDRAM PC3-10600 spdmem2 at iic0 addr 0x52: 4GB DDR3 SDRAM PC3-10600 spdmem3 at iic0 addr 0x54: 4GB DDR3 SDRAM PC3-10600, BIOS overclocking memory a bit up to 1333MHz Current tuning, apart from kernel defaults: and also modified /etc/sysctl somewhat - most changes below are from https://calomel.org/network_performance.html: hlk@xpc02:hlk$ grep -v "^#" /etc/sysctl.conf net.inet6.icmp6.rediraccept=1 # 1=Accept IPv6 ICMP redirects (for hosts) kern.pool_debug=0 # 0=Disable pool corruption checks (faster) machdep.allowaperture=2 # See xf86(4) ddb.panic=0# do not enter ddb console on kernel panic, reboot if possible kern.bufcachepercent=90# Allow the kernel to use up to 90% of the RAM for cache (default 10%) machdep.allowaperture=2# Access the X Window System (if you use X on the system) net.inet.ip.forwarding=1 # Permit forwarding (routing) of packets through the firewall net.inet.ip.ifq.maxlen=512 # Maximum allowed input queue length (256*number of physical interfaces) net.inet.ip.mtudisc=0 # TCP MTU (Maximum Transmission Unit) discovery off since our mss is small enough net.inet.tcp.rfc3390=2 # RFC3390 increasing TCP's Initial Congestion Window to 14600 for SPDY net.inet.tcp.mssdflt=1440 # maximum segment size (1440 from scrub pf.conf match statement) net.inet.udp.recvspace=131072 # Increase UDP "receive" buffer size. Good for 200Mbit without packet drop. net.inet.udp.sendspace=131072 # Increase UDP "send" buffer size. Good for 200Mbit without packet drop. I have a more detailed pastebin at: http://pastebin.com/iNpk3DgV The devices are NOT intended for production and I will be happy to allow access to these systems, only requirement is that you have IPv6 (I am moving these systems around a bit, and NAT sucks) - email me your SSH public key and I will create an account. I have this week until saturday dedicated for working on this and other related projects, so contact me if you want to join. Skype: kramshoej, EFnet: hlkv6, freenode: hlkv6 Current cabling is using direct attach copper cable, I only have direct attach cables, but multiple switches, did not make any significant difference. I got a Force10 switch to report the following in testing, so the hardware looks fine: force10-2#sh interfaces Tengigabitethernet 0/0 | grep Mbits Input 35.00 Mbits/sec, 62616 packets/sec, 0.45% of line-rate Output 9869.00 Mbits/sec, 812987 packets/sec, 99.90% of line-rate The main steps for getting FreeBSD tuned were: kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 sysctl hw.intr_storm_threshold=9000 - note I see LOTS and lots of interrupts, so maybe some /boot/loader.conf ixgbe_load="YES" hw.ixgbe.txd=4096 hw.ixgbe.rxd=4096 - this made a huge difference Killing xhci from kernel, was using lots of interrupts, buggy driver? Best regards Henrik -- Henrik Lund Kramshøj, Follower of the Great Way of Unix h...@kramse.org h...@solidonetworks.com +45 2026 6000 cand.scient CISSP http://solidonetworks.com/ Network Security is a business enabler
