Re: Slow Intel 10GbE CX4 adapter behaviour
On 02/09/2011 07:27 PM, Sergey Kandaurov wrote: On 9 February 2011 18:15, rihad wrote: On 02/09/2011 05:47 PM, Sergey Kandaurov wrote: On 9 February 2011 12:37, rihadwrote: Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above with ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 was because dummynet wasn't being loaded at all... so no traffic could pass on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it "kldload dummynet" in order to do without a reboot. Works like a charm so far. Thanks to all! Looks like loading dummynet.ko via /boot/loader.conf doesn't work because dummynet.ko depends on dummynet.ko but of the different version. Would dummynet_enable="YES" in rc.conf still work? We haven't yet had a chance to reboot to test that. Yes, it would. Note that it depends on firewall_enable="YES" also present in rc.conf. Thanks, I see. Now I think that changing through rc.conf is the "official", or "supported", way of enabling dummynet upon reboot, but loader.conf is a little way under the hood. I always asked myself why it was settable in two places, and not one. But now I know. The fact that dummynet can be set to load in loader.conf is more like an undesired effect of generality. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Slow Intel 10GbE CX4 adapter behaviour
On 9 February 2011 18:15, rihad wrote: > On 02/09/2011 05:47 PM, Sergey Kandaurov wrote: >> >> On 9 February 2011 12:37, rihad wrote: >>> >>> Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above >>> with >>> ixgbe (tons of "fragmentation failed" errors) was real. The issue in >>> 8.3-RC3 >>> was because dummynet wasn't being loaded at all... so no traffic could >>> pass >>> on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I >>> turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it >>> "kldload >>> dummynet" in order to do without a reboot. Works like a charm so far. >>> Thanks >>> to all! >> >> Looks like loading dummynet.ko via /boot/loader.conf doesn't work because >> dummynet.ko depends on dummynet.ko but of the different version. >> > Would dummynet_enable="YES" in rc.conf still work? We haven't yet had a > chance to reboot to test that. > Yes, it would. Note that it depends on firewall_enable="YES" also present in rc.conf. -- wbr, pluknet ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Slow Intel 10GbE CX4 adapter behaviour
On 02/09/2011 05:47 PM, Sergey Kandaurov wrote: On 9 February 2011 12:37, rihad wrote: Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above with ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 was because dummynet wasn't being loaded at all... so no traffic could pass on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it "kldload dummynet" in order to do without a reboot. Works like a charm so far. Thanks to all! Looks like loading dummynet.ko via /boot/loader.conf doesn't work because dummynet.ko depends on dummynet.ko but of the different version. Would dummynet_enable="YES" in rc.conf still work? We haven't yet had a chance to reboot to test that. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Slow Intel 10GbE CX4 adapter behaviour
On 9 February 2011 12:37, rihad wrote: > Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above with > ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 > was because dummynet wasn't being loaded at all... so no traffic could pass > on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I > turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it "kldload > dummynet" in order to do without a reboot. Works like a charm so far. Thanks > to all! Looks like loading dummynet.ko via /boot/loader.conf doesn't work because dummynet.ko depends on dummynet.ko but of the different version. There are even more strange things: 1) dummynet.ko declares itself as version 1: /sys/netinet/ipfw/ip_dummynet.c: MODULE_VERSION(dummynet, 1); 2) dummynet.ko compiles into itself the various schedulers: fifo, prio, rr, etc; 3) these schedulers presumably think they are compiled standalone, so they are explicitly and strongly depend on dummynet of version 3 (why?): /sys/netinet/ipfw/dn_sched.h: MODULE_DEPEND(name, dummynet, 3, 3, 3); * That makes loader to error like "dummynet: loading required module 'dummynet'". and, if loading dummynet.ko in loader prompt manually, then "module 'dummynet' exists but with wrong version"] This shall fix the problem: rebuilding only dummynet should be enough. %%% Index: /sys/netinet/ipfw/ip_dummynet.c === --- /sys/netinet/ipfw/ip_dummynet.c (revision 218026) +++ /sys/netinet/ipfw/ip_dummynet.c (working copy) @@ -2294,7 +2294,7 @@ #defineDN_MODEV_ORD(SI_ORDER_ANY - 128) /* after ipfw */ DECLARE_MODULE(dummynet, dummynet_mod, DN_SI_SUB, DN_MODEV_ORD); MODULE_DEPEND(dummynet, ipfw, 2, 2, 2); -MODULE_VERSION(dummynet, 1); +MODULE_VERSION(dummynet, 3); /* * Starting up. Done in order after dummynet_modevent() has been called. %%% -- wbr, pluknet ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Slow Intel 10GbE CX4 adapter behaviour
Problem solved, I'm so embarrassed :) The issue on 7.2 mentioned above with ixgbe (tons of "fragmentation failed" errors) was real. The issue in 8.3-RC3 was because dummynet wasn't being loaded at all... so no traffic could pass on it, despite dummynet_load="YES" being set in /boot/loader.conf. So I turned it on in /etc/rc.conf : dummynet_enable="YES" and loaded it "kldload dummynet" in order to do without a reboot. Works like a charm so far. Thanks to all! ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Slow Intel 10GbE CX4 adapter behaviour
On 02/09/2011 10:43 AM, Nikolay Denev wrote: I don't know if it's the same issue, but I had severe performance issues with ixgbe cards until I disable LRO (ifconfig ix0 -lro). That was on 7.2 too. So did I, with LR-based cards. But CX4 cards don't mention LRO in their ifconfig enabled flags. Besides, ixgbe.c doesn't enable it by default: /* Don't enable LRO by default */ ifp->if_capabilities |= IFCAP_LRO; ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Slow Intel 10GbE CX4 adapter behaviour
On 9 Feb, 2011, at 07:29 , rihad wrote:
> Hi, we're a medium sized ISP that need to pass all incoming user traffic
> through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up until
> yesterday it had two 1 gb em cards, one for input, one for output. As we were
> approaching the bandwidth limitation we switched the cards for a two-port
> Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD 7.2 and the
> built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow, and at only
> about 300-400 mbps load (~30-50 IP kpps) the internet access was very slow.
> Also, there were many "IP fragmentation failed" errors (1-30 kpps in "systat
> -ip"). So I decided to source-upgrade the world to 8.3-RC3 (ixgbe 2.3.8).
> Late in the night yesterday I didn't have enough opportunity to test the
> newer FreeBSD under load, but from the time we did and I know, the same
> slowness started happening at about 300-400 mbps load. There are no more
> fragmentation failed errors. No evident drops as per "netstat -s | fgrep
> drop". Only the speed is slooow. Even the ssh console lags a bit. Both ix0
> and ix1 are configured at their default settings.
>
> Then I read something about the number of ixgbe device descriptors
> (hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up to
> 4096 permittable. But after some grepping on the source tree I saw that
> contrary to what the old docs say they are both set to an optimal value:
>
> /sys/dev/ixgbe/ixgbe.c:
> /*
> ** Number of TX descriptors per ring,
> ** setting higher than RX as this seems
> ** the better performing choice.
> */
> static int ixgbe_txd = PERFORM_TXD;
> TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd);
>
> /* Number of RX descriptors per ring */
> static int ixgbe_rxd = PERFORM_RXD;
> TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd)
>
>
> /sys/dev/ixgbe/ixgbe.h:
> /*
> * TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the
> * number of transmit descriptors allocated by the driver. Increasing this
> * value allows the driver to queue more transmits. Each descriptor is 16
> * bytes. Performance tests have show the 2K value to be optimal for top
> * performance.
> */
> #define DEFAULT_TXD 1024
> #define PERFORM_TXD 2048
> #define MAX_TXD 4096
> #define MIN_TXD 64
>
>
>
> So, here's my kernel config for your viewing pleasure:
> include GENERIC
>
> ident SHAPER
>
> nomakeoptions DEBUG
>
> nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4
> nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5
> nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6
> options COMPAT_FREEBSD7 # Compatible with FreeBSD7
> nooptions COMPAT_FREEBSD32# Compatible with i386 binaries
>
> nooptions INET6 # IPv6 communications protocols
> options ZERO_COPY_SOCKETS
> # XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts to work
> #optionsDEVICE_POLLING
> options IPFIREWALL #firewall
> options IPFIREWALL_DEFAULT_TO_ACCEPT#allow everything by default
>
>
> Here's /etc/sysctl.conf:
>
> net.inet.ip.fw.verbose=0
>
> kern.ipc.shmall=65536
> kern.ipc.shmmax=268435456
> kern.ipc.semmap=1024
> kern.ipc.nmbclusters=11
>
> net.inet.ip.fastforwarding=1
> net.inet.ip.dummynet.io_fast=1 #XXX no longer used in 8.3??
> net.isr.direct=0
> net.inet.ip.intr_queue_maxlen=5000
>
> hw.intr_storm_threshold=9000
> #dev.em.0.rx_processing_limit=-1 # device not used any more
>
>
>
>
> Any tips? I'll be happy to try and add some more info upon request.
>
>
> Thanks.
> ___
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
I don't know if it's the same issue, but I had severe performance issues
with ixgbe cards until I disable LRO (ifconfig ix0 -lro). That was on 7.2 too.
Regards,
Nikolay___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Slow Intel 10GbE CX4 adapter behaviour
Hi, we're a medium sized ISP that need to pass all incoming user traffic
through a Intel Server Systems FreeBSD PC and its dummynet pipes. Up
until yesterday it had two 1 gb em cards, one for input, one for output.
As we were approaching the bandwidth limitation we switched the cards
for a two-port Intel 10GbE CX4 PCI-E adapter. With the then used FreeBSD
7.2 and the built-in FreeBSD ixgbe driver 1.7.3 (IIRC) it was very slow,
and at only about 300-400 mbps load (~30-50 IP kpps) the internet access
was very slow. Also, there were many "IP fragmentation failed" errors
(1-30 kpps in "systat -ip"). So I decided to source-upgrade the world to
8.3-RC3 (ixgbe 2.3.8). Late in the night yesterday I didn't have enough
opportunity to test the newer FreeBSD under load, but from the time we
did and I know, the same slowness started happening at about 300-400
mbps load. There are no more fragmentation failed errors. No evident
drops as per "netstat -s | fgrep drop". Only the speed is slooow. Even
the ssh console lags a bit. Both ix0 and ix1 are configured at their
default settings.
Then I read something about the number of ixgbe device descriptors
(hw.ixgbe.txd & hw.ixgbe.rxd) being set low at 256 by default, with up
to 4096 permittable. But after some grepping on the source tree I saw
that contrary to what the old docs say they are both set to an optimal
value:
/sys/dev/ixgbe/ixgbe.c:
/*
** Number of TX descriptors per ring,
** setting higher than RX as this seems
** the better performing choice.
*/
static int ixgbe_txd = PERFORM_TXD;
TUNABLE_INT("hw.ixgbe.txd", &ixgbe_txd);
/* Number of RX descriptors per ring */
static int ixgbe_rxd = PERFORM_RXD;
TUNABLE_INT("hw.ixgbe.rxd", &ixgbe_rxd)
/sys/dev/ixgbe/ixgbe.h:
/*
* TxDescriptors Valid Range: 64-4096 Default Value: 256 This value is the
* number of transmit descriptors allocated by the driver. Increasing this
* value allows the driver to queue more transmits. Each descriptor is 16
* bytes. Performance tests have show the 2K value to be optimal for top
* performance.
*/
#define DEFAULT_TXD 1024
#define PERFORM_TXD 2048
#define MAX_TXD 4096
#define MIN_TXD 64
So, here's my kernel config for your viewing pleasure:
include GENERIC
ident SHAPER
nomakeoptions DEBUG
nooptions COMPAT_FREEBSD4 # Compatible with FreeBSD4
nooptions COMPAT_FREEBSD5 # Compatible with FreeBSD5
nooptions COMPAT_FREEBSD6 # Compatible with FreeBSD6
options COMPAT_FREEBSD7 # Compatible with FreeBSD7
nooptions COMPAT_FREEBSD32# Compatible with i386 binaries
nooptions INET6 # IPv6 communications protocols
options ZERO_COPY_SOCKETS
# XXX 20091227: em(4) wants DEVICE_POLLING off for its fast-interrupts
to work
#optionsDEVICE_POLLING
options IPFIREWALL #firewall
options IPFIREWALL_DEFAULT_TO_ACCEPT#allow everything by default
Here's /etc/sysctl.conf:
net.inet.ip.fw.verbose=0
kern.ipc.shmall=65536
kern.ipc.shmmax=268435456
kern.ipc.semmap=1024
kern.ipc.nmbclusters=11
net.inet.ip.fastforwarding=1
net.inet.ip.dummynet.io_fast=1 #XXX no longer used in 8.3??
net.isr.direct=0
net.inet.ip.intr_queue_maxlen=5000
hw.intr_storm_threshold=9000
#dev.em.0.rx_processing_limit=-1 # device not used any more
Any tips? I'll be happy to try and add some more info upon request.
Thanks.
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
