On 13 June 2013 00:15, John Jasen <jja...@realityfailure.org> wrote: > (We've seen this problem through a few OpenBSD releases. 5.2 is much > improved over 4.8 in this regard. Using BIOS to disable multiple cores > on each physical CPU also yielded greater throughput. Hyperthreading and > virtualization BIOS extensions are off.) > > We have two Dell poweredge 710 systems in an active/passive carp > failover configuration, acting as firewall systems, with another one > currently being used for testing. > > Various tests show the system can handle close to 10GbE speeds in and > out, but I've not been able to push much beyond that, hitting a wall at > about 11-12Gb/s. >
That is more or less what you will get on OpenBSD. > As I try to push the test system, interrupts climb above 12-15k/second > (seen via systat), consuming more and more of the first CPU until after > about 50% utilisation (according to top), it just hits a wall and > refuses to spit out any more bandwidth. > > A coworker was able to drive it up to close to 60k interrupts/second, > but was not able to get much more through it. > > Comparison tests, booting a Debian "Squeeze" live cd, and booting > FreeBSD 9.x, indicate that out of the box, they can push 15-20Gb/s -- > which, while lower than what I would expect, is an improvement. Here your test is probably botched, you're probably just stressing one queue from the card, which gives you more or less what you're seeing, you need to send multiple tcp/udp streams, then you can have an idea how much linux/freebsd can do. obs: "we usually count forwarding rate, so when you say 15gbit/s, most people say 7.5gbit/s =)" > > Are there tuning options that we've not seen yet? Googling, reading the > ix(4) manpage have not exposed a clear "go faster" option, and I'm > concerned about the load and interrupts concentrating on one CPU so heavily. OpenBSD kernel is "single threaded" so this is the expected behaviour. > > ifconfig (from test box, minux carp and vlan interfaces) and dmesg enclosed. > > > OpenBSD 5.2 (GENERIC.MP) #368: Wed Aug 1 10:04:49 MDT 2012 > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 12870651904 (12274MB) > avail mem = 12505665536 (11926MB) > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xbf49c000 (84 entries) > bios0: vendor Dell Inc. version "6.3.0" date 07/24/2012 > bios0: Dell Inc. PowerEdge R710 > acpi0 at bios0: rev 2 > acpi0: sleep states S0 S4 S5 > acpi0: tables DSDT FACP APIC SPCR HPET DM__ MCFG WDAT SLIC ERST HEST > BERT EINJ TCPA > acpi0: wakeup devices PCI0(S5) > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 16 (boot processor) > cpu0: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz, 2926.44 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF > ,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: apic clock running at 132MHz > cpu1 at mainbus0: apid 0 (application processor) > cpu1: Intel(R) Xeon(R) CPU X5570 @ 2.93GHz, 2926.00 MHz > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF > ,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF > cpu1: 256KB 64b/line 8-way L2 cache > ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins > ioapic1 at mainbus0: apid 1 pa 0xfec80000, version 20, 24 pins > ioapic1: misconfigured as apic 0, remapped to apid 1 > acpihpet0 at acpi0: 14318179 Hz > acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 > acpiprt0 at acpi0: bus 0 (PCI0) > acpiprt1 at acpi0: bus 1 (PEX1) > acpiprt2 at acpi0: bus 2 (PEX3) > acpiprt3 at acpi0: bus 3 (PEX4) > acpiprt4 at acpi0: bus 4 (PEX5) > acpiprt5 at acpi0: bus 5 (PEX6) > acpiprt6 at acpi0: bus 6 (PEX7) > acpiprt7 at acpi0: bus 7 (PEX9) > acpiprt8 at acpi0: bus -1 (PEXA) > acpiprt9 at acpi0: bus -1 (SBEX) > acpiprt10 at acpi0: bus 8 (COMP) > acpicpu0 at acpi0 > acpicpu1 at acpi0 > ipmi at mainbus0 not configured > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "Intel 5520 Host" rev 0x13 > ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x13 > pci1 at ppb0 bus 1 > bnx0 at pci1 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 1 int 4 > bnx1 at pci1 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 1 int 16 > ppb1 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x13 > pci2 at ppb1 bus 2 > bnx2 at pci2 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 1 int 0 > bnx3 at pci2 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 1 int 10 > ppb2 at pci0 dev 4 function 0 "Intel X58 PCIE" rev 0x13 > pci3 at ppb2 bus 3 > mpi0 at pci3 dev 0 function 0 "Symbios Logic SAS1068E" rev 0x08: msi > scsibus0 at mpi0: 112 targets > sd0 at scsibus0 targ 0 lun 0: <Dell, VIRTUAL DISK, 1028> SCSI3 0/direct > fixed naa.600508e000000000a54b752495c4c504 > sd0: 237952MB, 512 bytes/sector, 487325696 sectors > ses0 at scsibus0 targ 8 lun 0: <DP, BACKPLANE, 1.07> SCSI3 13/enclosure > services fixed t10.DP_BACKPLANE000000 > ppb3 at pci0 dev 5 function 0 "Intel X58 PCIE" rev 0x13: msi > pci4 at ppb3 bus 4 > ppb4 at pci0 dev 6 function 0 "Intel X58 PCIE" rev 0x13: msi > pci5 at ppb4 bus 5 > ix0 at pci5 dev 0 function 0 "Intel 10GbE SR Dual (82598AF)" rev 0x01: > msi, address 00:1b:21:3f:f0:e3 > ix1 at pci5 dev 0 function 1 "Intel 10GbE SR Dual (82598AF)" rev 0x01: > msi, address 00:1b:21:3f:f0:e2 > ppb5 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x13: msi > pci6 at ppb5 bus 6 > ix2 at pci6 dev 0 function 0 "Intel 10GbE SR Dual (82598AF)" rev 0x01: > msi, address 00:1b:21:41:b3:43 > ix3 at pci6 dev 0 function 1 "Intel 10GbE SR Dual (82598AF)" rev 0x01: > msi, address 00:1b:21:41:b3:42 > ppb6 at pci0 dev 9 function 0 "Intel X58 PCIE" rev 0x13: msi > pci7 at ppb6 bus 7 > ix4 at pci7 dev 0 function 0 "Intel 10GbE LR (82598EB)" rev 0x01: msi, > address 00:1b:21:3f:b6:02 > "Intel X58 Misc" rev 0x13 at pci0 dev 20 function 0 not configured > "Intel X58 GPIO" rev 0x13 at pci0 dev 20 function 1 not configured > "Intel X58 RAS" rev 0x13 at pci0 dev 20 function 2 not configured > uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 0 int 17 > uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 0 int 18 > ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 0 int 19 > usb0 at ehci0: USB revision 2.0 > uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 > uhci2 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 0 int 21 > uhci3 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 0 int 20 > ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 0 int 21 > usb1 at ehci1: USB revision 2.0 > uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1 > ppb7 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92 > pci8 at ppb7 bus 8 > vga1 at pci8 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02 > pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA, > channel 0 configured to native-PCI, channel 1 configure > d to native-PCI > pciide0: using apic 0 int 23 for native-PCI interrupt > atapiscsi0 at pciide0 channel 0 drive 0 > scsibus1 at atapiscsi0: 2 targets > cd0 at scsibus1 targ 0 lun 0: <TEAC, DVD-ROM DV28SV, D.0K> ATAPI 5/cdrom > removable > cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 > usb2 at uhci0: USB revision 1.0 > uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb3 at uhci1: USB revision 1.0 > uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb4 at uhci2: USB revision 1.0 > uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > usb5 at uhci3: USB revision 1.0 > uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1 > isa0 at pcib0 > isadma0 at isa0 > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo > com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo > pckbc0 at isa0 port 0x60/5 > pckbd0 at pckbc0 (kbd slot) > pckbc0: using irq 1 for kbd slot > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > pcppi0 at isa0 port 0x61 > spkr0 at pcppi0 > mtrr: Pentium Pro MTRR support > uhub6 at uhub0 port 3 "Standard Microsystems product 0x2514" rev > 2.00/0.00 addr 2 > uhidev0 at uhub4 port 1 configuration 1 interface 0 "Avocent Dell > 03R874" rev 1.10/1.00 addr 2 > uhidev0: iclass 3/1 > ukbd0 at uhidev0: 8 variable keys, 6 key codes, country code 33 > wskbd1 at ukbd0 mux 1 > wskbd1: connecting to wsdisplay0 > uhidev1 at uhub4 port 1 configuration 1 interface 1 "Avocent Dell > 03R874" rev 1.10/1.00 addr 2 > uhidev1: iclass 3/1, 3 report ids > ums0 at uhidev1 reportid 1: 5 buttons, Z dir > wsmouse0 at ums0 mux 0 > uhid0 at uhidev1 reportid 2: input=2, output=0, feature=0 > uhid1 at uhidev1 reportid 3: input=1, output=0, feature=0 > vscsi0 at root > scsibus2 at vscsi0: 256 targets > softraid0 at root > scsibus3 at softraid0: 256 targets > root on sd0a (91d0defb08274310.a) swap on sd0b dump on sd0b > bnx0: address 00:26:b9:36:ee:fa > brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 > bnx1: address 00:26:b9:36:ee:fc > brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 > bnx2: address 00:26:b9:36:ee:fe > brgphy2 at bnx2 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 > bnx3: address 00:26:b9:36:ef:00 > brgphy3 at bnx3 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 > > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33152 > priority: 0 > groups: lo > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 > inet 127.0.0.1 netmask 0xff000000 > ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500 > lladdr 00:1b:21:3f:f0:e3 > priority: 0 > media: Ethernet autoselect > status: no carrier > ix1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > mtu 9000 > lladdr 00:1b:21:3f:f0:e2 > priority: 0 > media: Ethernet autoselect (10GbaseSR full-duplex) > status: active > inet6 fe80::21b:21ff:fe3f:f0e2%ix1 prefixlen 64 scopeid 0x2 > ix2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > mtu 9000 > lladdr 00:1b:21:41:b3:43 > priority: 0 > media: Ethernet autoselect (10GbaseSR full-duplex) > status: active > inet 172.16.3.253 netmask 0xffffff00 broadcast 172.16.3.255 > inet6 fe80::21b:21ff:fe41:b343%ix2 prefixlen 64 scopeid 0x3 > ix3: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > mtu 9000 > lladdr 00:1b:21:41:b3:42 > priority: 0 > media: Ethernet autoselect (10GbaseSR full-duplex) > status: active > inet 172.16.145.253 netmask 0xfffffe00 broadcast 172.16.145.255 > inet6 fe80::21b:21ff:fe41:b342%ix3 prefixlen 64 scopeid 0x4 > ix4: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> > mtu 9000 > lladdr 00:1b:21:3f:b6:02 > priority: 0 > groups: egress > media: Ethernet autoselect (10GbaseLR full-duplex) > status: active > inet 172.16.254.77 netmask 0xfffffff8 broadcast 172.16.254.79 > inet6 fe80::21b:21ff:fe3f:b602%ix4 prefixlen 64 scopeid 0x5 > enc0: flags=0<> > priority: 0 > groups: enc > status: active > bnx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500 > lladdr 00:26:b9:36:ee:fa > priority: 0 > media: Ethernet autoselect (1000baseT full-duplex) > status: active > inet 192.168.10.253 netmask 0xffffff00 broadcast 192.168.10.255 > inet6 fe80::226:b9ff:fe36:eefa%bnx0 prefixlen 64 scopeid 0x8 > bnx1: > flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 > lladdr 00:26:b9:36:ee:fc > priority: 0 > media: Ethernet autoselect (1000baseT full-duplex) > status: active > inet 172.16.242.125 netmask 0xffffff80 broadcast 172.16.242.127 > inet6 fe80::226:b9ff:fe36:eefc%bnx1 prefixlen 64 scopeid 0x9 > bnx2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500 > lladdr 00:26:b9:36:ee:fe > priority: 0 > media: Ethernet autoselect (none) > status: no carrier > bnx3: > flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500 > lladdr 00:26:b9:36:ef:00 > priority: 0 > media: Ethernet autoselect (1000baseT full-duplex) > status: active > inet 172.16.254.66 netmask 0xfffffff8 broadcast 172.16.254.71 > inet6 fe80::226:b9ff:fe36:ef00%bnx3 prefixlen 64 scopeid 0xb > lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33152 > priority: 0 > groups: lo > inet6 fe80::1%lo1 prefixlen 64 scopeid 0xc > inet 172.16.242.129 netmask 0xffffffff > > pfsync0: flags=41<UP,RUNNING> mtu 1500 > priority: 0 > pfsync: syncdev: bnx0 syncpeer: 192.168.10.254 maxupd: 128 > defer: off > groups: carp pfsync > pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33152 > priority: 0 > groups: pflog > > -- > -- John Jasen (jja...@realityfailure.org) > --
