Hello all,
Last saturday during a service window, we installed a new firewall as a
replacement to one of our oldest firewalls that had been running at a
very high load for a long time.
The old fw ran 3.9 (i386, SP) on a Pentium4 3.2Ghz and was able to
forward data at about 300 Mbit/s at 100% CPu usage (interrupts). The
NICs in this machine was Intel 1000/Pro cards on the PCI bus.
We were hoping that the new hardware would be able to achieve a higher
throughput, perhaps near 1 Gbit/s. The result, however was even worse
than the old fw. We can only get about 70-80 Mbit/s at 100% CPU...
The new system has a Xeon 4-core CPU at 2,5 Ghz and Intel 1000/Pro cards
on the PCI-E bus. It's running 4.8 GENERIC.MP amd64.
What we noticed when checking systat was a high number of livelocks on
the fw interfaces, em0 and em2:
IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
System 256 50 458
2k 30 2142
lo0
em0 313390 2k 84 256 8
em1
em2 379051 2k 11 4 256 11
em3
em4 4962 2k 4 4 256 4
This seems extremely high compared to some of our other fws where the
number of livelocks are a couple of hundreds after months of uptime.
What could be the reason for this huge drop in performance? I know that
the kernel only uses on CPU core and the clock frequency is a little
lower, but it seems the difference is too high.
Could it be the PCI-E NICs? The bus speed is much faster than the old
PCI cards, so we hoped they would perform better.
Could it be the OS? 64 bits instead of 32 and MP kernel instead of SP?
What we do know is that it's not PF, as we disabled it for a couple of
minutes and still had 100% interrupts.
Do any of you have the same issues with high interrupts and low
throughput?
We really don't know where to start... :(
I'm very grateful for any kind of input regarding this matter.
Brgds, Peter Hallin, Lund University Sweden
dmesg:
OpenBSD 4.8 (GENERIC.MP) #335: Mon Aug 16 09:09:20 MDT 2010
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
RTC BIOS diagnostic error 2
real mem = 2139598848 (2040MB)
avail mem = 2068819968 (1972MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x7fa32000 (60 entries)
bios0: vendor Intel Corporation version
"S5000.86B.11.00.0096.011320091422" date 01/13/2009
bios0: Intel S5000VSA
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET MCFG SSDT SSDT HEST BERT
ERST EINJ
acpi0: wakeup devices SLPB(S5) P32_(S5) PS2M(S1) PS2K(S1)
UAR1(S5) UAR2(S5) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) PCIE(S5)
PCIO(S5) PCIP(S5) PCIQ(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2494.09 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG
cpu0: 6MB 64b/line 16-way L2 cache
cpu0: apic clock running at 332MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG
cpu1: 6MB 64b/line 16-way L2 cache
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG
cpu2: 6MB 64b/line 16-way L2 cache
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG
cpu3: 6MB 64b/line 16-way L2 cache
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 9 (P32_)
acpiprt2 at acpi0: bus 8 (PEX0)
acpiprt3 at acpi0: bus -1 (PEX1)
acpiprt4 at acpi0: bus -1 (PEX2)
acpiprt5 at acpi0: bus -1 (PEX3)
acpiprt6 at acpi0: bus 1 (PCIE)
acpiprt7 at acpi0: bus 6 (PCIE)
acpiprt8 at acpi0: bus 2 (PCIW)
acpiprt9 at acpi0: bus 3 (PCIO)
acpiprt10 at acpi0: bus 4 (PCIP)
acpiprt11 at acpi0: bus 5 (PCIQ)
acpicpu0 at acpi0: C2, C1
acpicpu1 at acpi0: C2, C1
acpicpu2 at acpi0: C2, C1
acpicpu3 at acpi0: C2, C1
acpibtn0 at acpi0: SLPB
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5000V Host" rev 0xb1
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0xb1
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01:
apic 8 int 16 (irq 10)
pci3 at ppb2 bus 3
em0 at pci3 dev 0 function 0 "Intel PRO/1000 (82576)" rev 0x01:
apic 8 int 16 (irq 10), address 00:1b:21:9e:36:cc
em1 at pci3 dev 0 function 1 "Intel PRO/1000 (82576)" rev 0x01:
apic 8 int 17 (irq 11), address 00:1b:21:9e:36:cd
ppb3 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01:
apic 8 int 17 (irq 11)
pci4 at ppb3 bus 4
em2 at pci4 dev 0 function 0 "Intel PRO/1000 (82576)" rev 0x01:
apic 8 int 17 (irq 11), address 00:1b:21:9e:35:ba
em3 at pci4 dev 0 function 1 "Intel PRO/1000 (82576)" rev 0x01:
apic 8 int 18 (irq 5), address 00:1b:21:9e:35:bb
ppb4 at pci2 dev 2 function 0 "Intel 6321ESB PCIE" rev 0x01
pci5 at ppb4 bus 5
em4 at pci5 dev 0 function 0 "Intel PRO/1000 PT (80003ES2)" rev
0x01: apic 8 int 18 (irq 5), address 00:15:17:89:eb:fc
em5 at pci5 dev 0 function 1 "Intel PRO/1000 PT (80003ES2)" rev
0x01: apic 8 int 19 (irq 11), address 00:15:17:89:eb:fd
ppb5 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci6 at ppb5 bus 6
ppb6 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0xb1
pci7 at ppb6 bus 7
"Intel I/OAT" rev 0xb1 at pci0 dev 8 function 0 not configured
pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting"
rev 0xb1
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting"
rev 0xb1
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting"
rev 0xb1
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0xb1
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0xb1
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0xb1
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0xb1
ppb7 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09:
apic 8 int 16 (irq 10)
pci8 at ppb7 bus 8
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09:
apic 8 int 23 (irq 11)
uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09:
apic 8 int 22 (irq 5)
uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09:
apic 8 int 23 (irq 11)
uhci3 at pci0 dev 29 function 3 "Intel 6321ESB USB" rev 0x09:
apic 8 int 22 (irq 5)
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09:
apic 8 int 23 (irq 11)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb8 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev
0xd9
pci9 at ppb8 bus 9
vga1 at pci9 dev 12 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 8 int 17 (irq 11)
drm0 at radeondrm0
pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09:
DMA, channel 0 configured to compatibility, channel 1 configured to
compatibility
wd0 at pciide0 channel 0 drive 0: <WDC WD2500AAJB-00J3A0>
wd0: 16-sector PIO, LBA48, 238475MB, 488397168 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
pciide0: channel 1 disabled (no drives)
pciide1 at pci0 dev 31 function 2 "Intel 6321ESB SATA" rev
0x09: DMA, channel 0 configured to native-PCI, channel 1 configured to
native-PCI
pciide1: using apic 8 int 20 (irq 10) for native-PCI interrupt
atapiscsi0 at pciide1 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TSSTcorp, CDDVDW SH-S223F, SB02>
ATAPI 5/cdrom removable
cd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev
0x09: apic 8 int 20 (irq 10)
iic0 at ichiic0
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
mtrr: Pentium Pro MTRR support
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
END dmesg.
