Re: Firewall, high interrupt load, is this a driver problem (dc) ?
On 7/31/07, Henning Brauer <[EMAIL PROTECTED]> wrote: > > numbers above are from 4.1. -current should behave significantly better. > Have hardware recommendations changed at all? Any specific north/southbridges or NICs person should be looking at? thanks, aaron
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
* Aaron Glenn <[EMAIL PROTECTED]> [2007-07-31 19:01]: > > the install with the highest forwarding rate I know of uses a > > Supermicro X6DH8-XB, a 3.2 GHz Xeon and a bunch of em(4. I have > > seen it doing 750 MBit/s of real-world traffic at approx 150k pps. > > With a full routing table (~205k entries) and a GENERIC kernel it was > > running at roughly 80..90% CPU load; the slightly optimized for the task > > kernel I have in place there now gives quite some extra headroom. Also, > > I expect sk/msk(4) to perform better than em(4), but that has yet to be > > proven in real-world conditions. > > Does this still hold true for 4.1? I'm in the same predicament -- > carte blanche on hardware, need high PPS and throughput with as low > CPU usage as possible. I know there have been some signifigant chances > in the tree; but scouring interesting commit messages and grepping the > lists hasn't enlightened me much. numbers above are from 4.1. -current should behave significantly better. -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
> the install with the highest forwarding rate I know of uses a > Supermicro X6DH8-XB, a 3.2 GHz Xeon and a bunch of em(4. I have > seen it doing 750 MBit/s of real-world traffic at approx 150k pps. > With a full routing table (~205k entries) and a GENERIC kernel it was > running at roughly 80..90% CPU load; the slightly optimized for the task > kernel I have in place there now gives quite some extra headroom. Also, > I expect sk/msk(4) to perform better than em(4), but that has yet to be > proven in real-world conditions. Does this still hold true for 4.1? I'm in the same predicament -- carte blanche on hardware, need high PPS and throughput with as low CPU usage as possible. I know there have been some signifigant chances in the tree; but scouring interesting commit messages and grepping the lists hasn't enlightened me much. Thanks, Aaron
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
Here is usefull details from Henning (thanks!) Message original Sujet: Re: Firewall, high interrupt load, is this a driver problem (dc) ? Date: Tue, 23 Jan 2007 11:42:22 +0100 De: Henning Brauer <[EMAIL PROTECTED]> Pour: Ronnie Garcia <[EMAIL PROTECTED]> Rifirences: <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> * Ronnie Garcia <[EMAIL PROTECTED]> [2007-01-23 11:19]: > Hey Henning, > > Henning Brauer a icrit : > >* Ronnie Garcia <[EMAIL PROTECTED]> [2007-01-22 21:10]: > > >>I'm graphing a lot of kernel/pf variables with cacti, and i'm clearly > >>seeing the box maxing at 15k interrupts/s. > > > >that is not necessarily a problem. > > > >>I'm raising 15k interrupts/s when the box is routing approx 13k pps and > >>then the CPU is at 50-55%. > > > >at 13k pps you definately want good nics which have proper interrupt > >mitigation. most gigE NICs fall into that category; sk, msk and em fall > >definately into that category. > > Thanks for your detailled reply. > > I guess that you are using (or used) obsd routers/firewalls at BS Web > Services. They might also handle a high packets rate. yup > May i ask what kind of hardware you are using ? Motherboard, CPU, NIC, > PCI type ? varying. > I'm considering buying new hardware for these firewalls, and i'd like > them to handle a bunch of pps ;) the install with the highest forwarding rate I know of uses a Supermicro X6DH8-XB, a 3.2 GHz Xeon and a bunch of em(4. I have seen it doing 750 MBit/s of real-world traffic at approx 150k pps. With a full routing table (~205k entries) and a GENERIC kernel it was running at roughly 80..90% CPU load; the slightly optimized for the task kernel I have in place there now gives quite some extra headroom. Also, I expect sk/msk(4) to perform better than em(4), but that has yet to be proven in real-world conditions. -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
Hey Henning, Henning Brauer a icrit : * Ronnie Garcia <[EMAIL PROTECTED]> [2007-01-22 21:10]: I'm graphing a lot of kernel/pf variables with cacti, and i'm clearly seeing the box maxing at 15k interrupts/s. that is not necessarily a problem. I'm raising 15k interrupts/s when the box is routing approx 13k pps and then the CPU is at 50-55%. at 13k pps you definately want good nics which have proper interrupt mitigation. most gigE NICs fall into that category; sk, msk and em fall definately into that category. Thanks for your detailled reply. I guess that you are using (or used) obsd routers/firewalls at BS Web Services. They might also handle a high packets rate. May i ask what kind of hardware you are using ? Motherboard, CPU, NIC, PCI type ? I'm considering buying new hardware for these firewalls, and i'd like them to handle a bunch of pps ;) Regards, -- Ronnie Garcia Directeur ovea Til : +33 4 6767 Gsm : +33 6 29500295 http://www.ovea.com
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
* Ronnie Garcia <[EMAIL PROTECTED]> [2007-01-22 21:10]: > Ronnie Garcia a icrit : > >I recently switched one of our firewalls from Linux to oBSD 4.0. > >Its handling approx 8-9 kpps (in+out) on both interfaces. It has a > >D-Link DFE-570TX quad ports NIC (dc driver), two ports are used. > >On Linux, the CPU was loaded at approx 20% when, and on oBSD, its > >actually loaded at ~30%. No big deal, but on Linux we had queueing > >(shaping) with TC/HTB, whereas ALTQ is not (yet) enabled on oBSD. > > > >The CPU usage is almost only "interrupt", as you can see on this top > >output : > > [The rest of the message is left bellow for the record.] > > I can now tell that i have the exact same behaviour with bsd.mp. > > I'm graphing a lot of kernel/pf variables with cacti, and i'm clearly > seeing the box maxing at 15k interrupts/s. that is not necessarily a problem. > I'm raising 15k interrupts/s when the box is routing approx 13k pps and > then the CPU is at 50-55%. at 13k pps you definately want good nics which have proper interrupt mitigation. most gigE NICs fall into that category; sk, msk and em fall definately into that category. > When i disable pf (pfctl -d), the CPU downs to ~40% but the interrupts > rate does not decrease. This means that the high interrupts rate is due > to network activity, and not to pf. the whole fwding and pf run in (partially soft-) interrupt context and are counted as interrupt time. > I might try with an Intel Pro/1000MT quad instead of the D-Link > DFE-570TX quad to see if my problem is the NIC or the PCI bus/chipset. that will do. -- Henning Brauer, [EMAIL PROTECTED], [EMAIL PROTECTED] BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
Ronnie Garcia a icrit :
I recently switched one of our firewalls from Linux to oBSD 4.0.
Its handling approx 8-9 kpps (in+out) on both interfaces. It has a
D-Link DFE-570TX quad ports NIC (dc driver), two ports are used.
On Linux, the CPU was loaded at approx 20% when, and on oBSD, its
actually loaded at ~30%. No big deal, but on Linux we had queueing
(shaping) with TC/HTB, whereas ALTQ is not (yet) enabled on oBSD.
The CPU usage is almost only "interrupt", as you can see on this top
output :
[The rest of the message is left bellow for the record.]
I can now tell that i have the exact same behaviour with bsd.mp.
I'm graphing a lot of kernel/pf variables with cacti, and i'm clearly
seeing the box maxing at 15k interrupts/s.
I'm raising 15k interrupts/s when the box is routing approx 13k pps and
then the CPU is at 50-55%.
When i disable pf (pfctl -d), the CPU downs to ~40% but the interrupts
rate does not decrease. This means that the high interrupts rate is due
to network activity, and not to pf.
The interrupts rate is higher than the packets rate !
I might try with an Intel Pro/1000MT quad instead of the D-Link
DFE-570TX quad to see if my problem is the NIC or the PCI bus/chipset.
Again, my dmesg :
syncing disks... 0
OpenBSD 4.0 (GENERIC) #1107: Sat Sep 16 19:15:58 MDT 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID
real mem = 528035840 (515660K)
avail mem = 473710592 (462608K)
using 4256 buffers containing 26505216 bytes (25884K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(fc) BIOS, date 06/11/03, BIOS32 rev. 0 @
0xf10a0, SMBIOS rev. 2.3 @ 0xf2d10 (44 entries)
bios0: ASUSTeK Computer INC. P4S533MX
apm0 at bios0: Power Management spec V1.2 (BIOS mgmt disabled)
apm0: APM power management enable: unrecognized device ID (9)
apm0: APM engage (device 1): power management disabled (1)
apm0: AC on, battery charge unknown
apm0: flags b0102 dobusy 0 doidle 1
pcibios0 at bios0: rev 2.1 @ 0xf/0x16d2
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf1640/144 (7 entries)
pcibios0: PCI Interrupt Router at 000:02:0 ("SiS 962 ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus
bios0: ROM list: 0xc/0xc000
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "SiS 651 PCI" rev 0x02
ppb0 at pci0 dev 1 function 0 "SiS 86C201 AGP" rev 0x00
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "SiS 650 VGA" rev 0x00: aperture at
0xf000, size 0x40
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 2 function 0 "SiS 962 ISA" rev 0x25
pciide0 at pci0 dev 2 function 5 "SiS 5513 EIDE" rev 0x00: 651: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0:
wd0: 16-sector PIO, LBA48, 76319MB, 156301488 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
atapiscsi0 at pciide0 channel 1 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <, 54X CD-ROM, 6.53> SCSI0 5/cdrom removable
cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2
auich0 at pci0 dev 2 function 7 "SiS 7012 AC97" rev 0xa0: irq 12,
SiS7012 AC97
ac97: codec id 0x41445370 (Analog Devices AD1980)
ac97: codec features headphone, 20 bit DAC, No 3D Stereo
audio0 at auich0
ohci0 at pci0 dev 3 function 0 "SiS 5597/5598 USB" rev 0x0f: irq 5,
version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: SiS OHCI root hub, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1 at pci0 dev 3 function 1 "SiS 5597/5598 USB" rev 0x0f: irq 9,
version 1.0, legacy support
usb1 at ohci1: USB revision 1.0
uhub1 at usb1
uhub1: SiS OHCI root hub, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
ehci0 at pci0 dev 3 function 3 "SiS 7002 USB" rev 0x00: irq 9
usb2 at ehci0: USB revision 2.0
uhub2 at usb2
uhub2: SiS EHCI root hub, rev 2.00/1.00, addr 1
uhub2: 6 ports with 6 removable, self powered
sis0 at pci0 dev 4 function 0 "SiS 900 10/100BaseTX" rev 0x91: irq 3,
address 00:0c:6e:d8:4a:59
rlphy0 at sis0 phy 1: RTL8201L 10/100 PHY, rev. 1
ppb1 at pci0 dev 14 function 0 "Intel S21152BB PCI-PCI" rev 0x00
pci2 at ppb1 bus 2
dc0 at pci2 dev 4 function 0 "DEC 21142/3" rev 0x41: irq 10, address
00:80:c8:cd:c8:21
nsphyter0 at dc0 phy 1: DP83843 10/100 PHY, rev. 0
dc1 at pci2 dev 5 function 0 "DEC 21142/3" rev 0x41: irq 12, address
00:80:c8:cd:c8:22
nsphyter1 at dc1 phy 1: DP83843 10/100 PHY, rev. 0
dc2 at pci2 dev 6 function 0 "DEC 21142/3" rev 0x41: irq 3, address
00:80:c8:cd:c8:23
nsphyter2 at dc2 phy 1: DP83843 10/100 PHY, rev. 0
dc3 at pci2 dev 7 function 0 "DEC 21142/3" rev 0x41: irq 11, address
00:80:c8:cd:c8:24
nsphyter3 at dc3 ph
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
Daniel Ouellet a icrit :
Ronnie Garcia wrote:
The CPU usage is almost only "interrupt", as you can see on this top
output :
Instead of showing part of your DMESG, all of it would have been better.
You can find it below, for the record.
Anyway, as far as Interrupts are concern, you can try to run the bsd.mp
kernel as the interrupt processing is different in the mp version oppose
to the single core one.
Thats what someone just told me in a private mail. I will try ASAP.
Thanks.
dmesg :
syncing disks... 0
OpenBSD 4.0 (GENERIC) #1107: Sat Sep 16 19:15:58 MDT 2006
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID
real mem = 528035840 (515660K)
avail mem = 473710592 (462608K)
using 4256 buffers containing 26505216 bytes (25884K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(fc) BIOS, date 06/11/03, BIOS32 rev. 0 @
0xf10a0, SMBIOS rev. 2.3 @ 0xf2d10 (44 entries)
bios0: ASUSTeK Computer INC. P4S533MX
apm0 at bios0: Power Management spec V1.2 (BIOS mgmt disabled)
apm0: APM power management enable: unrecognized device ID (9)
apm0: APM engage (device 1): power management disabled (1)
apm0: AC on, battery charge unknown
apm0: flags b0102 dobusy 0 doidle 1
pcibios0 at bios0: rev 2.1 @ 0xf/0x16d2
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf1640/144 (7 entries)
pcibios0: PCI Interrupt Router at 000:02:0 ("SiS 962 ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus
bios0: ROM list: 0xc/0xc000
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "SiS 651 PCI" rev 0x02
ppb0 at pci0 dev 1 function 0 "SiS 86C201 AGP" rev 0x00
pci1 at ppb0 bus 1
vga1 at pci1 dev 0 function 0 "SiS 650 VGA" rev 0x00: aperture at
0xf000, size 0x40
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 2 function 0 "SiS 962 ISA" rev 0x25
pciide0 at pci0 dev 2 function 5 "SiS 5513 EIDE" rev 0x00: 651: DMA,
channel 0 configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0:
wd0: 16-sector PIO, LBA48, 76319MB, 156301488 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
atapiscsi0 at pciide0 channel 1 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <, 54X CD-ROM, 6.53> SCSI0 5/cdrom removable
cd0(pciide0:1:1): using PIO mode 4, Ultra-DMA mode 2
auich0 at pci0 dev 2 function 7 "SiS 7012 AC97" rev 0xa0: irq 12,
SiS7012 AC97
ac97: codec id 0x41445370 (Analog Devices AD1980)
ac97: codec features headphone, 20 bit DAC, No 3D Stereo
audio0 at auich0
ohci0 at pci0 dev 3 function 0 "SiS 5597/5598 USB" rev 0x0f: irq 5,
version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: SiS OHCI root hub, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1 at pci0 dev 3 function 1 "SiS 5597/5598 USB" rev 0x0f: irq 9,
version 1.0, legacy support
usb1 at ohci1: USB revision 1.0
uhub1 at usb1
uhub1: SiS OHCI root hub, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
ehci0 at pci0 dev 3 function 3 "SiS 7002 USB" rev 0x00: irq 9
usb2 at ehci0: USB revision 2.0
uhub2 at usb2
uhub2: SiS EHCI root hub, rev 2.00/1.00, addr 1
uhub2: 6 ports with 6 removable, self powered
sis0 at pci0 dev 4 function 0 "SiS 900 10/100BaseTX" rev 0x91: irq 3,
address 00:0c:6e:d8:4a:59
rlphy0 at sis0 phy 1: RTL8201L 10/100 PHY, rev. 1
ppb1 at pci0 dev 14 function 0 "Intel S21152BB PCI-PCI" rev 0x00
pci2 at ppb1 bus 2
dc0 at pci2 dev 4 function 0 "DEC 21142/3" rev 0x41: irq 10, address
00:80:c8:cd:c8:21
nsphyter0 at dc0 phy 1: DP83843 10/100 PHY, rev. 0
dc1 at pci2 dev 5 function 0 "DEC 21142/3" rev 0x41: irq 12, address
00:80:c8:cd:c8:22
nsphyter1 at dc1 phy 1: DP83843 10/100 PHY, rev. 0
dc2 at pci2 dev 6 function 0 "DEC 21142/3" rev 0x41: irq 3, address
00:80:c8:cd:c8:23
nsphyter2 at dc2 phy 1: DP83843 10/100 PHY, rev. 0
dc3 at pci2 dev 7 function 0 "DEC 21142/3" rev 0x41: irq 11, address
00:80:c8:cd:c8:24
nsphyter3 at dc3 phy 1: DP83843 10/100 PHY, rev. 0
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0:
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
it0 at isa0 port 0x290/8: IT87
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask e365 netmask ff6d ttymask ffef
pctr: user-level cycle counter enabled
dkcsum: wd0 matches BIOS drive 0x80
root on wd0a
rootdev=0x0 rrootdev=0x300 rawdev=0x302
Thanks,
--
Ronnie Garcia
Directeur
ovea
Til
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
Ronnie Garcia wrote: The CPU usage is almost only "interrupt", as you can see on this top output : Instead of showing part of your DMESG, all of it would have been better. Anyway, as far as Interrupts are concern, you can try to run the bsd.mp kernel as the interrupt processing is different in the mp version oppose to the single core one. This is all in the archive as well. Hope this help you or make a difference to you. Daniel
Re: Firewall, high interrupt load, is this a driver problem (dc) ?
Okay, here is a good suggestion from H.B.: almost all should work, my suggestion would be em(4) or sk(4) based ones, dc(4) is also excellent as long as you stick with real 21143s and not the wannabe-clones. You should see what chip that D-link uses. Also, this thread discusses the card you're currently using: http://marc.theaimsgroup.com/?l=openbsd-misc&m=98933188300353&w=2 On Sun, 07 Jan 2007 18:54:07 +0100 Ronnie Garcia <[EMAIL PROTECTED]> wrote: > Hi, > > I recently switched one of our firewalls from Linux to oBSD 4.0. > Its handling approx 8-9 kpps (in+out) on both interfaces. It has a > D-Link DFE-570TX quad ports NIC (dc driver), two ports are used. > On Linux, the CPU was loaded at approx 20% when, and on oBSD, its > actually loaded at ~30%. No big deal, but on Linux we had queueing > (shaping) with TC/HTB, whereas ALTQ is not (yet) enabled on oBSD.
