Hey guys, I'm running two HPDL360 G5 servers with OpenBSD 4.6+carp+pf+pfsync as an active/passive firewall pair.
Both are running: (full dmesg at bottom, along with edited pf.conf, in case it's relevant) j...@f2:/home/joe> uname -a OpenBSD f2 4.6 GENERIC.MP#81 amd64 I've had a weird problem happen twice now. It seems after about 4 - 6 weeks of running very happily, both servers lock up completely at the same time. Both consoles show no error messages, but the cursor is blinking away happily. Neither console will take any input and the only remedy is to power cycle them. There is nothing unusual in any of the logfiles. I'm planning on updating them to 4.7 anyway, but is this a problem that people are aware of? Is there a fix? Kind regards DMESG ====================== OpenBSD 4.6 (GENERIC.MP) #81: Thu Jul 9 21:26:19 MDT 2009 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 3746754560 (3573MB) avail mem = 3624001536 (3456MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.6 @ 0xdf7fe000 (127 entries) bios0: vendor HP version "P64" date 07/24/2009 bios0: HP ProLiant DL360 G6 acpi0 at bios0: rev 2 acpi0: tables DSDT FACP SPCR MCFG HPET FFFF SPMI ERST APIC SRAT FFFF BERT HEST DMAR SSDT SSDT SSDT SSDT SSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, 2400.39 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu0: 256KB 64b/line 8-way L2 cache cpu0: apic clock running at 133MHz cpu1 at mainbus0: apid 4 (application processor) cpu1: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, 2400.09 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu1: 256KB 64b/line 8-way L2 cache cpu2 at mainbus0: apid 2 (application processor) cpu2: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, 2400.09 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu2: 256KB 64b/line 8-way L2 cache cpu3 at mainbus0: apid 6 (application processor) cpu3: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, 2400.09 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu3: 256KB 64b/line 8-way L2 cache cpu4 at mainbus0: apid 1 (application processor) cpu4: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, 2400.09 MHz cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu4: 256KB 64b/line 8-way L2 cache cpu5 at mainbus0: apid 5 (application processor) cpu5: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, 2400.09 MHz cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu5: 256KB 64b/line 8-way L2 cache cpu6 at mainbus0: apid 3 (application processor) cpu6: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, 2400.09 MHz cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu6: 256KB 64b/line 8-way L2 cache cpu7 at mainbus0: apid 7 (application processor) cpu7: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz, 2400.09 MHz cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,CX16,xTPR,NXE,LONG cpu7: 256KB 64b/line 8-way L2 cache ioapic0 at mainbus0 apid 8 pa 0xfec00000, version 20, 24 pins ioapic1 at mainbus0 apid 0 pa 0xfec80000, version 20, 24 pins acpiprt0 at acpi0: bus 1 (IP2P) acpiprt1 at acpi0: bus -1 (IPT1) acpiprt2 at acpi0: bus 3 (PT01) acpiprt3 at acpi0: bus 10 (PT02) acpiprt4 at acpi0: bus 7 (PT03) acpiprt5 at acpi0: bus 11 (PT04) acpiprt6 at acpi0: bus 12 (PT05) acpiprt7 at acpi0: bus 13 (PT06) acpiprt8 at acpi0: bus 14 (PT07) acpiprt9 at acpi0: bus 2 (PT08) acpiprt10 at acpi0: bus 4 (PT09) acpiprt11 at acpi0: bus 15 (PT0A) acpiprt12 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0: C3, C3, C1 acpicpu1 at acpi0: C3, C3, C1 acpicpu2 at acpi0: C3, C3, C1 acpicpu3 at acpi0: C3, C3, C1 acpicpu4 at acpi0: C3, C3, C1 acpicpu5 at acpi0: C3, C3, C1 acpicpu6 at acpi0: C3, C3, C1 acpicpu7 at acpi0: C3, C3, C1 acpitz0 at acpi0: critical temperature 31 degC ipmi at mainbus0 not configured cpu0: unknown i686 model 0x1a, can't get bus clock cpu0: EST: unknown system bus clock pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 vendor "Intel", unknown product 0x3406 rev 0x13 ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x13 pci1 at ppb0 bus 3 ciss0 at pci1 dev 0 function 0 "Hewlett-Packard Smart Array" rev 0x01: apic 0 int 4 (irq 7) ciss0: 1 LD, HW rev 2, FW 2.50/2.50, 64bit fifo rro scsibus0 at ciss0: 1 targets sd0 at scsibus0 targ 0 lun 0: <HP, LOGICAL VOLUME, 2.50> SCSI3 0/direct fixed sd0: 139979MB, 512 bytes/sec, 286677120 sec total ppb1 at pci0 dev 2 function 0 "Intel X58 PCIE" rev 0x13 pci2 at ppb1 bus 10 ppb2 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x13 pci3 at ppb2 bus 7 ppb3 at pci0 dev 4 function 0 "Intel X58 PCIE" rev 0x13 pci4 at ppb3 bus 11 ppb4 at pci0 dev 5 function 0 "Intel X58 PCIE" rev 0x13 pci5 at ppb4 bus 12 ppb5 at pci0 dev 6 function 0 "Intel X58 PCIE" rev 0x13 pci6 at ppb5 bus 13 ppb6 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x13 pci7 at ppb6 bus 14 ppb7 at pci0 dev 8 function 0 "Intel X58 PCIE" rev 0x13 pci8 at ppb7 bus 2 bnx0 at pci8 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 0 int 7 (irq 7) bnx1 at pci8 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 0 int 15 (irq 11) ppb8 at pci0 dev 9 function 0 "Intel X58 PCIE" rev 0x13 pci9 at ppb8 bus 4 ppb9 at pci0 dev 10 function 0 "Intel X58 PCIE" rev 0x13 pci10 at ppb9 bus 15 pchb1 at pci0 dev 13 function 0 vendor "Intel", unknown product 0x343a rev 0x13 pchb2 at pci0 dev 13 function 1 vendor "Intel", unknown product 0x343b rev 0x13 pchb3 at pci0 dev 13 function 2 vendor "Intel", unknown product 0x343c rev 0x13 pchb4 at pci0 dev 13 function 3 vendor "Intel", unknown product 0x343d rev 0x13 pchb5 at pci0 dev 13 function 4 vendor "Intel", unknown product 0x3418 rev 0x13 pchb6 at pci0 dev 13 function 5 vendor "Intel", unknown product 0x3419 rev 0x13 pchb7 at pci0 dev 13 function 6 vendor "Intel", unknown product 0x341a rev 0x13 pchb8 at pci0 dev 14 function 0 vendor "Intel", unknown product 0x341c rev 0x13 pchb9 at pci0 dev 14 function 1 vendor "Intel", unknown product 0x341d rev 0x13 pchb10 at pci0 dev 14 function 2 vendor "Intel", unknown product 0x341e rev 0x13 pchb11 at pci0 dev 14 function 3 vendor "Intel", unknown product 0x341f rev 0x13 pchb12 at pci0 dev 14 function 4 vendor "Intel", unknown product 0x3439 rev 0x13 "Intel X58 Misc" rev 0x13 at pci0 dev 20 function 0 not configured "Intel X58 GPIO" rev 0x13 at pci0 dev 20 function 1 not configured "Intel X58 RAS" rev 0x13 at pci0 dev 20 function 2 not configured uhci0 at pci0 dev 29 function 0 "Intel 82801JI USB" rev 0x00: apic 8 int 20 (irq 5) uhci1 at pci0 dev 29 function 1 "Intel 82801JI USB" rev 0x00: apic 8 int 23 (irq 7) uhci2 at pci0 dev 29 function 2 "Intel 82801JI USB" rev 0x00: apic 8 int 22 (irq 10) uhci3 at pci0 dev 29 function 3 "Intel 82801JI USB" rev 0x00: apic 8 int 23 (irq 7) ehci0 at pci0 dev 29 function 7 "Intel 82801JI USB" rev 0x00: apic 8 int 20 (irq 5) usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb10 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x90 pci11 at ppb10 bus 1 vga1 at pci11 dev 3 function 0 "ATI ES1000" rev 0x02 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) radeondrm0 at vga1: apic 8 int 23 (irq 7) drm0 at radeondrm0 "Compaq iLO" rev 0x03 at pci11 dev 4 function 0 not configured "Compaq iLO" rev 0x03 at pci11 dev 4 function 2 not configured uhci4 at pci11 dev 4 function 4 "Hewlett-Packard USB" rev 0x00: apic 8 int 22 (irq 10) "Hewlett-Packard IPMI" rev 0x00 at pci11 dev 4 function 6 not configured usb1 at uhci4: USB revision 1.0 uhub1 at usb1 "Hewlett-Packard UHCI root hub" rev 1.00/1.00 addr 1 pcib0 at pci0 dev 31 function 0 "Intel 82801JIB LPC" rev 0x00 usb2 at uhci0: USB revision 1.0 uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo com1: probed fifo depth: 0 bytes pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: <PC speaker> spkr0 at pcppi0 mtrr: Pentium Pro MTRR support uhidev0 at uhub1 port 1 configuration 1 interface 0 "HP Virtual Keyboard" rev 1.10/0.02 addr 2 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 modifier keys, 6 key codes, country code 33 wskbd1 at ukbd0 mux 1 wskbd1: connecting to wsdisplay0 uhidev1 at uhub1 port 1 configuration 1 interface 1 "HP Virtual Keyboard" rev 1.10/0.02 addr 2 uhidev1: iclass 3/1 ums0 at uhidev1: 3 buttons wsmouse0 at ums0 mux 0 softraid0 at root root on sd0a swap on sd0b dump on sd0b bnx0: address 18:a9:05:76:9c:c8 brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 bnx1: address 18:a9:05:76:9c:ca brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 ======================== pf.conf ======================== # Let's trust localhost set skip on lo # Define our interfaces extif="bnx0" intif="bnx1" # Define our networks intnet = "10.10.0.0/16" pubnet = "XXXX/27" # Define some trusted hosts and networks officenet = "XXXXXXXXXX/28" joeshosts = "XXXXXX" httpvips = "XXXXXXXXX" # Upstream package servers dpkgsrv = "xxxxxxx" archubunt = "XXXXXX" # Martians! - CAREFUL where we use this, it includes our internal 1918 nonroutable = "{ 192.168.0.0/16, 127.0.0.0/8, 172.16.0.0/12, 10.0.0.0/8, 0.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, 204.152.64.0/23, 255.255.255.255/32 }" # Set up some settings and configure for highish-load # note we can up these if we need to, loads of RAM! set block-policy return set loginterface $extif set limit { states 100000, frags 100000, src-nodes 50000 } set optimization normal # loads of RAM, no need to be aggressive set ruleset-optimization basic # Clean stuff up match in all scrub ( reassemble tcp no-df random-id ) # Nat permitted traffic for the rfc1918 only nat on $extif from $intnet to any -> $extif # Block everything in block in log on $extif all block in log on $intif all # Block nonroutables block in quick log on $extif from $nonroutable to any block out quick log on $extif from any to $nonroutable # Enable antispoof antispoof for $extif antispoof for $intif # Once it is in, it is cleared for transit pass out keep state pass in proto icmp keep state # Permit CARPing. pass on { $intif $extif } inet proto carp keep state pass quick on $intif proto pfsync keep state # Pass HTTP stuff in quick pass in quick on $extif proto tcp from any to { $httpvips } \ port { 80 , 443 } keep state # SSH connections pass in on $extif proto tcp from { $officenet, $joeshosts } \ to $extif port 22 keep state pass in on $extif proto tcp from { $officenet, $joeshosts } \ to $pubnet port 22 keep state pass in on $intif proto tcp from any to $intif port 22 keep state pass in on $intif proto tcp from any to $intnet port 22 keep state pass in on $intif proto tcp from any to { $officenet, $joeshosts } \ port 22 keep state # Outbound DNS -- To be removed once we have a caching nameserver pass proto { tcp, udp } from any to any port 53 keep state # Pass in joeshosts to zeus pass in on $extif proto tcp from { $joeshosts } to any \ port { 9090 , 8082 } keep state # Permit access to Ubuntu package server pass in on $intif proto tcp from any to { $dpkgsrv } port 80 keep state pass in on $intif proto tcp from any to { $archubunt } port 80 keep state # Permit NTP out pass in on $intif proto udp from any to any port 123 keep state # Permit SMTP out from the pubnet pass in on $intif proto tcp from any to any port 25 keep state # Permit access to DB from internal networks pass in on $intif proto {tcp,udp} from {$intnet,$pubnet} \ to XXXXXXXX port 3306 keep state # Allow the Zeus AFMs to download updates and new rulesets pass in on $intif proto {tcp,udp} from {$pubnet} \ to XXXXXXXXXX port 80 keep state # Permit access from internal to the office pass in on $intif proto tcp from {$pubnet,$intnet} \ to {$officenet} port {80,443} keep state