Hi again!
I forgot to mention that although carp is configured i do not use there
pfsync (even no pfsync0 interface). (There have been problems with
pfsync, at least in the past and for me).
Imre
On Fri, 2014-04-11 at 15:45 +0300, Imre Oolberg wrote:
> Hi!
>
> I have run two node active-passive cluster with OpenBSD successfully
> over several years, there have been problems too but at least now i know
> how to escape them. Now i run into major issue and i would like to ask
> for help, maybe its software bug or i just manage my system
> inappropriately, i welcome a suggestion what to do differently.
>
> Hardware is
> # sysctl hw
> ..
> hw.machine=amd64
> hw.model=Intel(R) Xeon(R) CPU E5520 @ 2.27GHz
> hw.cpuspeed=2267
> hw.vendor=IBM
> hw.product=49Y6498
> hw.version=00
> hw.serialno=KD8300D
> hw.uuid=ad7604f6-7846-3f90-8cc8-776d95280e29
> hw.physmem=12857503744
> hw.usermem=12857462784
>
> and operating system OpenBSD 5.4 (patched until 005_sha512.patch), dmesg
> is below.
>
> It has about 20 vlan interfaces over four physical interfaces, and 19
> carp interfaces, two vethers, three rdomains (most traffic running in
> default domain). Mostly ipv4 but very little ipv6 also. Besides so to
> say usual processes are
>
> bgpd (at the time problem occured it run only on active node, ipv4 and
> ipv6)
> isakmpd (only on active)
> spamd (only on active)
> dhcpd (only on active)
> ftp-proxy
>
> and from packages are symon and zabbix_agentd. Usually system has 30k
> states and traffic around couple on MBait/s flowing thru it. Number of
> pf rules is 555. Multicore 64 bit kernel.
>
> Being in the middle of transfering one internal vlan over to the other
> firewall system (fortigate) i did the following
>
> 0. killed dhcpd daemon that was using vlan901
> 1. said on the passive ifconfig carp201 destroy
> 2. on the active ifconfig carp201 destroy
> 3. on the passive ifconfig vlan901 destroy
> 4. on the active ifconfig vlan901 destroy
>
> (For afterwards i left to check which rules to delete with pfctl
> -nf /etc/pf.conf and then deleting-rechecking until all right, to that
> part i didnt get).
>
> All good, four interfaces gone, firewalls humming but in five minutes or
> so i lost connection to both firewalls and at the console they said
>
> 1. on active
>
> panic: Data modified on freelist: word 12 of object 0xffff800000efe780
> size 0x28 previous type xform_data (0xdeafbeac != deafbeac)
>
> Stopped at Debugger+0x5: leave
> RUN AT LEAST 'trace' ...
>
> ddb{2}>
>
> 2. on passive
>
> panic: Data modified on freelist: word 12 of object 0xffff800000b5c940
> size 0x38 previous type vnodes (0xdeafbeac != deafbeac)
>
> Stopped at Debugger+0x5: leave
> RUN AT LEAST 'trace' ...
>
> ddb{0}>
>
> They are java remote consoles and they usually get keyboard input but
> unfortunately it wast possible to ask trace, ps etc.
>
> Resolution was to power down passive and recycle power and check
> filesystems on active and it works again, i was lucky it didnt happen at
> noon but in the evening.
>
> I would like to ask if it is a known problem that while configuring carp
> and vlan cerntain conditions are met system may get unstable? And in
> general, what is suggested procedure in removing carp and vlan
> interfaces? Actually i have done it like this (and added interfcases
> too) without problems many times before (at least in erarlier OpenBSD
> versions). Unfortunately i dont have other logs at hand
> (/var/log/messages etc). But from other sources i see there was no
> sudden massive network activity etc.
>
>
> Best regards,
>
> Imre
>
>
> OpenBSD 5.4 (GENERIC.MP) #0: Wed Mar 12 17:16:34 EET 2014
> root@xxx:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> RTC BIOS diagnostic error 80<clock_battery>
> real mem = 12857503744 (12261MB)
> avail mem = 12507508736 (11928MB)
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.5 @ 0x7f6bd000 (74 entries)
> bios0: vendor IBM Corp. version "-[D6E126AUS-1.02]-" date 06/26/2009
> bios0: IBM 49Y6498
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S1 S5
> acpi0: tables DSDT FACP TCPA APIC MCFG SLIC HPET SSDT ERST DMAR
> acpi0: wakeup devices UHC1(S4) UHC2(S4) UHC3(S4) UHC4(S4) UHC5(S4)
> EHC1(S4) EHC2(S4)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz, 2267.02 MHz
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> cpu0: apic clock running at 133MHz
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz, 2266.75 MHz
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 4 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz, 2266.75 MHz
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 6 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz, 2266.75 MHz
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 3, package 0
> cpu4 at mainbus0: apid 1 (application processor)
> cpu4: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz, 2266.75 MHz
> cpu4:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 1, core 0, package 0
> cpu5 at mainbus0: apid 3 (application processor)
> cpu5: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz, 2266.75 MHz
> cpu5:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 1, core 1, package 0
> cpu6 at mainbus0: apid 5 (application processor)
> cpu6: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz, 2266.75 MHz
> cpu6:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC
> cpu6: 256KB 64b/line 8-way L2 cache
> cpu6: smt 1, core 2, package 0
> cpu7 at mainbus0: apid 7 (application processor)
> cpu7: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz, 2266.75 MHz
> cpu7:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,SSE4.2,POPCNT,NXE,LONG,LAHF,PERF,ITSC
> cpu7: 256KB 64b/line 8-way L2 cache
> cpu7: smt 1, core 3, package 0
> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
> acpimcfg0 at acpi0 addr 0x80000000, bus 0-255
> acpihpet0 at acpi0: 14318179 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 11 (PCI1)
> acpiprt2 at acpi0: bus 21 (PCI3)
> acpiprt3 at acpi0: bus -1 (PCI6)
> acpiprt4 at acpi0: bus 26 (PCI7)
> acpiprt5 at acpi0: bus -1 (PCI9)
> acpicpu0 at acpi0
> acpicpu1 at acpi0
> acpicpu2 at acpi0
> acpicpu3 at acpi0
> acpicpu4 at acpi0
> acpicpu5 at acpi0
> acpicpu6 at acpi0
> acpicpu7 at acpi0
> ipmi at mainbus0 not configured
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 5520 Host" rev 0x13
> ppb0 at pci0 dev 1 function 0 "Intel X58 PCIE" rev 0x13
> pci1 at ppb0 bus 11
> bnx0 at pci1 dev 0 function 0 "Broadcom BCM5709" rev 0x20: apic 9 int 4
> bnx1 at pci1 dev 0 function 1 "Broadcom BCM5709" rev 0x20: apic 9 int 16
> ppb1 at pci0 dev 2 function 0 "Intel X58 PCIE" rev 0x13
> pci2 at ppb1 bus 16
> ppb2 at pci0 dev 3 function 0 "Intel X58 PCIE" rev 0x13: msi
> pci3 at ppb2 bus 21
> ppb3 at pci0 dev 7 function 0 "Intel X58 PCIE" rev 0x13: msi
> pci4 at ppb3 bus 26
> em0 at pci4 dev 0 function 0 "Intel 82580" rev 0x01: msi, address
> 90:e2:ba:52:f6:28
> em1 at pci4 dev 0 function 1 "Intel 82580" rev 0x01: msi, address
> 90:e2:ba:52:f6:29
> em2 at pci4 dev 0 function 2 "Intel 82580" rev 0x01: msi, address
> 90:e2:ba:52:f6:2a
> em3 at pci4 dev 0 function 3 "Intel 82580" rev 0x01: msi, address
> 90:e2:ba:52:f6:2b
> "Intel X58 QuickPath" rev 0x13 at pci0 dev 16 function 0 not configured
> "Intel X58 QuickPath" rev 0x13 at pci0 dev 16 function 1 not configured
> "Intel X58 QuickPath" rev 0x13 at pci0 dev 17 function 0 not configured
> "Intel X58 QuickPath" rev 0x13 at pci0 dev 17 function 1 not configured
> "Intel X58 Misc" rev 0x13 at pci0 dev 20 function 0 not configured
> "Intel X58 GPIO" rev 0x13 at pci0 dev 20 function 1 not configured
> "Intel X58 RAS" rev 0x13 at pci0 dev 20 function 2 not configured
> "Intel X58 Throttle" rev 0x13 at pci0 dev 20 function 3 not configured
> vendor "Intel", unknown product 0x342f (class system subclass interrupt,
> rev 0x13) at pci0 dev 21 function 0 not configured
> "Intel X58 QuickData" rev 0x13 at pci0 dev 22 function 0 not configured
> "Intel X58 QuickData" rev 0x13 at pci0 dev 22 function 1 not configured
> "Intel X58 QuickData" rev 0x13 at pci0 dev 22 function 2 not configured
> "Intel X58 QuickData" rev 0x13 at pci0 dev 22 function 3 not configured
> "Intel X58 QuickData" rev 0x13 at pci0 dev 22 function 4 not configured
> "Intel X58 QuickData" rev 0x13 at pci0 dev 22 function 5 not configured
> "Intel X58 QuickData" rev 0x13 at pci0 dev 22 function 6 not configured
> "Intel X58 QuickData" rev 0x13 at pci0 dev 22 function 7 not configured
> uhci0 at pci0 dev 26 function 0 "Intel 82801JI USB" rev 0x00: apic 8 int
> 17
> uhci1 at pci0 dev 26 function 1 "Intel 82801JI USB" rev 0x00: apic 8 int
> 18
> ehci0 at pci0 dev 26 function 7 "Intel 82801JI USB" rev 0x00: apic 8 int
> 19
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb4 at pci0 dev 28 function 0 "Intel 82801JI PCIE" rev 0x00: msi
> pci5 at ppb4 bus 1
> mfi0 at pci5 dev 0 function 0 "Symbios Logic SAS1078" rev 0x04: apic 8
> int 16
> mfi0: "IBM ServeRAID-MR10i SAS/SATA Controller", firmware 11.0.1-0014,
> 256MB cache
> scsibus0 at mfi0: 64 targets
> sd0 at scsibus0 targ 0 lun 0: <IBM, ServeRAID-MR10i, 1.40> SCSI3
> 0/direct fixed naa.600605b001107910123bc5a2187a4e68
> sd0: 139236MB, 512 bytes/sector, 285155328 sectors
> ppb5 at pci0 dev 28 function 4 "Intel 82801JI PCIE" rev 0x00
> pci6 at ppb5 bus 6
> ppb6 at pci6 dev 0 function 0 unknown vendor 0x101b product 0x0452 rev
> 0x01
> pci7 at ppb6 bus 7
> vga1 at pci7 dev 0 function 0 vendor "Matrox", unknown product 0x0530
> rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> uhci2 at pci0 dev 29 function 0 "Intel 82801JI USB" rev 0x00: apic 8 int
> 17
> uhci3 at pci0 dev 29 function 1 "Intel 82801JI USB" rev 0x00: apic 8 int
> 18
> uhci4 at pci0 dev 29 function 2 "Intel 82801JI USB" rev 0x00: apic 8 int
> 19
> ehci1 at pci0 dev 29 function 7 "Intel 82801JI USB" rev 0x00: apic 8 int
> 17
> usb1 at ehci1: USB revision 2.0
> uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb7 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x90
> pci8 at ppb7 bus 31
> pcib0 at pci0 dev 31 function 0 "Intel 82801JIB LPC" rev 0x00
> pciide0 at pci0 dev 31 function 2 "Intel 82801JI SATA" rev 0x00: DMA,
> channel 0 configured to native-PCI, channel 1 configured to native-PCI
> pciide0: using apic 8 int 16 for native-PCI interrupt
> ichiic0 at pci0 dev 31 function 3 "Intel 82801JI SMBus" rev 0x00: apic 8
> int 22
> iic0 at ichiic0
> iic0: addr 0x2e 00=40 words 00=4040 01=0000 02=0000 03=0000 04=0000
> 05=0000 06=0000 07=0000
> pciide1 at pci0 dev 31 function 5 "Intel 82801JI SATA" rev 0x00: DMA,
> channel 0 wired to native-PCI, channel 1 wired to native-PCI
> pciide1: using apic 8 int 21 for native-PCI interrupt
> usb2 at uhci0: USB revision 1.0
> uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb3 at uhci1: USB revision 1.0
> uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb4 at uhci2: USB revision 1.0
> uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb5 at uhci3: USB revision 1.0
> uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb6 at uhci4: USB revision 1.0
> uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> mtrr: Pentium Pro MTRR support
> uhidev0 at uhub1 port 1 configuration 2 interface 0 "IBM IBM Composite
> Device-0" rev 2.00/0.00 addr 2
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> wskbd1: connecting to wsdisplay0
> uhidev1 at uhub1 port 1 configuration 2 interface 1 "IBM IBM Composite
> Device-0" rev 2.00/0.00 addr 2
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhidev2 at uhub1 port 1 configuration 2 interface 2 "IBM IBM Composite
> Device-0" rev 2.00/0.00 addr 2
> uhidev2: iclass 3/1
> ums1 at uhidev2: 3 buttons, Z dir
> wsmouse1 at ums1 mux 0
> urndis0 at uhub4 port 2 configuration 2 interface 0 "IBM RNDIS/CDC
> ETHER" rev 2.00/2.15 addr 2
> urndis0: using Vendor: interface alternate setting 0 failed
> vscsi0 at root
> scsibus1 at vscsi0: 256 targets
> softraid0 at root
> scsibus2 at softraid0: 256 targets
