Hi, I have a pair of OpenBSD 5.2 VMs running on KVM, they have a carp interface and are running relayd to load balancer http traffic into two webservers (also VMs). While benchmarking the setup with ab, I noticed that the OpenBSD VMs panic'd, I can easily reproduce the panics. Here is a typical stack trace:
uvm_fault(0xfffffe807d0c62a8, 0x0, 0, 1) -> e kernel: page fault trap, code=0 Stopped at somove+0x22: movq 0x78(%rdi),%r14 ddb> somove() at somove+0x22 sowwakeup() at sowwakeup+0x26 tcp_input() at tcp_input+0x2a37 ipv4_input() at ipv4_input+0x584 ipintr() at ipintr+0x7f netintr() at netintr+0xd5 softintr_dispatch() at softintr_dispatch+0x5d Xsoftnet() at Xsoftnet+0x28 --- interrupt --- (null)() at 0xffff800021454e30 end of kernel end trace frame: 0x4043c748, count: -9 ddb> PID PPID PGRP UID S FLAGS WAIT COMMAND 13819 1 13819 0 3 0x80 select sendmail 15713 1 15713 0 3 0x80 ttyin getty 3077 1 3077 0 3 0x80 ttyin getty 1982 1 1982 0 3 0x80 ttyin getty 12235 1 12235 0 3 0x80 ttyin getty 17057 1 17057 0 3 0x80 ttyin getty 23271 1 23271 0 3 0x80 select cron 4619 1 4619 0 3 0x80 select ruby18 13722 1 13722 99 3 0x80 poll sndiod 22844 18069 18069 89 3 0x80 kqread relayd 19323 18069 18069 89 3 0x80 kqread relayd 1643 18069 18069 89 3 0x80 kqread relayd *26499 18069 18069 89 7 0 relayd 18069 9864 18069 89 3 0x80 kqread relayd 10272 9864 10272 89 3 0x80 kqread relayd 13354 9864 13354 89 3 0x80 kqread relayd 9864 1 9864 0 3 0x80 kqread relayd 22085 1 22085 0 3 0x80 select sshd 18165 18463 19253 83 3 0x80 poll ntpd 18463 19253 19253 83 3 0x80 poll ntpd 19253 1 19253 0 3 0x80 poll ntpd 26963 18156 18156 74 3 0x80 bpf pflogd 18156 1 18156 0 3 0x80 netio pflogd 30594 10090 10090 73 2 0x80 syslogd 10090 1 10090 0 3 0x80 netio syslogd 3510 1 3510 77 3 0x80 poll dhclient 20348 1 22482 0 3 0x80 poll dhclient 25124 1 25124 77 3 0x80 poll dhclient 12672 1 22482 0 3 0x80 poll dhclient 13 0 0 0 3 0x100200 aiodoned aiodoned 12 0 0 0 3 0x100200 syncer update 11 0 0 0 3 0x100200 cleaner cleaner 10 0 0 0 3 0x100200 reaper reaper 9 0 0 0 3 0x100200 pgdaemon pagedaemon 8 0 0 0 3 0x100200 bored crypto 7 0 0 0 3 0x100200 pftm pfpurge 6 0 0 0 3 0x100200 usbtsk usbtask 5 0 0 0 3 0x100200 usbatsk usbatsk 4 0 0 0 3 0x100200 acpi0 acpi0 3 0 0 0 3 0x100200 bored syswq 2 0 0 0 3 0x40100200 idle0 1 0 1 0 3 0x80 wait init 0 -1 0 0 3 0x200 scheduler swapper ddb> rebooting... dmesg from same machine: OpenBSD 5.2 (GENERIC) #309: Wed Aug 1 09:58:55 MDT 2012 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC real mem = 2146369536 (2046MB) avail mem = 2066952192 (1971MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfbc4f (10 entries) bios0: vendor QEMU version "QEMU" date 01/01/2007 acpi0 at bios0: rev 0 acpi0: sleep states S3 S4 S5 acpi0: tables DSDT FACP SSDT APIC acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 mpbios at bios0 not configured vmt0 at mainbus0 vmware: open failed, eax=564d5868, ecx=0000001e, edx=00005658 vmt0: failed to open backdoor RPC channel (TCLO protocol) cpu0 at mainbus0: (uniprocessor) cpu0: QEMU Virtual CPU version 0.10.50, 2200.26 MHz cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,NXE,LONG cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility wd0 at pciide0 channel 0 drive 0: <QEMU HARDDISK> wd0: 16-sector PIO, LBA48, 10240MB, 20971520 sectors wd0(pciide0:0:0): using PIO mode 0, DMA mode 2 atapiscsi0 at pciide0 channel 1 drive 0 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 0.10> ATAPI 5/cdrom removable cd0(pciide0:1:0): using PIO mode 0 uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: irq 11 piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: irq 10 iic0 at piixpm0 iic0: addr 0x4c 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000 iic0: addr 0x4e 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000 vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) em0 at pci0 dev 3 function 0 "Intel PRO/1000MT (82540EM)" rev 0x03: irq 11, address 52:54:00:cd:f0:c2 em1 at pci0 dev 4 function 0 "Intel PRO/1000MT (82540EM)" rev 0x03: irq 11, address 52:54:00:2d:c1:8d em2 at pci0 dev 5 function 0 "Intel PRO/1000MT (82540EM)" rev 0x03: irq 10, address 52:54:00:49:2c:e8 "Qumranet Virtio Memory" rev 0x00 at pci0 dev 6 function 0 not configured isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: probed fifo depth: 0 bytes pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) pckbc0: using irq 12 for aux slot wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 0: density unknown fd1 at fdc0 drive 1: density unknown usb0 at uhci0: USB revision 1.0 uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 nvram: invalid checksum mtrr: Pentium Pro MTRR support vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root scsibus2 at softraid0: 256 targets root on wd0a (7e2867c9d162711b.a) swap on wd0b dump on wd0b WARNING: / was not properly unmounted clock: unknown CMOS layout carp0: state transition: BACKUP -> MASTER After advice from my provider I had to do a "bsd -c" and "disable mpbios" to get OpenBSD to boot. relayd.conf: ext_addr="10.201.0.3" www1="192.168.0.4" www2="192.168.0.5" table <webhosts> { $www1 $www2 } relay www { listen on $ext_addr port http forward to <webhosts> port http mode roundrobin check http "/" code 200 } pf.conf: set skip on lo anchor "relayd/*" pass quick on em1 proto pfsync keep state (no-sync) pass on em1 proto carp keep state pass # to establish keep-state block in on ! lo0 proto tcp to port 6000:6010 Thanks, Peter