Hi,
i have a problem with mysql connections between a java app server and a
mysql server (both running linux) and an openbsd loadbalancer/firewall
in between. The loadbalancer is using a relayd redirect to switch to a mysql
backup server.
A couple of times per hour we see dropped database connections on the app
server:
java.net.SocketException MESSAGE: java.net.NoRouteToHostException: No route to
host
with the usual verbose java stack trace...
On the loadbalancer i see these messages (first three octets of the ip-address
are modified):
messages.0.gz:Aug 19 10:20:38 lbmyc-p1 /bsd: pf: BAD state: TCP out wire:
(0) 1.2.3.138:3306 1.2.3.146:52876 stack: (0) - [lo=1716543793
high=1716549681 win=501 modulator=0 wscale=7] [lo=1645585568 high=1645649696
win=46 modulator=0 wscale=7] 4:4 @-1 S seq=1154168068 (1154168068)
ack=1645585568 len=0 ackskew=0 pkts=0:0 dir=out,fwd
messages.0.gz:Aug 19 10:20:39 lbmyc-p1 /bsd: pf: BAD state: TCP out wire:
(0) 1.2.3.138:3306 1.2.3.146:52892 stack: (0) - [lo=1746884471
high=1746890359 win=501 modulator=0 wscale=7] [lo=1665945826 high=1665997794
win=46 modulator=0 wscale=7] 4:4 @-1 S seq=1174090100 (1174090100)
ack=1665945826 len=0 ackskew=0 pkts=0:0 dir=out,fwd
The machine has a carp interface shared with a second machine with identical
setup.
dmesg below, the pfsync message is because i upgraded to the Aug.8 snapshot
only on the master machine. I had the problems before that with 4.9 as well.
I tried reducing the pf tcp.closing timeout without conclusive results.
/Benno
-----------------------------------
OpenBSD 5.0 (GENERIC) #39: Mon Aug 8 14:53:43 MDT 2011
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE
real mem = 2146471936 (2047MB)
avail mem = 2101305344 (2003MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 05/26/10, BIOS32 rev. 0 @ 0xf0010, SMBIOS
rev. 2.6 @ 0x9ac00 (19 entries)
bios0: vendor American Megatrends Inc. version "1.0c" date 05/26/2010
bios0: Supermicro X7SPA-HF
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP MCFG SLIC OEMB HPET EINJ BERT ERST HEST
acpi0: wakeup devices P0P1(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4) USB2(S4)
USB5(S4) EUSB(S4) USB3(S4) USB4(S4) USB6(S4) USBE(S4) P0P4(S4) P0P5(S4)
P0P6(S4) P0P7(S4) P0P8(S4) P0P9(S4) GBE_(S4) SLPB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (P0P1)
acpiprt2 at acpi0: bus 1 (P0P4)
acpiprt3 at acpi0: bus -1 (P0P5)
acpiprt4 at acpi0: bus -1 (P0P6)
acpiprt5 at acpi0: bus -1 (P0P7)
acpiprt6 at acpi0: bus 2 (P0P8)
acpiprt7 at acpi0: bus 3 (P0P9)
acpicpu0 at acpi0
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
mpbios0 at bios0: Intel MP Specification 1.4
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 166MHz
cpu at mainbus0: not configured
mpbios0: bus 0 is type PCI
mpbios0: bus 1 is type PCI
mpbios0: bus 2 is type PCI
mpbios0: bus 3 is type PCI
mpbios0: bus 4 is type PCI
mpbios0: bus 5 is type ISA
ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 24 pins
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 1 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 1 int 21
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 1 int 19
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 1 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: apic 1 int 17
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev 0x06: msi,
address 00:15:17:f0:28:1e
ppb1 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: apic 1 int 17
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi,
address 00:25:90:0a:3f:b2
ppb2 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: apic 1 int 16
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi,
address 00:25:90:0a:3f:b3
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 1 int 23
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 1 int 19
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 1 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 1 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci4 at ppb3 bus 4
vga1 at pci4 dev 4 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02: PM disabled
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.2
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, KINGSTON SS100S2, D100> SCSI3 0/direct
fixed t10.ATA_KINGSTON_SS100S28G_08GB30014990_
sd0: 7641MB, 512 bytes/sector, 15649200 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 1 int 18
iic0 at ichiic0
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627DHG rev 0x25
lm1 at wbsio0 port 0xca0/8: W83627DHG
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
mtrr: Pentium Pro MTRR support
uhidev0 at uhub4 port 2 configuration 1 interface 0 "Winbond Electronics Corp
Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ums0 at uhidev0: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev1 at uhub4 port 2 configuration 1 interface 1 "Winbond Electronics Corp
Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ukbd0 at uhidev1: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (6fc6fe2b8617688e.a) swap on sd0b dump on sd0b
carp: carp0 demoted group carp by 1 to 129 (carpdev)
carp: carp1 demoted group carp by 1 to 130 (carpdev)
carp: pfsync0 demoted group carp by 1 to 131 (pfsync bulk start)
carp: pfsync0 demoted group pfsync by 1 to 1 (pfsync bulk start)
pfsync: requesting bulk update
pfsync: received bulk update start
pfsync: pfsync_input: PFSYNC_ACT_UPD: invalid value
pfsync: pfsync_input: PFSYNC_ACT_UPD: invalid value
--
Sebastian Benoit <benoit-li...@fb12.de>
