Dear list, Recently I built a new VPN hub and it seems I reached a limit in ospfd. The configuration is the following :
2 central OpenBSD (4.7 on production, 4.8 and latest snapshot in our lab). they both run ospfd on LAN side. 49 OpenBSD clients, running IPSEC + gif encapsulation over to each central server. Each client is running ospfd too. Everyone is in area 0.0.0.0. On the 50th client, the central daemon stop to function normally and emit a *LOT* of traffic to each client. The only solution is to kill simultaneously ospfd on each central server and restart the daemon after the packet storm ended. I was able to reproduce the problem with 2 servers : - the first one has a single ospfd daemon for all 50 gif, - the second one has 50 rdomains and each rdomain contains one gif and an ospfd daemon - pf was configured with "pass all" - no IPSEC Note also that the problem only occurs if the ospf states are FULL/P2P, We had to establish each 50 peering in order to reproduce the problem. Nothing useful can be found on the log files, ("ospfd -vd") Here is a sample of what is emitted continuously (look at the timestamp to see how aggressive the flood is) : 17:49:46.220024 10.10.254.140 > 172.16.0.138: 192.168.200.153 > 224.0.0.6: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 58146, len 48) [tos 0xc0] (ttl 64, id 34204, len 68) 17:49:46.220035 10.10.254.140 > 172.16.0.106: 192.168.200.25 > 192.168.200.26: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 9168, len 48) [tos 0xc0] (ttl 64, id 53652, len 68) 17:49:46.220047 10.10.254.140 > 172.16.0.111: 192.168.200.45 > 224.0.0.6: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 34368, len 48) [tos 0xc0] (ttl 64, id 57261, len 68) 17:49:46.220066 10.10.254.140 > 172.16.0.100: 192.168.200.1 > 192.168.200.2: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 22329, len 48) [tos 0xc0] (ttl 64, id 44263, len 68) 17:49:46.220077 10.10.254.140 > 172.16.0.147: 192.168.200.189 > 192.168.200.190: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 39764, len 48) [tos 0xc0] (ttl 64, id 21228, len 68) 17:49:46.220093 10.10.254.140 > 172.16.0.115: 192.168.200.61 > 224.0.0.6: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 65435, len 48) [tos 0xc0] (ttl 64, id 43562, len 68) 17:49:46.220105 10.10.254.140 > 172.16.0.102: 192.168.200.9 > 224.0.0.6: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 21586, len 48) [tos 0xc0] (ttl 64, id 38683, len 68) 17:49:46.220118 10.10.254.140 > 172.16.0.144: 192.168.200.177 > 224.0.0.6: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 8955, len 48) [tos 0xc0] (ttl 64, id 2926, len 68) 17:49:46.220135 10.10.254.140 > 172.16.0.126: 192.168.200.105 > 192.168.200.106: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 52430, len 48) [tos 0xc0] (ttl 64, id 27209, len 68) 17:49:46.220146 10.10.254.140 > 172.16.0.134: 192.168.200.137 > 192.168.200.138: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 18572, len 48) [tos 0xc0] (ttl 64, id 46924, len 68) 17:49:46.220157 10.10.254.140 > 172.16.0.102: 192.168.200.9 > 192.168.200.10: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 26438, len 48) [tos 0xc0] (ttl 64, id 51262, len 68) 17:49:46.220168 10.10.254.140 > 172.16.0.129: 192.168.200.117 > 192.168.200.118: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 8920, len 48) [tos 0xc0] (ttl 64, id 38270, len 68) 17:49:46.220187 10.10.254.140 > 172.16.0.124: 192.168.200.97 > 192.168.200.98: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 61062, len 48) [tos 0xc0] (ttl 64, id 50506, len 68) 17:49:46.220198 10.10.254.140 > 172.16.0.120: 192.168.200.81 > 192.168.200.82: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 38498, len 48) [tos 0xc0] (ttl 64, id 50045, len 68) 17:49:46.220213 10.10.254.140 > 172.16.0.143: 192.168.200.173 > 192.168.200.174: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 28513, len 48) [tos 0xc0] (ttl 64, id 45727, len 68) 17:49:46.220226 10.10.254.140 > 172.16.0.117: 192.168.200.69 > 224.0.0.6: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 53678, len 48) [tos 0xc0] (ttl 64, id 49607, len 68) 17:49:46.220237 10.10.254.140 > 172.16.0.133: 192.168.200.133 > 192.168.200.134: OSPFv2-ls_upd 28: rtrid 10.8.2.53 backbone auth "XXXXXX" [tos 0xc0] [ttl 1] (id 30748, len 48) [tos 0xc0] (ttl 64, id 15801, len 68) Sample master ospfd.conf : password="XXXXXX" router-id 10.8.2.53 auth-key $password auth-type simple no redistribute 10.8.2.48/30 redistribute 10.0.0.0/8 redistribute 10.10.250.128/25 # areas area 0.0.0.0 { interface em1 { metric 1 } interface gif100 interface gif101 interface gif102 interface gif103 interface gif104 interface gif105 interface gif106 interface gif107 interface gif108 interface gif109 ... interface gif148 interface gif149 } Sample remote ospfd.conf : # cat /etc/rdom/ospfd_100.conf rdomain 100 router-id 172.16.0.100 auth-key XXXXXX auth-type simple area 0.0.0.0 { interface gif100 { metric 40 } } Dmesg follows OpenBSD 4.9-beta (GENERIC) #644: Fri Jan 28 16:09:13 MST 2011 t...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel(R) Xeon(R) CPU X3430 @ 2.40GHz ("GenuineIntel" 686-class) 2.40 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,SMX,EST,T M2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT real mem = 2137194496 (2038MB) avail mem = 2092097536 (1995MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 10/15/10, BIOS32 rev. 0 @ 0xf9ea0, SMBIOS rev. 2.6 @ 0x7f79c000 (66 entries) bios0: vendor Dell Inc. version "1.5.2" date 10/15/2010 bios0: Dell Inc. PowerEdge R310 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET DM__ MCFG WD__ SLIC ERST HEST BERT EINJ TCPA SSDT acpi0: wakeup devices PCI0(S5) USBA(S0) USBB(S0) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 133MHz cpu at mainbus0: not configured cpu at mainbus0: not configured cpu at mainbus0: not configured ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins acpihpet0 at acpi0: 14318179 Hz acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 4 (LYD0) acpiprt2 at acpi0: bus 8 (LYD2) acpiprt3 at acpi0: bus -1 (HVD0) acpiprt4 at acpi0: bus -1 (HVD2) acpiprt5 at acpi0: bus 3 (PEX0) acpiprt6 at acpi0: bus -1 (PEX2) acpiprt7 at acpi0: bus -1 (PEX3) acpiprt8 at acpi0: bus 2 (PEX4) acpiprt9 at acpi0: bus 1 (COMP) acpicpu0 at acpi0: C3, C1 bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xc9000/0x2e00 0xec000/0x4000! ipmi at mainbus0 not configured pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 "Intel Core DMI" rev 0x11 ppb0 at pci0 dev 3 function 0 "Intel Core PCIE" rev 0x11: apic 0 int 16 (irq 0) pci1 at ppb0 bus 4 ppb1 at pci1 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0e pci2 at ppb1 bus 5 ppb2 at pci2 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0e pci3 at ppb2 bus 6 em0 at pci3 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int 18 (irq 11), address 00:1b:21:6a:33:28 em1 at pci3 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int 19 (irq 11), address 00:1b:21:6a:33:29 ppb3 at pci2 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0e pci4 at ppb3 bus 7 em2 at pci4 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int 16 (irq 15), address 00:1b:21:6a:33:2c em3 at pci4 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int 17 (irq 10), address 00:1b:21:6a:33:2d ppb4 at pci0 dev 5 function 0 "Intel Core PCIE" rev 0x11: apic 0 int 16 (irq 0) pci5 at ppb4 bus 8 ppb5 at pci5 dev 0 function 0 "IDT 89HPES12N3A" rev 0x0c pci6 at ppb5 bus 9 ppb6 at pci6 dev 2 function 0 "IDT 89HPES12N3A" rev 0x0c pci7 at ppb6 bus 10 em4 at pci7 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int 18 (irq 11), address 00:1b:21:7d:5f:2c em5 at pci7 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int 19 (irq 11), address 00:1b:21:7d:5f:2d ppb7 at pci6 dev 4 function 0 "IDT 89HPES12N3A" rev 0x0c pci8 at ppb7 bus 11 em6 at pci8 dev 0 function 0 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int 16 (irq 15), address 00:1b:21:7d:5f:2e em7 at pci8 dev 0 function 1 "Intel PRO/1000 QP (82576)" rev 0x01: apic 0 int 17 (irq 10), address 00:1b:21:7d:5f:2f "Intel Core Management" rev 0x11 at pci0 dev 8 function 0 not configured "Intel Core Scratch" rev 0x11 at pci0 dev 8 function 1 not configured "Intel Core Control" rev 0x11 at pci0 dev 8 function 2 not configured "Intel Core Misc" rev 0x11 at pci0 dev 8 function 3 not configured "Intel Core QPI Link" rev 0x11 at pci0 dev 16 function 0 not configured "Intel Core QPI Routing" rev 0x11 at pci0 dev 16 function 1 not configured ehci0 at pci0 dev 26 function 0 "Intel 3400 USB" rev 0x05: apic 0 int 22 (irq 14) usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb8 at pci0 dev 28 function 0 "Intel 3400 PCIE" rev 0x05 pci9 at ppb8 bus 3 ppb9 at pci0 dev 28 function 4 "Intel 3400 PCIE" rev 0x05 pci10 at ppb9 bus 2 bnx0 at pci10 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16 (irq 15) bnx1 at pci10 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17 (irq 10) ehci1 at pci0 dev 29 function 0 "Intel 3400 USB" rev 0x05: apic 0 int 22 (irq 14) usb1 at ehci1: USB revision 2.0 uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb10 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5 pci11 at ppb10 bus 1 vga1 at pci11 dev 3 function 0 "Matrox MGA G200eW" rev 0x0a wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 31 function 0 "Intel 3420 LPC" rev 0x05 pciide0 at pci0 dev 31 function 2 "Intel 3400 SATA" rev 0x05: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI pciide0: using apic 0 int 20 (irq 11) for native-PCI interrupt wd0 at pciide0 channel 0 drive 0: <WDC WD1602ABKS-18N8A0> wd0: 16-sector PIO, LBA48, 152587MB, 312500000 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6 wd1 at pciide0 channel 1 drive 0: <WDC WD1602ABKS-18N8A0> wd1: 16-sector PIO, LBA48, 152587MB, 312500000 sectors wd1(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 6 pciide1 at pci0 dev 31 function 5 "Intel 3400 SATA" rev 0x05: DMA, channel 0 wired to native-PCI, channel 1 wired to native-PCI pciide1: using apic 0 int 21 (irq 10) for native-PCI interrupt atapiscsi0 at pciide1 channel 0 drive 0 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: <TSSTcorp, DVD+-RW TS-L633C, D250> ATAPI 5/cdrom removable cd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 kbc: cmd word write error pcppi0 at isa0 port 0x61 spkr0 at pcppi0 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 mtrr: Pentium Pro MTRR support uhub2 at uhub0 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2 uhub3 at uhub2 port 1 "Standard Microsystems product 0x2514" rev 2.00/0.00 addr 3 uhidev0 at uhub3 port 1 configuration 1 interface 0 "Dell Dell USB Keyboard" rev 1.10/1.00 addr 4 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 modifier keys, 6 key codes wskbd0 at ukbd0 mux 1 wskbd0: connecting to wsdisplay0 uhub4 at uhub1 port 1 "Intel Rate Matching Hub" rev 2.00/0.00 addr 2 vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root root on wd0a swap on wd0b dump on wd0b bnx0: address 84:2b:2b:71:65:c4 brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 bnx1: address 84:2b:2b:71:65:c5 brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8