Hi list, I'm setting up a replacement for a customers' current Alteon Load Balancers, using OpenBSD, pf, and relayd.
First of all: Thanks, guys, this is faboulous stuff! Having experiences with Linux' LVS and stuff, this is like a very nice, fresh breeze... I remember the Haiku that was posted when pf was born. :) Now, the setup I have has some noteworthinesses. First, I have to create about 600 tables (one for each VIP) that in turn redirect to about 2,200 IPs in the backend (those are not real hosts, most of the hosts have several inet aliases set due to Alteon config necessities). In the meanwhile I tuned relayd.conf massively using parenting, so that there are no unnecessary checks being done (most of them are relatively 'expensive', e.g. 'check ssl' or 'check script', with scripts doing RADIUS logins, etc). The interval is set to 10 seconds, what works for me at a load of about 4 (yes, I read the recent discussion on this). Is that 'too high'? The machine is mostly idling, vmstat output below. There's a problem that 'check send' does not seem to work correctly, for instance. Thusly, I had to code some scripts that check for POP, IMAP, or FTP banners and give an appropriate return code, the same was done for SSL wrapped services. When I use 'check send' [ssl] for one defined service and look at the tcpdump, I can see that it works correctly, i.e. the request is sent to the client and the answer of the client (FTP banner, e.g.) is received by the OpenBSD machine, but relayd says that the check didn't work correctly. Using shell scripts, it works like a charm, but is expensive. For SSL checks (using a shell script that invokes OpenSSL's s_client) I get the following error message (watching 'relayd -v -n') 21415:error:0906D06C:PEM routines:PEM_read_bio:no start line:/usr/src/lib/libssl/src/crypto/pem/pem_lib.c:650:Expecting: TRUSTED CERTIFICATE The script itself is: POP3S_SERVER=$1 POP3S_OKAY=`echo "GET /" | openssl s_client -connect $1:995 2>/dev/null | \ awk '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/{ print }' | \ openssl x509 -text -noout | grep Issuer | wc -l` if [ $POP3S_OKAY -eq 1 ] then exit 1 else exit 0 fi The certificate is okay, and the test also succeeds. So, this is more a 'cosmetic' question... Any hints? Thank in advance and best regards, Joe vstat -m: Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 2761 3383 15390751 1280 102 32 308 332 4576010 640 0 64 1666 1534 1336669 320 95100 128 1148 36 697939 160 10000 256 208 224 3219368 80 15826 512 273 151 1343754 40 24349 1024 245 35 561499 20 55420 2048 1220 4 534068 10 49136 4096 25 3 18484 5 0 8192 6 1 15069580 5 0 16384 6 0 23 5 0 32768 2 0 3 5 0 65536 1 0 1 5 0 Memory usage type by bucket size Size Type(s) 16 devbuf, pcb, routetbl, sysctl, vnodes, UFS mount, dirhash, ACPI, in_multi, exec, xform_data, VM swap, UVM amap, UVM aobj, temp 32 devbuf, pcb, routetbl, ifaddr, UFS mount, sem, dirhash, ACPI, proc, VFS cluster, in_multi, ether_multi, xform_data, VM swap, UVM amap, temp 64 devbuf, pcb, routetbl, vnodes, sem, dirhash, ACPI, in_multi, pfkey data, UVM amap, NDP, temp 128 devbuf, routetbl, ifaddr, iov, vnodes, dirhash, ACPI, NFS srvsock, ttys, inodedep, UVM amap, NDP, temp 256 devbuf, routetbl, ifaddr, sysctl, ioctlops, iov, vnodes, shm, VM map, dirhash, file desc, NFS daemon, exec, newblk, UVM amap, temp 512 devbuf, pcb, ifaddr, ioctlops, iov, UFS mount, shm, dirhash, file desc, proc, ttys, exec, UVM amap, temp 1024 devbuf, ioctlops, iov, mount, ACPI, ttys, exec, UVM amap, UVM aobj, crypto data, temp 2048 devbuf, ifaddr, ioctlops, iov, namecache, UFS mount, proc, VM swap, UVM amap, temp 4096 devbuf, ioctlops, iov, pagedep, UVM amap, memdesc, temp 8192 devbuf, iov, MSDOSFS mount, temp 16384 NFS node, namecache, UFS quota, UFS mount, ISOFS mount, inodedep, indirdep 32768 devbuf 65536 namecache Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) devbuf 2019 2324K 2324K 39322K 2090 0 0 16,32,64,128,256,512,1024,2048,4096,8192,32768 pcb 38 4K 5K 39322K 65124 0 0 16,32,64,512 routetbl 865 98K 107K 39322K 25779 0 0 16,32,64,128,256 ifaddr 72 14K 14K 39322K 73 0 0 32,128,256,512,2048 sysctl 2 1K 1K 39322K 2 0 0 16,256 ioctlops 0 0K 4K 39322K 173421 0 0 256,512,1024,2048,4096 iov 0 0K 8K 39322K 61642 0 0 128,256,512,1024,2048,4096,8192 mount 4 4K 4K 39322K 4 0 0 1024 NFS node 1 16K 16K 39322K 1 0 0 16384 vnodes 49 8K 97K 39322K 5894 0 0 16,64,128,256 namecache 3 82K 82K 39322K 3 0 0 2048,16384,65536 UFS quota 1 16K 16K 39322K 1 0 0 16384 UFS mount 17 35K 35K 39322K 17 0 0 16,32,512,2048,16384 shm 2 1K 1K 39322K 2 0 0 256,512 VM map 2 1K 1K 39322K 2 0 0 256 sem 2 1K 1K 39322K 2 0 0 32,64 dirhash 186 35K 43K 39322K 447 0 0 16,32,64,128,256,512 ACPI 701 41K 45K 39322K 2540 0 0 16,32,64,128,1024 file desc 1 1K 3K 39322K 53838 0 0 256,512 proc 12 5K 5K 39322K 12 0 0 32,512,2048 VFS cluster 0 0K 1K 39322K 34 0 0 32 NFS srvsock 1 1K 1K 39322K 1 0 0 128 NFS daemon 1 1K 1K 39322K 1 0 0 256 in_multi 43 2K 2K 39322K 43 0 0 16,32,64 ether_multi 12 1K 1K 39322K 12 0 0 32 ISOFS mount 1 16K 16K 39322K 1 0 0 16384 MSDOSFS mount 1 8K 8K 39322K 1 0 0 8192 ttys 414 259K 259K 39322K 414 0 0 128,512,1024 exec 0 0K 2K 39322K 729130 0 0 16,256,512,1024 pfkey data 1 1K 1K 39322K 2 0 0 64 xform_data 0 0K 1K 39322K 156310 0 0 16,32 pagedep 1 4K 4K 39322K 1 0 0 4096 inodedep 1 16K 17K 39322K 34 0 0 128,16384 newblk 1 1K 1K 39322K 1 0 0 256 indirdep 0 0K 16K 39322K 17 0 0 16384 VM swap 1 1K 3K 39322K 4 0 0 16,32,2048 UVM amap 2988 167K 387K 39322K 21333380 0 0 16,32,64,128,256,512,1024,2048,4096 UVM aobj 2 2K 2K 39322K 2 0 0 16,1024 memdesc 1 4K 4K 39322K 1 0 0 4096 crypto data 1 1K 1K 39322K 1 0 0 1024 NDP 11 1K 1K 39322K 13 0 0 64,128 temp 415 389K 397K 39322K 20137886 0 0 16,32,64,128,256,512,1024,2048,4096,8192 Memory Totals: In Use Free Requests 3547K 359K 42748183 Memory resource pool statistics Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle extentpl 20 239 0 39 1 0 1 1 0 8 0 phpool 40 961 0 415 5 0 5 5 0 8 0 pmappl 80 439814 0 39 3 0 3 3 0 8 2 vmsppl 188 439814 0 39 7 0 7 7 0 8 5 vmmpepl 88 72975166 0 3692 208 0 208 208 0 179 126 vmmpekpl 88 1551605 0 14 2 0 2 2 0 8 1 aobjpl 52 1 0 1 1 0 1 1 0 8 0 amappl 44 20847488 0 2914 75 0 75 75 0 45 42 anonpl 16 33329130 0 4014 40 0 40 40 0 125 17 bufpl 144 58582 0 12224 469 9 460 469 0 8 8 mbpl 256 7496735 0 160 23 0 23 23 1 384 10 mcl2k 2048 1432711 0 49 47 0 47 47 4 3072 20 sockpl 212 353804 0 127 18 7 11 16 0 8 3 procpl 348 439825 0 50 13 0 13 13 0 8 8 processpl 24 439825 0 50 1 0 1 1 0 8 0 zombiepl 72 439775 0 0 2 0 2 2 0 8 2 ucredpl 80 105414 0 17 1 0 1 1 0 8 0 pgrppl 24 2035 0 28 1 0 1 1 0 8 0 sessionpl 48 1215 0 22 1 0 1 1 0 8 0 pcredpl 24 439825 0 50 1 0 1 1 0 8 0 lockfpl 56 166 0 2 1 0 1 1 0 8 0 filepl 88 4702237 0 122 9 0 9 9 0 8 5 fdescpl 300 439815 0 40 10 0 10 10 0 8 6 pipepl 72 666370 0 12 4 0 4 4 0 8 3 kqueuepl 192 96 0 3 1 0 1 1 0 8 0 knotepl 64 3516384 0 15 2 0 2 2 0 8 1 sigapl 316 439814 0 39 11 0 11 11 0 8 7 wdcspl 96 50646 0 0 1 0 1 1 0 8 1 namei 1024 7359254 0 0 2 0 2 2 0 8 2 vnodes 148 5927 0 5927 220 0 220 220 0 8 0 nchpl 72 2963 0 2963 53 0 53 53 0 8 0 ffsino 184 122944 0 5922 270 0 270 270 0 8 0 dino1pl 128 122944 0 5922 192 0 192 192 0 8 0 pagedeppl 68 149 0 0 1 0 1 1 0 8 1 inodedeppl 84 363 0 0 1 0 1 1 0 8 1 newblkpl 36 653 0 0 1 0 1 1 0 8 1 bmsafemappl 32 193 0 0 1 0 1 1 0 8 1 allocdirectpl 76 627 0 0 2 0 2 2 0 8 2 indirdeppl 28 22 0 0 1 0 1 1 0 8 1 allocindirpl 60 26 0 0 1 0 1 1 0 8 1 freefragpl 36 91 0 0 1 0 1 1 0 8 1 freeblkspl 168 133 0 0 1 0 1 1 0 8 1 freefilepl 28 186 0 0 1 0 1 1 0 8 1 diraddpl 32 209 0 0 1 0 1 1 0 8 1 mkdirpl 28 10 0 0 1 0 1 1 0 8 1 dirrempl 32 199 0 0 1 0 1 1 0 8 1 dirhash 1024 612 0 252 84 0 84 84 0 128 20 pfrulepl 852 4156 0 179 536 491 45 45 0 8 0 pfstatepl 216 163055 0 452 50 0 50 50 0 556 24 pfstatekeypl 72 163055 0 452 26 16 10 16 0 8 1 pfstateitempl 12 163055 0 452 3 0 3 3 0 8 1 pfpooladdrpl 68 4067 0 172 3 0 3 3 0 8 0 pfrktable 1240 11043 0 356 119 0 119 119 0 5000 0 pfrkentry 92 10894 0 541 37 24 13 13 0 8 0 pfosfpen 108 8352 0 696 140 121 19 19 0 8 0 pfosfp 28 4884 0 407 3 0 3 3 0 8 0 rtentpl 116 1659 0 75 6 0 6 6 0 8 1 tcpcbpl 400 230742 0 92 68 55 13 27 0 8 1 tcpqepl 16 19029 0 0 1 0 1 1 0 13 1 sackhlpl 20 2 0 0 1 0 1 1 0 163 1 synpl 184 1035 0 0 1 0 1 1 0 8 1 plimitpl 152 178 0 11 1 0 1 1 0 8 0 inpcbpl 224 288751 0 100 17 7 10 15 0 8 3 In use 6522K, total allocated 32912K; utilization 19.8% dmesg: OpenBSD 4.6-current (GENERIC) #86: Tue Jul 28 23:51:07 MDT 2009 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel(R) Pentium(R) 4 CPU 2.80GHz ("GenuineIntel" 686-class) 2.81 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,CNXT-ID,xTPR real mem = 2146795520 (2047MB) avail mem = 2067046400 (1971MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 01/04/06, BIOS32 rev. 0 @ 0xffe90, SMBIOS rev. 2.3 @ 0xfb030 (83 entries) bios0: vendor Dell Computer Corporation version "A06" date 01/04/2006 bios0: Dell Computer Corporation PowerEdge 750 acpi0 at bios0: rev 0 acpi0: tables DSDT FACP APIC SPCR acpi0: wakeup devices PCI0(S5) PCI1(S5) PCI2(S5) PCI3(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 200MHz cpu at mainbus0: not configured ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins ioapic0: misconfigured as apic 0, remapped to apid 2 ioapic1 at mainbus0: apid 3 pa 0xfec10000, version 20, 24 pins ioapic1: misconfigured as apic 0, remapped to apid 3 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 3 (PCI1) acpiprt2 at acpi0: bus 2 (PCI2) acpiprt3 at acpi0: bus 1 (PCI3) acpicpu0 at acpi0 bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xec000/0x4000! pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 "Intel 82875P Host" rev 0x02 ppb0 at pci0 dev 3 function 0 "Intel 82875P CSA" rev 0x02 pci1 at ppb0 bus 1 em0 at pci1 dev 1 function 0 "Intel PRO/1000CT (82547GI)" rev 0x00: apic 2 int 18 (irq 10), address 00:c0:9f:46:39:87 ppb1 at pci0 dev 28 function 0 "Intel 6300ESB PCIX" rev 0x02 pci2 at ppb1 bus 2 ppb2 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x0a pci3 at ppb2 bus 3 em1 at pci3 dev 2 function 0 "Intel PRO/1000MT (82541GI)" rev 0x00: apic 2 int 21 (irq 7), address 00:c0:9f:46:39:88 xl0 at pci3 dev 3 function 0 "3Com 3c905B 100Base-TX" rev 0x64: apic 2 int 22 (irq 5), address 00:50:da:43:cf:11 bmtphy0 at xl0 phy 24: 3C905B internal PHY, rev. 0 vga1 at pci3 dev 14 function 0 "ATI Rage XL" rev 0x27 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) ichpcib0 at pci0 dev 31 function 0 "Intel 6300ESB LPC" rev 0x02 pciide0 at pci0 dev 31 function 2 "Intel 6300ESB SATA" rev 0x02: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: <TS8GSSD25-S> wd0: 1-sector PIO, LBA, 7627MB, 15621984 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 ichiic0 at pci0 dev 31 function 3 "Intel 6300ESB SMBus" rev 0x02: SMBus disabled isa0 at ichpcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: <PC speaker> spkr0 at pcppi0 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec mtrr: Pentium Pro MTRR support softraid0 at root root on wd0a swap on wd0b dump on wd0b