On 14/11/13 15:21, Leonardo Santagostini wrote:
> Hello misc,
>
> Im doing my final approach to put a production system with
> carp+pfsync+relayd on production.
>
> The point is that im facing some trouble setting more than one ip alias
> address with different vhid and different passwd.
>
> So, this is the scenario.
>
> Im trying to relayd more or less 15 sites so i have conceptual doubts.
>
> 1) is it nesessary to create one carp interface for each one of my
> internals VIP address
> 2) my understanding is that i have to work with pf on my carp interfaces.
>
> I have tried to put two different VIP's on my carp, but whitout lucky.
>
> Here is the homework.
>
> [root@server ~]# uname -a
> OpenBSD server.internaldomain.com 5.4 GENERIC#37 amd64
> [root@server ~]#
>
> [root@server ~]# cat /etc/hostname.em0
> inet 172.19.224.180 255.255.255.0
>
> [root@server ~]# cat /etc/hostname.em1
> inet 172.19.226.231 255.255.255.0 172.19.226.255
>
> [root@server ~]# cat /etc/hostname.carp0
> # inet alias 172.19.224.16 255.255.255.255 172.19.224.255 vhid 1 advskew 10
> carpdev em0 pass Ahsooqu3
> inet alias 172.19.224.131 255.255.255.0 172.19.224.255 vhid 2 advskew 10
> carpdev em0 pass Meixo9oe
> # inet alias 172.19.224.41 255.255.255.255 172.19.224.255 vhid 3 advskew 10
> carpdev em0 pass av5eG9Gi
> # inet alias 172.19.224.40 255.255.255.255 172.19.224.255 vhid 4 advskew 10
> carpdev em0 pass Rei6thai
> # inet alias 172.19.224.181 255.255.255.0 172.19.224.255 vhid 5 advskew 10
> carpdev em0 pass Toobohz3
> # inet alias 172.19.224.182 255.255.255.255 172.19.224.255 vhid 6 adskew 10
> carpdev em0 pass Quahng6U
CARP should look like this (master);
inet 172.19.224.16 255.255.255.0 172.19.224.255 vhid 1 carpdev em0 pass
Ahsooqu3 advskew 0
inet alias 172.19.224.131 255.255.255.255
inet alias 172.19.224.41 255.255.255.255
inet alias 172.19.224.40 255.255.255.255
inet alias 172.19.224.181 255.255.255.255
inet alias 172.19.224.182 255.255.255.255
And (backup);
inet 172.19.224.16 255.255.255.0 172.19.224.255 vhid 1 carpdev em0 pass
Ahsooqu3 advskew 200
inet alias 172.19.224.131 255.255.255.255
inet alias 172.19.224.41 255.255.255.255
inet alias 172.19.224.40 255.255.255.255
inet alias 172.19.224.181 255.255.255.255
inet alias 172.19.224.182 255.255.255.255
And yes the subnet masks for the alias' should be /32 and you will see a
warning in the logs during fail-over. This is fine, the devs just
haven't muted the check warning yet.
You've done it right if 'netstat -rn' shows;
172.19.224.131 127.0.0.1 UGHS 0 0 33152 8 lo0
172.19.224.131/32 172.19.224.131 U 0 0 - 4 carp0
> [root@server ~]# cat /etc/hostname.pfsync0
> up syncdev em1
>
> [root@server ~]# cat /etc/pf.conf
> ext_if="carp0"
You don't refer to CARP as an interface, it is simply a VRRP watchdog
interface (for example you cannot set the MTU on a CARP interface as it
is not really an interface.
Use the physical..
ext_if="em0"
>
> set fingerprints "/etc/pf.os"
> set optimization aggressive
> set limit states 90000
Definitely needs to be higher! try 1 million..
> set limit src-nodes 65000
>
> table <bad_ip> persist
> table <internat_net> persist file "/etc/internal_net"
> table <admitted_net> persist file "/etc/admitted.txt"
>
> # vip1_address = "172.19.224.181"
> # vip2_address = "172.19.224.16"
> vip3_address = "172.19.224.131"
> # vip4_address = "172.19.224.41"
> # vip5_address = "172.19.224.40"
Just to keep you sane remember these rules;
# (SNAT) NATing is done before filtering, 'pass out on $if_ext from
$external_carp_ip1' (public address as src for outbound).
# (DNAT) RDRing is done before filtering, 'pass in on $if_ext from any
to $internal_ip1' (private address as dst for inbound).
OpenBSD_PF_flow
>
> # Dejo de procesar cuando se trata de las redes internas
> pass in quick from <internat_net> to any
>
> # Dejo pasar las ips desde las redes permitidas
> # pass in quick from <admitted_net> to $vip1_address
> pass in quick from <admitted_net> to $vip3_address
>
> # Genero el block
> block in quick from <bad_ip>
Your 'block in quick's should be above your 'pass in quick's!
quick means stop evaluating and do this action now..
> block in log quick on $ext_if proto tcp from any os "NMAP" to any label
> ExtNMAPScan
>
> # Proteccion contra nmap y herramientas similares
> # block in quick on $ext_if proto tcp flags FUP/WEUAPRSF
> block in quick on $ext_if proto tcp flags WEUAPRSF/WEUAPRSF
> block in quick on $ext_if proto tcp flags SRAFU/WEUAPRSF
> block in quick on $ext_if proto tcp flags /WEUAPRSF
> block in quick on $ext_if proto tcp flags SR/SR
> block in quick on $ext_if proto tcp flags SF/SF
> block in quick from urpf-failed
>
>
> # Aplico reglas de DoS y Syn Flood en site1
> # pass in log on $mob_if proto tcp to $vip1_address port www keep state
> (sloppy, max 10000, max-src-nodes 5000, max-src-conn 100, max-src-conn-rate
> 95/2, adaptive .start 6000, adaptive.end 12000, tcp.first
> 15, tcp.opening 5, tcp.established 3600, tcp.closing 5, tcp.finwait 15,
> tcp.closed 15, tcp.tsdiff 5)
Be careful, Direct Server Return does require sloppy states but be aware
that this totally undermines state security!
You still need a firewall on the outside of the load balancer to
sanitize the flows.. Catch 22 as you cannot have the load balancer (with
DSR) on the same box as your PF filtering if you want load balancing
(with DSR) and full security..
I.e. DSR and full state modulation/security are mutually exclusive..
>
> # Aplico reglas de DoS y Syn Flood en site2
> # pass in on $ext_if proto tcp to $vip2_address port www keep state
> (sloppy, max 10000, max-src-nodes 5000, max-src-conn 150, max-src-conn-rate
> 150/3)
>
> # Aplico reglas para site3
> pass in on $ext_if proto tcp to $vip3_address port www keep state (sloppy,
> max 10000, max-src-nodes 5000, max-src-conn 150, max-src-conn-rate 100/3)
>
> # Aplico reglas de DoS y Syn Flood en site4
> # pass in on $ext_if proto tcp to $vip4_address port www keep state
> (sloppy, max 10000, max-src-nodes 5000, max-src-conn 150, max-src-conn-rate
> 100/3)
>
> # Aplico reglas de DoS y Syn Flood en site5
> # pass in on $ext_if proto tcp to $vip5_address port www keep state
> (sloppy, max 10000, max-src-nodes 5000, max-src-conn 150, max-src-conn-rate
> 100/3)
>
> # Anchor Para relayd
> anchor "relayd/*"
>
>
> [root@server ~]# cat /etc/relayd.conf
> # Archivo de configuracion de balanceo
>
> ## Opciones globales
> interval 5
> timeout 500
> prefork 15
> log all
>
> ## Direcciones de las vip
> # address1="172.19.224.16"
> # address2="172.19.224.181"
> address3="172.19.224.131"
> # address4="172.19.224.41"
> # address5="172.19.224.40"
>
> ## Direcciones de los servidores
> wsapp1="172.19.224.200"
> wsapp2="172.19.224.201"
> webcache01="172.19.224.70"
> webcache02="172.19.224.71"
> webcache03="172.19.224.72"
> webcache04="172.19.224.73"
>
> ## Definicion de Tablas
> table <mobileweb> { $wsapp1 $wsapp2 }
> table <webcaches> { $webcache01 $webcache02 $webcache03 $webcache04 }
> table <webcaches1> { $webcache01 }
>
> ## Definicion de protocolos (Filtros)
>
> http protocol "httpSite1" {
>
> header change "Connection" to "close"
> header append "$REMOTE_ADDR" to "X-Forwarded-For"
> cookie hash "sessid"
>
> }
>
> http protocol "httpSite2" {
>
> header change "Connection" to "close"
> header append "$REMOTE_ADDR" to "X-Forwarded-For"
> cookie hash "sessid"
>
> }
>
> http protocol "httpSite3" {
>
> header change "Connection" to "close"
> header append "$REMOTE_ADDR" to "X-Forwarded-For"
> }
>
> http protocol "httpSite4" {
>
> header append "$REMOTE_ADDR" to "X-Forwarded-For"
>
> }
>
> http protocol "httpSite5" {
>
> header append "$REMOTE_ADDR" to "X-Forwarded-For"
>
> }
>
> ## Definicion de los relays
>
> #relay site1 {
> # listen on $address2 port 80
> # protocol "httpSite1"
> # forward to <mobileweb> port 80 mode roundrobin check http
> "/relaycheck/index.php" code 200
> #}
>
> relay site2 {
> listen on $address3 port 80
> protocol "httpSite2"
> forward to <webcaches> port 80 mode roundrobin check http
> "/monitoreo/relayd.txt" code 200
> }
>
> #relay site3 {
> # listen on $address1 port 80
> # protocol "httpSite3"
> # forward to <webcaches> port 80 mode roundrobin check http
> "/monitoreo/relayd.txt" code 200
> #}
>
> #relay site4 {
> # listen on $address4 port 80
> # protocol "httpSite4"
> # forward to <webcaches> port 80 mode roundrobin check http
> "/monitoreo/relayd.txt" code 200
> #}
>
> #relay site5 {
> # listen on $address5 port 80
> # protocol "httpSite5"
> # forward to <webcaches> port 80 mode roundrobin check http
> "/monitoreo/relayd.txt" code 200
> #}
Haven't implemented this myself so can't comment!
Good luck :)
Andy.
>
> [root@server ~]# dmesg
> OpenBSD 5.4 (GENERIC) #37: Tue Jul 30 15:24:05 MDT 2013
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
> real mem = 2130694144 (2031MB)
> avail mem = 2066341888 (1970MB)
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.4 @ 0x7ffffea0 (11 entries)
> bios0: vendor Seabios version "0.5.1" date 01/01/2007
> bios0: Red Hat KVM
> acpi0 at bios0: rev 0
> acpi0: sleep states S5
> acpi0: tables DSDT FACP SSDT APIC
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0
> mpbios0 at bios0: Intel MP Specification 1.4
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Opteron or Athlon 64, 2659.01 MHz
> cpu0:
> FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,PGE,CMOV,PAT,MMX,FXSR,SSE,SSE2,SSE3,POPCNT
> cpu0: smt 0, core 0, package 0
> cpu0: apic clock running at 1000MHz
> cpu at mainbus0: not configured
> mpbios0: bus 0 is type PCI
> mpbios0: bus 1 is type ISA
> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins
> ioapic0: misconfigured as apic 0, remapped to apid 2
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel
> 0 wired to compatibility, channel 1 wired to compatibility
> wd0 at pciide0 channel 0 drive 0: <QEMU HARDDISK>
> wd0: 16-sector PIO, LBA48, 5000MB, 10240000 sectors
> wd0(pciide0:0:0): using PIO mode 0, DMA mode 2
> atapiscsi0 at pciide0 channel 1 drive 0
> scsibus0 at atapiscsi0: 2 targets
> cd0 at scsibus0 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 0.12> ATAPI 5/cdrom
> removable
> cd0(pciide0:1:0): using PIO mode 0
> uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 2 int 11
> piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 2 int
> 9
> iic0 at piixpm0
> iic0: addr 0x4c 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000
> 06=0000 07=0000
> iic0: addr 0x4e 48=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000
> 06=0000 07=0000
> vga1 at pci0 dev 2 function 0 "Cirrus Logic CL-GD5446" rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> em0 at pci0 dev 3 function 0 "Intel 82540EM" rev 0x03: apic 2 int 11,
> address 52:54:00:4a:77:22
> virtio0 at pci0 dev 4 function 0 "Qumranet Virtio Memory" rev 0x00: Virtio
> Memory Balloon Device
> viomb0 at virtio0
> virtio0: apic 2 int 11
> em1 at pci0 dev 5 function 0 "Intel 82540EM" rev 0x03: apic 2 int 10,
> address 52:54:00:1e:57:27
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: probed fifo depth: 0 bytes
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> pckbc0: using irq 12 for aux slot
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> fd0 at fdc0 drive 0: density unknown
> fd1 at fdc0 drive 1: density unknown
> usb0 at uhci0: USB revision 1.0
> uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> nvram: invalid checksum
> vscsi0 at root
> scsibus1 at vscsi0: 256 targets
> softraid0 at root
> scsibus2 at softraid0: 256 targets
> root on wd0a (bf96a0273abe52bf.a) swap on wd0b dump on wd0b
> WARNING: / was not properly unmounted
> clock: unknown CMOS layout
> carp: pfsync0 demoted group carp by 32 to 160 (pfsync init)
> carp: pfsync0 demoted group pfsync by 32 to 32 (pfsync init)
> carp: pfsync0 demoted group carp by 1 to 161 (pfsync bulk start)
> carp: pfsync0 demoted group pfsync by 1 to 33 (pfsync bulk start)
> carp: pfsync0 demoted group carp by -1 to 32 (pfsync bulk fail)
> carp: pfsync0 demoted group pfsync by -1 to 32 (pfsync bulk fail)
> carp: pfsync0 demoted group carp by -32 to 0 (pfsync init)
> carp: pfsync0 demoted group pfsync by -32 to 0 (pfsync init)
> pfsync: failed to receive bulk update
> carp0: state transition: BACKUP -> MASTER
> carp0: state transition: BACKUP -> MASTER
> nd6_na_input: duplicate IP6 address fe80:0005::0200:5eff:fe00:0102
> carp0: state transition: BACKUP -> MASTER
> nd6_na_input: duplicate IP6 address fe80:0005::0200:5eff:fe00:0102
>
>
> Two more things
>
> 1) Sorry for my english, is not my mothers tongue
> 2) Thank you for doing this great operantig system
>
> Saludos / Regards
> Leonardo Santagostini
