I must add to this that I have seen a clear pattern now but I can't explain
it.
Like I said, round-robin redirection for certain ports to certain
hosts. It's very predictable, if netcat worked, next connect won't.
But I'm less sure this is because of the LAN machines now, because of
this tcpdump output here.
lb01 $ tcpdump -nettti pflog0 'tcp'
Oct 17 09:58:44.006881 rule 30/(match) pass in on vic2:
XX.XX.XX.234.52143 > 10.221.181.22.25: S 3081755618:3081755618(0) win
65535
(DF)
tcpdump: WARNING: compensating for unaligned libpcap packets
Oct 17 09:58:44.006909 rule 38/(match) pass out on vic3:
XX.XX.XX.234.52143 > 10.221.181.22.25: S 3081755618:3081755618(0) win
65535
(DF)
Oct 17 09:58:45.261752 rule 30/(match) pass in on vic2:
XX.XX.XX.234.52144 > 10.221.181.21.25: S 523320814:523320814(0) win
65535
(DF)
Oct 17 09:58:45.261786 rule 38/(match) pass out on vic3:
XX.XX.XX.234.52144 > 10.221.181.21.25: S 523320814:523320814(0) win
65535
(DF)
This is when it works, then CARP makes the packets come in on lb02 the
next connection attempt and I get this.
lb02 $ tcpdump -nettti pflog0 'tcp'
Oct 17 09:59:03.349586 rule 12/(match) block in on vic3:
10.221.181.21.25 > XX.XX.XX.234.52144: S 3675863197:3675863197(0) ack
523320815 win 5792 (DF)
And this always happens on lb02, so after adding multiple variations
of pass in on vic3 from 10.221.181.0/24 to any I of course made sure
to check the configurations themselves and they're identical.
I have this script setup now to regularly check that they're identical
but so far they have been without fail.
$ cat bin/compare
#!/bin/sh
ssh lb02 sudo pfctl -sr > /tmp/pf.compare.lb02
sudo pfctl -sr > /tmp/pf.compare.lb01
diff /tmp/pf.compare.lb01 /tmp/pf.compare.lb02
The remote source I'm testing from has been censored by the EPA. ;)
2011/10/17 Stefan Midjich :
> I had this gateway with NAT working fine until I added another for
> load balancing using carp. So now I've been slowly discovering the ins
> and outs of carp in PF rules. Namely that packets seem to be going in
> and out of the physical interfaces, but in on the carp interfaces at
> the same time. Only a detail I've noted with tcpdump.
>
> vic0 is management only, works fine.
> vic1 is pfsync only, set skip
> vic2 is external public ipv4 address, seen here below as XX.XX.XX.59.
> vic3 is the first internal network 10.221.181.0/24
> vic4 is the second internal network 10.221.182.0/24
> carp0 is vic2 on both machines, there is no IP-information on the
> physical carpdev's, only on the carps.
> carp1 is vic3
> carp2 is vic4
>
> There are no pfsync update errors in syslog, I can see carp traffic
> pass between the carpdevs using proto carp.
>
> Both loadbalancers are identical in configuration and I use git,
> public ssh keys and bash scripts to update the configuration on the
> git-server, update it on both loadbalancers and then run pfctl -vf
> /etc/pf.conf on both.
>
> Here is output of pfctl -vvsr, please excuse the mail formatting. I'm
> hoping this will shed light on my ruleset.
>
> @0 block drop on vic0 all
> [ Evaluations: 353 Packets: 0 Bytes: 0 States: 0
]
> [ Inserted: uid 0 pid 3940 State Creations: 0 ]
> @1 pass in quick on vic0 inet proto tcp from any to 10.220.100.0/24
> port = 1022 flags S/SA keep state label "PassInMGMTSSH"
> [ Evaluations: 347 Packets: 0 Bytes: 0 States: 0
]
> [ Inserted: uid 0 pid 3940 State Creations: 0 ]
> @2 pass in quick on vic0 inet proto tcp from any to 10.220.100.0/24
> port = ssh flags S/SA keep state label "PassInMGMTSSH" [ Evaluations:
> 0 Packets: 0 Bytes: 0 States: 0 ]
> [ Inserted: uid 0 pid 3940 State Creations: 0 ]
> @3 pass out quick on vic0 inet proto tcp from 10.220.100.0/24 to any
> port = 1022 flags S/SA keep state label "PassOutMGMTSSH"
> [ Evaluations: 16Packets: 136 Bytes: 19978 States: 0
]
> [ Inserted: uid 0 pid 3940 State Creations: 1 ]
> @4 pass out quick on vic0 inet proto tcp from 10.220.100.0/24 to any
> port = ssh flags S/SA keep state label "PassOutMGMTSSH"
> [ Evaluations: 15Packets: 1020 Bytes: 441024 States: 4
]
> [ Inserted: uid 0 pid 3940 State Creations: 30]
> @5 pass on vic0 proto udp from any to any port = domain keep state
> label "PassMGMTDNS"
> [ Evaluations: 331 Packets: 0 Bytes: 0 States: 0
]
> [ Inserted: uid 0 pid 3940 State Creations: 0 ]
> @6 pass on vic0 inet proto icmp all icmp-type echorep keep state label
> "PassMGMTICMP"
> [ Evaluations: 331 Packets: 0 Bytes: 0 States: 0
]
> [ Inserted: uid 0 pid 3940 State Creations: 0 ]
> @7 pass on vic0 inet proto icmp all icmp-type echoreq keep state label
> "PassMGMTICMP"
> [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0
]
> [ Inserted: uid 0 pid 3940 State Creations: 0 ]
> @8 pass on vic0 inet proto icmp all icmp-type unr
