Re: Problem filtering CARP in PF
On Fri, Mar 02, 2012 at 09:23:38AM +0100, Marios Makassikis wrote: > > The demotion counter is decremented when you lose connectivity (ip_output > errors for instance), but shouldn't it be reincremented when you regain > connectivity? Well, that's a chicken and egg problem there. It won't send out any further advertisements, because its demote count is higher. But without doing so, it cannot know that the error is gone. We could try to solve this by sending some probe carp advertisements using an unused/reserved vhid, but I'm not sure if that's worth the trouble..
Re: Problem filtering CARP in PF
On 2-3-2012 9:23, Marios Makassikis wrote: >> I just thought of something that bit me recently as well. >> >> With a real IPv6 address CARP will send out advertisements via IPv4 >> _and_ IPv6. It's the same CARP message so if either one reaches the >> backup it's ok. >> >> Your block rule had "inet" so you were probably blocking IPv4 only. But >> because of the send errors (due to pf blocking) fw1 started to demote >> itself. >> >> Anyway, you have to block inet6 too if you want to block carp completely. >> > > > Hi, > > I thought of this yesterday shortly after leaving ... > Indeed, after bumping the net.inet.carp.log value even more, there > are messages such as this one in /var/log/messages : > fw1 /bsd: carp0: ip_output failed: 65 > fw1 /bsd: carp0: ip6_output failed: 65 > fw1 /bsd: carp1: ip_output failed: 65 > fw1 /bsd: carp1: ip6_output failed: 65 > > After clearing the states, reloading PF to allow CARP packets, fw1 says it > transitioned from master to backup. I take it's because of the demote > counter: > as soon as I set it back up to 0, fw1 goes back to master and fw2 to backup. Ok! > The demotion counter is decremented when you lose connectivity (ip_output > errors for instance), but shouldn't it be reincremented when you regain > connectivity? Yup, it should. Maybe inet4 clears the errorcounter for inet6 or vice versa. I'll take a look. > I also tested the issue Frederic was having, and it seems to be fixed when > you > flush the rules and states after disabling PF. Hmm, could be an issue as well. It looks like disabling pf does not clear out all states etc. > @Russell > I think either > > pass quick proto carp keep state (no-sync) > > or > > pass quick proto carp no state > > is more appropriate, as it also takes into account pfsync presence and saves > you from debugging issues like the one I was having. As it trips up many people maybe pfctl and/or pf should apply "no-sync" automatically for carp.
Re: Problem filtering CARP in PF
> I just thought of something that bit me recently as well. > > With a real IPv6 address CARP will send out advertisements via IPv4 > _and_ IPv6. It's the same CARP message so if either one reaches the > backup it's ok. > > Your block rule had "inet" so you were probably blocking IPv4 only. But > because of the send errors (due to pf blocking) fw1 started to demote > itself. > > Anyway, you have to block inet6 too if you want to block carp completely. > Hi, I thought of this yesterday shortly after leaving ... Indeed, after bumping the net.inet.carp.log value even more, there are messages such as this one in /var/log/messages : fw1 /bsd: carp0: ip_output failed: 65 fw1 /bsd: carp0: ip6_output failed: 65 fw1 /bsd: carp1: ip_output failed: 65 fw1 /bsd: carp1: ip6_output failed: 65 After clearing the states, reloading PF to allow CARP packets, fw1 says it transitioned from master to backup. I take it's because of the demote counter: as soon as I set it back up to 0, fw1 goes back to master and fw2 to backup. The demotion counter is decremented when you lose connectivity (ip_output errors for instance), but shouldn't it be reincremented when you regain connectivity? I'm guessing the user is expected to do this with something like ifstated if it isn't done automatically. I also tested the issue Frederic was having, and it seems to be fixed when you flush the rules and states after disabling PF. @Russell I think either pass quick proto carp keep state (no-sync) or pass quick proto carp no state is more appropriate, as it also takes into account pfsync presence and saves you from debugging issues like the one I was having. Thank you all for your help. Marios.
Re: Problem filtering CARP in PF
In the spirit of K.I.S.S. I use: pass quick proto carp Since that should match the number on 4 and 6 packets. > Your block rule had "inet" so you were probably blocking IPv4 only. But > because of the send errors (due to pf blocking) fw1 started to demote > itself.
Re: Problem filtering CARP in PF
On 1-3-2012 18:20, Camiel Dobbelaar wrote: > On 1-3-2012 18:10, Marios Makassikis wrote: >> Here you go: >> carp: >> 45808 packets received (IPv4) >> 74835 packets received (IPv6) >> 0 packets discarded for bad interface >> 0 packets discarded for wrong TTL >> 0 packets shorter than header >> 0 discarded for bad checksums >> 0 discarded packets with a bad version >> 0 discarded because packet too short >> 0 discarded for bad authentication >> 32062 discarded for unknown vhid >> 0 discarded because of a bad address list >> 1582 packets sent (IPv4) >> 1582 packets sent (IPv6) I just thought of something that bit me recently as well. With a real IPv6 address CARP will send out advertisements via IPv4 _and_ IPv6. It's the same CARP message so if either one reaches the backup it's ok. Your block rule had "inet" so you were probably blocking IPv4 only. But because of the send errors (due to pf blocking) fw1 started to demote itself. Anyway, you have to block inet6 too if you want to block carp completely. Does that explain it? -- Cam
Re: Problem filtering CARP in PF
On 1-3-2012 18:10, Marios Makassikis wrote:
> Here you go:
> carp:
> 45808 packets received (IPv4)
> 74835 packets received (IPv6)
> 0 packets discarded for bad interface
> 0 packets discarded for wrong TTL
> 0 packets shorter than header
> 0 discarded for bad checksums
> 0 discarded packets with a bad version
> 0 discarded because packet too short
> 0 discarded for bad authentication
> 32062 discarded for unknown vhid
> 0 discarded because of a bad address list
> 1582 packets sent (IPv4)
> 1582 packets sent (IPv6)
> 0 send failed due to mbuf memory error
> 923 transitions to master
>
> The unknown vhid is because there another set of hosts using CARP
> on the same link. The fw1 and fw2 are using the same vhid.
Ok.
>> If I look at the code netinet/ip_carp.c:
>>
>>> error = ip_output(m, NULL, NULL, IP_RAWOUTPUT,
> &sc->sc_imo,
>>> NULL);
>>> if (error) {
>>> if (error == ENOBUFS)
>>> carpstats.carps_onomem++;
>>> else
>>> CARP_LOG(LOG_WARNING, sc,
>>> ("ip_output failed: %d", error));
>>> sc->sc_if.if_oerrors++;
>>> if (sc->sc_sendad_errors < INT_MAX)
>>> sc->sc_sendad_errors++;
>>> if (sc->sc_sendad_errors ==
> CARP_SENDAD_MAX_ERRORS(sc))
>>> carp_group_demote_adj(&sc->sc_if, 1,
>>> "> snderrors");
>>
>>
>> Either the "send failed due to mbuf memory error" counter should be
>> increasing, or an "ip_output failed" error should appear in the log.
>>
>
> As I said before, there's nothing besides state transitions and the snderrors
> I pasted and incorrectly interpreted. From your input, I take it the
> first demotion
> was because I put a 'block all', and as such the CARP packets were
> correctly blocked.
> The value going back to one is most likely due to the fact I reloaded
> PF with a rule
> to let CARP traffic pass. However, that doesn't explain why it isn't
> going back to 0.
Sorry, you need to bump net.inet.carp.log even more to see the
ip_output() error. Can you try net.inet.carp.log=6 ?
It's probably pf blocking the carp packets indeed, but maybe another
error will show up as well.
Re: Problem filtering CARP in PF
Hello,
> No, that's not from your manual commands. It says there are send errors
> when sending out the carp packets.
My bad.
>
> Just paste the output instead of interpreting...
>
Here you go:
carp:
45808 packets received (IPv4)
74835 packets received (IPv6)
0 packets discarded for bad interface
0 packets discarded for wrong TTL
0 packets shorter than header
0 discarded for bad checksums
0 discarded packets with a bad version
0 discarded because packet too short
0 discarded for bad authentication
32062 discarded for unknown vhid
0 discarded because of a bad address list
1582 packets sent (IPv4)
1582 packets sent (IPv6)
0 send failed due to mbuf memory error
923 transitions to master
The unknown vhid is because there another set of hosts using CARP
on the same link. The fw1 and fw2 are using the same vhid.
> If I look at the code netinet/ip_carp.c:
>
>> error = ip_output(m, NULL, NULL, IP_RAWOUTPUT,
&sc->sc_imo,
>> NULL);
>> if (error) {
>> if (error == ENOBUFS)
>> carpstats.carps_onomem++;
>> else
>> CARP_LOG(LOG_WARNING, sc,
>> ("ip_output failed: %d", error));
>> sc->sc_if.if_oerrors++;
>> if (sc->sc_sendad_errors < INT_MAX)
>> sc->sc_sendad_errors++;
>> if (sc->sc_sendad_errors ==
CARP_SENDAD_MAX_ERRORS(sc))
>> carp_group_demote_adj(&sc->sc_if, 1,
>> "> snderrors");
>
>
> Either the "send failed due to mbuf memory error" counter should be
> increasing, or an "ip_output failed" error should appear in the log.
>
As I said before, there's nothing besides state transitions and the snderrors
I pasted and incorrectly interpreted. From your input, I take it the
first demotion
was because I put a 'block all', and as such the CARP packets were
correctly blocked.
The value going back to one is most likely due to the fact I reloaded
PF with a rule
to let CARP traffic pass. However, that doesn't explain why it isn't
going back to 0.
Marios
Re: Problem filtering CARP in PF
On 1-3-2012 16:32, Marios Makassikis wrote:
> Bumping net.inet.carp.log value only reports the demotion:
> carp:carp0 demoted group carp by 1 to 2 (> snderrors)
> carp:carp1 demoted group carp by 1 to 2 (> snderrors)
>
> And then, a few state transitions later:
> carp: carp0 demoted group carp by -1 to 1 (< snderrors)
>
> which corresponds to me trying to reset the demote counter back to 0.
No, that's not from your manual commands. It says there are send errors
when sending out the carp packets.
> 'netstat -sp carp' doesn't give any information I consider useful, besides
> the
> number of IPv4/IPv6 packets sent and received, as well as the number of
> transitions to master.
Just paste the output instead of interpreting...
If I look at the code netinet/ip_carp.c:
> error = ip_output(m, NULL, NULL, IP_RAWOUTPUT, &sc->sc_imo,
> NULL);
> if (error) {
> if (error == ENOBUFS)
> carpstats.carps_onomem++;
> else
> CARP_LOG(LOG_WARNING, sc,
> ("ip_output failed: %d", error));
> sc->sc_if.if_oerrors++;
> if (sc->sc_sendad_errors < INT_MAX)
> sc->sc_sendad_errors++;
> if (sc->sc_sendad_errors ==
> CARP_SENDAD_MAX_ERRORS(sc))
> carp_group_demote_adj(&sc->sc_if, 1,
> "> snderrors");
Either the "send failed due to mbuf memory error" counter should be
increasing, or an "ip_output failed" error should appear in the log.
--
Cam
Re: Problem filtering CARP in PF
Hi, > Are you sure that fw1 is sending and not receiving those? The only way > to be really sure is to use "tcpdump -D out". The sender IP was the one I assigned to fw1, but I retested it anyway with -D out and I can confirm that there is a difference between the demote count displayed by ifconfig and the one transmitted over to fw2. > Not sure what's going on yet, but the following may provide more hints: > - bump net.inet.carp.log to 3 > - check "netstat -s -p carp" > - if you use pfsync, use "no-sync" on the carp pass rules The no-sync shouldn't change anything, as I had previously set 'no state' on the carp rule. pfsync can't sync states that don't exist, can it? :-) Anyway, using either 'no state' or 'no-sync' doesn't change anything. Bumping net.inet.carp.log value only reports the demotion: carp:carp0 demoted group carp by 1 to 2 (> snderrors) carp:carp1 demoted group carp by 1 to 2 (> snderrors) And then, a few state transitions later: carp: carp0 demoted group carp by -1 to 1 (< snderrors) which corresponds to me trying to reset the demote counter back to 0. 'netstat -sp carp' doesn't give any information I consider useful, besides the number of IPv4/IPv6 packets sent and received, as well as the number of transitions to master. Marios
Re: Problem filtering CARP in PF
On 1-3-2012 10:08, Marios Makassikis wrote: > Hello, > No, I'm using hardware machines. > > I tested what Imre suggested, i.e.: flushing PF states with > 'pfctl -F states'. > With a freshly booted machine, CARP packets are allowed to pass. > I then disabled pf, flushed the states and reloaded pf with the > 'block log' rule. At this point, CARP is effectively blocked, and > remains so until a reboot happens: adding a pass rule, clearing states > or even completely disabling pf had no effects. > > I tried using the following rule to let CARP pass: > pass quick log inet proto carp no state > > which leads me to a very odd CARP configuration: fw2 remains master > although it should go back to backup state. advskew on fw1 is 50, and > 100 on fw2. > A closer look at tcpdump revealed that CARP packets sent by fw1 have a > demote value of 1, which explains why fw2 remains master as its demote > value is 0. > What I can't explain is this: > > fw1~# ifconfig -g carp > carp: carp demote count 0 > > In other words, fw1 says it's demotion counter is null, but the messages > sent have a demote value of 1. At the same time, fw1 transitions from > backup to master and back again repeatedly, as I saw in > /var/log/messages. Are you sure that fw1 is sending and not receiving those? The only way to be really sure is to use "tcpdump -D out". > I tried setting the demotion counter to a higher value, say 10, using > 'ifconfig -g carp carpdemote 10' and then I get the expected behavior: > fw2 remains master, fw1 remains backup and the demote value displayed by > ifconfig is the same as the one that is sent to the other machine. Not sure what's going on yet, but the following may provide more hints: - bump net.inet.carp.log to 3 - check "netstat -s -p carp" - if you use pfsync, use "no-sync" on the carp pass rules
Re: Problem filtering CARP in PF
On 2012-03-01, Camiel Dobbelaar wrote: > On 29-2-2012 23:01, Fridiric URBAN wrote: >> Hello, >> >> Confirmed on a fresh and very simple virtual environnement with 2 >> firewall using latest snapshot (amd64). >> pf.conf containt a single line "block log", nothing is logged on pflog >> and the other firewall on the sharing the link layer still catch carp >> advertisement ! > > Virtual eh? I was wondering where the dmesg was. :-) > > If this is esxi you have to allow promiscious mode on the vswitch. The packets must be getting through OK if the other VM is seeing the carp adv's
Re: Problem filtering CARP in PF
Hello, No, I'm using hardware machines. I tested what Imre suggested, i.e.: flushing PF states with 'pfctl -F states'. With a freshly booted machine, CARP packets are allowed to pass. I then disabled pf, flushed the states and reloaded pf with the 'block log' rule. At this point, CARP is effectively blocked, and remains so until a reboot happens: adding a pass rule, clearing states or even completely disabling pf had no effects. I tried using the following rule to let CARP pass: pass quick log inet proto carp no state which leads me to a very odd CARP configuration: fw2 remains master although it should go back to backup state. advskew on fw1 is 50, and 100 on fw2. A closer look at tcpdump revealed that CARP packets sent by fw1 have a demote value of 1, which explains why fw2 remains master as its demote value is 0. What I can't explain is this: fw1~# ifconfig -g carp carp: carp demote count 0 In other words, fw1 says it's demotion counter is null, but the messages sent have a demote value of 1. At the same time, fw1 transitions from backup to master and back again repeatedly, as I saw in /var/log/messages. I tried setting the demotion counter to a higher value, say 10, using 'ifconfig -g carp carpdemote 10' and then I get the expected behavior: fw2 remains master, fw1 remains backup and the demote value displayed by ifconfig is the same as the one that is sent to the other machine. I am not sure how useful a dmesg is in this case, but here is the dmesg.boot from the firewall (with the public IPv6 addresses obscured): http://pastebin.ca/2123109 Marios On 1 March 2012 07:03, Camiel Dobbelaar wrote: > On 29-2-2012 23:01, Fridiric URBAN wrote: >> Hello, >> >> Confirmed on a fresh and very simple virtual environnement with 2 >> firewall using latest snapshot (amd64). >> pf.conf containt a single line "block log", nothing is logged on pflog >> and the other firewall on the sharing the link layer still catch carp >> advertisement ! > > Virtual eh? I was wondering where the dmesg was. :-) > > If this is esxi you have to allow promiscious mode on the vswitch. > > Marios, are you using virtual machinery too? Can you post a dmesg > otherwise? > > > -- > Cam
Re: Problem filtering CARP in PF
On 29-2-2012 23:01, Fridiric URBAN wrote: > Hello, > > Confirmed on a fresh and very simple virtual environnement with 2 > firewall using latest snapshot (amd64). > pf.conf containt a single line "block log", nothing is logged on pflog > and the other firewall on the sharing the link layer still catch carp > advertisement ! Virtual eh? I was wondering where the dmesg was. :-) If this is esxi you have to allow promiscious mode on the vswitch. Marios, are you using virtual machinery too? Can you post a dmesg otherwise? -- Cam
Re: Problem filtering CARP in PF
Hi! On 02/29/12 19:16, Marios Makassikis wrote: A last test prior to posting got me the following results: The pf.conf file contained this rule at the top: block quick log inet proto carp And CARP was effectively blocked. Changing the 'block' to 'pass' allowed the packets to flow, as expected. Changing it back again to block has no effect. I must confess i didnt grasp everything about your setup but this part remindid me of the time i was perplexed about something similar. And my line of thought was then like this 1. test with block rule blocks carp packets 2. test with pass rule passes carp packets, states are created 3. new test with block rule seems to take no effect because packet filter runs stateful and carp packets are passed thru based on states as they should I believe you can control this behaviour how you load new rules i.e. you could flush states first. You could follow states in effect with systat, pftop, and of course with pfctl. Imre PS Using carp you must be attentive which node actually emits carp packets and which one is silent.
Re: Problem filtering CARP in PF
Hello, Confirmed on a fresh and very simple virtual environnement with 2 firewall using latest snapshot (amd64). pf.conf containt a single line "block log", nothing is logged on pflog and the other firewall on the sharing the link layer still catch carp advertisement ! Another interessting information: - If you set pf policy to block log, reboot the system, all carp packet are properly dropped and logged in pflog0! - If you turn off pf "pfctl -d", carp is still filtered (omg !!!). Even if you set pf.conf to "pass" and turn it on again (pfctf -e) you won't be able to allow/see carp again on the system until next reboot ! Fred ! Le 29/02/2012 18:16, Marios Makassikis a icrit : Hi all, I am in the process of setting up a lab to test a IPv6 setup, and I'm having some issues with filtering CARP traffic. The configuration looks like this: +| WAN/Internet |+ || em0||em0 +-+ +-+ | fw1 |-xl0--xl0-| fw2 | +-+ +-+ em1||em1 || || || em0||em0 +-+ +-+ | br1 |-rl0--rl0-| br2 | +-+ +-+ || || ---+---Shared LAN---+--- All four machines above are running OpenBSD. The firewalls (fw1 and fw2) are running OpenBSD 5.0, while the bridges br1 and br2 are running the latest snapshot available on /pub/OpenBSD/snapshots/ (dated 12/02/2012). I configured two CARP devices on the firewalls: - carp0 for the em0 interfaces - carp1 for the em1 interfaces I added the following rule in the pf.conf file, as the default policy is to block everything pass quick inet proto carp At this point, CARP seems to be working fine, but the rule isn't actually working. If I add the 'log' keyword, and run tcpdump -netvi pflog0 ip proto 112 there is no output at all. On the other and, running the following command on fw2 gives the expected output: tcpdump -netvi em1 ip proto 112 At this point I thought maybe some other rule in my ruleset is letting CARP traffic pass, so I replaced the whole ruleset with the following: block log all Sure enough, I still don't have any output on fw1, but fw2 receives CARP packets correctly. I should mention that originally, all machines were running -current, which made me think a regression may have been introduced. After reinstalling 5.0 on the firewalls and copying back the configuration files, the issue seemed to have disappeared, as I could match CARP packets: rule 25/(match) [uid 0, pid 28901] pass out on em1: carp 192.168.200.253 224.0.0.18: CARPv2-advertise 36: vhid=48 advbase=1 advskew=50 demote=2 (DF) [tos 0x10] (ttl 255, id 57018, len 56, bad cksum 0!) rule 25/(match) [uid 0, pid 28901] pass in on em1: carp 192.168.200.252 224.0.0.18: CARPv2-advertise 36: vhid=48 advbase=1 advskew=100 demote=0 (DF) (ttl 255, id 31275, len 56) ... Additionally, I should add that all the machines are dual-stacked. Perhaps this has to do something with the problem, although I have the exact same issue. For instance, 'block all' doesn't actually block, and the one time I had PF matching the IPv4 CARP packets, it also matched the IPv6 ones: rule 34/(match) [uid 0, pid 7854] pass out on em1: fe80::20e:cff:fe68:aad2> ff02::12: CARPv2-advertise 36: vhid=48 advbase=1 advskew=50 demote=0 (len 36, hlim 255) rule 34/(match) [uid 0, pid 7854] pass in on em1: fe80::202:b3ff:feb2:e6ce> ff02::12: CARPv2-advertise 36: vhid=48 advbase=1 advskew=100 demote=0 (len 36, hlim 255) ... Attempts at blocking CARP traffic on the bridge were equally unsuccessful. A last test prior to posting got me the following results: The pf.conf file contained this rule at the top: block quick log inet proto carp And CARP was effectively blocked. Changing the 'block' to 'pass' allowed the packets to flow, as expected. Changing it back again to block has no effect. Can anyone explain this strange behaviour? Thanks, Marios.
Problem filtering CARP in PF
Hi all, I am in the process of setting up a lab to test a IPv6 setup, and I'm having some issues with filtering CARP traffic. The configuration looks like this: +| WAN/Internet |+ || em0||em0 +-+ +-+ | fw1 |-xl0--xl0-| fw2 | +-+ +-+ em1||em1 || || || em0||em0 +-+ +-+ | br1 |-rl0--rl0-| br2 | +-+ +-+ || || ---+---Shared LAN---+--- All four machines above are running OpenBSD. The firewalls (fw1 and fw2) are running OpenBSD 5.0, while the bridges br1 and br2 are running the latest snapshot available on /pub/OpenBSD/snapshots/ (dated 12/02/2012). I configured two CARP devices on the firewalls: - carp0 for the em0 interfaces - carp1 for the em1 interfaces I added the following rule in the pf.conf file, as the default policy is to block everything pass quick inet proto carp At this point, CARP seems to be working fine, but the rule isn't actually working. If I add the 'log' keyword, and run tcpdump -netvi pflog0 ip proto 112 there is no output at all. On the other and, running the following command on fw2 gives the expected output: tcpdump -netvi em1 ip proto 112 At this point I thought maybe some other rule in my ruleset is letting CARP traffic pass, so I replaced the whole ruleset with the following: block log all Sure enough, I still don't have any output on fw1, but fw2 receives CARP packets correctly. I should mention that originally, all machines were running -current, which made me think a regression may have been introduced. After reinstalling 5.0 on the firewalls and copying back the configuration files, the issue seemed to have disappeared, as I could match CARP packets: rule 25/(match) [uid 0, pid 28901] pass out on em1: carp 192.168.200.253 > 224.0.0.18: CARPv2-advertise 36: vhid=48 advbase=1 advskew=50 demote=2 (DF) [tos 0x10] (ttl 255, id 57018, len 56, bad cksum 0!) rule 25/(match) [uid 0, pid 28901] pass in on em1: carp 192.168.200.252 > 224.0.0.18: CARPv2-advertise 36: vhid=48 advbase=1 advskew=100 demote=0 (DF) (ttl 255, id 31275, len 56) ... Additionally, I should add that all the machines are dual-stacked. Perhaps this has to do something with the problem, although I have the exact same issue. For instance, 'block all' doesn't actually block, and the one time I had PF matching the IPv4 CARP packets, it also matched the IPv6 ones: rule 34/(match) [uid 0, pid 7854] pass out on em1: fe80::20e:cff:fe68:aad2 > ff02::12: CARPv2-advertise 36: vhid=48 advbase=1 advskew=50 demote=0 (len 36, hlim 255) rule 34/(match) [uid 0, pid 7854] pass in on em1: fe80::202:b3ff:feb2:e6ce > ff02::12: CARPv2-advertise 36: vhid=48 advbase=1 advskew=100 demote=0 (len 36, hlim 255) ... Attempts at blocking CARP traffic on the bridge were equally unsuccessful. A last test prior to posting got me the following results: The pf.conf file contained this rule at the top: block quick log inet proto carp And CARP was effectively blocked. Changing the 'block' to 'pass' allowed the packets to flow, as expected. Changing it back again to block has no effect. Can anyone explain this strange behaviour? Thanks, Marios.
