Re: 4.7 ospfd FIB/RIB synchronization
On Sun, Jul 24 2011 at 27:21, David Gwynne wrote: On 24/07/2011, at 8:27 PM, Jonathan Lassoff wrote: On Wed, Apr 20, 2011 at 7:10 AM, David Gwynne l...@animata.net wrote: On 20/04/2011, at 11:08 PM, Jonathan Lassoff wrote: On Wed, Apr 20, 2011 at 4:22 AM, David Gwynne l...@animata.net wrote: you might be able to upgrade your passive firewall to 4.9 next to the active 4.7 one. it looks like the protocol stayed the same so they should be able to talk to each other. This would seem to be the case. This (http://undeadly.org/cgi?action=articlesid=20090301211402) is an absolutely excellent bit of writing about the improvements to pfsync, BTW. Thanks for letting that be shared. however, it looks like bulk updates were broken in 4.7, which would explain your failover problems. you can work around that by going pfctl -S /dev/stdout | ssh activefw pfctl -L /dev/stdin as root on the passive fw. As an initial seeding of state? It seems to me that only some of my flows get affected when failing over (not everything is reset and traffic can still flow). yes. the pfctl commands will do a bulk update since the in kernel implementation was unreliable back then. It appears that both firewalls have an approximately congruent set of states, but usually a pfctl -ss | wc -l can be off by several hundred, to several thousand states at times. My hunch is that state creation and counter updates are not updated synchronously, so when failing over there are still some updates in-flight, and for flows that are moving their sequence numbers at a decent clip I could see why they might get reset. pf has a bit of fuzz when it does its tcp window matching, so packets can get ahead of the firewall and be ok. Do you know if there is a way to see how much this fuzz is or if there's an offset? from memory its 1000 bytes. If dropped for being out of a window, will (or can) it get logged to pflog? again, from memory its just dropped. i wrote defer, so yes... on my boxes the increase in latency is about .2 to .3ms. if a firewall is missing its peer(s) it will go up to about 1/100th of a second. So does defer wait for a peer to acknowledge a new state just at the time of creation, or does it include state updates about sequence numbers as well? defer only delays the first packet. I suspect I'm hitting a similar issue as you were with long-lived flows getting reset at failover. i think my problem is that i run both firewalls with the carp demotion counter set low. when a box is rebooted the carp default is at 0 or 1, which means it takes over traffic before it gets all the states. later code in rc.local demotes it, but by that time some packets have been eaten by the new box. i should fix it, but im lazy. thats exactly how i have my stuff configured. Have you ever had trouble when re-numbering an interface? It seems to me like ospfd doesn't pick up changes in interface numbering if changed out from under it. Most other OSPF daemons I use would pick this up as it changes, but as far I as can tell there's no way to tell ospfd to reload interface addressing. interfaces and addresses moving around hurts me too. I'm often needing to add more and more interfaces and ospf interfaces, necessitating failing over so as to make it safe to kill and re-start ospfd -- in the process it just seems to nip some flows from flowing. i do that too. lets annoy claudio together! In my world, it happends to change interface numbering. The solution we found is - remove the interface from ospfd.conf, - reload configuration with ospfctl reload - destroy the interface (our ospf interfaces are mainly gif ones), - recreate interface with new IPs - add conf to ospfd.conf, - reload configuration with ospfctl reload This may sound a bit too much, but it works and seems to be reliable for the moment and it does not require to kill and restart the daemon :) Claer
Re: 4.7 ospfd FIB/RIB synchronization
On Wed, Apr 20, 2011 at 7:10 AM, David Gwynne l...@animata.net wrote: On 20/04/2011, at 11:08 PM, Jonathan Lassoff wrote: On Wed, Apr 20, 2011 at 4:22 AM, David Gwynne l...@animata.net wrote: you might be able to upgrade your passive firewall to 4.9 next to the active 4.7 one. it looks like the protocol stayed the same so they should be able to talk to each other. This would seem to be the case. This (http://undeadly.org/cgi?action=articlesid=20090301211402) is an absolutely excellent bit of writing about the improvements to pfsync, BTW. Thanks for letting that be shared. however, it looks like bulk updates were broken in 4.7, which would explain your failover problems. you can work around that by going pfctl -S /dev/stdout | ssh activefw pfctl -L /dev/stdin as root on the passive fw. As an initial seeding of state? It seems to me that only some of my flows get affected when failing over (not everything is reset and traffic can still flow). yes. the pfctl commands will do a bulk update since the in kernel implementation was unreliable back then. It appears that both firewalls have an approximately congruent set of states, but usually a pfctl -ss | wc -l can be off by several hundred, to several thousand states at times. My hunch is that state creation and counter updates are not updated synchronously, so when failing over there are still some updates in-flight, and for flows that are moving their sequence numbers at a decent clip I could see why they might get reset. pf has a bit of fuzz when it does its tcp window matching, so packets can get ahead of the firewall and be ok. Do you know if there is a way to see how much this fuzz is or if there's an offset? If dropped for being out of a window, will (or can) it get logged to pflog? i wrote defer, so yes... on my boxes the increase in latency is about .2 to .3ms. if a firewall is missing its peer(s) it will go up to about 1/100th of a second. So does defer wait for a peer to acknowledge a new state just at the time of creation, or does it include state updates about sequence numbers as well? I suspect I'm hitting a similar issue as you were with long-lived flows getting reset at failover. thats exactly how i have my stuff configured. Have you ever had trouble when re-numbering an interface? It seems to me like ospfd doesn't pick up changes in interface numbering if changed out from under it. Most other OSPF daemons I use would pick this up as it changes, but as far I as can tell there's no way to tell ospfd to reload interface addressing. I'm often needing to add more and more interfaces and ospf interfaces, necessitating failing over so as to make it safe to kill and re-start ospfd -- in the process it just seems to nip some flows from flowing.
Re: 4.7 ospfd FIB/RIB synchronization
On 24/07/2011, at 8:27 PM, Jonathan Lassoff wrote: On Wed, Apr 20, 2011 at 7:10 AM, David Gwynne l...@animata.net wrote: On 20/04/2011, at 11:08 PM, Jonathan Lassoff wrote: On Wed, Apr 20, 2011 at 4:22 AM, David Gwynne l...@animata.net wrote: you might be able to upgrade your passive firewall to 4.9 next to the active 4.7 one. it looks like the protocol stayed the same so they should be able to talk to each other. This would seem to be the case. This (http://undeadly.org/cgi?action=articlesid=20090301211402) is an absolutely excellent bit of writing about the improvements to pfsync, BTW. Thanks for letting that be shared. however, it looks like bulk updates were broken in 4.7, which would explain your failover problems. you can work around that by going pfctl -S /dev/stdout | ssh activefw pfctl -L /dev/stdin as root on the passive fw. As an initial seeding of state? It seems to me that only some of my flows get affected when failing over (not everything is reset and traffic can still flow). yes. the pfctl commands will do a bulk update since the in kernel implementation was unreliable back then. It appears that both firewalls have an approximately congruent set of states, but usually a pfctl -ss | wc -l can be off by several hundred, to several thousand states at times. My hunch is that state creation and counter updates are not updated synchronously, so when failing over there are still some updates in-flight, and for flows that are moving their sequence numbers at a decent clip I could see why they might get reset. pf has a bit of fuzz when it does its tcp window matching, so packets can get ahead of the firewall and be ok. Do you know if there is a way to see how much this fuzz is or if there's an offset? from memory its 1000 bytes. If dropped for being out of a window, will (or can) it get logged to pflog? again, from memory its just dropped. i wrote defer, so yes... on my boxes the increase in latency is about .2 to .3ms. if a firewall is missing its peer(s) it will go up to about 1/100th of a second. So does defer wait for a peer to acknowledge a new state just at the time of creation, or does it include state updates about sequence numbers as well? defer only delays the first packet. I suspect I'm hitting a similar issue as you were with long-lived flows getting reset at failover. i think my problem is that i run both firewalls with the carp demotion counter set low. when a box is rebooted the carp default is at 0 or 1, which means it takes over traffic before it gets all the states. later code in rc.local demotes it, but by that time some packets have been eaten by the new box. i should fix it, but im lazy. thats exactly how i have my stuff configured. Have you ever had trouble when re-numbering an interface? It seems to me like ospfd doesn't pick up changes in interface numbering if changed out from under it. Most other OSPF daemons I use would pick this up as it changes, but as far I as can tell there's no way to tell ospfd to reload interface addressing. interfaces and addresses moving around hurts me too. I'm often needing to add more and more interfaces and ospf interfaces, necessitating failing over so as to make it safe to kill and re-start ospfd -- in the process it just seems to nip some flows from flowing. i do that too. lets annoy claudio together!
Re: 4.7 ospfd FIB/RIB synchronization
you might be able to upgrade your passive firewall to 4.9 next to the active 4.7 one. it looks like the protocol stayed the same so they should be able to talk to each other. however, it looks like bulk updates were broken in 4.7, which would explain your failover problems. you can work around that by going pfctl -S /dev/stdout | ssh activefw pfctl -L /dev/stdin as root on the passive fw. as a matter of interest, are you using ospf for failover on one side of your firewalls? dlg On 20/04/2011, at 2:45 PM, Jonathan Lassoff wrote: On Tue, Apr 19, 2011 at 7:14 PM, David Gwynne l...@animata.net wrote: i had this same problem and fixed it in time for the 4.8 release. is it possible you can upgrade? Do you mean that this was an issue in 4.7 that was fixed in 4.8? I most definitely plan to upgrade (all the way to 4.9, most likely), but am stuck with 4.7 for now, since there's not a hitless way for me to upgrade right now (mostly due to pfsync causing sessions to reset when failing over). Thanks for the pointer. Cheers, jof
Re: 4.7 ospfd FIB/RIB synchronization
On Wed, Apr 20, 2011 at 4:22 AM, David Gwynne l...@animata.net wrote: you might be able to upgrade your passive firewall to 4.9 next to the active 4.7 one. it looks like the protocol stayed the same so they should be able to talk to each other. This would seem to be the case. This (http://undeadly.org/cgi?action=articlesid=20090301211402) is an absolutely excellent bit of writing about the improvements to pfsync, BTW. Thanks for letting that be shared. however, it looks like bulk updates were broken in 4.7, which would explain your failover problems. you can work around that by going pfctl -S /dev/stdout | ssh activefw pfctl -L /dev/stdin as root on the passive fw. As an initial seeding of state? It seems to me that only some of my flows get affected when failing over (not everything is reset and traffic can still flow). It appears that both firewalls have an approximately congruent set of states, but usually a pfctl -ss | wc -l can be off by several hundred, to several thousand states at times. My hunch is that state creation and counter updates are not updated synchronously, so when failing over there are still some updates in-flight, and for flows that are moving their sequence numbers at a decent clip I could see why they might get reset. Have you ever used pfsync with the defer option set? I can imagine that it just takes longer for sessions to start since each firewall would have to wait for the insertion of the state on the other firewall, but I wonder how much latency that adds in practice. Another open question would be what to do in the case of multiple firewalls receiving the multicast update (not applicable for me, but something I'm considering trying). I wonder if there ought to be a hook for defer to count the number of related received state insertion messages it gets before starting. as a matter of interest, are you using ospf for failover on one side of your firewalls? I'm hooking CARP interfaces up into ospfd to signal to my IGP which firewall is active at a given time. ospfd seems to have hooks into CARP which will change LSA metrics based on the CARP state. For the interfaces that these firewalls are announcing into the IGP, CARP is used to direct upstream traffic at the active router. Cheers, jof
Re: 4.7 ospfd FIB/RIB synchronization
On 20/04/2011, at 11:08 PM, Jonathan Lassoff wrote: On Wed, Apr 20, 2011 at 4:22 AM, David Gwynne l...@animata.net wrote: you might be able to upgrade your passive firewall to 4.9 next to the active 4.7 one. it looks like the protocol stayed the same so they should be able to talk to each other. This would seem to be the case. This (http://undeadly.org/cgi?action=articlesid=20090301211402) is an absolutely excellent bit of writing about the improvements to pfsync, BTW. Thanks for letting that be shared. however, it looks like bulk updates were broken in 4.7, which would explain your failover problems. you can work around that by going pfctl -S /dev/stdout | ssh activefw pfctl -L /dev/stdin as root on the passive fw. As an initial seeding of state? It seems to me that only some of my flows get affected when failing over (not everything is reset and traffic can still flow). yes. the pfctl commands will do a bulk update since the in kernel implementation was unreliable back then. It appears that both firewalls have an approximately congruent set of states, but usually a pfctl -ss | wc -l can be off by several hundred, to several thousand states at times. My hunch is that state creation and counter updates are not updated synchronously, so when failing over there are still some updates in-flight, and for flows that are moving their sequence numbers at a decent clip I could see why they might get reset. pf has a bit of fuzz when it does its tcp window matching, so packets can get ahead of the firewall and be ok. also, pf will drop out of window packets rather than send RSTs and such. pfsync will also make a good effort to merge state updates with local changes and will aggressively send updates to its peers when it thinks traffic has recently gone over both legs of a firewall. however, if the bulk update didnt work properly then you can have some missing after failover. if the state doesnt exist then you fall through to the ruleset, pfsync doesnt ask its peers for missing states. this used to affect me with very long lived connections that could be idle for a while (eg, nfs). Have you ever used pfsync with the defer option set? I can imagine that it just takes longer for sessions to start since each firewall would have to wait for the insertion of the state on the other firewall, but I wonder how much latency that adds in practice. i wrote defer, so yes... on my boxes the increase in latency is about .2 to .3ms. if a firewall is missing its peer(s) it will go up to about 1/100th of a second. Another open question would be what to do in the case of multiple firewalls receiving the multicast update (not applicable for me, but something I'm considering trying). I wonder if there ought to be a hook for defer to count the number of related received state insertion messages it gets before starting. the code assumes that if one peer got and acked the update, then all your peers got the update. as a matter of interest, are you using ospf for failover on one side of your firewalls? I'm hooking CARP interfaces up into ospfd to signal to my IGP which firewall is active at a given time. ospfd seems to have hooks into CARP which will change LSA metrics based on the CARP state. For the interfaces that these firewalls are announcing into the IGP, CARP is used to direct upstream traffic at the active router. thats exactly how i have my stuff configured. dlg
4.7 ospfd FIB/RIB synchronization
I'm having a bit of an issue with OpenOSPFd on 4.7 running on i386 hardware. The gist of the problem is that it seems that changes to the kernel routing table and/or interfaces are not being synchronized into the OSPF RIB and LSDB. As an example, I have a CARP interface called carp17 that is configured in /etc/ospfd.conf, and routed like so: # ifconfig carp17 carp17: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr 00:00:5e:00:01:11 description: Foobarbaz CARP priority: 0 carp: BACKUP carpdev vlan17 vhid 17 advbase 1 advskew 100 groups: carp inet6 fe80::200:5eff:fe00:111%carp17 prefixlen 64 scopeid 0x2f inet X.X.186.161 netmask 0xfff0 broadcast X.X.186.175 # netstat -rn | grep X.X.186.160 | grep carp X.X.186.160/28 link#47C 00 - 4 carp17 # ospfctl show fib | grep .186.160 C4 X.X.186.160/28link#47 *O 32 X.X.186.160/28X.X.191.21 # ospfctl show rib | grep .186.160 X.X.186.160/28X.X.191.21 Intra-Area Network 20 11:50:38 So, it's configured with the network X.X.186.160/28. If I try and re-configure this to be a /29: # ifconfig carp17 inet X.X.186.161 netmask 0xfff8 ospfd's FIB reflects the change: # ospfctl show fib | grep 186.160 *O 32 X.X.186.160/28X.X.191.21 C4 X.X.186.160/29link#47 But the RIB does not: # ospfctl show rib | grep 186.160 X.X.186.160/28X.X.191.21 Intra-Area Network 20 12:09:25 I've tried an ospfctl fib reload to no avail. The RIB still doesn't reflect the change: # ospfctl show rib | grep 186.160 X.X.186.160/28X.X.191.21 Intra-Area Network 20 12:11:39 Is there something I could be missing or doing wrong? Should FIB synchronization into OSPF work with 4.7? I'm going off of the changelog and this mailinglist entry: http://marc.info/?l=openbsd-miscm=127616167503271 Cheers, jof
Re: 4.7 ospfd FIB/RIB synchronization
i had this same problem and fixed it in time for the 4.8 release. is it possible you can upgrade? On 20/04/2011, at 9:10 AM, Jonathan Lassoff wrote: I'm having a bit of an issue with OpenOSPFd on 4.7 running on i386 hardware. The gist of the problem is that it seems that changes to the kernel routing table and/or interfaces are not being synchronized into the OSPF RIB and LSDB. As an example, I have a CARP interface called carp17 that is configured in /etc/ospfd.conf, and routed like so: # ifconfig carp17 carp17: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr 00:00:5e:00:01:11 description: Foobarbaz CARP priority: 0 carp: BACKUP carpdev vlan17 vhid 17 advbase 1 advskew 100 groups: carp inet6 fe80::200:5eff:fe00:111%carp17 prefixlen 64 scopeid 0x2f inet X.X.186.161 netmask 0xfff0 broadcast X.X.186.175 # netstat -rn | grep X.X.186.160 | grep carp X.X.186.160/28 link#47C 00 - 4 carp17 # ospfctl show fib | grep .186.160 C4 X.X.186.160/28link#47 *O 32 X.X.186.160/28X.X.191.21 # ospfctl show rib | grep .186.160 X.X.186.160/28X.X.191.21 Intra-Area Network 20 11:50:38 So, it's configured with the network X.X.186.160/28. If I try and re-configure this to be a /29: # ifconfig carp17 inet X.X.186.161 netmask 0xfff8 ospfd's FIB reflects the change: # ospfctl show fib | grep 186.160 *O 32 X.X.186.160/28X.X.191.21 C4 X.X.186.160/29link#47 But the RIB does not: # ospfctl show rib | grep 186.160 X.X.186.160/28X.X.191.21 Intra-Area Network 20 12:09:25 I've tried an ospfctl fib reload to no avail. The RIB still doesn't reflect the change: # ospfctl show rib | grep 186.160 X.X.186.160/28X.X.191.21 Intra-Area Network 20 12:11:39 Is there something I could be missing or doing wrong? Should FIB synchronization into OSPF work with 4.7? I'm going off of the changelog and this mailinglist entry: http://marc.info/?l=openbsd-miscm=127616167503271 Cheers, jof
