Re: PF sync doesn't not work very well
On 03/07/2013, at 10:11 PM, Mark Felder f...@feld.me wrote: On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: Hello, no carp is used at this time. pfsync needs to be used with carp... without it you're just playing whack-a-mole with your session table. no it doesnt. pfsync just does its best to keep the state table in sync, it in no way relies on carp to achieve that. however, it does provide feedback to carp to try and avoid the box becoming a master and therefore taking traffic until it either thinks it has the whole state table from a peer or it is alone.
Re: PF sync doesn't not work very well
On 03/07/2013, at 6:23 PM, Loïc Blot loic.b...@unix-experience.fr wrote: Okay, defer is now enabled on pfsync interface (sorry for my last idea, i haven't the man on me :) ). It seems the problem isn't resolved. The transfer starts but blocked at random time. i have hit this too, despite being the person most responsible for trying to make pfsync work in active-active (hi bob!) configurations. the problem is the tcp window tracking pf does, and how pfsync tries to cope with different routers being responsible for different halves of the packet flow. pfsync tries to merge each side of the tcp windows and tries to detect split paths to exchange updates more rapidly for those states. however, i find at some point the actual tcp windows move too fast for pfsync to keep up and all the real packets fall out of the window, causing the stalls you're talking about. my solution is to try and prefer one half of the firewalls for all traffic, and use the second for handling failure. the split path handling works well enough that we can support traffic while we change roles (moving master to slave and slave to master) and the upstream hasnt figured it out yet via ospf. sorry for the bad news. i might try and have a look at the state merge code again and see if there's something obvious i am missing. cheers, dlg -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit : Hi, Thanks for your reply. I wasn't careful about this section. If i understand i must add defer option to my WAN iface (or i'm wrong i must add it to my vlan995 iface ?) ? I will test it this morning, and i return back to misc :) -- Best regards, Loc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le mercredi 03 juillet 2013 02:02 +0200, mxb a crit : pfsync(4) explains this: The pfsync interface will attempt to collapse multiple state updates into a single packet where possible. The maximum number of times a single state can be updated before a pfsync packet will be sent out is controlled by the maxupd parameter and Where more than one firewall might actively handle packets, e.g. with certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to defer transmission of the initial packet of a connection. The pfsync state insert message is sent immediately; the packet is queued until either this message is acknowledged by another system, or a timeout has expired. This behaviour is enabled with the defer parameter to ifconfig(8). Eg. defer: on, yours is off. //mxb On 2 jul 2013, at 21:54, Loc BLOT loic.b...@unix-experience.fr wrote: Hi all I have a strange issue (or i haven't read pfsync correctly but i don't think this is the problem :D) I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site. Those BGP routers are secure with strong PF in stateful mode, and the stateful is working very well on each router. Because of my full mesh BGP configuration, the outgoing layer 7 sessions can leave my network by one router and responses can income by the other. To resolve this issue, i have created a dedidated VLAN for the pfsync traffic and attached pfsync to this VLAN. Here is a sample output of ifconfig on my first router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:10:4a:a6 priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.129 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync And here on my second router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:17:e2:1e priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.130 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync As you see in next tcpdump capture, there is some discussions between the two routers: # tcpdump -nni vlan995 tcpdump: listening on vlan995, link-type EN10MB tcpdump: WARNING: compensating for unaligned libpcap packets 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0
Re: PF sync doesn't not work very well
Hello, I don't know if this may help you, but I have a working BGP setup with two routers active/active. I don't use pfsync, but keep state (sloppy). This is less secure according to pf.conf(5), but that's not really a concern for me as those routers are not my border firewalls... But maybe I am mistaking doing this ? -- Cordialement, Pierre BARDOU -Message d'origine- De : David Gwynne [mailto:da...@gwynne.id.au] Envoyé : jeudi 4 juillet 2013 09:47 À : loic.b...@unix-experience.fr Cc : misc@openbsd.org Objet : Re: PF sync doesn't not work very well On 03/07/2013, at 6:23 PM, Loïc Blot loic.b...@unix-experience.fr wrote: Okay, defer is now enabled on pfsync interface (sorry for my last idea, i haven't the man on me :) ). It seems the problem isn't resolved. The transfer starts but blocked at random time. i have hit this too, despite being the person most responsible for trying to make pfsync work in active-active (hi bob!) configurations. the problem is the tcp window tracking pf does, and how pfsync tries to cope with different routers being responsible for different halves of the packet flow. pfsync tries to merge each side of the tcp windows and tries to detect split paths to exchange updates more rapidly for those states. however, i find at some point the actual tcp windows move too fast for pfsync to keep up and all the real packets fall out of the window, causing the stalls you're talking about. my solution is to try and prefer one half of the firewalls for all traffic, and use the second for handling failure. the split path handling works well enough that we can support traffic while we change roles (moving master to slave and slave to master) and the upstream hasnt figured it out yet via ospf. sorry for the bad news. i might try and have a look at the state merge code again and see if there's something obvious i am missing. cheers, dlg -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit : Hi, Thanks for your reply. I wasn't careful about this section. If i understand i must add defer option to my WAN iface (or i'm wrong i must add it to my vlan995 iface ?) ? I will test it this morning, and i return back to misc :) -- Best regards, Loc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le mercredi 03 juillet 2013 02:02 +0200, mxb a crit : pfsync(4) explains this: The pfsync interface will attempt to collapse multiple state updates into a single packet where possible. The maximum number of times a single state can be updated before a pfsync packet will be sent out is controlled by the maxupd parameter and Where more than one firewall might actively handle packets, e.g. with certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to defer transmission of the initial packet of a connection. The pfsync state insert message is sent immediately; the packet is queued until either this message is acknowledged by another system, or a timeout has expired. This behaviour is enabled with the defer parameter to ifconfig(8). Eg. defer: on, yours is off. //mxb On 2 jul 2013, at 21:54, Loc BLOT loic.b...@unix-experience.fr wrote: Hi all I have a strange issue (or i haven't read pfsync correctly but i don't think this is the problem :D) I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site. Those BGP routers are secure with strong PF in stateful mode, and the stateful is working very well on each router. Because of my full mesh BGP configuration, the outgoing layer 7 sessions can leave my network by one router and responses can income by the other. To resolve this issue, i have created a dedidated VLAN for the pfsync traffic and attached pfsync to this VLAN. Here is a sample output of ifconfig on my first router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:10:4a:a6 priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.129 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync And here on my second router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:17:e2:1e priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.130 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer
Re: PF sync doesn't not work very well
[pfsync w/o carp] * Mark Felder f...@feld.me [2013-07-03 16:37]: First of all, the states of node 1 being synced to node 2 and vice versa is worthless because they have different IP addresses; the states wont match anything. orly. have you actually LOOKED at your state table? pfctl -vvss to the rescue. a tcp connection from 81.209.180.1 to 129.128.5.194 cares about wether the intermediate firewall has 80.86.183.252 or 80.86.183.253 how exactly? how do bridges work at all? miracles all over the place! Secondly, you'll probably end up dealing with the nodes fighting each other as they sync back and forth. If a state from node1 is synced to node2 and node2 decides to expire that session because it hasn't been used it will tell node1 to remove that session as well. sigh. this is completely wrong, too. I've never even attempted to set this up in a lab and I know nothing of the pfsync/pf code, yeah, you made that obvious. but I assume aha. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: PF sync doesn't not work very well
* mxb m...@alumni.chalmers.se [2013-07-03 17:33]: States ARE synced. IPs are not the same on node1 and node2 for external. The you initiated connection to ftp.fr, you done it via node1 with its external IP. On node2 those packets will be DROPPED as those do not belong to external NIC on node2 (IP) again, WRONG. you are caught in your tiny little NATing home firewall scenario. and since the OP said BGP, NAT isn't all that likely there. surprise, surprise: OpenBSD and pf are capable of, good for and actually used (a lot!) for MUCH more, including way bigger setups. Including many that I manage or have helped with, besides the 99.999% I've never seen or heard of. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: PF sync doesn't not work very well
* BARDOU Pierre bardo...@mipih.fr [2013-07-04 14:38]: I don't know if this may help you, but I have a working BGP setup with two routers active/active. I don't use pfsync, but keep state (sloppy). This is less secure according to pf.conf(5), but that's not really a concern for me as those routers are not my border firewalls... But maybe I am mistaking doing this ? no. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: PF sync doesn't not work very well
My apologies for just being noise; I missed his first full post with much more detail. I was picturing him trying to run redundant servers without CARP and running into issues of states disappearing.
Re: PF sync doesn't not work very well
Henning, with all respect(!), I'd cut you off with this home NATing. My home is far more simple than need of active-active CARP (IT IS NOT as of writing) With all respect to ALL devs working and pushing new code upstreams, we still have MP-problems. For sure, I'm not the one to fix this - I take simpler approach to donate my hw and test time. But there are bug to be FIXED //mxb On 4 jul 2013, at 20:07, Henning Brauer lists-open...@bsws.de wrote: * mxb m...@alumni.chalmers.se [2013-07-03 17:33]: States ARE synced. IPs are not the same on node1 and node2 for external. The you initiated connection to ftp.fr, you done it via node1 with its external IP. On node2 those packets will be DROPPED as those do not belong to external NIC on node2 (IP) again, WRONG. you are caught in your tiny little NATing home firewall scenario. and since the OP said BGP, NAT isn't all that likely there. surprise, surprise: OpenBSD and pf are capable of, good for and actually used (a lot!) for MUCH more, including way bigger setups. Including many that I manage or have helped with, besides the 99.999% I've never seen or heard of. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de, Full-Service ISP Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed Henning Brauer Consulting, http://henningbrauer.com/
Re: PF sync doesn't not work very well
Hello all, thanks for this interesting debate about pf syncing. To remember my initial question: pfsync seems to sync states but not correctly on my BGP+OSPF routers. Because each BGP router is master/standby to 2 neighbors (full meshed bgp) packets which are outgoing by one router can income by the other router, then if i want to use pf as a stateful firewall i must use pfsync to sync created states from router A to router B. If you tell me it's not possible, then i will use pf as a stateless firewall. -- Best regards, Loïc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le jeudi 04 juillet 2013 à 13:17 -0500, Mark Felder a écrit : My apologies for just being noise; I missed his first full post with much more detail. I was picturing him trying to run redundant servers without CARP and running into issues of states disappearing. [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]
Re: PF sync doesn't not work very well
On Thu, 04 Jul 2013 21:30:56 +0200 Loïc BLOT loic.b...@unix-experience.fr wrote: Hello all, thanks for this interesting debate about pf syncing. To remember my initial question: pfsync seems to sync states but not correctly on my BGP+OSPF routers. Because each BGP router is master/standby to 2 neighbors (full meshed bgp) packets which are outgoing by one router can income by the other router, then if i want to use pf as a stateful firewall i must use pfsync to sync created states from router A to router B. If you tell me it's not possible, then i will use pf as a stateless firewall. If you speak Serbian, I have just documented my BGP / OSPF / CARP / PFSYNC setup: https://www.mimar.rs/openbsd-na-obodu-korporacijske-mreze/ Maybe google translate is good enough to make it understandable.
Re: PF sync doesn't not work very well
you could try using sloppy states like henning suggested. you'll still get to write stateful rules and get the tcp state machine checks but not the tcp window checks. if it works with sloppy states it narrows the issue down to the pfsync state merge code. at the moment im kind of guessing thats where the problem is. dlg On 05/07/2013, at 5:30 AM, Loïc BLOT loic.b...@unix-experience.fr wrote: Hello all, thanks for this interesting debate about pf syncing. To remember my initial question: pfsync seems to sync states but not correctly on my BGP+OSPF routers. Because each BGP router is master/standby to 2 neighbors (full meshed bgp) packets which are outgoing by one router can income by the other router, then if i want to use pf as a stateful firewall i must use pfsync to sync created states from router A to router B. If you tell me it's not possible, then i will use pf as a stateless firewall. -- Best regards, Loïc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le jeudi 04 juillet 2013 à 13:17 -0500, Mark Felder a écrit : My apologies for just being noise; I missed his first full post with much more detail. I was picturing him trying to run redundant servers without CARP and running into issues of states disappearing. [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]
Re: PF sync doesn't not work very well
Hi, Thanks for your reply. I wasn't careful about this section. If i understand i must add defer option to my WAN iface (or i'm wrong i must add it to my vlan995 iface ?) ? I will test it this morning, and i return back to misc :) -- Best regards, Loïc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 02:02 +0200, mxb a écrit : pfsync(4) explains this: ⦠The pfsync interface will attempt to collapse multiple state updates into a single packet where possible. The maximum number of times a single state can be updated before a pfsync packet will be sent out is controlled by the maxupd parameter ⦠and ⦠Where more than one firewall might actively handle packets, e.g. with certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to defer transmission of the initial packet of a connection. The pfsync state insert message is sent immediately; the packet is queued until either this message is acknowledged by another system, or a timeout has expired. This behaviour is enabled with the defer parameter to ifconfig(8). ⦠Eg. defer: on, yours is off. //mxb On 2 jul 2013, at 21:54, Loïc BLOT loic.b...@unix-experience.fr wrote: Hi all I have a strange issue (or i haven't read pfsync correctly but i don't think this is the problem :D) I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site. Those BGP routers are secure with strong PF in stateful mode, and the stateful is working very well on each router. Because of my full mesh BGP configuration, the outgoing layer 7 sessions can leave my network by one router and responses can income by the other. To resolve this issue, i have created a dedidated VLAN for the pfsync traffic and attached pfsync to this VLAN. Here is a sample output of ifconfig on my first router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:10:4a:a6 priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.129 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync And here on my second router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:17:e2:1e priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.130 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync As you see in next tcpdump capture, there is some discussions between the two routers: # tcpdump -nni vlan995 tcpdump: listening on vlan995, link-type EN10MB tcpdump: WARNING: compensating for unaligned libpcap packets 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36 act DEL ST COMP count 1 id: 51d16a356c33 creatorid: a10bbd21 (DF) [tos 0x10] 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116 act UPD ST COMP count 13 ... (DF) [tos 0x10] The problem is simple, when i initiate a stateful connection from one server, the return (by second router) is blocked by PF (i see the return with pflog0) To be precise here is an example (and tested path): OBSD NTP - OBSD router 1 - WAN...ftp.fr.openbsd.org...WAN - OBSD router 2 || blocked PF allow in/out routing traffic from this server but incoming from WAN is blocked by default Can you confirm to me that pfsync may add a state for outgoing tcp connection in the second router when the first router add it ? Have you got any idea on this issue ? -- Best regards, Loïc BLOT, UNIX systems, security and network
Re: PF sync doesn't not work very well
Okay, defer is now enabled on pfsync interface (sorry for my last idea, i haven't the man on me :) ). It seems the problem isn't resolved. The transfer starts but blocked at random time. -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit : Hi, Thanks for your reply. I wasn't careful about this section. If i understand i must add defer option to my WAN iface (or i'm wrong i must add it to my vlan995 iface ?) ? I will test it this morning, and i return back to misc :) -- Best regards, Loc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le mercredi 03 juillet 2013 02:02 +0200, mxb a crit : pfsync(4) explains this: The pfsync interface will attempt to collapse multiple state updates into a single packet where possible. The maximum number of times a single state can be updated before a pfsync packet will be sent out is controlled by the maxupd parameter and Where more than one firewall might actively handle packets, e.g. with certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to defer transmission of the initial packet of a connection. The pfsync state insert message is sent immediately; the packet is queued until either this message is acknowledged by another system, or a timeout has expired. This behaviour is enabled with the defer parameter to ifconfig(8). Eg. defer: on, yours is off. //mxb On 2 jul 2013, at 21:54, Loc BLOT loic.b...@unix-experience.fr wrote: Hi all I have a strange issue (or i haven't read pfsync correctly but i don't think this is the problem :D) I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site. Those BGP routers are secure with strong PF in stateful mode, and the stateful is working very well on each router. Because of my full mesh BGP configuration, the outgoing layer 7 sessions can leave my network by one router and responses can income by the other. To resolve this issue, i have created a dedidated VLAN for the pfsync traffic and attached pfsync to this VLAN. Here is a sample output of ifconfig on my first router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:10:4a:a6 priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.129 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync And here on my second router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:17:e2:1e priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.130 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync As you see in next tcpdump capture, there is some discussions between the two routers: # tcpdump -nni vlan995 tcpdump: listening on vlan995, link-type EN10MB tcpdump: WARNING: compensating for unaligned libpcap packets 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36 act DEL ST COMP count 1 id: 51d16a356c33 creatorid: a10bbd21 (DF) [tos 0x10] 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116 act UPD ST COMP count 13 ... (DF) [tos 0x10] The problem is simple, when i initiate a stateful connection from one server, the return (by second router) is blocked by PF (i see the return with pflog0) To be precise here
Re: PF sync doesn't not work very well
How does your CARP setup looks like. On both machines? Can you send your ifconfig output? What is your environment/setup for this 2-node CARP? How interfaces (ext/int) are connected? What switches do you use? On 3 jul 2013, at 10:23, Loïc Blot loic.b...@unix-experience.fr wrote: Okay, defer is now enabled on pfsync interface (sorry for my last idea, i haven't the man on me :) ). It seems the problem isn't resolved. The transfer starts but blocked at random time. -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit : Hi, Thanks for your reply. I wasn't careful about this section. If i understand i must add defer option to my WAN iface (or i'm wrong i must add it to my vlan995 iface ?) ? I will test it this morning, and i return back to misc :) -- Best regards, Loc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le mercredi 03 juillet 2013 02:02 +0200, mxb a crit : pfsync(4) explains this: The pfsync interface will attempt to collapse multiple state updates into a single packet where possible. The maximum number of times a single state can be updated before a pfsync packet will be sent out is controlled by the maxupd parameter and Where more than one firewall might actively handle packets, e.g. with certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to defer transmission of the initial packet of a connection. The pfsync state insert message is sent immediately; the packet is queued until either this message is acknowledged by another system, or a timeout has expired. This behaviour is enabled with the defer parameter to ifconfig(8). Eg. defer: on, yours is off. //mxb On 2 jul 2013, at 21:54, Loc BLOT loic.b...@unix-experience.fr wrote: Hi all I have a strange issue (or i haven't read pfsync correctly but i don't think this is the problem :D) I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site. Those BGP routers are secure with strong PF in stateful mode, and the stateful is working very well on each router. Because of my full mesh BGP configuration, the outgoing layer 7 sessions can leave my network by one router and responses can income by the other. To resolve this issue, i have created a dedidated VLAN for the pfsync traffic and attached pfsync to this VLAN. Here is a sample output of ifconfig on my first router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:10:4a:a6 priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.129 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync And here on my second router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:17:e2:1e priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.130 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync As you see in next tcpdump capture, there is some discussions between the two routers: # tcpdump -nni vlan995 tcpdump: listening on vlan995, link-type EN10MB tcpdump: WARNING: compensating for unaligned libpcap packets 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36 act DEL ST COMP count 1 id: 51d16a356c33 creatorid: a10bbd21 (DF) [tos 0x10] 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116 act UPD ST COMP count 13 ... (DF) [tos 0x10] The problem is simple, when i initiate a stateful connection from one server, the return (by second router) is
Re: PF sync doesn't not work very well
Hello, no carp is used at this time. My configuration on each router is simple: em0 + em3 = trunk0 em1 + em2 = trunk1 4 interco vlan (at this time, only 2 are active, 1 for a BGP neighbor IPv4, 1 for a BGP neighbor IPv6) on trunk0 vlan 50 + vlan 90 + vlan995 on trunk1 pfsync on vlan 995 -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 12:47 +0200, mxb a écrit : How does your CARP setup looks like. On both machines? Can you send your ifconfig output? What is your environment/setup for this 2-node CARP? How interfaces (ext/int) are connected? What switches do you use? On 3 jul 2013, at 10:23, Loïc Blot loic.b...@unix-experience.fr wrote: Okay, defer is now enabled on pfsync interface (sorry for my last idea, i haven't the man on me :) ). It seems the problem isn't resolved. The transfer starts but blocked at random time. -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit : Hi, Thanks for your reply. I wasn't careful about this section. If i understand i must add defer option to my WAN iface (or i'm wrong i must add it to my vlan995 iface ?) ? I will test it this morning, and i return back to misc :) -- Best regards, Loc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr Le mercredi 03 juillet 2013 02:02 +0200, mxb a crit : pfsync(4) explains this: The pfsync interface will attempt to collapse multiple state updates into a single packet where possible. The maximum number of times a single state can be updated before a pfsync packet will be sent out is controlled by the maxupd parameter and Where more than one firewall might actively handle packets, e.g. with certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to defer transmission of the initial packet of a connection. The pfsync state insert message is sent immediately; the packet is queued until either this message is acknowledged by another system, or a timeout has expired. This behaviour is enabled with the defer parameter to ifconfig(8). Eg. defer: on, yours is off. //mxb On 2 jul 2013, at 21:54, Loc BLOT loic.b...@unix-experience.fr wrote: Hi all I have a strange issue (or i haven't read pfsync correctly but i don't think this is the problem :D) I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site. Those BGP routers are secure with strong PF in stateful mode, and the stateful is working very well on each router. Because of my full mesh BGP configuration, the outgoing layer 7 sessions can leave my network by one router and responses can income by the other. To resolve this issue, i have created a dedidated VLAN for the pfsync traffic and attached pfsync to this VLAN. Here is a sample output of ifconfig on my first router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:10:4a:a6 priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.129 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync And here on my second router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:17:e2:1e priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.130 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync As you see in next tcpdump capture, there is some discussions between the two routers: # tcpdump -nni vlan995 tcpdump: listening on vlan995, link-type EN10MB tcpdump: WARNING: compensating for unaligned libpcap packets 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.949617 10.117.1.130:
Re: PF sync doesn't not work very well
On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: Hello, no carp is used at this time. pfsync needs to be used with carp... without it you're just playing whack-a-mole with your session table.
Re: PF sync doesn't not work very well
It's not possible to sync pf table without CARP ? I must use it in some case, then those case will be fixed but the other (OSPFd routing) may fail i think ? -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 07:11 -0500, Mark Felder a écrit : On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: Hello, no carp is used at this time. pfsync needs to be used with carp... without it you're just playing whack-a-mole with your session table.
Re: PF sync doesn't not work very well
On Wed, 03 Jul 2013 07:40:08 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: It's not possible to sync pf table without CARP ? In order to answer that I'll need to understand what you believe the pf table is.
Re: PF sync doesn't not work very well
Sure it syncs, but node1 has completely different IP addresses than node2(both external and internal ??), if no CARP. So storing states from node1, which passes/initiated connection to ftp.fr , on node2 does not help. In your case, you'd probably to decide to ever have MASTER-BACKUP or to have MASTER-MASTER CARP setup. On 3 jul 2013, at 14:40, Loïc Blot loic.b...@unix-experience.fr wrote: It's not possible to sync pf table without CARP ? I must use it in some case, then those case will be fixed but the other (OSPFd routing) may fail i think ? -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 07:11 -0500, Mark Felder a écrit : On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: Hello, no carp is used at this time. pfsync needs to be used with carp... without it you're just playing whack-a-mole with your session table.
Re: PF sync doesn't not work very well
For me pf table is (sorry for the missing precisions) the pf state stable for stateful operations -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 08:22 -0500, Mark Felder a écrit : On Wed, 03 Jul 2013 07:40:08 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: It's not possible to sync pf table without CARP ? In order to answer that I'll need to understand what you believe the pf table is.
Re: PF sync doesn't not work very well
Le Wed, 03 Jul 2013 07:11:08 -0500, Mark Felder f...@feld.me a écrit : On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: Hello, no carp is used at this time. pfsync needs to be used with carp... without it you're just playing whack-a-mole with your session table. I don't see why as states are not attached on carp interfaces but to real interfaces (if-bounded) or even by default to no interface at all ? Regards
Re: PF sync doesn't not work very well
On Wed, 03 Jul 2013 09:24:54 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: For me pf table is (sorry for the missing precisions) the pf state stable for stateful operations First of all, the states of node 1 being synced to node 2 and vice versa is worthless because they have different IP addresses; the states wont match anything. Secondly, you'll probably end up dealing with the nodes fighting each other as they sync back and forth. If a state from node1 is synced to node2 and node2 decides to expire that session because it hasn't been used it will tell node1 to remove that session as well. Now your session that was working on node1 has stopped functioning. This is probably the hanging/stalling behavior you were experiencing before. I've never even attempted to set this up in a lab and I know nothing of the pfsync/pf code, but I assume this is what is happening to you. I'm actually quite surprised it will even accept any changes to states for IPs that don't exist on the server, but I suppose it doesn't seem worthwhile to put such strict validation on it.
Re: PF sync doesn't not work very well
I don't understand why they can't be synced because if i have this scheme: server 1 - | Router 1 + Router 2 | remote server 1 contact remote, outgoing by Router 1 and the return traffic comes from Router 2. The state may have server 1 port A to remote port B, then the virtual IP is useless in this configuration, no ? -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 09:36 -0500, Mark Felder a écrit : On Wed, 03 Jul 2013 09:24:54 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: For me pf table is (sorry for the missing precisions) the pf state stable for stateful operations First of all, the states of node 1 being synced to node 2 and vice versa is worthless because they have different IP addresses; the states wont match anything. Secondly, you'll probably end up dealing with the nodes fighting each other as they sync back and forth. If a state from node1 is synced to node2 and node2 decides to expire that session because it hasn't been used it will tell node1 to remove that session as well. Now your session that was working on node1 has stopped functioning. This is probably the hanging/stalling behavior you were experiencing before. I've never even attempted to set this up in a lab and I know nothing of the pfsync/pf code, but I assume this is what is happening to you. I'm actually quite surprised it will even accept any changes to states for IPs that don't exist on the server, but I suppose it doesn't seem worthwhile to put such strict validation on it.
Re: PF sync doesn't not work very well
States ARE synced. IPs are not the same on node1 and node2 for external. The you initiated connection to ftp.fr, you done it via node1 with its external IP. On node2 those packets will be DROPPED as those do not belong to external NIC on node2 (IP) On 3 jul 2013, at 17:16, Loïc Blot loic.b...@unix-experience.fr wrote: I don't understand why they can't be synced because if i have this scheme: server 1 - | Router 1 + Router 2 | remote server 1 contact remote, outgoing by Router 1 and the return traffic comes from Router 2. The state may have server 1 port A to remote port B, then the virtual IP is useless in this configuration, no ? -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 09:36 -0500, Mark Felder a écrit : On Wed, 03 Jul 2013 09:24:54 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: For me pf table is (sorry for the missing precisions) the pf state stable for stateful operations First of all, the states of node 1 being synced to node 2 and vice versa is worthless because they have different IP addresses; the states wont match anything. Secondly, you'll probably end up dealing with the nodes fighting each other as they sync back and forth. If a state from node1 is synced to node2 and node2 decides to expire that session because it hasn't been used it will tell node1 to remove that session as well. Now your session that was working on node1 has stopped functioning. This is probably the hanging/stalling behavior you were experiencing before. I've never even attempted to set this up in a lab and I know nothing of the pfsync/pf code, but I assume this is what is happening to you. I'm actually quite surprised it will even accept any changes to states for IPs that don't exist on the server, but I suppose it doesn't seem worthwhile to put such strict validation on it.
Re: PF sync doesn't not work very well
The connection is not done by my routers themselves but by DMZ servers behind them ! -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 17:32 +0200, mxb a écrit : States ARE synced. IPs are not the same on node1 and node2 for external. The you initiated connection to ftp.fr, you done it via node1 with its external IP. On node2 those packets will be DROPPED as those do not belong to external NIC on node2 (IP) On 3 jul 2013, at 17:16, Loïc Blot loic.b...@unix-experience.fr wrote: I don't understand why they can't be synced because if i have this scheme: server 1 - | Router 1 + Router 2 | remote server 1 contact remote, outgoing by Router 1 and the return traffic comes from Router 2. The state may have server 1 port A to remote port B, then the virtual IP is useless in this configuration, no ? -- Best regards, Loïc BLOT, Engineering UNIX Systems, Security and Networks http://www.unix-experience.fr Le mercredi 03 juillet 2013 à 09:36 -0500, Mark Felder a écrit : On Wed, 03 Jul 2013 09:24:54 -0500, Loïc Blot loic.b...@unix-experience.fr wrote: For me pf table is (sorry for the missing precisions) the pf state stable for stateful operations First of all, the states of node 1 being synced to node 2 and vice versa is worthless because they have different IP addresses; the states wont match anything. Secondly, you'll probably end up dealing with the nodes fighting each other as they sync back and forth. If a state from node1 is synced to node2 and node2 decides to expire that session because it hasn't been used it will tell node1 to remove that session as well. Now your session that was working on node1 has stopped functioning. This is probably the hanging/stalling behavior you were experiencing before. I've never even attempted to set this up in a lab and I know nothing of the pfsync/pf code, but I assume this is what is happening to you. I'm actually quite surprised it will even accept any changes to states for IPs that don't exist on the server, but I suppose it doesn't seem worthwhile to put such strict validation on it.
Re: PF sync doesn't not work very well
pfsync(4) explains this: … The pfsync interface will attempt to collapse multiple state updates into a single packet where possible. The maximum number of times a single state can be updated before a pfsync packet will be sent out is controlled by the maxupd parameter … and … Where more than one firewall might actively handle packets, e.g. with certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to defer transmission of the initial packet of a connection. The pfsync state insert message is sent immediately; the packet is queued until either this message is acknowledged by another system, or a timeout has expired. This behaviour is enabled with the defer parameter to ifconfig(8). … Eg. defer: on, yours is off. //mxb On 2 jul 2013, at 21:54, Loïc BLOT loic.b...@unix-experience.fr wrote: Hi all I have a strange issue (or i haven't read pfsync correctly but i don't think this is the problem :D) I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site. Those BGP routers are secure with strong PF in stateful mode, and the stateful is working very well on each router. Because of my full mesh BGP configuration, the outgoing layer 7 sessions can leave my network by one router and responses can income by the other. To resolve this issue, i have created a dedidated VLAN for the pfsync traffic and attached pfsync to this VLAN. Here is a sample output of ifconfig on my first router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:10:4a:a6 priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.129 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync And here on my second router: vlan995: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST mtu 1500 lladdr a0:36:9f:17:e2:1e priority: 0 vlan: 995 parent interface: trunk1 groups: vlan status: active inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid 0x10 inet 10.117.1.130 netmask 0xfff8 broadcast 10.117.1.135 pfsync0: flags=41UP,RUNNING mtu 1500 priority: 0 pfsync: syncdev: vlan995 maxupd: 255 defer: off groups: carp pfsync As you see in next tcpdump capture, there is some discussions between the two routers: # tcpdump -nni vlan995 tcpdump: listening on vlan995, link-type EN10MB tcpdump: WARNING: compensating for unaligned libpcap packets 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124 act UPD ST COMP count 1 ... (DF) [tos 0x10] 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36 act DEL ST COMP count 1 id: 51d16a356c33 creatorid: a10bbd21 (DF) [tos 0x10] 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3 bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0 hello=2/0 fwdelay=15/0 pvid=995 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116 act UPD ST COMP count 13 ... (DF) [tos 0x10] The problem is simple, when i initiate a stateful connection from one server, the return (by second router) is blocked by PF (i see the return with pflog0) To be precise here is an example (and tested path): OBSD NTP - OBSD router 1 - WAN...ftp.fr.openbsd.org...WAN - OBSD router 2 || blocked PF allow in/out routing traffic from this server but incoming from WAN is blocked by default Can you confirm to me that pfsync may add a state for outgoing tcp connection in the second router when the first router add it ? Have you got any idea on this issue ? -- Best regards, Loïc BLOT, UNIX systems, security and network expert http://www.unix-experience.fr [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]
