Re: Checking Routes/Gateways For Good Connection
On Aug 30, 2010 at 3:36 PM, Henning Brauer wrote: > > why don't you look at the real interfaces instead of speculating. > > pflog is a bit messy, but that's another story hopefully solved soon. > > -- > Henning Brauer, h...@bsws.de, henn...@openbsd.org > BS Web Services, http://bsws.de > Full-Service ISP - Secure Hosting, Mail and DNS Services > Dedicated Servers, Rootservers, Application Hosting > I took your advice and ran a log on both external interfaces in addition to pflog0. What I found was that there was no problem with my pass rules. I failed to realize that the match rules for the NAT were affecting the packet prior to the pass rules (duh) and after translation I wasn't seeing the output in the log because I was logging for only the addresses of the internal interfaces. What I found _IS_ the problem is that my match rules for NAT are failing to match every time. They are as follows: match out on $ext_if_1 from (vether0) nat-to ($ext_if_1) match out on $ext_if_2 from (vether1) nat-to ($ext_if_2) What I'm seeing by logging the two external interfaces is the result of: $ traceroute -s 172.16.0.1 -n google.com (172.16.0.1 is the IP for vether0) ...sometimes gets matched by the nat-to rule, which correctly shows up in tcpdump on the external interface with the source being the external interface's IP address. The result of this command of course gets the expected traceroute replies. The problem is, when running the exact same command over and over, is that half the time I'm seeing the traceroute requests going out on the external interface with the source being 172.16.0.1. This of course gets no replies because the NAT isn't happening. Any clues as to why my match rules work only half the time?
Re: Checking Routes/Gateways For Good Connection
* Don Tek [2010-08-30 20:35]: > On 8/29/2010 11:54 AM, Henning Brauer wrote: > >when pf sees that packet it is outbound on em0. you are logging that > >fact. then pf re-routes via em1. when pf sees it again on em1, you log > >that fact. > > > My tcpdump output does not seem to confirm this. For instance: > tcpdump: listening on pflog0, link-type PFLOG why don't you look at the real interfaces instead of speculating. pflog is a bit messy, but that's another story hopefully solved soon. -- Henning Brauer, h...@bsws.de, henn...@openbsd.org BS Web Services, http://bsws.de Full-Service ISP - Secure Hosting, Mail and DNS Services Dedicated Servers, Rootservers, Application Hosting
Re: Checking Routes/Gateways For Good Connection
On 8/29/2010 11:54 AM, Henning Brauer wrote: when pf sees that packet it is outbound on em0. you are logging that fact. then pf re-routes via em1. when pf sees it again on em1, you log that fact. My tcpdump output does not seem to confirm this. For instance: $ traceroute -m 4 -s 172.16.0.1 -n google.com $ sudo tcpdump -n -e -ttt -i pflog0 host 172.16.0.1 tcpdump: listening on pflog0, link-type PFLOG Aug 30 13:13:09.622700 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33435: udp 12 [ttl 1] Aug 30 13:13:14.630584 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33436: udp 12 [ttl 1] Aug 30 13:13:19.639902 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33437: udp 12 [ttl 1] Aug 30 13:13:24.649161 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33438: udp 12 Aug 30 13:13:29.658493 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33439: udp 12 Aug 30 13:13:34.667819 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33440: udp 12 Aug 30 13:13:39.677161 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33441: udp 12 Aug 30 13:13:44.686542 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33442: udp 12 Aug 30 13:13:49.695834 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33443: udp 12 Aug 30 13:13:54.705161 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33444: udp 12 Aug 30 13:13:59.714426 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33445: udp 12 Aug 30 13:14:04.723664 rule 42/(match) pass out on em1: 172.16.0.1.58471 > 209.85.225.104.33446: udp 12 Traceroute's defaults dictate a 5 second wait between probes and 3 probes per hop. I forced my max_ttl to 4 (hops). I get exactly 12 lines of log corresponding to (3 probes * 4 hops) and the log shows 5 seconds between hops. So where are the log lines for the re-routes? It appears to me PF is ignoring my route-to(s), or it is ignoring the source and not matching, and the request is simply going out whichever of my two multipath default routes gets chosen at the time.
Re: Checking Routes/Gateways For Good Connection
On August 28, 2010 11:35 PM Bret S. Lambert wrote: >> 39: pass out quick log from 172.16.0.1 route-to (em0 192.168.0.1) >> $ traceroute -s 172.16.0.1 -n google.com >> Tcpdump pflog0 output: >> Aug 28 21:41:11.225656 rule 39/(match) pass out on em1: 172.16.0.1.48096 > 74.125.45.147.33449: udp 12 >from pf.conf: > When a route-to rule > creates state, only packets that pass in the same direction as the > filter rule specifies will be routed in this way. Packets passing > in the opposite direction (replies) are not affected and are routed > normally. My rule is for outgoing packets from 172.16.0.1, which should be routed to em0. My tcpdump output is showing outgoing packets from 172.16.0.1, which are being routed to both em0 and unexpectedly, em1. How are you suggesting that "opposite direction (replies)" are coming into play here?
Re: Checking Routes/Gateways For Good Connection
On Sat, Aug 28, 2010 at 09:50:30PM -0500, dontek wrote: > This is even more strange to me. If I change rule 39 and 40 by taking out > the "on" interface to the following: > > PF Rules: (rule number prepended, these are the _last_ 6 lines in my > pf.conf) > > 39: pass out quick log from 172.16.0.1 route-to (em0 192.168.0.1) > 40: pass out quick log from 172.16.1.1 route-to (em1 10.10.0.1) > 41:pass out log on em0 42:pass out log on em1 > 43:pass out log on em0 from em1 route-to (em1 10.10.0.1) 44:pass out log on > em1 from em0 route-to (em0 192.168.0.1) > > > Tests: > > $ traceroute -s 172.16.0.1 -n google.com > > > Tcpdump pflog0 output: > > Aug 28 21:41:11.215660 rule 40/(match) pass out on em0: 172.16. 1.1.63306 > > 74.125.45.147.33449: udp 12 > Aug 28 21:41:11.225656 rule 39/(match) pass out on em1: 172.16.0.1.48096 > > 74.125.45.147.33449: udp 12 > > > Now these packets are being caught by my rule 39 and 40, but it appears the > route-to is just being ignored. Am I reading the tcpdump output wrong? I > just don't get it..? > from pf.conf: When a route-to rule creates state, only packets that pass in the same direction as the filter rule specifies will be routed in this way. Packets passing in the opposite direction (replies) are not affected and are routed normally.
Re: Checking Routes/Gateways For Good Connection
This is even more strange to me. If I change rule 39 and 40 by taking out the "on" interface to the following: PF Rules: (rule number prepended, these are the _last_ 6 lines in my pf.conf) 39: pass out quick log from 172.16.0.1 route-to (em0 192.168.0.1) 40: pass out quick log from 172.16.1.1 route-to (em1 10.10.0.1) 41:pass out log on em0 42:pass out log on em1 43:pass out log on em0 from em1 route-to (em1 10.10.0.1) 44:pass out log on em1 from em0 route-to (em0 192.168.0.1) Tests: $ traceroute -s 172.16.0.1 -n google.com Tcpdump pflog0 output: Aug 28 21:41:11.215660 rule 40/(match) pass out on em0: 172.16. 1.1.63306 > 74.125.45.147.33449: udp 12 Aug 28 21:41:11.225656 rule 39/(match) pass out on em1: 172.16.0.1.48096 > 74.125.45.147.33449: udp 12 Now these packets are being caught by my rule 39 and 40, but it appears the route-to is just being ignored. Am I reading the tcpdump output wrong? I just don't get it..?
Re: Checking Routes/Gateways For Good Connection
Well, I thought I had this issue worked out, but my pf rules aren't evaluating as I expected them to: PF Rules: (rule number prepended, these are the _last_ 6 lines in my pf.conf) 39:pass out quick log on em0 from 172.16.0.1 route-to (em0 192.168.0.1) 40:pass out quick log on em1 from 172.16.1.1 route-to (em1 10.10.0.1) 41:pass out log on em0 42:pass out log on em1 43:pass out log on em0 from em1 route-to (em1 10.10.0.1) 44:pass out log on em1 from em0 route-to (em0 192.168.0.1) Tests: $ traceroute -s 172.16.0.1 -n google.com Tcpdump pflog0 output: Aug 27 15:35:16.418090 rule 42/(match) pass out on em1: 172.16.0.1.34561> 74.125.45.106.33438: udp 12 Aug 27 15:50:01.658596 rule 41/(match) pass out on em0: 172.16.0.1.63615> 74.125.45.103.33444: udp 12 Why are these packets not being caught by rule 39 and always going out the em0 gateway?
Re: Checking Routes/Gateways For Good Connection
On August 26, 2010 4:14 Pete Vickers wrote: >On 26. aug. 2010, at 00.18, Don Tek wrote: >> I've recently implemented a firewall with two internet connections using multipath routing and round-robin outbound load balancing. >> >>I am looking for a solution from the shell to detect failure of these two internet gateways so I can force routing and pf changes from a script. >> >> I need something more robust than simply checking to see if the interface is up or down. >> >> I have managed a solution using traceroute that allows me to accomplish half of my goal. I can detect a failure and "down" that route, however, once I delete the default route from the routing table for the failed connection, I can no longer test it with traceroute. This is because it doesn't appear to me that OpenBSD's traceroute allows forcing an interface to work on. >> >> I am looking for better solutions from some of you more experienced users. Any suggestions are welcome. >> >> don.. >> >Taking a look at the bigger picture, the 'correct' way to do this is to have redundancy at the firewall level as well at ISP link level. This gives higher availability, and makes your problem much easier. If you have a single ISP link per firewall then link testing is simple. Redundancy/LB is then managed by CARP between the two firewalls' _inside_ interfaces. >/Pete I wish I could do this, but this particular client is as stingy as it gets with IT spending. They also aren't very concerned about redundancy. The only reason they even have two ISP's is because they don't employ any internal IT personnel and we try and do almost everything remotely, and, they insist on hosting file sharing for their road warriors internally and it was the cheapest route to more speed. Of course, the two ISP's are nothing more than DSL and DOCIS from two different companies which seem to flake out fairly often. At least if I can test on the one firewall, I can send notification and adjust routes when one of them goes down.
Re: Checking Routes/Gateways For Good Connection
On August 25, 2010 8:20 PM, Stuart Henderson wrote: >On 2010-08-26, David Gwynne wrote: >> relayd can do this i think. > >afaik, it just uses the normal routing table, you can't force a specific interface/nexthop. if that's correct, to have relayd check connectivity beyond the gateways, you'll need static routes (e.g. host 8.8.8.8 via connection A, 208.67.222.222 via connection B) and use those hosts as an indicator. >> On 26/08/2010, at 9:10 AM, dontek wrote: > I have managed a solution using traceroute that allows me to > accomplish half of my goal. I can detect a failure and "down" that > route, however, once I delete the default route from the routing > table for the failed connection, I can no longer test it with traceroute. > This is because it doesn't appear to me that OpenBSD's traceroute > allows forcing an interface to work on. >it may work to set the source address (ping -I, traceroute -s) and use a route-to pf rule to make sure packets with a specific source address are directed out of the relevant interface...though if you only have one address to play with for each connection you might be unable to distinguish "check" packets from normal packets. I tried out Stuart's suggestion with a couple of vether interfaces and it seems to have accomplished what I was looking for. Details: $ cat /etc/hostname.vether0 inet 172.16.0.1 255.255.255.0 NONE $ cat /etc/hostname.vether1 inet 172.16.1.1 255.255.255.0 NONE $ cat /etc/pf.conf (additions only) match out on $ext_if_1 from (vether0:network) nat-to ($ext_if_1) match out on $ext_if_2 from (vether1:network) nat-to ($ext_if_2) pass out on $ext_if_1 from (vether0) route-to ($ext_if_1 $ext_gate_1) pass out on $ext_if_2 from (vether1) route-to ($ext_if_2 $ext_gate_2) Tests: $ traceroute -s 172.16.0.1 -n google.com $ traceroute -s 172.16.1.1 -n google.com Once the above was done I could apply my traceroute scheme of testing to a script and/or integrate into ifstated. Thanks Stuart. >otherwise maybe you can do something with multiple routing tables...
Re: Checking Routes/Gateways For Good Connection
On 26. aug. 2010, at 00.18, Don Tek wrote: > I've recently implemented a firewall with two internet connections using multipath routing and round-robin outbound load balancing. > > I am looking for a solution from the shell to detect failure of these two internet gateways so I can force routing and pf changes from a script. > > I need something more robust than simply checking to see if the interface is up or down. > > I have managed a solution using traceroute that allows me to accomplish half of my goal. I can detect a failure and "down" that route, however, once I delete the default route from the routing table for the failed connection, I can no longer test it with traceroute. This is because it doesn't appear to me that OpenBSD's traceroute allows forcing an interface to work on. > > I am looking for better solutions from some of you more experienced users. Any suggestions are welcome. > > don.. > Taking a look at the bigger picture, the 'correct' way to do this is to have redundancy at the firewall level as well at ISP link level. This gives higher availability, and makes your problem much easier. If you have a single ISP link per firewall then link testing is simple. Redundancy/LB is then managed by CARP between the two firewalls' _inside_ interfaces. /Pete
Re: Checking Routes/Gateways For Good Connection
On 2010-08-26, David Gwynne wrote: > relayd can do this i think. afaik, it just uses the normal routing table, you can't force a specific interface/nexthop. if that's correct, to have relayd check connectivity beyond the gateways, you'll need static routes (e.g. host 8.8.8.8 via connection A, 208.67.222.222 via connection B) and use those hosts as an indicator. > On 26/08/2010, at 9:10 AM, dontek wrote: I have managed a solution using traceroute that allows me to accomplish half of my goal. I can detect a failure and "down" that route, however, once I delete the default route from the routing table for the failed connection, I can no longer test it with traceroute. This is because it doesn't appear to me that OpenBSD's traceroute allows forcing an interface to work on. it may work to set the source address (ping -I, traceroute -s) and use a route-to pf rule to make sure packets with a specific source address are directed out of the relevant interface...though if you only have one address to play with for each connection you might be unable to distinguish "check" packets from normal packets. otherwise maybe you can do something with multiple routing tables...
Re: Checking Routes/Gateways For Good Connection
relayd can do this i think. On 26/08/2010, at 9:10 AM, dontek wrote: > Jacob Yocom-Piatt wrote: >> Don Tek wrote: >>> I've recently implemented a firewall with two internet connections >>> using multipath routing and round-robin outbound load balancing. >>> >>> I am looking for a solution from the shell to detect failure of these >>> two internet gateways so I can force routing and pf changes from a >>> script. >>> >>> I need something more robust than simply checking to see if the >>> interface is up or down. >>> >>> I have managed a solution using traceroute that allows me to >>> accomplish half of my goal. I can detect a failure and "down" that >>> route, however, once I delete the default route from the routing table >>> for the failed connection, I can no longer test it with traceroute. >>> This is because it doesn't appear to me that OpenBSD's traceroute >>> allows forcing an interface to work on. >>> >>> I am looking for better solutions from some of you more experienced >>> users. Any suggestions are welcome. >> >> ifstated > > > As stated, I need something more robust than simple link state testing. I > realize ifstated can utilize external tests, an example of an one of these > external tests is what I'm looking for.
Re: Checking Routes/Gateways For Good Connection
Jacob Yocom-Piatt wrote: >Don Tek wrote: >> I've recently implemented a firewall with two internet connections >> using multipath routing and round-robin outbound load balancing. >> >> I am looking for a solution from the shell to detect failure of these >> two internet gateways so I can force routing and pf changes from a >> script. >> >> I need something more robust than simply checking to see if the >> interface is up or down. >> >> I have managed a solution using traceroute that allows me to >> accomplish half of my goal. I can detect a failure and "down" that >> route, however, once I delete the default route from the routing table >> for the failed connection, I can no longer test it with traceroute. >> This is because it doesn't appear to me that OpenBSD's traceroute >> allows forcing an interface to work on. >> >> I am looking for better solutions from some of you more experienced >> users. Any suggestions are welcome. > >ifstated As stated, I need something more robust than simple link state testing. I realize ifstated can utilize external tests, an example of an one of these external tests is what I'm looking for.
Re: Checking Routes/Gateways For Good Connection
Don Tek wrote: I've recently implemented a firewall with two internet connections using multipath routing and round-robin outbound load balancing. I am looking for a solution from the shell to detect failure of these two internet gateways so I can force routing and pf changes from a script. I need something more robust than simply checking to see if the interface is up or down. I have managed a solution using traceroute that allows me to accomplish half of my goal. I can detect a failure and "down" that route, however, once I delete the default route from the routing table for the failed connection, I can no longer test it with traceroute. This is because it doesn't appear to me that OpenBSD's traceroute allows forcing an interface to work on. I am looking for better solutions from some of you more experienced users. Any suggestions are welcome. ifstated
Checking Routes/Gateways For Good Connection
I've recently implemented a firewall with two internet connections using multipath routing and round-robin outbound load balancing. I am looking for a solution from the shell to detect failure of these two internet gateways so I can force routing and pf changes from a script. I need something more robust than simply checking to see if the interface is up or down. I have managed a solution using traceroute that allows me to accomplish half of my goal. I can detect a failure and "down" that route, however, once I delete the default route from the routing table for the failed connection, I can no longer test it with traceroute. This is because it doesn't appear to me that OpenBSD's traceroute allows forcing an interface to work on. I am looking for better solutions from some of you more experienced users. Any suggestions are welcome. don..
