On Mon, Oct 19, 2020 at 09:34:31AM +0100, Stuart Henderson wrote:
> On 2020/10/19 15:35, David Gwynne wrote:
> > every few years i try and use route-to in pf, and every time it
> > goes badly. i tried it again last week in a slightly different
> > setting, and actually tried to understand the sharp edges i hit
> > this time instead of giving up. it turns out there are 2 or 3
> > different things together that have cause me trouble, which is why
> > the diff below is so big.
>
> I used to route-to/reply-to quite a lot at places with poor internet
> connections to split traffic between lines (mostly those have better
> connections now so I don't need it as often). It worked as I expected -
> but I only ever used it with the interface specified.
cool. did it work beyond the first packet in a connection?
> I mostly used it with pppoe interfaces so the peer address was unknown
> at ruleset load time. (I was lucky and had static IPs my side, but the
> ISP side was variable). I relied on the fact that once packets are
> directed at a point-point interface there's only one place for them to
> go. I didn't notice that ":peer" might be useful here (and the syntax
> 'route-to pppoe1:peer@pppoe1' is pretty awkward so I probably wouldn't
> have come up with it), I had 0.0.0.1@pppoe1, 0.0.0.2@pppoe2 etc
> (though actually I think it works with $any_random_address@pppoeX).
yes. i was trying to use it with peers over ethernet, and always
struggled with the syntax.
> > the first and i would argue most fundamental problem is a semantic
> > problem. if you ask a random person who has some clue about networks
> > and routing what they would expect the "argument" to route-to or
> > reply-to to be, they would say "a nexthop address" or "a gateway
> > address". eg, say i want to force packets to a specific backend
> > server without using NAT, i would write a rule like this:
> >
> > n_servers="192.0.2.128/27"
> > pass out on $if_internal to $n_servers route-to 192.168.0.1
> >
> > pfctl will happily parse this, shove it into the kernel, let you read
> > the rules back out again with pfctl -sr, and it all looks plausible, but
> > it turns out that it's using the argument to route-to as an interface
> > name. because rulesets can refer to interfaces that don't exist yet, pf
> > just passes the IP address around as a string, hoping i'll plug in an
> > interface with a driver name that looks like an ip address. i spent
> > literally a day trying to figure out why a rule like this wasn't
> > working.
>
> I don't think I tried this, but the pf.conf(5) BNF syntax suggests it's
> supposed to work. So either doc or implementation bug there.
im leaning toward implementation bug.
> route = ( "route-to" | "reply-to" | "dup-to" )
> ( routehost | "{" routehost-list "}" )
> [ pooltype ]
>
> routehost-list = routehost [ [ "," ] routehost-list ]
>
> routehost = host | host "@" interface-name |
> "(" interface-name [ address [ "/" mask-bits ] ] ")"
>
> > the second problem is that the pf_route calls from pfsync don't
> > have all the information it is supposed to have. more specifically,
> > an ifp pointer isn't set which leads to a segfault. the ifp pointer
> > isn't set because pfsync doesnt track which interface a packet is
> > going out, it assumes the ip layer will get it right again later, or a
> > rule provided something usable.
> >
> > the third problem is that pf_route relies on information from rules to
> > work correctly. this is a problem in a pfsync environment because you
> > cannot have the same ruleset on both firewalls 100% of the time, which
> > means you cannot have route-to/reply-to behave consistently on a pair of
> > firwalls 100% of the time.
>
> I didn't run into this because pppoe(4) and pfsync/carp don't really
> go well together, but ouch!
>
> > all of this together makes things work pretty obviously and smoothly.
> > in my opinion anyway. route-to now works more like rdr-to, it just
> > feels like it changes the address used for the route lookup rather
> > than changing the actual IP address in the packet. it also works
> > predictably in a pfsync pair, which is great from the point of view of
> > high availability.
> >
> > the main caveat is that it's not backward compatible. if you're already
> > using route-to, you will need to tweak your rules to have them parse.
> > however, i doubt anyone is using this stuff because it feels very broken
> > to me.
>
> Do you expect this to work with a bracketed "address" to defer lookup
> until rule evaluation time? i.e.
>
> pass out proto tcp to any port 22 route-to (pppoe1:peer)
in my opinion route-to should be able to take whatever rdr-to accepts.
however, i just tried it and it doesnt currently work, but i'm sure i
can figure it out.
> I think that will be all that's needed to allow converting the pppoe
> use case. I don't have a multiple pppoe setup handy but I can probably
> hack together some sort of test.
>
> I've also used route-to with squid "transparent" proxying (shown in
> the pkg-readme), I don't do that any more but I can put a squid test
> together easily enough.
it looks doable.
my changes will break route-to with "no state" rules though. should
i try and keep those working too?
>
> > @@ -1842,37 +1833,18 @@ pfrule : action dir logquick interface
> > decide_address_family($7.src.host, &r.af);
> > decide_address_family($7.dst.host, &r.af);
> >
> > - if ($8.route.rt) {
> > + if ($8.rt) {
> > + if ($8.rt != PF_DUPTO && !r.direction) {
> > + yyerror("direction must be explicit"
> > + " with rules that specify routing");
> > + YYERROR;
> > + }
> > if (!r.direction) {
> > yyerror("direction must be explicit "
> > "with rules that specify routing");
> > YYERROR;
> > }
>
> this stood out on reading the diff - the added if block doesn't change
> any outcome, should it have actually been like this?
possibly :)
>
> - if (!r.direction) {
> + if ($8.rt != PF_DUPTO && !r.direction) {
> yyerror("direction must be explicit"
> " with rules that specify routing");
> YYERROR;
> }
>
