Re: Weird pf NAT failure on apu2
On Sat, 24 Jun 2023 07:33 -0600, Zack Newman wrote: > On 6/2l/23 9:01, Stephan Neuhaus wrote: > > I'm not sure about the Configuring NAT section being > > correct. I still maintain that the documentation and > > observed behaviour are different. > > I was lazy when I said that. I meant the example I quoted from that > section in the original reply is correct. Everything else that says > otherwise (including the two people that said that part was wrong) is > incorrect. Explicitly the following rule _is_ correct: > > match out on interface [af] \ > from src_addr to dst_addr \ > nat-to ext_addr [pool_type] [static-port] > [...] > pass out [log] on interface [af] [proto protocol] \ > from ext_addr [port src_port] \ > to dst_addr [port dst_port] > > There is only so many ways that this can be shown. If the pass out rule > had "src_addr" instead of "ext_addr", it would be wrong. The diff that > "fixes" that example needs to be rejected. It is the _other_ example > that is wrong. > > If you tried using the above example to get NAT to work, you will find > that it will work. The last e-mail I sent clearly follows the above > example except I chose to use stateless rules to help show more > thoroughly all the rules that are necessary. Additionally, I prefer > "quick" rules; but the conclusion is very clear: the match out rule > applies and "sticks" which in turn means "src_addr" is replaced with > "ext_addr" which in turn means the pass out rule must have "ext_addr". > I forgot about that diff until I read your mail today. You're right, I just tested that and your logic is sound. This is a good reminder to me to test before sending diffs. I'm glad it wasn't committed. Thank you for investigating that.
Re: Weird pf NAT failure on apu2
On 6/24/23 13:14, Stuart Henderson wrote: On 2023-06-24, Stephan Neuhaus wrote: I now think that either the documentation is wrong, or pf is wrong. At any rate, there seems to be a rather serious disconnect between the two. The FAQ clearly says: When a packet is selected by a match rule, parameters (e.g. nat-to) in that rule are remembered and are applied to the packet when a pass rule matching the packet is reached. Yes that's wrong. Address changes take effect at the time the match rule is processed when traversing the ruleset; rules processed later "see" the rewritten address. As pf.conf(5) says, Translation Translation options modify either the source or destination address and port of the packets associated with a stateful connection. pf(4) modifies the specified address and/or port in the packet and recalculates IP, TCP, and UDP checksums as necessary. If specified on a match rule, subsequent rules will see packets as they look after any addresses and ports have been translated. These rules will therefore have to filter based on the translated address and port number. OK, that's clear and unambiguous, thanks! So it's a bug in the FAQ, not in pf itself. Cheers Stephan
Re: Weird pf NAT failure on apu2
On 6/2l/23 9:01, Stephan Neuhaus wrote: I'm not sure about the Configuring NAT section being correct. I still maintain that the documentation and observed behaviour are different. I was lazy when I said that. I meant the example I quoted from that section in the original reply is correct. Everything else that says otherwise (including the two people that said that part was wrong) is incorrect. Explicitly the following rule _is_ correct: match out on interface [af] \ from src_addr to dst_addr \ nat-to ext_addr [pool_type] [static-port] [...] pass out [log] on interface [af] [proto protocol] \ from ext_addr [port src_port] \ to dst_addr [port dst_port] There is only so many ways that this can be shown. If the pass out rule had "src_addr" instead of "ext_addr", it would be wrong. The diff that "fixes" that example needs to be rejected. It is the _other_ example that is wrong. If you tried using the above example to get NAT to work, you will find that it will work. The last e-mail I sent clearly follows the above example except I chose to use stateless rules to help show more thoroughly all the rules that are necessary. Additionally, I prefer "quick" rules; but the conclusion is very clear: the match out rule applies and "sticks" which in turn means "src_addr" is replaced with "ext_addr" which in turn means the pass out rule must have "ext_addr".
Re: Weird pf NAT failure on apu2
On 2023-06-24, Stephan Neuhaus wrote:
> Hi Zack
>
> On 6/24/23 03:39, Zack Newman wrote:
>> There do appear to be contradictions in documentation as well as the pf
>> book. The Configuring NAT section is correct as you have seen with your
>> own rules.
>
> I'm not sure about the Configuring NAT section being
> correct. I still maintain that the documentation and
> observed behaviour are different.
The pf.conf(5) manual page is the best documentation for PF and that
is expected to be correct.
The pf faq could well be wrong, it's not really had a full
review/update/rewrite since before nat-to was introduced - a relatively
light conversion to the "new" syntax and mostly smaller changes -
no integration of new features in the existing text e.g. the very
useful tag/tagged is relegated to an added on "advanced features"
section).
> I now have the following small ruleset. The rules for
> ports 22 and 53 are just so that I can do my
> experiments over ssh. (Port 53 may not be necessary
> but the blocked packets mess up my logs.) The em0
You can always follow a general "block log" with a "block on port XX"
if you want to log most blocked psckets except for certain ones.
> interface has the IPv4 address 192.168.0.2.
>
> set skip on lo
>
> block log all
>
> pass in on em0 proto tcp to any port 22
> pass out on em0 proto tcp from any port 22
> pass on em0 proto udp from any port 53
> pass on em0 proto udp to any port 53
This allows someone on the internet to bypass your filter rules and
send UDP traffic to any port by simply setting the source port to 53.
Just have "to port 53", reply ("from port 53") packets will match
the state created by the packet that was passed by the "to port 53"
rule.
(You would have the same with TCP except for the direction on the
rule; still "pass out proto tcp from port 22" is probably not quite
what you want).
> I now think that either the documentation is wrong, or
> pf is wrong. At any rate, there seems to be a rather
> serious disconnect between the two. The FAQ clearly
> says:
>
> When a packet is selected by a match rule, parameters (e.g. nat-to) in
> that rule are remembered and are applied to the packet when a pass rule
> matching the packet is reached.
Yes that's wrong. Address changes take effect at the time the match rule
is processed when traversing the ruleset; rules processed later "see"
the rewritten address. As pf.conf(5) says,
Translation
Translation options modify either the source or destination address
and port of the packets associated with a stateful connection. pf(4)
modifies the specified address and/or port in the packet and
recalculates IP, TCP, and UDP checksums as necessary.
If specified on a match rule, subsequent rules will see packets as
they look after any addresses and ports have been translated. These
rules will therefore have to filter based on the translated address
and port number.
> I know that the documentation (at this time) ALSO
> says:
>
> match out on interface [af] \
> from src_addr to dst_addr \
> nat-to ext_addr [pool_type] [static-port]
> [...]
> pass out [log] on interface [af] [proto protocol] \
> from ext_addr [port src_port] \
> to dst_addr [port dst_port]
>
> where the pass rule has ext_addr instead of src_addr,
> but at least two people on t...@openbsd.org have
> confirmed that this is a bug and have already
> submitted a diff that replaces ext_addr with src_addr
> in the pass rule, see
> https://marc.info/?l=openbsd-tech=168714686620055=2
I think that this bit _is_ correct as written, although it's not
particularly useful as it doesn't allow passing just the specific
translated packets.
I think it's the example rules which are wrong, i.e.
match out on tl0 from 192.168.1.0/24 to any nat-to 198.51.100.1
pass on tl0 from 192.168.1.0/24 to any
directly fixing this would be
match out on tl0 from 192.168.1.0/24 to any nat-to 198.51.100.1
pass on tl0 from 198.51.100.1 to any
however this would be a good place to instead show how to use
match...nat-to...tag and pass...tagged, maybe also received-on. Then a)
it's clear what traffic is passed and b) that fits better with the very
common use-case where the external (nat-rewritten) address is dynamic so
you need to use an interface name with parentheses in the nat-to rule.
But when you pull at that thread, the current structure of the pf faq
section starts to unravel, i.e. it doesn't make sense to separate tags
into "advanced", there's a fair bit of information which really isn't
useful at the faq level (flag matching? the bit about using "block
return" instead of a set of 6 different block rules that nobody learning
from the faq would have thought of writing? syn proxy, esoteric scrub
rules, being originally written from a point-of-view of "keep state"
being a special thing and tweaked from that to the current state, ...)
and it becomes a fairly big authoring job.
> Should I be taking this to another mailing
Re: Weird pf NAT failure on apu2
Hi Zack On 6/24/23 03:39, Zack Newman wrote: There do appear to be contradictions in documentation as well as the pf book. The Configuring NAT section is correct as you have seen with your own rules. I'm not sure about the Configuring NAT section being correct. I still maintain that the documentation and observed behaviour are different. I now have the following small ruleset. The rules for ports 22 and 53 are just so that I can do my experiments over ssh. (Port 53 may not be necessary but the blocked packets mess up my logs.) The em0 interface has the IPv4 address 192.168.0.2. set skip on lo block log all pass in on em0 proto tcp to any port 22 pass out on em0 proto tcp from any port 22 pass on em0 proto udp from any port 53 pass on em0 proto udp to any port 53 pass in on athn0 match out log on em0 from athn0:network to any nat-to (em0) #pass out log on em0 from athn0:network to any pass out log on em0 from 192.168.0.2 to any This causes packets from athn0 to pass and to be correctly NATed. If I uncomment the penultimate line and comment out the last line, the packets get matched (as before), but then blocked by the catch-all rule 0. This is consistent with the nat-to being applied immediately after the match rule matches, EXCEPT that the blocked packet is logged with a source address inside athn0:network (192.168.3.0/24 in this case) and not the NATed-to 192.168.0.2. For example: Jun 24 10:24:46.498327 rule 0/(match) block out on em0: 192.168.3.32.54459 > 172.217.168.68.443: S 2479117865:2479117865(0) win 65535 (DF) I now think that either the documentation is wrong, or pf is wrong. At any rate, there seems to be a rather serious disconnect between the two. The FAQ clearly says: When a packet is selected by a match rule, parameters (e.g. nat-to) in that rule are remembered and are applied to the packet when a pass rule matching the packet is reached. This seems to me to imply that the nat-to in my match rule should not be applied BEFORE the pass rule is attempted, but only AFTER. This seems also to be the point of these lines in the documentation: match When a packet traverses the ruleset and matches a match rule, any optional parameters specified in that rule are remembered for future use (made "sticky"). pass This rule allows the packet to be transmitted. If the packet was previously matched by a match rule where parameters were specified, they will be applied to this packet. [...] I know that the documentation (at this time) ALSO says: match out on interface [af] \ from src_addr to dst_addr \ nat-to ext_addr [pool_type] [static-port] [...] pass out [log] on interface [af] [proto protocol] \ from ext_addr [port src_port] \ to dst_addr [port dst_port] where the pass rule has ext_addr instead of src_addr, but at least two people on t...@openbsd.org have confirmed that this is a bug and have already submitted a diff that replaces ext_addr with src_addr in the pass rule, see https://marc.info/?l=openbsd-tech=168714686620055=2 Should I be taking this to another mailing list? Should I be submitting a bug report? Or am I just really really dense and am just too stupid to read the documentation correctly? Cheers Stephan
Re: Weird pf NAT failure on apu2
There do appear to be contradictions in documentation as well as the pf book. The Configuring NAT section is correct as you have seen with your own rules. Here is the minimum set of stateless rules that allows ICMP traffic between my laptop and Cloudflare. # Options. set block-policy drop # Macros. wan = em0 wifi = vlan4 external = 76.154.165.21 laptop = 192.168.6.2 cflare = 1.1.1.1 # NAT. match out on $wan inet proto icmp from $laptop nat-to $external static-port # Filtering rules. pass in quick on $wifi inet proto icmp from $laptop to $cflare no state pass out quick on $wan inet proto icmp from $external to $cflare keep state (if-bound) pass out quick on $wifi inet proto icmp from $cflare to $laptop no state block quick The second filter rule _must_ be stateful in order for the router to map the ICMP Query ID back to the original source IP (i.e., my laptop). When relying on only stateful rules, we can remove the last pass rule- since the first filter rule when hit will allow the returning Echo Response. If instead of using match rules for NAT, you use pass out; then pf will fail to load pf.conf(5) if the second rule is stateless: /etc/pf.conf:13: nat-to and rdr-to require keep state /etc/pf.conf:13: skipping rule due to errors /etc/pf.conf:13: rule expands to no valid combination pfctl: Syntax error in config file: pf rules not loaded If you want to easily distinguish traffic sourced from your router leaving em0 from traffic sourced from your (V)LAN devices, then you will need to have a separate rule. For example: pass out quick log on em0 inet from (em0) match out log on em0 inet from athn0:network nat-to (em0) pass out log quick on em0 inet
Re: Weird pf NAT failure on apu2
Just wanted to reply that that was an excellent rebuttal. Looks like I should have put my foot in my mouth. I am now keenly interested-and disappointed in my (lack) of knowledge. I will practice with pf on my machine to better understand what is happening. If/when I have something meaningful to say, I will send another e-mail. Thanks for the information.
Re: Weird pf NAT failure on apu2
On 6/23/23 18:29, Zack Newman wrote:
On 6/23/23 11:19, Stephan Neuhaus wrote:
# Rule 5
match out log on em0 from athn0:network to any nat-to (em0)
# Rule 6
pass out log on em0 from athn0:network to any
Rule 5 replaces the source IP address with the IP address assigned to
em0-as well as replaces the source port (for TCP and UDP) with an
ephemeral port. Rule 6 does _not_ pass traffic out of em0 from the IP
address assigned to em0 but instead passes traffic where the IP address
is from athn0:network, so of course it won't work.
Thanks for replying!
What you say may well be true, but if it is, it is in
conflict with the documentation and IMO the match/pass
combo is much less useful.
To make my first point, the pf FAQ says the following
about match/pass:
match
When a packet traverses the ruleset and matches a match rule, any
optional parameters specified in that rule are remembered for future use
(made "sticky").
pass
This rule allows the packet to be transmitted. If the packet was
previously matched by a match rule where parameters were specified, they
will be applied to this packet. [...]
This makes it very clear (to me at least) that the
nat-to in the match rule is "remembered for
future use" in the match, and is applied to the packet
only when it is finally subject to a pass rule.
The FAQ also has the following example:
match out on tl0 from 192.168.1.0/24 to any nat-to 198.51.100.1
pass on tl0 from 192.168.1.0/24 to any
This is what I meant when I said that I took my code
"almost verbatim" from the FAQ.
Also, "The Book of pf", 3rd ed., has a similar match/pass combo on p.53:
match out on $ext_if inet from $localnet nat-to ($ext_if)
pass quick inet proto { tcp, udp } from $localnet to port $udp_services
(See how the "from" address in the pass rule is
$localnet, which would match the packet's original
source address, and not $(ext_if)?)
To make my second point, If the packet would
immediately change its effective source address to
that of the outgoing interface, it would for example
become very hard in later rules to distinguish between
NATed packets and packets originating on the firewall, at least in pass
out rules.
It would be a lot easier to just have a pass out rule and only filter
ingress traffic.
Agreed, but at this stage, I'm only trying to understand what's going on.
Is there anything you see in these rules, especially
in rules 5 and 6, that is not correct? I don't think
so, I've taken this almost verbatim from the pf FAQ
https://www.openbsd.org/faq/pf/nat.html.
You did not read that FAQ carefully enough, so I wouldn't say you
have followed it "almost verbatim". Under Configuring NAT, the example
shows
match out on interface [af] \
from src_addr to dst_addr \
nat-to ext_addr [pool_type] [static-port]
[...]
pass out [log] on interface [af] [proto protocol] \
from ext_addr [port src_port] \
to dst_addr [port dst_port]
Notice the pass out rule which states "from *ext_addr*" _not_ "from
src_addr".
Agreed! These very lines were the subject of a previous post of mine,
and two people from t...@openbsd.org said that it was a typo and that
the ext_addr in the pass rule should be changed to src_addr. See
https://marc.info/?l=openbsd-tech=168714686620055=2 Note also that
these lines are in conflict with the example that comes later in the
FAQ, and which I have quoted above.
You are also not specifying the IP version which you likely should since
you probably don't want to rely on NAT for IPv6. For example
match out log on em0 inet from athn0:network nat-to (em0)
pass out
Fair point, thanks.
Cheers
Stephan
Re: Weird pf NAT failure on apu2
On 6/23/23 11:19, Stephan Neuhaus wrote: # Rule 5 match out log on em0 from athn0:network to any nat-to (em0) # Rule 6 pass out log on em0 from athn0:network to any Rule 5 replaces the source IP address with the IP address assigned to em0-as well as replaces the source port (for TCP and UDP) with an ephemeral port. Rule 6 does _not_ pass traffic out of em0 from the IP address assigned to em0 but instead passes traffic where the IP address is from athn0:network, so of course it won't work. It would be a lot easier to just have a pass out rule and only filter ingress traffic. Is there anything you see in these rules, especially in rules 5 and 6, that is not correct? I don't think so, I've taken this almost verbatim from the pf FAQ https://www.openbsd.org/faq/pf/nat.html. You did not read that FAQ carefully enough, so I wouldn't say you have followed it "almost verbatim". Under Configuring NAT, the example shows match out on interface [af] \ from src_addr to dst_addr \ nat-to ext_addr [pool_type] [static-port] [...] pass out [log] on interface [af] [proto protocol] \ from ext_addr [port src_port] \ to dst_addr [port dst_port] Notice the pass out rule which states "from *ext_addr*" _not_ "from src_addr". You are also not specifying the IP version which you likely should since you probably don't want to rely on NAT for IPv6. For example match out log on em0 inet from athn0:network nat-to (em0) pass out
Re: Weird pf NAT failure on apu2
On 6/23/23 13:19, Stephan Neuhaus wrote:
[...]
Some people have replied to this post off-list and
have made the entirely reasonable conjecture that the
packet changes its effective source address the moment
the match rule matches. With the changed source
address, the pass rule no longer matches. That is
entirely possible and agrees with all the experimental
evidence I have.
Still, I don't think that this is what's going on, for
the following reasons.
1. It is in conflict with the documentation. The FAQ
http://www.openbsd.org/faq/pf/nat.html says
match
When a packet traverses the ruleset and matches a match rule, any
optional parameters specified in that rule are remembered for future use
(made "sticky").
pass
This rule allows the packet to be transmitted. If the packet was
previously matched by a match rule where parameters were specified, they
will be applied to this packet. [...]
This makes it very clear that the nat-to in the match
rule is only "remembered for future use", and is
applied to the packet only when it is finally subject
to a pass rule.
Similarly, "The Book of pf", 3rd ed., has a match/pass
combo on p.53 that goes
match out on $ext_if inet from $localnet nat-to ($ext_if)
pass quick inet proto { tcp, udp } from $localnet to port $udp_services
As you can see, the pass rule has the packet's
original source specification $localnet, not the
natted-to $(ext_if).
2. The match/pass combo would be much less useful. If
the packet would immediately change its effective
source address to that of the outgoing interface, it
would for example become very hard in later rules to
distinguish between NATed packets and packets
originating on the firewall.
Cheers
Stephan
Re: Weird pf NAT failure on apu2
On 6/23/23 13:19, Stephan Neuhaus wrote: Hi list [...] In other words, now the same packets that weren't passed using the match/pass combo are not passed when the nat-to is part of the pass rule. That should have been "...combo are NOW passed...". Sorry. Cheers Stephan
Weird pf NAT failure on apu2
Hi list I am using a PC Engines apu2 board as a firewall. Or rather, I want to use it as one, but it doesn't work as I think it should. First up, some information about my system. It has three gigabit wired Ethernet interfaces, em0, em1, and em2, as well as an 802.11n interface, athn0. Only em0 and athn0 will be relevant for this case. # uname -a OpenBSD my.host.name 7.3 GENERIC.MP#1125 amd64 # ifconfig em0 # I've zeroed out the lladdr em0: flags=8843 mtu 1500 lladdr 00:00:00:00:00:00 description: egress interface index 1 priority 0 llprio 3 groups: egress media: Ethernet autoselect (1000baseT full-duplex,master,rxpause,txpause) status: active inet 192.168.0.2 netmask 0xff00 broadcast 192.168.0.255 # ifconfig athn0 # I've zeroed out the lladdr, bssid and changed the nwid. athn0: flags=8843 mtu 1500 lladdr 00:00:00:00:00:00 description: wireless interface index 4 priority 4 llprio 3 groups: wlan media: IEEE802.11 autoselect mode 11n hostap status: active ieee80211: nwid mynw chan 56 bssid 00:00:00:00:00:00 -67dBm wpakey wpaprotos wpa2 wpaakms psk wpaciphers ccmp wpagroupcipher ccmp inet 192.168.3.2 netmask 0xff00 broadcast 192.168.3.255 What I want to do is NAT the wireless interface to the egress interface. I have this experimental pf setup, which has many problems, and which therefore has a big comment at the top: # PF configuration file to test matching and NAT # # DO NOT USE IN PRODUCTION # set skip on lo block log all # Rule 0 pass in on em0 proto tcp to any port 22# Rule 1 pass out on em0 proto tcp from any port 22 # Rule 2 pass out on em0 proto udp to any port 53 # Rule 3 pass in on athn0 # Rule 4 # Rule 5 match out log on em0 from athn0:network to any nat-to (em0) # Rule 6 pass out log on em0 from athn0:network to any Rules 1--3 are there so I can do my experiments over SSH. These rules affect the rule numbering as shown by pflog, but are otherwise not the point. Is there anything you see in these rules, especially in rules 5 and 6, that is not correct? I don't think so, I've taken this almost verbatim from the pf FAQ https://www.openbsd.org/faq/pf/nat.html. When I connect my phone to the wireless network on athn0, this is what I see: # doas tcpdump -n -e -ttt -i pflog0 tcpdump: WARNING: snaplen raised from 116 to 160 tcpdump: listening on pflog0, link-type PFLOG Jun 23 12:48:28.349710 rule def/(ip-option) pass in on athn0: :: > ff02::16: HBH multicast listener report v2, 1 group record(s) [hlim 1] Jun 23 12:48:28.714929 rule 5/(match) match out on em0: 192.168.3.32.54151 > 172.217.168.68.443: S 940430546:940430546(0) win 65535 (DF) Jun 23 12:48:28.714932 rule 0/(match) block out on em0: 192.168.3.32.54151 > 172.217.168.68.443: S 940430546:940430546(0) win 65535 (DF) Jun 23 12:48:28.716461 rule 5/(match) match out on em0: 192.168.3.32.37973 > 216.58.215.227.80: S 2291102750:2291102750(0) win 65535 (DF) Jun 23 12:48:28.716463 rule 0/(match) block out on em0: 192.168.3.32.37973 > 216.58.215.227.80: S 2291102750:2291102750(0) win 65535 (DF) Jun 23 12:48:29.728732 rule 5/(match) match out on em0: 192.168.3.32.54151 > 172.217.168.68.443: S 940430546:940430546(0) win 65535 (DF) Jun 23 12:48:29.728736 rule 0/(match) block out on em0: 192.168.3.32.54151 > 172.217.168.68.443: S 940430546:940430546(0) win 65535 (DF) As you can see, the packet is being matched by the match rule (rule 5) but then NOT matched by the pass rule (rule 6) and consequently by the block-all rule (rule 0). When I remove rule 5 and change rule 6 to this (making it the new rule 5): pass out log on em0 from athn0:network to any nat-to (em0) this is what I see in the logs: Jun 23 12:50:59.791736 rule def/(ip-option) pass in on athn0: :: > ff02::16: HBH multicast listener report v2, 1 group record(s) [hlim 1] Jun 23 12:51:00.091647 rule def/(ip-option) pass in on athn0: :: > ff02::16: HBH multicast listener report v2, 1 group record(s) [hlim 1] Jun 23 12:51:00.152530 rule 5/(match) pass out on em0: 192.168.3.32.37988 > 216.58.215.227.80: S 749016608:749016608(0) win 65535 (DF) Jun 23 12:51:00.152614 rule 5/(match) pass out on em0: 192.168.3.32.54168 > 172.217.168.68.443: S 2019795291:2019795291(0) win 65535 (DF) In other words, now the same packets that weren't passed using the match/pass combo are not passed when the nat-to is part of the pass rule. No matter how I read the docs, there is no way I can explain what's happening. Yet my setup is so simple that I MUST be doing something wrong, right? Can you help me see what's going on? Cheers Stephan PS: I didn't want to make this message even longer by including a dmesg, but it is of course available on request.
