PF/NAT UDP fragment problem
I hope somebody here can help me with a PF/NAT problem I'm having. I'm
moving a machine of ours from OpenBSD 2.9 to 3.2. This was all working
with IPF running on the 2.9 install.
The machine in question (nat) is working as a front for a DNS(SEC)
server. It takes UDP or TCP packets to port 53 on its external
interface (xl0) and forwards them out its internal interface (xl1) to
the real server running on a different. The machine the real server is
running on (dennis) is further back in our network, behind a few
firewalls and in private IP space. Finally, nat is not along the
default path of dennis. Because of this, the source address of the
redirected packets needs to be rewritten to nat's internal IP address.
One side note about this. Normal DNS (UDP) queries do not fragment, but
it's not unusual to see DNSSEC queries fragment. This is the whole
reason we set up nat in the first place. The older BigIP boxes we were
using were dropping the UDP fragments returned by dennis. Non-DNSSEC
queries would work and TCP-based DNSSEC queries would work, but the
UDP-based DNSSEC queries would fail.
With OpenBSD 2.9, I was able to get this set up working without any
problems using the following:
rdr xl0 PP.PP.PP.PP/32 port 53 - DD.DD.DD.DD port 5353 tcp/udp
map xl1 from 0.0.0.0/0 to DD.DD.DD.DD port = 5353 - II.II.II.II/32 portmap tcp/udp
1:65000
Where PP.PP.PP.PP == nat's Public IP
II.II.II.II == nat's Internal IP
DD.DD.DD.DD == dennis' IP
With OpenBSD 3.2, I'm back to the same problem I was having with the
BigIPs. It looks like nat's dropping the UDP fragments returned from
dennis. This is the first time I've used pf, so I'm fairly certain that
it's just a problem with configuration of pf. I'm hoping someone can
help me.
This is the entire pf.conf file I'm using:
ExtIf=xl0
IntIf=xl1
DNS=PP.PP.PP.PP
Dennis=DD.DD.DD.DD
#scrub in all
scrub in on $ExtIf all fragment reassemble
scrub in on $IntIf all fragment reassemble
rdr on $ExtIf proto {udp,tcp} from any to $DNS port 53 - $Dennis port 5353
nat on $IntIf proto {udp,tcp} from any to $Dennis port 5353 - $IntIf
pass in all
pass out all
I tried the commented out scrub in all vs the other two scrub
commands, but that didn't help.
With this configuration loaded, normal UDP queries to PP.PP.PP.PP work
fine. TCP-based DNSSEC queries also work fine. The UDP-based DNSSEC
queris are failing. From tcpdumps on nat's two interfaces and on
dennis' interface, here's the patch I stitched together... (dennis' time
is off slightly.)
= DNSSEC query comes into nat's external interface
14:33:26.917445 XX.XX.XX.XX.43263 PP.PP.PP.PP.53: 18333+ [1au] A? foo.com. (36)
(DF)
= Nat RDRs it to dennis (DD.DD.DD.DD) and NATs it to its
internal interface (II.II.II.II)
14:33:26.917596 II.II.II.II.56175 DD.DD.DD.DD.5353: udp 36 (DF)
= Dennis receives the query...
14:33:26.752437 II.II.II.II.56175 DD.DD.DD.DD.5353: udp 36 (DF)
= ...and replies to it. This is where the frag begins.
14:33:26.784417 DD.DD.DD.DD II.II.II.II: (frag 55914:[EMAIL PROTECTED])
14:33:26.784437 DD.DD.DD.DD.5353 II.II.II.II.56175: udp 1643 (frag 55914:[EMAIL
PROTECTED])
= Nat receives the two reply fragments
14:33:26.951181 DD.DD.DD.DD.5353 II.II.II.II.56175: udp 1643 (frag 55914:[EMAIL
PROTECTED])
14:33:26.951195 DD.DD.DD.DD II.II.II.II: (frag 55914:[EMAIL PROTECTED])
= Nothing goes out Nat's external interface
Anybody have any ideas? Am I using scrub incorrectly? Should I be
using scrub? Is there something else I'm doing wrong? Is there any
other potentially useful information I forgot to give?
BTW, this machine is running -STABLE, patched up to Wednesday night.
Thanks,
pete
Re: PF/NAT UDP fragment problem
On Fri, Mar 07, 2003 at 11:45:16AM -0500, Pete Toscano wrote: Anybody have any ideas? Am I using scrub incorrectly? Should I be using scrub? Is there something else I'm doing wrong? Is there any other potentially useful information I forgot to give? Your ruleset looks fine, that's exactly how it should work (rdr on external, nat on internal, scrub on both). = ...and replies to it. This is where the frag begins. 14:33:26.784417 DD.DD.DD.DD II.II.II.II: (frag 55914:[EMAIL PROTECTED]) 14:33:26.784437 DD.DD.DD.DD.5353 II.II.II.II.56175: udp 1643 (frag 55914:[EMAIL PROTECTED]) = Nat receives the two reply fragments 14:33:26.951181 DD.DD.DD.DD.5353 II.II.II.II.56175: udp 1643 (frag 55914:[EMAIL PROTECTED]) 14:33:26.951195 DD.DD.DD.DD II.II.II.II: (frag 55914:[EMAIL PROTECTED]) = Nothing goes out Nat's external interface It must be somehow related to the fragmentation. For some reason, the pf box is not reassembling the fragments. To determine the reason, can you a) enable debug logging with pfctl -x m, and check /var/log/messages for entries related to pf fragment reassembly? Ideally, quote all lines related to one packet's fragments being reassembled. b) get a tcpdump -nvvvXSpi $IntIF output from the pf box for all fragments of a single packet. One possible explanation would be if the fragments have the DF (don't fragment) flag set. pf, prior to -current as of a few weeks ago, drops them unconditionally. If that's the problem, you could try a snapshot (which is stable, now that we approach 3.3-release). If not, hopefully the additional output from above shows something. Daniel
Re: PF/NAT UDP fragment problem
On Fri, Mar 07, 2003 at 03:27:06PM -0500, Pete Toscano wrote:
That's good to know. Would scrub in all work just as well as scrub
in on {$ExtIf, $IntIf} all fragment reassemble?
Yes, 'fragment reassemble' is the default, so both do the same thing
(unless you have additional interfaces that you don't want to scrub on,
of course). If you load 'scrub in all' and then display the loaded rules
(with pfctl -sr), you'll see that it gets loaded as 'fragment
reassemble'.
Mar 7 15:20:02 reflect /bsd: pf_normalize_ip: IP_DF
Mar 7 15:20:02 reflect /bsd: pf_normalize_ip: dropping bad fragment
Ok, that's it, then.
Excellent. Thank you for the help. I'll try -current and see how that
turns out. If it's still a problem, I'll include the dumped packets,
but I think you found the issue.
With a recent snapshot, add 'no-df' to your scrub rule(s), and pf will
strip the IP_DF flag before reassembly, so the packets can get
reassembled. The snapshots on the ftp mirrors already include that
change.
Daniel
Re: PF/NAT UDP fragment problem
On Fri, 07 Mar 2003, Daniel Hartmeier wrote:
Your ruleset looks fine, that's exactly how it should work (rdr on
external, nat on internal, scrub on both).
That's good to know. Would scrub in all work just as well as scrub
in on {$ExtIf, $IntIf} all fragment reassemble?
It must be somehow related to the fragmentation. For some reason, the pf
box is not reassembling the fragments. To determine the reason, can you
a) enable debug logging with pfctl -x m, and check /var/log/messages
for entries related to pf fragment reassembly? Ideally, quote all
lines related to one packet's fragments being reassembled.
A few of these lines were repeated in /var/log/messages. Here they are
without the repeats.
pf_normalize_ip: IP_DF
pf_normalize_ip: dropping bad fragment
Mar 7 15:20:02 reflect /bsd: pf_normalize_ip: IP_DF
Mar 7 15:20:02 reflect /bsd: pf_normalize_ip: dropping bad fragment
b) get a tcpdump -nvvvXSpi $IntIF output from the pf box for all
fragments of a single packet.
One possible explanation would be if the fragments have the DF (don't
fragment) flag set.
Indeed, it does. I took a look at the tcpdump and the fragments do have
the DF flag set.
pf, prior to -current as of a few weeks ago, drops
them unconditionally. If that's the problem, you could try a snapshot
(which is stable, now that we approach 3.3-release). If not, hopefully
the additional output from above shows something.
Excellent. Thank you for the help. I'll try -current and see how that
turns out. If it's still a problem, I'll include the dumped packets,
but I think you found the issue.
Thanks again,
pete
