In-kernel NAT [ipfw] dropping large UDP return packets
When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC
tunnel to the T-Mobile provisioning servers, the reassembled, 4640-byte
return packet is silently dropped by the in-kernel NAT, even though it
"matches" the outbound packet from less than 100 ms prior.
All other operations of the firewall seem to be functioning as expected.
This includes iPhones using "WiFi Calling" which utilizes similar IPSEC
connections to T-Mobile servers (though fragmentation has not been seen
on those connections). The connection for the femto-cell can be handled
by a Linux/netfilter NAT. Proper reassembly of the packet fragments
within the firewall ("reass ip from any to any"), at the start of the
rule set, prior to NAT, has been confirmed with ngtee and wireshark.
Is there a known issue with large packets and in-kernel NAT?
The only sysctl that I found that seemed related was the UDP timeout.
For good measure I upped it to 30 (seconds), but that did not change the
behavior.
Are there known causes and/or resolutions for this behavior?
Is there a way to be able to "monitor" the NAT table?
(I didn't see anything obvious in the ipfw, natd, or libalias man pages.)
Thanks!
Jeff
11.1-RELEASE-p9 FreeBSD 11.1-RELEASE-p9 #0: Tue Apr 3 16:59:16 UTC 2018
r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
Additional details at
https://forums.freebsd.org/threads/in-kernel-nat-dropping-large-udp-return-packets.66262/
pcapng files and ipfw rule set available on request to freebsd {a}
wagsky {.} com
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On Wed, Jun 13, 2018 at 10:16 AM, Jeff Kletsky wrote: When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC tunnel > to the T-Mobile provisioning servers, the reassembled, 4640-byte return > packet is silently dropped by the in-kernel NAT, even though it "matches" > the outbound packet from less than 100 ms prior. Do you have a 'reass' rule before applying nat on inbound traffic? - M -- "Well," Brahma said, "even after ten thousand explanations, a fool is no wiser, but an intelligent person requires only two thousand five hundred." - The Mahābhārata ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On 6/13/18 10:22 AM, Michael Sierchio wrote: On Wed, Jun 13, 2018 at 10:16 AM, Jeff Kletsky wrote: When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC tunnel to the T-Mobile provisioning servers, the reassembled, 4640-byte return packet is silently dropped by the in-kernel NAT, even though it "matches" the outbound packet from less than 100 ms prior. Do you have a 'reass' rule before applying nat on inbound traffic? - M Yes, at the start of the rule set. Reassembly confirmed to be working by wireshark examination of the ngtee "taps" shown $ sudo ipfw list 1 deny ip from any to any recv ng* 4 ngtee 100 ip from any to any proto udp dst-port 500,4500 in 4 ngtee 100 ip from any to any proto udp frag in 4 ngtee 110 ip from any to any proto udp dst-port 500,4500 out 4 ngtee 110 ip from any to any proto udp frag out 5 reass ip from any to any 6 ngtee 101 ip from any to any proto udp dst-port 500,4500 in // reassembled in 6 ngtee 101 ip from any to any proto udp frag in // never should be frags after reass 6 ngtee 111 ip from any to any proto udp dst-port 500,4500 out // reass out 6 ngtee 111 ip from any to any proto udp frag out // never should be frage after reass [...] ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
Re: In-kernel NAT [ipfw] dropping large UDP return packets
I see you have a case of Netgraph. Perhaps Julian will chime in. On Wed, Jun 13, 2018 at 10:32 AM, Jeff Kletsky wrote: > On 6/13/18 10:22 AM, Michael Sierchio wrote: > > On Wed, Jun 13, 2018 at 10:16 AM, Jeff Kletsky wrote: >> >> When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC tunnel >> >>> to the T-Mobile provisioning servers, the reassembled, 4640-byte return >>> packet is silently dropped by the in-kernel NAT, even though it "matches" >>> the outbound packet from less than 100 ms prior. >>> >> >> >> Do you have a 'reass' rule before applying nat on inbound traffic? >> >> - M >> > Yes, at the start of the rule set. > > Reassembly confirmed to be working by wireshark examination of the ngtee > "taps" shown > > $ sudo ipfw list > 1 deny ip from any to any recv ng* > 4 ngtee 100 ip from any to any proto udp dst-port 500,4500 in > 4 ngtee 100 ip from any to any proto udp frag in > 4 ngtee 110 ip from any to any proto udp dst-port 500,4500 out > 4 ngtee 110 ip from any to any proto udp frag out > 5 reass ip from any to any > 6 ngtee 101 ip from any to any proto udp dst-port 500,4500 in // > reassembled in > 6 ngtee 101 ip from any to any proto udp frag in // never should be > frags after reass > 6 ngtee 111 ip from any to any proto udp dst-port 500,4500 out // > reass out > 6 ngtee 111 ip from any to any proto udp frag out // never should be > frage after reass > [...] > > > ___ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org" > -- "Well," Brahma said, "even after ten thousand explanations, a fool is no wiser, but an intelligent person requires only two thousand five hundred." - The Mahābhārata ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On 13.06.2018 20:16, Jeff Kletsky wrote: > When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC > tunnel to the T-Mobile provisioning servers, the reassembled, 4640-byte > return packet is silently dropped by the in-kernel NAT, even though it > "matches" the outbound packet from less than 100 ms prior. > Are there known causes and/or resolutions for this behavior? > > Is there a way to be able to "monitor" the NAT table? > > (I didn't see anything obvious in the ipfw, natd, or libalias man pages.) The kernel version of libalias uses m_megapullup() function to make single contiguous buffer. m_megapullup() uses m_get2() function to allocate mbuf of appropriate size. If size of packet greater than 4k it will fail. So, if you use MTU greater than 4k or if after fragments reassembly you get a packet with length greater than 4k, ipfw_nat() function will drop this packet. -- WBR, Andrey V. Elsukov signature.asc Description: OpenPGP digital signature
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On 6/13/18 12:01 PM, Andrey V. Elsukov wrote: On 13.06.2018 20:16, Jeff Kletsky wrote: When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC tunnel to the T-Mobile provisioning servers, the reassembled, 4640-byte return packet is silently dropped by the in-kernel NAT, even though it "matches" the outbound packet from less than 100 ms prior. Are there known causes and/or resolutions for this behavior? Is there a way to be able to "monitor" the NAT table? (I didn't see anything obvious in the ipfw, natd, or libalias man pages.) The kernel version of libalias uses m_megapullup() function to make single contiguous buffer. m_megapullup() uses m_get2() function to allocate mbuf of appropriate size. If size of packet greater than 4k it will fail. So, if you use MTU greater than 4k or if after fragments reassembly you get a packet with length greater than 4k, ipfw_nat() function will drop this packet. Thanks!! Mystery solved... /usr/src/sys/netinet/libalias/alias.c #ifdef _KERNEL /* * m_megapullup() - this function is a big hack. * Thankfully, it's only used in ng_nat and ipfw+nat. suggests that the "old school" approach of natd might resolve this. I'll give it a try when I'm close enough to the box to resolve it when I make a configuration error. Jeff ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On 13.06.2018 23:04, Jeff Kletsky wrote: >> The kernel version of libalias uses m_megapullup() function to make >> single contiguous buffer. m_megapullup() uses m_get2() function to >> allocate mbuf of appropriate size. If size of packet greater than 4k it >> will fail. So, if you use MTU greater than 4k or if after fragments >> reassembly you get a packet with length greater than 4k, ipfw_nat() >> function will drop this packet. >> > Thanks!! > > Mystery solved... > > /usr/src/sys/netinet/libalias/alias.c > > #ifdef _KERNEL > /* > * m_megapullup() - this function is a big hack. > * Thankfully, it's only used in ng_nat and ipfw+nat. > > suggests that the "old school" approach of natd might resolve this. I'll > give it a try when I'm close enough to the box to resolve it when I make > a configuration error. I didn't look at the rest of libalias, but you, probably, can improve this hack to use 9k or 16k mbufs. You can replace m_get2() call in m_megapullup() with the following code: if (len <= MJUMPAGESIZE) mcl = m_get2(len, M_NOWAIT, MT_DATA, M_PKTHDR); else if (len <= MJUM9BYTES) mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM9BYTES); else if (len <= MJUM16BYTES) mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM16BYTES); else goto bad; -- WBR, Andrey V. Elsukov signature.asc Description: OpenPGP digital signature
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On 6/13/18 1:28 PM, Andrey V. Elsukov wrote: On 13.06.2018 23:04, Jeff Kletsky wrote: The kernel version of libalias uses m_megapullup() function to make single contiguous buffer. m_megapullup() uses m_get2() function to allocate mbuf of appropriate size. If size of packet greater than 4k it will fail. So, if you use MTU greater than 4k or if after fragments reassembly you get a packet with length greater than 4k, ipfw_nat() function will drop this packet. Thanks!! Mystery solved... /usr/src/sys/netinet/libalias/alias.c #ifdef _KERNEL /* * m_megapullup() - this function is a big hack. * Thankfully, it's only used in ng_nat and ipfw+nat. suggests that the "old school" approach of natd might resolve this. I'll give it a try when I'm close enough to the box to resolve it when I make a configuration error. I didn't look at the rest of libalias, but you, probably, can improve this hack to use 9k or 16k mbufs. You can replace m_get2() call in m_megapullup() with the following code: if (len <= MJUMPAGESIZE) mcl = m_get2(len, M_NOWAIT, MT_DATA, M_PKTHDR); else if (len <= MJUM9BYTES) mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM9BYTES); else if (len <= MJUM16BYTES) mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM16BYTES); else goto bad; Tested and "works for me" on 11.1-RELEASE-p10 with GENERIC kernconf 8< --- alias.c.orig 2017-07-20 16:42:02.0 -0700 +++ alias.c 2018-06-13 15:41:46.862121000 -0700 @@ -1758,7 +1758,14 @@ if (m->m_next == NULL && M_WRITABLE(m)) return (m); - mcl = m_get2(len, M_NOWAIT, MT_DATA, M_PKTHDR); + if (len <= MJUMPAGESIZE) + mcl = m_get2(len, M_NOWAIT, MT_DATA, M_PKTHDR); + else if (len <= MJUM9BYTES) + mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM9BYTES); + else if (len <= MJUM16BYTES) + mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM16BYTES); + else + goto bad; if (mcl == NULL) goto bad; m_align(mcl, len); >8 Thanks again! Jeff ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On 14/6/18 1:41 am, Michael Sierchio wrote: I see you have a case of Netgraph. Perhaps Julian will chime in. well I'm reading but not got any specific ideas at the moment.. Netgraph itself has no requirements on packet size or even contents. a node may however have some. On Wed, Jun 13, 2018 at 10:32 AM, Jeff Kletsky wrote: On 6/13/18 10:22 AM, Michael Sierchio wrote: On Wed, Jun 13, 2018 at 10:16 AM, Jeff Kletsky wrote: When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC tunnel to the T-Mobile provisioning servers, the reassembled, 4640-byte return packet is silently dropped by the in-kernel NAT, even though it "matches" the outbound packet from less than 100 ms prior. Do you have a 'reass' rule before applying nat on inbound traffic? - M Yes, at the start of the rule set. Reassembly confirmed to be working by wireshark examination of the ngtee "taps" shown $ sudo ipfw list 1 deny ip from any to any recv ng* 4 ngtee 100 ip from any to any proto udp dst-port 500,4500 in 4 ngtee 100 ip from any to any proto udp frag in 4 ngtee 110 ip from any to any proto udp dst-port 500,4500 out 4 ngtee 110 ip from any to any proto udp frag out 5 reass ip from any to any 6 ngtee 101 ip from any to any proto udp dst-port 500,4500 in // reassembled in 6 ngtee 101 ip from any to any proto udp frag in // never should be frags after reass 6 ngtee 111 ip from any to any proto udp dst-port 500,4500 out // reass out 6 ngtee 111 ip from any to any proto udp frag out // never should be frage after reass [...] ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org" ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On 14/6/18 3:01 am, Andrey V. Elsukov wrote: On 13.06.2018 20:16, Jeff Kletsky wrote: When a T-Mobile "femto-cell" is trying to establish its IPv4, IPSEC tunnel to the T-Mobile provisioning servers, the reassembled, 4640-byte return packet is silently dropped by the in-kernel NAT, even though it "matches" the outbound packet from less than 100 ms prior. Are there known causes and/or resolutions for this behavior? Is there a way to be able to "monitor" the NAT table? (I didn't see anything obvious in the ipfw, natd, or libalias man pages.) The kernel version of libalias uses m_megapullup() function to make single contiguous buffer. m_megapullup() uses m_get2() function to allocate mbuf of appropriate size. If size of packet greater than 4k it will fail. So, if you use MTU greater than 4k or if after fragments reassembly you get a packet with length greater than 4k, ipfw_nat() function will drop this packet. hmmm that sounds like a bug to me.. why does it fail? ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
Re: In-kernel NAT [ipfw] dropping large UDP return packets
On 14/6/18 7:44 am, Jeff Kletsky wrote: On 6/13/18 1:28 PM, Andrey V. Elsukov wrote: On 13.06.2018 23:04, Jeff Kletsky wrote: The kernel version of libalias uses m_megapullup() function to make single contiguous buffer. m_megapullup() uses m_get2() function to allocate mbuf of appropriate size. If size of packet greater than 4k it will fail. So, if you use MTU greater than 4k or if after fragments reassembly you get a packet with length greater than 4k, ipfw_nat() function will drop this packet. Thanks!! Mystery solved... /usr/src/sys/netinet/libalias/alias.c #ifdef _KERNEL /* * m_megapullup() - this function is a big hack. * Thankfully, it's only used in ng_nat and ipfw+nat. suggests that the "old school" approach of natd might resolve this. I'll give it a try when I'm close enough to the box to resolve it when I make a configuration error. I didn't look at the rest of libalias, but you, probably, can improve this hack to use 9k or 16k mbufs. You can replace m_get2() call in m_megapullup() with the following code: if (len <= MJUMPAGESIZE) mcl = m_get2(len, M_NOWAIT, MT_DATA, M_PKTHDR); else if (len <= MJUM9BYTES) mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM9BYTES); else if (len <= MJUM16BYTES) mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM16BYTES); else goto bad; Tested and "works for me" on 11.1-RELEASE-p10 with GENERIC kernconf 8< --- alias.c.orig 2017-07-20 16:42:02.0 -0700 +++ alias.c 2018-06-13 15:41:46.862121000 -0700 @@ -1758,7 +1758,14 @@ if (m->m_next == NULL && M_WRITABLE(m)) return (m); - mcl = m_get2(len, M_NOWAIT, MT_DATA, M_PKTHDR); + if (len <= MJUMPAGESIZE) + mcl = m_get2(len, M_NOWAIT, MT_DATA, M_PKTHDR); + else if (len <= MJUM9BYTES) + mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM9BYTES); + else if (len <= MJUM16BYTES) + mcl = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, MJUM16BYTES); + else + goto bad; if (mcl == NULL) goto bad; m_align(mcl, len); >8 Thanks again! Hi Andey, please commit.. Jeff ___ freebsd-...@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" ___ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
