Hi Daniel, Thank you so much for taking the time to recreate my (rather large) setup, and for posting it. I double checked my setup compared to your examples here to make sure all was equivalent. I dumbed down my pf.conf as you suggested. Still the same symptoms occur.
Then, as a last resort, I went and looked at my sysctl.conf file and started playing with some of my custom settings. Voila! net.inet.ip.fastforwarding is the culprit. Switching net.inet.ip.fastforwarding: 1 -> 0 fixes the issue and allows all packets to match and be encrypted. Switching net.inet.ip.fastforwarding: 0 -> 1 and the encryption breaks again. My use of this sysctl dates back to some optimization I'd done when running FreeBSD v7.1. Does anyone here have a current understanding/opinion of its role and necessity on a FreeBSD v8.3 router/gateway? Best, Daniel On May 17, 2013, at 4:34 AM, Daniel Hartmeier <[email protected]> wrote: > I rebuilt your setup but can't reproduce the problem. > > I picked A.A.A.A=3.3.3.3 and B.B.B.B=4.4.4.4 and used FreeBSD 8.3-STABLE > i386 with GENERIC plus IPSEC, and installed ipsec-tools-0.8.0_3. > > ------------------------------ gatewayA ------------------------------ > > /etc/rc.conf > ifconfig_em0="inet 1.1.1.254 netmask 255.255.255.0" > ifconfig_em1="inet 3.3.3.3 netmask 255.255.255.0" > gif_interfaces="gif0" > gifconfig_gif0="3.3.3.3 4.4.4.4" > ifconfig_gif0="1.1.1.254 2.2.2.254 netmask 255.255.255.0" > defaultrouter="3.3.3.1" > static_routes="gif" > route_gif="-net 2.2.2.0/24 2.2.2.254" > gateway_enable="YES" > racoon_enable="YES" > pf_enable="YES" > > # ifconfig gif0 > gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280 > tunnel inet 3.3.3.3 --> 4.4.4.4 > inet 1.1.1.254 --> 2.2.2.254 netmask 0xffffff00 > options=1<ACCEPT_REV_ETHIP_VER> > > # netstat -anr > Destination Gateway Flags Refs Use Netif Expire > default 3.3.3.1 UGS 0 1948 em1 > 1.1.1.0/24 link#1 U 0 1270 em0 > 1.1.1.254 link#1 UHS 1 0 lo0 > 2.2.2.0/24 2.2.2.254 UGS 0 1873 gif0 > 2.2.2.254 link#5 UH 0 39 gif0 > 3.3.3.0/24 link#2 U 0 0 em1 > 3.3.3.3 link#2 UHS 0 0 lo0 > 127.0.0.1 link#4 UH 0 0 lo0 > > /etc/pf.conf is a simple (and identical on gatewayB) > set state-policy if-bound > set skip on { lo } > scrub in log all fragment reassemble > block log > pass > > # pfctl -ss > em0 icmp 2.2.2.2:25352 <- 1.1.1.1 0:0 > em1 esp 3.3.3.3 -> 4.4.4.4 MULTIPLE:MULTIPLE > > /usr/local/etc/racoon/psk.txt > 4.4.4.4 topsecret > > /usr/local/etc/racoon/ipsec.conf > flush; > spdflush; > spdadd 1.1.1.0/24 2.2.2.0/24 any -P out ipsec esp/tunnel/3.3.3.3-4.4.4.4/use; > spdadd 2.2.2.0/24 1.1.1.0/24 any -P in ipsec esp/tunnel/4.4.4.4-3.3.3.3/use; > > /usr/local/etc/racoon/racoon.conf > (exact copy of handbook example, only differences:) > listen > isakmp 3.3.3.3 [500]; > isakmp_natt 3.3.3.3 [4500]; > remote 4.4.4.4 [500] > my_identifier address 3.3.3.3; > peers_identifier address 4.4.4.4; > sainfo (address 1.1.1.0/24 any address 2.2.2.0/24 any) > > ------------------------------ gatewayB ------------------------------ > > ifconfig_em0="inet 2.2.2.254 netmask 255.255.255.0" > ifconfig_em1="inet 4.4.4.4 netmask 255.255.255.0" > gif_interfaces="gif0" > gifconfig_gif0="4.4.4.4 3.3.3.3" > ifconfig_gif0="2.2.2.254 1.1.1.254 netmask 255.255.255.0" > defaultrouter="4.4.4.1" > static_routes="gif" > route_gif="-net 1.1.1.0/24 1.1.1.254" > gateway_enable="YES" > racoon_enable="NO" > pf_enable="YES" > > # ifconfig gif0 > gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280 > tunnel inet 4.4.4.4 --> 3.3.3.3 > inet 2.2.2.254 --> 1.1.1.254 netmask 0xffffff00 > options=1<ACCEPT_REV_ETHIP_VER> > > # netstat -anr > Destination Gateway Flags Refs Use Netif Expire > default 4.4.4.1 UGS 0 2066 em1 > 1.1.1.0/24 1.1.1.254 UGS 0 2023 gif0 > 1.1.1.254 link#5 UH 0 0 gif0 > 2.2.2.0/24 link#1 U 0 1984 em0 > 2.2.2.254 link#1 UHS 1 0 lo0 > 4.4.4.0/24 link#2 U 0 0 em1 > 4.4.4.4 link#2 UHS 0 0 lo0 > 127.0.0.1 link#4 UH 0 0 lo0 > > # pfctl -ss > em1 esp 4.4.4.4 <- 3.3.3.3 MULTIPLE:MULTIPLE > em0 icmp 1.1.1.1:25352 -> 2.2.2.2 0:0 > > /usr/local/etc/racoon/psk.txt > 3.3.3.3 topsecret > > /usr/local/etc/racoon/ipsec.conf > flush; > spdflush; > spdadd 2.2.2.0/24 1.1.1.0/24 any -P out ipsec esp/tunnel/4.4.4.4-3.3.3.3/use; > spdadd 1.1.1.0/24 2.2.2.0/24 any -P in ipsec esp/tunnel/3.3.3.3-4.4.4.4/use; > > /usr/local/etc/racoon/racoon.conf > listen > isakmp 4.4.4.4 [500]; > isakmp_natt 4.4.4.4 [4500]; > remote 3.3.3.3 [500] > my_identifier address 4.4.4.4; > peers_identifier address 3.3.3.3; > sainfo (address 2.2.2.0/24 any address 1.1.1.0/24 any) > > ------------------------------ router ------------------------------ > > When I ping from gatewayB to 1.1.1.1 (or from 1.1.1.1 to 2.2.2.2), > I see only encrypted packets: > > 13:23:52.800285 IP (tos 0x0, ttl 63, id 6391, offset 0, flags [none], proto > ESP (50), length 136) > 4.4.4.4 > 3.3.3.3: ESP(spi=0x016bdbe7,seq=0x5e), length 116 > 13:23:52.801401 IP (tos 0x0, ttl 64, id 5827, offset 0, flags [none], proto > ESP (50), length 136) > 3.3.3.3 > 4.4.4.4: ESP(spi=0x04049e8b,seq=0x5e), length 116 > 13:23:53.820296 IP (tos 0x0, ttl 63, id 6394, offset 0, flags [none], proto > ESP (50), length 136) > 4.4.4.4 > 3.3.3.3: ESP(spi=0x016bdbe7,seq=0x5f), length 116 > 13:23:53.821230 IP (tos 0x0, ttl 64, id 5829, offset 0, flags [none], proto > ESP (50), length 136) > 3.3.3.3 > 4.4.4.4: ESP(spi=0x04049e8b,seq=0x5f), length 116 > > There must be something in your setup that causes the difference. > > If there's a non-trivial pf.conf, maybe try with a trivial one first. > > Kind regards, > Daniel -- daniel duerr | president | ouido.net [email protected] | +1 (831) 531-2272 x103 Managed hosting services for Business
