Hi. I found a weird artifacts on one of my FreeBSD installations. Here comes the traceroute and tcpdumps outputs, mostly self-explanatory. Problem: some of the ICMP exceeded in transit replies have source IP translated to the original traceroute destination IP (i.e. I traceroute host A, and some of the packets on the third hop returns with the source IP of the host A, which is impossible). As you can see below, originating host receives a traceroute picture that is really weird. In the same time the border passes clearly valid packets. Something bad happens on the NAT itself. All the three hosts run FreeBSD with pf, different releases, mostly 10.x branch.
I have a border configuration with two borders in CARP. First I thought that this could be explaining if the traceroute session is somehow split between borders, but, as you can see below, the session is handled by only one border, from first to the last packet. All the outputs are captured during the same traceroute pass/ Host one - ICMP originator: traceroute -P icmp 153.92.28.82 traceroute to 153.92.28.82 (153.92.28.82), 64 hops max, 48 byte packets 1 192.168.7.7 (192.168.7.7) 0.129 ms 0.227 ms 0.116 ms 2 wizard.hq.norma.perm.ru (128.127.144.1) 0.379 ms 153.92.28.82 (153.92.28.82) 0.313 ms wizard.hq.norma.perm.ru (128.127.144.1) 0.246 ms 3 153.92.28.82 (153.92.28.82) 1.153 ms 0.999 ms prm01.prm28.transtelecom.net (188.43.17.174) 0.923 ms 4 153.92.28.82 (153.92.28.82) 69.619 ms rtr01.da-rz.net (80.81.194.157) 64.087 ms 153.92.28.82 (153.92.28.82) 60.011 ms 5 153.92.28.82 (153.92.28.82) 60.124 ms 60.004 ms 59.983 ms it's tcpdump: # tcpdump -npi re0 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on re0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:08:49.703343 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 1, length 28 12:08:49.703434 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.712355 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 2, length 28 12:08:49.712505 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.712548 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 3, length 28 12:08:49.712644 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.712668 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 4, length 28 12:08:49.713032 IP 128.127.144.1 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.713552 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 5, length 28 12:08:49.713818 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.714239 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 6, length 28 12:08:49.714468 IP 128.127.144.1 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.714948 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 7, length 28 12:08:49.716088 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.716716 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 8, length 28 12:08:49.717654 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.717718 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 9, length 28 12:08:49.718581 IP 188.43.17.174 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.718982 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 10, length 28 12:08:49.788448 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.789403 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 11, length 28 12:08:49.853330 IP 80.81.194.157 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.854609 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 12, length 28 12:08:49.914486 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.915685 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 13, length 28 12:08:49.975603 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602, seq 13, length 28 12:08:49.976377 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 14, length 28 12:08:50.036233 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602, seq 14, length 28 12:08:50.036381 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 15, length 28 12:08:50.096203 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602, seq 15, length 28 Host with NAT - tcpdump on the LAN interface (facing ICMP originator; spolier: some ICMP replies are translated to the IP of the destination host): # tcpdump -npi vlan15 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan15, link-type EN10MB (Ethernet), capture size 65535 bytes 12:08:42.108537 IP 192.168.7.253 > 192.168.3.9: ICMP 192.168.7.253 udp port 623 unreachable, length 57 12:08:42.200953 IP 192.168.7.138 > 192.168.142.220: ICMP echo request, id 512, seq 61241, length 19 12:08:44.501117 IP 192.168.7.123 > 192.168.7.6: ICMP echo request, id 39200, seq 0, length 64 12:08:44.501132 IP 192.168.7.6 > 192.168.7.123: ICMP echo reply, id 39200, seq 0, length 64 12:08:47.108923 IP 192.168.7.253 > 192.168.3.9: ICMP 192.168.7.253 udp port 623 unreachable, length 57 12:08:47.684410 IP 192.168.7.138 > 192.168.142.220: ICMP echo request, id 512, seq 61497, length 19 12:08:49.694248 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 1, length 28 12:08:49.694267 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.703258 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 2, length 28 12:08:49.703266 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.703454 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 3, length 28 12:08:49.703461 IP 192.168.7.7 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.703573 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 4, length 28 12:08:49.703874 IP 128.127.144.1 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.704453 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 5, length 28 12:08:49.704659 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.705141 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 6, length 28 12:08:49.705309 IP 128.127.144.1 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.705864 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 7, length 28 12:08:49.706929 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.707656 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 8, length 28 12:08:49.708495 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.708625 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 9, length 28 12:08:49.709421 IP 188.43.17.174 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.709884 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 10, length 28 12:08:49.779249 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.780345 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 11, length 28 12:08:49.844153 IP 80.81.194.157 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.845512 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 12, length 28 12:08:49.905325 IP 153.92.28.82 > 192.168.7.96: ICMP time exceeded in-transit, length 36 12:08:49.906601 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 13, length 28 12:08:49.966390 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602, seq 13, length 28 12:08:49.967282 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 14, length 28 12:08:50.027041 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602, seq 14, length 28 12:08:50.027284 IP 192.168.7.96 > 153.92.28.82: ICMP echo request, id 46602, seq 15, length 28 12:08:50.086991 IP 153.92.28.82 > 192.168.7.96: ICMP echo reply, id 46602, seq 15, length 28 border interface facing host with NAT (spolier: everything is normal ): # tcpdump -npi vlan23 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan23, link-type EN10MB (Ethernet), capture size 65535 bytes 12:08:49.704074 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 4, length 28 12:08:49.704086 IP 128.127.144.1 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.704879 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 5, length 28 12:08:49.704887 IP 128.127.144.1 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.705523 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 6, length 28 12:08:49.705532 IP 128.127.144.1 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.706324 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 7, length 28 12:08:49.707132 IP 188.43.17.174 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.708122 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 8, length 28 12:08:49.708660 IP 188.43.17.174 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.709110 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 9, length 28 12:08:49.709571 IP 188.43.17.174 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.710234 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 10, length 28 12:08:49.779444 IP 80.81.194.157 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.780816 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 11, length 28 12:08:49.844373 IP 80.81.194.157 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.845883 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 12, length 28 12:08:49.905547 IP 80.81.194.157 > 128.127.144.3: ICMP time exceeded in-transit, length 36 12:08:49.906993 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 13, length 28 12:08:49.966620 IP 153.92.28.82 > 128.127.144.3: ICMP echo reply, id 44550, seq 13, length 28 12:08:49.967736 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 14, length 28 12:08:50.027266 IP 153.92.28.82 > 128.127.144.3: ICMP echo reply, id 44550, seq 14, length 28 12:08:50.027746 IP 128.127.144.3 > 153.92.28.82: ICMP echo request, id 44550, seq 15, length 28 12:08:50.087208 IP 153.92.28.82 > 128.127.144.3: ICMP echo reply, id 44550, seq 15, length 28 Eugene. _______________________________________________ freebsd-pf@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-pf To unsubscribe, send any mail to "freebsd-pf-unsubscr...@freebsd.org"