Hi All,
I have a question regarding ‘reply-to’ with pf.conf and its expected
behavior. I have tested this on 5.2, 5.3, 5.4 and latest snapshot from
20140228. It could well be that I am doing something wrong but after
reading pf.conf manpage, FAQs on openbsd.org and searching the mailing
lists I cannot see anything that would stand out as obvious straight
away. Even stranger, is that I remember doing the same thing before, and
if I remember correctly it worked without issues. So any help would be
greatly appreciated.
Below is a very simple setup that allows me to reproduce this behavior
in a self contained environment. I am using VirtualBox to demonstrate
this behavior, but it works identical on real hardware.
Both vm00 and vm01 have 4 vio interfaces:
* vio0 - NAT (disabled)
* vio1 - vm host net for control (192.168.56.0/24, vm00 is .1, and vm01 is .2)
* vio2 - interface under test (10.0.0.0/24, vm00 is .1 and vm01 is .2)
* vio3 - disabled
vm01 also has a loopback1 interface (172.16.0.1/32) used a source
address for testing. Below is some relevant output from vm00 that shows
its configuration (this is the host under test).
------------------------------------------------------------------------
# cat /etc/hostname.vio1
inet 192.168.56.4 255.255.255.0
# cat /etc/hostname.vio2
inet 10.0.0.1 255.255.255.0
------------------------------------------------------------------------
# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33144
priority: 0
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
inet 127.0.0.1 netmask 0xff000000
vio0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 08:00:27:05:4c:cf
priority: 0
media: Ethernet autoselect
status: no carrier
vio1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 08:00:27:95:47:81
priority: 0
media: Ethernet autoselect
status: active
inet 192.168.56.4 netmask 0xffffff00 broadcast 192.168.56.255
inet6 fe80::a00:27ff:fe95:4781%vio1 prefixlen 64 scopeid 0x2
vio2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 08:00:27:77:9f:07
priority: 0
media: Ethernet autoselect
status: active
inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
inet6 fe80::a00:27ff:fe77:9f07%vio2 prefixlen 64 scopeid 0x3
vio3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
lladdr 08:00:27:e8:4e:af
priority: 0
media: Ethernet autoselect
status: no carrier
enc0: flags=0<>
priority: 0
groups: enc
status: active
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33144
priority: 0
groups: pflog
------------------------------------------------------------------------
# sysctl net.inet.ip.forwarding
net.inet.ip.forwarding=1
------------------------------------------------------------------------
# route -n show -inet
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
10.0.0/24 link#3 UC 1 0 - 4 vio2
10.0.0.2 08:00:27:46:57:e4 UHLc 0 12 - 4 vio2
127/8 127.0.0.1 UGRS 0 0 33144 8 lo0
127.0.0.1 127.0.0.1 UH 1 0 33144 4 lo0
192.168.56/24 link#2 UC 1 0 - 4 vio1
192.168.56.1 08:00:27:00:1c:ce UHLc 3 617 - 4 vio1
224/4 127.0.0.1 URS 0 0 33144 8 lo0
------------------------------------------------------------------------
# cat /etc/pf.conf
set skip on lo
set skip on vio0
set skip on vio1
block log all
block quick inet6 all
pass inet proto icmp
pass out proto tcp from any to any port ssh
pass in on vio2 inet proto tcp from any to vio2 port ssh reply-to
(vio2 10.0.0.2)
------------------------------------------------------------------------
# pfctl -f /etc/pf.conf -vve
Loaded 710 passive OS fingerprints
set skip on { lo }
set skip on { vio0 }
set skip on { vio1 }
@0 block drop log all
@1 block drop quick inet6 all
@2 pass inet proto icmp all
@3 pass out proto tcp from any to any port = 22 flags S/SA
@4 pass in on vio2 inet proto tcp from any to 10.0.0.1 port = 22 flags
S/SA reply-to 10.0.0.2@vio2
pfctl: pf already enabled
------------------------------------------------------------------------
# pfctl -vvsr
@0 block drop log all
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: uid 0 pid 32125 State Creations: 0 ]
@1 block drop quick inet6 all
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: uid 0 pid 32125 State Creations: 0 ]
@2 pass inet proto icmp all
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: uid 0 pid 32125 State Creations: 0 ]
@3 pass out proto tcp from any to any port = 22 flags S/SA
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: uid 0 pid 32125 State Creations: 0 ]
@4 pass in on vio2 inet proto tcp from any to 10.0.0.1 port = 22 flags
S/SA reply-to 10.0.0.2@vio2
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: uid 0 pid 32125 State Creations: 0 ]
------------------------------------------------------------------------
# pfctl -vvss
------------------------------------------------------------------------
# ping -c 4 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=255 time=11.436 ms
64 bytes from 10.0.0.2: icmp_seq=1 ttl=255 time=10.121 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=255 time=11.034 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=255 time=1.563 ms
--- 10.0.0.2 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 1.563/8.538/11.436/4.056 ms
------------------------------------------------------------------------
>From the above it can be seen that there is no default route installed,
pf rule @4 (if I am doing everything correctly) should send response
packets back to 172.16.0.1 through 10.0.0.2 on vio2 back to vm01.
Now from vm01, I can attempt a TCP connect test to 10.0.0.1 on port 22
on vm00 from interface lo1 on vm01. At this point I can see a packet
from 172.16.0.1 coming in on vio2 on vm00, but there is no response
that is sent back on this interface.
------------------------------------------------------------------------
# telnet -b 172.16.0.1 10.0.0.1 22
Trying 10.0.0.1...
^C
------------------------------------------------------------------------
# tcpdump -i vio2 -nlt
tcpdump: listening on vio2, link-type EN10MB
tcpdump: WARNING: compensating for unaligned libpcap packets
172.16.0.1.37218 > 10.0.0.1.22: S 1863915401:1863915401(0) win 16384
<mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 3580746694 0>
(DF) [tos 0x10]
------------------------------------------------------------------------
# pfctl -vvsr -R 4
@4 pass in on vio2 inet proto tcp from any to 10.0.0.1 port = 22 flags
S/SA reply-to 10.0.0.2@vio2
[ Evaluations: 1 Packets: 1 Bytes: 64 States: 1 ]
[ Inserted: uid 0 pid 32125 State Creations: 1 ]
------------------------------------------------------------------------
# pfctl -vvss
all tcp 10.0.0.1:22 <- 172.16.0.1:37218 CLOSED:SYN_SENT
[0 + 1] [1863915401 + 2]
age 00:01:36, expires in 00:00:24, 1:0 pkts, 64:0 bytes, rule 4
id: 531153bd00000006 creatorid: a099e0df
------------------------------------------------------------------------
It looks like on vm00, pf has evaluated rule @4 and created a state for
it, but the return packet did not exit via vio2.
Now if I add a default route to 127.0.0.1 on vm00 and repeat the test, I
can see the response packet from 10.0.0.1 being sent out on lo0 on vm00
(as if pf is ignoring the reply-to filter option).
------------------------------------------------------------------------
# route add default 127.0.0.1
add net default: gateway 127.0.0.1
# route -n show -inet
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 127.0.0.1 UGS 0 0 33144 8 lo0
10.0.0/24 link#3 UC 1 0 - 4 vio2
10.0.0.2 08:00:27:46:57:e4 UHLc 0 4 - 4 vio2
127/8 127.0.0.1 UGRS 0 0 33144 8 lo0
127.0.0.1 127.0.0.1 UH 2 0 33144 4 lo0
192.168.56/24 link#2 UC 1 0 - 4 vio1
192.168.56.1 08:00:27:00:1c:ce UHLc 3 1146 - 4 vio1
224/4 127.0.0.1 URS 0 0 33144 8 lo0
------------------------------------------------------------------------
Reset pf state again by re-enabling it and repeat the test.
------------------------------------------------------------------------
# tcpdump -i vio2 -nlt
tcpdump: listening on vio2, link-type EN10MB
tcpdump: WARNING: compensating for unaligned libpcap packets
172.16.0.1.10957 > 10.0.0.1.22: S 505808880:505808880(0) win 16384
<mss 1460,nop,nop,sackOK,nop,wscale 3,nop,nop,timestamp 921742020 0>
(DF) [tos 0x10]
------------------------------------------------------------------------
# tcpdump -i lo0 -nlt
tcpdump: listening on lo0, link-type LOOP
10.0.0.1.22 > 172.16.0.1.10957: S 1992123758:1992123758(0) ack
505808881 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
3,nop,nop,timestamp 2345352085 921742020> (DF)
10.0.0.1.22 > 172.16.0.1.10957: S 1992123758:1992123758(0) ack
505808881 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
3,nop,nop,timestamp 2345352091 921742020> (DF)
10.0.0.1.22 > 172.16.0.1.10957: S 1992123758:1992123758(0) ack
505808881 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
3,nop,nop,timestamp 2345352103 921742020> (DF)
10.0.0.1.22 > 172.16.0.1.10957: S 1992123758:1992123758(0) ack
505808881 win 16384 <mss 1460,nop,nop,sackOK,nop,wscale
3,nop,nop,timestamp 2345352127 921742020> (DF)
------------------------------------------------------------------------
# pfctl -vvsr -R 4
@4 pass in on vio2 inet proto tcp from any to 10.0.0.1 port = 22 flags
S/SA reply-to 10.0.0.2@vio2
[ Evaluations: 1 Packets: 1 Bytes: 64 States: 1 ]
[ Inserted: uid 0 pid 7419 State Creations: 1 ]
------------------------------------------------------------------------
# pfctl -vvss
all tcp 10.0.0.1:22 <- 172.16.0.1:10957 CLOSED:SYN_SENT
[0 + 1] [505808880 + 2]
age 00:00:59, expires in 00:01:01, 1:0 pkts, 64:0 bytes, rule 4
id: 531153bd00000007 creatorid: a099e0df
------------------------------------------------------------------------
And again, we can see that rule @4 was evaluated, the state created, but
the response packet is exiting on lo0 instead of vio2. If I add a
default route to a physical interface, say vio1 (192.168.56.1), the
response packet will exit on vio1 interface instead of vio2 as expected.
Am I doing something completely wrong (hooray for sleep depravation >:))
or is this really is an unexpected behavior?
Thank you!
--peter
