Greetings all,
Last week I described briefly a problem with *return* TCP/IP traffic
only, across a LAN-to-LAN VPN network bridge, only on the first
connection. I appreciate your responses and so now as you've requested
I have composed a detailed network topology and configuration document
in order to properly describe the problem in detail.
This email best viewed with a fixed-width font. (Pardon the ASCII art.)
#------------
#THE NETWORK:
#------------
PUBLIC INTERNET
| | |
| | |
------------- ------------- -------------
|firewall 3 |__link_A__|firewall 1 |__link_B__|firewall 2 |
|OpenBSD 4.0| |OpenBSD 3.9| |OpenBSD 4.0|
------------- ------------- -------------
| | | | |
172.18.1.0/24 | 192.168.248.0/21 192.168.254.0/24
| | | | -------------
172.18.2.0/24 | |_| router 1 |
| ------------- |OS X 10.4.8|
172.18.3.0/24 | router 2 | -------------
|OpenBSD 4.0| | | |
------------- 172.17.1.0/24 |
| | |
| 172.17.2.0/24
| |
X.X.X.X/X 172.17.3.0/24
#----------------------
#Internal IP addresses:
#----------------------
firewall 1: 192.168.250.1
firewall 2: 192.168.254.1
firewall 3: 172.18.1.1, 172.18.2.1, 172.18.3.1
router 1: 192.168.254.2, 172.17.1.1, 172.17.2.1, 172.17.3.1
router 2: 192.168.250.3, X.X.X.X
The network behind firewall 1 is the primary network. Software
developers sit within its private address space. Firewall 1 is also
running two OpenVPN server instances; one instance is configured as a
routed tunnel, one instance is configured as a bridged tunnel.
Firewalls 2 and 3 are both at remote locations. They each run an
OpenVPN client instance which connects back to firewall 1. In the
diagram above link A represents the routed OpenVPN tunnel and link B
represents the bridged tunnel. You'll notice that firewall 2, which
connects to the bridged tunnel, handles a private IP range that is in
fact a subset of firewall 1's. The art department's desktops lie within
that 192.168.254.0/24 range at firewall 2's remote site. It should be
noted here that developers also regularly connect to either tunnel when
working from home or the road. The bridged tunnel is configured to
forward all traffic while the routed tunnel is configured to only
forward appropriate traffic.
Router 1 sits within the range of addresses firewall 2's bridge to
firewall 1's network and routes traffic for 3 seperate subnets
(sub-subnets?) of intel mac mini development server clusters (God,
please don't ask why). Router 2 sits within the main part of firewall
1's local network and acts as a OpenVPN client just for routing traffic
to/from a network at another remote site whose administration is not
within my jurisdiction. Theoretically TCP/IP traffic should be able to
pass from any part of the network to any other part of the network and
back. For that matter, this all in fact seems to work correctly despite
the fact that I'm a total rookie network admin... with one notable
exception, which brings us to the problem.
#------------
#THE PROBLEM:
#------------
Now for the fun part. I've been adding routes and configurations and pf
rules and such as we build this network out. Some branches (notably the
172.17 and 172.18 subnets) are very new additions. The only part where
known problems persist (and here lies the point of this whole email) is
with access from developer desktops behind firewall 1 (also all OS X
10.4.8 machines, fwiw) to the mac mini dev cluster behind router 1. The
real odd part of this problem is that its only a problem for any given
client the first time it connects to a mac dev server in any given day.
Whats even weirder is that it doesn't appear to be a problem with the
macs themselves or with router 1, which is also a mac. Traffic reaches
the mac mini server and tcpdump verifies that it comes all the way back
through the tunnel and appears on tun1 of firewall 1 but fails to get
passed over the bridge (bridge0) with firewall 1's tunnel endpoint and
its local ethernet device (sis0). This first connection always times
out, however immediately retrying always works. Pinging always works
though and pinging first always eliminates the first-connection failure
of the following TCP/IP connections - but for only that client and the
strange little cycle seems to reset itself sometime between the end of
the business day and the start of the next. Weird huh?
The bridging code's interaction with OpenVPN and the sis ethernet driver
is suspect as are my pf configurations but the intermittent behavior has
pretty much stumped me at this point. Firewall states would seem to be
an obvious culprit except that its the first try thats failing, not
subsequent tries. Macs going to sleep would also seem a likely culprit
except that they're not and they're returning the traffic. The failure
itself is happening within firewall 1.
#---------------
#CONFIGURATIONS: (\ represents newlines not actually in the file)
#---------------
-----------------------------------------------------------------
firewall 1:
-----------------------------------------------------------------
dc0: external ip
sis0: 192.168.250.1 netmask 255.255.248.0 broadcast 192.168.255.255
tun1: no ip addresss. link B (bridged) vpn tunnel endpoint.
tun0: 172.16.0.1 -> 172.16.0.2 link A (routed) vpn tunnel endpoint.
bridge0: tun1, sis0
pf.conf:
--------
ext_if="dc0"
int_if="sis0"
table <local-servers> const { 192.168.250.0/24 }
table <local-desktops> const { 192.168.252.0/24, \
192.168.253.0/24 } # 192.168.255.0/24 unused but available
table <dhcp-clients> const { 192.168.251.0/24 }
table <development-clusters> { 172.17.1.0/24, 172.17.2.0/24, \
172.17.3.0/24 }
table <bridged-vpn-clients> const { 192.168.254.0/24 }
table <routed-vpn-clients> const { 172.16.0.0/24 } \
# we may expand this to the full 172.16.0.0/12 but only /24 is \
in use so far
table <downtown-office> const { 172.18.1.0/24, \
172.18.2.0/24, 172.18.3.0/24 }
set skip on { lo0, bridge0, tun1 }
scrub in all
nat on $ext_if inet from { <local-servers>, <local-desktops>, \
<dhcp-clients>, <bridged-vpn-clients>, <development-clusters> } \
to any -> $ext_if
block in all
block out all
pass in on $int_if from { <local-servers>, <local-desktops>, \
<dhcp-clients>, <bridged-vpn-clients>, <development-clusters> } \
to any modulate state
pass in on $ext_if inet proto udp from any to $ext_if port 1194:1195 \
modulate state
pass in on tun0 from { <routed-vpn-clients>, <downtown-office> } \
to any modulate state
pass out on { $int_if, $ext_if, tun0 } from any to any modulate state
hostname.tun0:
--------------
up
hostname.tun1:
--------------
link0 up
hostname.sis0:
--------------
inet 192.168.250.1 255.255.248.0 NONE
manual route additions to rc.local:
-----------------------------------
/sbin/route add X.X.X.X/X 192.168.250.3 255.255.255.0
/sbin/route add 172.17.2.0/24 192.168.254.2 255.255.255.0
/sbin/route add 172.17.1.0/24 192.168.254.2 255.255.255.0
/sbin/route add 172.17.2.0/24 192.168.254.2 255.255.255.0
openvpn routed server config:
-----------------------------
local X.X.X.X
port 1194
proto udp
dev tun0
ca /etc/openvpn/2.0/keys/ca.crt
cert /etc/openvpn/2.0/keys/server.crt
key /etc/openvpn/2.0/keys/server.key
dh /etc/openvpn/2.0/keys/dh2048.pem
server 172.16.0.0 255.255.255.0
ifconfig-pool-persist /var/openvpn/ipp.txt
push "route 192.168.248.0 255.255.248.0"
push "route 172.17.1.0 255.255.255.0"
push "route 172.17.2.0 255.255.255.0"
push "route 172.17.3.0 255.255.255.0"
push "route 172.18.1.0 255.255.255.0"
push "route 172.18.2.0 255.255.255.0"
push "route 172.18.3.0 255.255.255.0"
client-config-dir ccd
route 172.18.1.0 255.255.255.0
route 172.18.2.0 255.255.255.0
route 172.18.3.0 255.255.255.0
client-to-client
keepalive 10 120
tls-auth /etc/openvpn/2.0/keys/ta.key 0
tls-server
comp-lzo
max-clients 100
user nobody
group nobody
persist-key
persist-tun
status /var/openvpn/openvpn-status.log
verb 3
chroot /etc/openvpn/2.0
daemon
writepid /var/run/openvpn.pid
crl-verify /crl.pem
/etc/openvpn/2.0/ccd/firewall3:
-------------------------------
iroute 172.18.1.0 255.255.255.0
iroute 172.18.2.0 255.255.255.0
iroute 172.18.3.0 255.255.255.0
max-routes-per-client 768
openvpn bridged server config:
------------------------------
local X.X.X.X
mode server
port 1195
proto udp
dev-type tap
dev tun1
ca /etc/openvpn/2.0/keys/ca.crt
cert /etc/openvpn/2.0/keys/server.crt
key /etc/openvpn/2.0/keys/server.key
dh /etc/openvpn/2.0/keys/dh2048.pem
ifconfig-pool-persist /var/openvpn/ipp_bridge.txt
server-bridge 192.168.250.1 255.255.248.0 192.168.254.64 192.168.254.127
client-config-dir ccd_bridge
push "redirect-gateway def1"
push "dhcp-option DNS 192.168.250.1"
push "dhcp-option DOMAIN teamcanoga.com"
client-to-client
keepalive 10 120
tls-auth /etc/openvpn/2.0/keys/ta.key 0
tls-server
comp-lzo
max-clients 100
user nobody
group nobody
persist-key
persist-tun
status /var/openvpn/openvpn-status-bridge.log
verb 3
chroot /etc/openvpn/2.0
daemon
writepid /var/run/openvpn_bridge.pid
crl-verify /crl.pem
/etc/openvpn/2.0/ccd_bridge/firewall2:
--------------------------------------
ifconfig-push 192.168.254.1 255.255.248.0
max-routes-per-client 1024
-----------------------------------------------------------------
firewall 2:
-----------------------------------------------------------------
em0: external ip
em1: 192.168.254.1 netmask 255.255.248.0 broadcast 192.168.255.255
tun0: 192.168.254.1 netmask 255.255.248.0 broadcast 192.168.255.255 (<-
link B tunnel endpoint)
bridge0: tun0, em1
pf.conf:
--------
ext_if="em0"
int_if="em1"
table <local-network> const { 192.168.248.0/21 }
table <dev-network> const { 172.17.1.0/24, 172.17.2.0/24, \
172.17.3.0/24 }
set skip on { lo0, bridge0 }
scrub in all
block all
pass in on $int_if from { <local-network>, <dev-network> } to any \
modulate state
pass out on { $int_if, $ext_if } from any to any modulate state
pass on tun0 all modulate state
pass in on tun0 from any to <dev-network> modulate state
pass out on tun0 from <dev-network> to any modulate state
pass on tun0 proto { tcp, udp } from any port ssh modulate state
hostname.tun0:
--------------
link0 up
hostname.em1:
-------------
inet 192.168.254.1 255.255.248.0 NONE
openvpn client config:
----------------------
client
dev-type tap
dev tun0
proto udp
remote X.X.X.X 1195
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/firewall2.crt
key /etc/openvpn/firewall2.key
ns-cert-type server
tls-auth /etc/openvpn/ta.key 1
comp-lzo
verb 5
writepid /var/run/openvpn.pid
chroot /etc/openvpn
daemon
up /etc/openvpn/up.sh
/etc/openvpn/up.sh:
-------------------
#!/bin/sh
#
/sbin/brconfig bridge0 add tun0 # not sure if still needed.
/sbin/pfctl -f /etc/pf.conf # not even sure why it was needed
# when i discovered that it fixed
# an earlier problem with bridge0
# dismantling itself after boot.
-----------------------------------------------------------------
firewall 3:
-----------------------------------------------------------------
em0: external ip
em1: 172.18.3.1
vlan0: 172.18.1.1
vlan1: 172.18.2.1
tun0: link A (routed) vpn tunnel endpoint. ip address assigned on connect.
openvpn client config:
----------------------
client
dev tun0
proto udp
remote X.X.X.X 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/firewall3.crt
key /etc/openvpn/firewall3.key
ns-cert-type server
tls-auth /etc/openvpn/ta.key 1
comp-lzo
verb 3
writepid /var/run/openvpn.pid
daemon
hostname.em1:
-------------
inet 172.18.3.1 255.255.255.0 NONE
hostname.vlan0:
---------------
inet 172.18.1.1 255.255.255.0 NONE vlan 1000 vlandev em1
hostname.vlan1:
---------------
inet 172.18.2.1 255.255.255.0 NONE vlan 2000 vlandev em1
