Hi,
So using the information Stuart and Andreas provided, I have been testing
this (load balancing across multiple VPN servers to improve bandwidth).
And I have multiple VPNs working properly within there own rdomains.
* However 'route-to' is not load balancing with rdomains :(
I have not been able to use the more simple solution you highlighted Stuart
(using basic multipath routing), as the tunnel subnets overlap.
So I think this is a potential bug, but I need your wisdom to verify my
working first :)
Re; Load Balancing SSL VPNs using OpenBSD 6.4, with VPN TunX interfaces in
unique rdomains (overlapping tunnel subnets)
Configure sysctl's
# Ensure '/etc/sysctl.conf' contains;
net.inet.ip.forwarding=1 # Permit forwarding (routing) of packets
net.inet.ip.multipath=1 # 1=Enable IP multipath routing
# Active sysctl's now without reboot
sysctl net.inet.ip.forwarding=1
sysctl net.inet.ip.multipath=1
Pre-create tunX interfaces (in their respective rdomains)
# Ensure '/etc/hostname.tun1' contains;
up
rdomain 1
# Ensure '/etc/hostname.tun2' contains;
up
rdomain 2
# Bring up the new tunX interfaces
sh /etc/netstart
fw1# ifconfig
tun1
tun1: flags=8011<UP,POINTOPOINT,MULTICAST> rdomain 1 mtu 1500
index 8 priority 0 llprio 3
groups: tun
status: down
fw1# ifconfig tun2
tun2: flags=8011<UP,POINTOPOINT,MULTICAST> rdomain 2 mtu 1500
index 9 priority 0 llprio 3
groups: tun
status: down
# Start all SSL VPN tunnels (in unique VRF/rdomain's)
/usr/local/sbin/openvpn --config ./ch70.nordvpn.com.udp.ovpn --writepid
/var/run/openvpn.tun1.pid --dev tun1 &
/usr/local/sbin/openvpn --config ./ch71.nordvpn.com.udp.ovpn --writepid
/var/run/openvpn.tun2.pid --dev tun2 &
('auth-user-pass' updated in config files)
Each openvpn tunnel should start using 'rtable 0' for the VPN's outer
connection itself, but with each virtual tunnel TunX interface being placed
into a unique routing domain.
This results in the following tunX interface and rtable updates;
fw1# ifconfig
tun1
tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> rdomain 1 mtu 1500
index 6 priority 0 llprio 3
groups: tun
status: active
inet 10.8.8.128 --> 10.8.8.1 netmask 0xffffff00
fw1# ifconfig tun2
tun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> rdomain 2 mtu 1500
index 7 priority 0 llprio 3
groups: tun
status: active
inet 10.8.8.129 --> 10.8.8.1 netmask 0xffffff00
fw1# route -T 1 show
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio
Iface
10.8.8.1 10.8.8.128 UH 0 0 - 8
tun1
10.8.8.128 10.8.8.128 UHl 0 0 - 1
tun1
localhost localhost UHl 0 0 32768 1
lo1
fw1# route -T 2 show
Routing tables
Internet:
Destination Gateway Flags Refs Use Mtu Prio
Iface
10.8.8.1 10.8.8.129 UH 0 0 - 8
tun2
10.8.8.129 10.8.8.129 UHl 0 0 - 1
tun2
localhost localhost UHl 0 0 32768 1
lo2
# Test each tunnel - Ping the remote connected vpn peer within each rdomain
ping -V 1 10.8.8.1
ping -V 2 10.8.8.1
Shows both VPN tunnels are working independently with the overlapping
addressing :)
# To be able to test each tunnel beyond the peer IP, add some default
routes to the rdomains;
route -T 1 -n add default 10.8.8.1
route -T 2 -n add default 10.8.8.1
# Test each tunnel - Ping beyond the connected peer
ping -V 1 8.8.8.8
ping -V 2 8.8.8.8
Shows both VPN tunnels are definitely working independently with the
overlapping addressing :)
# Reverse routing - I have read in various places that PF's 'route-to' can
be used for jumping rdomains's in the forward path of the session, but the
reply packets need any matching route in the remote rdomain for the reply
destination (the matching route is to ensure in the reply packet is passed
through the routing table and gets into the PF processing, where PF can
manage the return back to the default rdomain etc.
But as I am using outbound NATing on the tunX interfaces, there is always a
matching route for the reply traffic. And so a route for the internal
subnet is not needed within rdomain 1 and 2.
# Finally ensure '/etc/pf.conf' contains something like;
if_ext = "em0"
if_int = "em1"
#CDR = 80 Down/20 Up
queue out_ext on $if_ext flows 1024 bandwidth 18M max 19M qlimit 1024
default
queue out_tun1 on tun1 flows 1024 bandwidth 17M max 18M qlimit 1024 default
queue out_tun2 on tun2 flows 1024 bandwidth 17M max 18M qlimit 1024 default
queue out_int on $if_srx flows 1024 bandwidth 74M max 78M qlimit 1024
default
#MTU = 1500
match proto tcp all scrub (no-df max-mss 1460) set prio (2,5)
match proto udp all scrub (no-df max-mss 1472) set prio (2,5)
match proto icmp all scrub (no-df max-mss 1472) set prio 7
#NAT all outbound traffic
match out on $if_ext from any to any nat-to ($if_ext)
match out on tun1 from any to any nat-to (tun1) rtable 1
match out on tun2 from any to any nat-to (tun2) rtable 2
#Allow outbound traffic on egress for vpn tunnel setup etc
pass out quick on { $if_ext } from self to any set prio (3,6)
#Load balance outbound traffic from internal network across tun1 and tun2 -
THIS IS NOT WORKING - IT ONLY USES FIRST TUNNEL
pass in quick on { $if_int } to any route-to { (tun1 10.8.8.1), (tun2
10.8.8.1) } round-robin set prio (3,6)
#Allow outbound traffic over vpn tunnels
pass out quick on tun1 to any set prio (3,6)
pass out quick on tun2 to any set prio (3,6)
# Verify which tunnels are being used
systat ifstat
*This command shows that all the traffic is only flowing over the first
tun1 interface, and the second tun2 is never ever used.*
# NB; I have tried with and without 'set state-policy if-bound'.
I have tried all the load balancing policies; round-robin, random,
least-states and source-hash
If I change the 'route-to' pool to "{ (tun2 10.8.8.1), (tun1 10.8.8.1) }",
then only tun2 is used instead.. :(
So 'route-to' seems to only use the first tunnel in the pool.
Any advice on what is going wrong here. I am wondering if I am falling
victim to some processing-order issue with PF, or if this is a real bug?
Thanks, Andy.
On Wed, Sep 12, 2018 at 5:58 PM Stuart Henderson <s...@spacehopper.org>
wrote:
> On 2018-09-11, Andrew Lemin <a_le...@hotmail.com> wrote:
> > Hi list,
> >
> > I use an OpenVPN based internet access service (like NordVPN, AirVPN
> etc).
> >
> > The issue with these public VPN services, is the VPN servers are always
> congested. The most I’ll get is maybe 10Mbits through one server.
> >
> > Local connection is a few hundred mbps..
> >
> > So I had the idea of running multiple openvpn tunnels to different
> servers, and load balancing outbound traffic across the tunnels.
> >
> > Sounds simple enough..
> >
> > However every vpn tunnel uses the same subnet and nexthop gw. This of
> course won’t work with normal routing.
>
> rtable/rdomain with openvpn might be a bit complex, I think it may need
> persist-tun and create the tun device in advance with the wanted rdomain.
> (you need the VPN to be in one, but the UDP/TCP connection in another).
>
> Assuming you are using tun (and so point-to-point connections) rather
> than tap, try one or other of these:
>
> - PF route-to and 'probability', IIRC it works to just use a junk
> address as long as the interface is correct ("route-to 10.10.10.10@tun0",
> "route-to 10.10.10.10@tun1").
>
> - ECMP (net.inet.ip.multipath=1) and multiple route entries with
> the same priority. Use -ifp to set the interface ("route add
> default -priority 8 -ifp $interface $dest").
>
> The "destination address" isn't really very relevant for routing
> on point-to-point interfaces (though current versions of OpenBSD
> do require that it matches the destination address on the interface,
> otherwise they won't allow the route to be added).
>
>
>
