Hi, > I have set charon.install_routes = 0 to avoid installing default route to > route table 220, so I can actually setup the second tunnel.
Set it to "no". > So it seems that the kernel's source address selection does not work > correctly in my case. I am able to workaround my troubles by specifying SNAT > rules like: > [...] > This got me thinking that the source ip selection probably happens sooner > than marking the packets. It is exactly like that. Take a look at the flow graph for netfilter[1]. > So I tried to insert my marking rules based on uid to OUTPUT chain of raw > table. When I did this, it did not help and moreover, even the case with > manual selection of source ip in curl stopped working. I never used the uid PBR selector, so I can't tell you if some parts of it are broken - or not. Keep it simple and/or test other kernels. Maybe you need to disable the return path filter (rp_filter setting for the interface, all or default). You could work around the problem by putting the processes in cgroups, setting the mark in the application's socket (maybe using some LD_PRELOAD hack) or something else. If you use cgroups, you again need marking rules. Kind regards Noel [1] inai.de/images/nf-packet-flow.png On 28.11.2017 07:55, Jiri Horky wrote: > Hi list, > > I am wondering if it is possible to create ~1000 of tunnels from a single > linux machine for testing purposes and route the traffic based on UID of the > processes. I was trying to do PoC with just two tunnels using the following > setup: > > strongswan-5.5.3, kernel 4.12.5-gentoo > > hotgeorge horky # cat /etc/ipsec.conf > conn node13 > auto=add > type=tunnel > keyexchange=ikev2 > ike=aes256-sha1-modp1024! > esp=aes256-sha1-noesn! > left=%defaultroute > leftid="SomeID" > leftsourceip=%config4 > right=node13 > rightid=myrightid > rightsubnet=0.0.0.0/0 <http://0.0.0.0/0> > authby=psk > mark_out=13 > leftupdown=/usr/bin/sudo -E /etc/ipsec_mark_updown > > conn node14 > auto=add > type=tunnel > keyexchange=ikev2 > ike=aes256-sha1-modp1024! > esp=aes256-sha1-noesn! > left=%defaultroute > leftid="SomeID" > leftsourceip=%config4 > right=node14 > rightid=myrightid > rightsubnet=0.0.0.0/0 <http://0.0.0.0/0> > authby=psk > mark_out=14 > leftupdown=/usr/bin/sudo -E /etc/ipsec_mark_updown > > I have set charon.install_routes = 0 to avoid installing default route to > route table 220, so I can actually setup the second tunnel. > > I use following static iptables rules to mark the traffic based on UID: > iptables -t mangle -A OUTPUT -m owner --uid 1013 -j MARK --set-xmark > 0xd/0xffffffff > iptables -t mangle -A OUTPUT -m owner --uid 1014 -j MARK --set-xmark > 0xe/0xffffffff > > The interesting content of /etc/ipsec_mark_updown script is the following: > MY_DEFAULT_GW=10.7.65.1 > case $PLUTO_CONNECTION in > *node13*) > MARK=13 > ;; > *node14*) > MARK=14 > ;; > esac > > ROUTE_TABLE=$((1000+MARK)) > > case $PLUTO_VERB in > up-client) > ip route flush table $ROUTE_TABLE > ip rule del fwmark $MARK table $ROUTE_TABLE 2>/dev/null > ip rule add priority 10 fwmark $MARK table $ROUTE_TABLE > ip route add default via $MY_DEFAULT_GW proto static src $PLUTO_MY_SOURCEIP > table $ROUTE_TABLE > ;; > > Now, if I fire up the two tunnels: > ipsec up node13 > ipsec up node14 > > I have the following routing rules: > hotgeorge horky # ip rule list > 0:from all lookup local > 10:from all fwmark 0xd lookup 1013 > 10:from all fwmark 0xe lookup 1014 > 220:from all lookup 220 > 32766:from all lookup main > 32767:from all lookup default > > And following routing table: > hotgeorge horky # ip route list table 1013 > default via 10.7.65.1 dev wlo1 proto static src 100.111.0.91 > hotgeorge horky # ip route list table 1014 > default via 10.7.65.1 dev wlo1 proto static src 100.111.0.167 > hotgeorge horky # ip route list table 220 > > Where table 220 is empty. > > The trouble is the if I execute curl under user test_1013 (with uid 1013), it > times out on sending a DNS query: > su test_1013 -c "curl http://ip-info.ff.avast.com/v1/info"; > > When I manually specify the source address, it works: > > su test_1013 -c "curl http://ipv4bot.whatismyipaddress.com --interface > 100.111.0.91 --dns-ipv4-addr 100.111.0.91"; echo > 77.234.40.153 > > su test_1014 -c "curl http://ipv4bot.whatismyipaddress.com --interface > 100.111.0.167 --dns-ipv4-addr 100.111.0.167"; echo > 77.234.40.182 > > So it seems that the kernel's source address selection does not work > correctly in my case. I am able to workaround my troubles by specifying SNAT > rules like: > > iptables -t nat -A POSTROUTING -m mark --mark $MARK ! -s $PLUTO_MY_SOURCEIP > -j SNAT --to-source $PLUTO_MY_SOURCEIP > > But I would like to avoid doing that. > > Could you please enlighten me what I am doing wrong? It seems that if I let > strongwan install the routes to the table 220 (without any restrictions to > marks), the source address is selected correctly. This got me thinking that > the source ip selection probably happens sooner than marking the packets. So > I tried to insert my marking rules based on uid to OUTPUT chain of raw > table. When I did this, it did not help and moreover, even the case with > manual selection of source ip in curl stopped working. > > I would be grateful for any tips. > > Note: I know that using network namespaces could sound like a better idea > than this. Unfortunatelly, even the newest kernel contain some race > conditions when destroying namespaces rapidly preventing us to use this way. > > Thanks > Jiri Horky
signature.asc
Description: OpenPGP digital signature
