Hi Noel,
Thanks for the help.
I have moved the route for IPSec back into the main routing table.
[root@arch-linux ~]# ip route
default via 192.168.45.1 dev ens18
10.10.10.0/30 dev ip_vti1 scope link src 10.10.10.2
192.168.45.0/24 dev ens18 proto kernel scope link src 192.168.45.30
[root@arch-linux ~]# ip route show table 220
[root@arch-linux ~]#
And the connmark plugin has been disabled.
[root@arch-linux ~]# ipsec statusall | grep -i connmark
[root@arch-linux ~]#
Also, I have noticed that even after the connmark plugin is disabled some mark
rules are added to iptables.
[root@arch-linux ~]# ipsec stop
Stopping strongSwan IPsec...
[root@arch-linux ~]# iptables-save | grep -i mark
[root@arch-linux ~]#
[root@arch-linux ~]# ipsec start
Starting strongSwan 5.9.3 IPsec [starter]...
[root@arch-linux ~]# swanctl --load-all
plugin 'mysql' failed to load: libmariadb.so.3: cannot open shared object file:
No such file or directory
loaded ike secret 'ike-0'
no authorities found, 0 unloaded
no pools found, 0 unloaded
loaded connection 'ipseclab'
successfully loaded 1 connections, 0 unloaded
[root@arch-linux ~]# iptables-save | grep -i mark
-A PREROUTING -d 10.10.10.0/30 -j MARK --set-xmark 0x2a/0xffffffff
-A PREROUTING -s 192.168.45.10/32 -d 192.168.45.30/32 -p esp -m esp --espspi
3419086685 -j MARK --set-xmark 0x2a/0xfffffff
f
-A OUTPUT -d 10.10.10.0/30 -j MARK --set-xmark 0x2a/0xffffffff
The scenario is the same at the moment :
pings from 10.10.10.1 ( pfSense ) to 10.10.10.2 ( Linux ) ✅ seem as received by
Linux
pings from 10.10.10.2 ( Linux ) to 10.10.10.1 ( pfSense ) ❌ seem as Errors
NoRoute
ip_vti1: ip/ip remote 192.168.45.10 local 192.168.45.30 ttl inherit nopmtudisc
key 42
RX: Packets Bytes Errors CsumErrs OutOfSeq Mcasts
66095 5551980 0 0 0 0
TX: Packets Bytes Errors DeadLoop NoRoute NoBufs
0 0 133788 0 133788 0
I am at work and not able to capture packets to further investigate but will do
as soon possible.
[root@arch-linux ~]# ipsec statusall
Status of IKE charon daemon (strongSwan 5.9.3, Linux 5.13.12-arch1-1, x86_64):
uptime: 9 minutes, since Sep 01 00:24:27 2021
malloc: sbrk 2936832, mmap 0, used 1328288, free 1608544
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled:
115
loaded plugins: charon ldap pkcs11 aes des rc2 sha2 sha3 sha1 md5 mgf1 random
nonce x509 revocation constraints pubkey p
kcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp curve25519
agent chapoly xcbc cmac hmac ntru drbg newho
pe bliss curl sqlite attr kernel-netlink resolve socket-default forecast farp
stroke vici updown eap-identity eap-sim eap-
aka eap-aka-3gpp2 eap-simaka-pseudonym eap-simaka-reauth eap-md5 eap-gtc
eap-mschapv2 eap-dynamic eap-radius eap-tls eap-t
tls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp radattr unity
counters
Listening IP addresses:
192.168.45.30
10.10.10.2
Connections:
ipseclab: 192.168.45.30...192.168.45.10 IKEv2, dpddelay=10s
ipseclab: local: [ipsec-lab-openwrt] uses pre-shared key authentication
ipseclab: remote: [ipsec-lab-pfsense] uses pre-shared key authentication
con1: child: 10.10.10.0/30 === 10.10.10.0/30 TUNNEL, dpdaction=restart
Security Associations (2 up, 0 connecting):
ipseclab[54]: ESTABLISHED 9 seconds ago,
192.168.45.30[ipsec-lab-openwrt]...192.168.45.10[ipsec-lab-pfsense]
ipseclab[54]: IKEv2 SPIs: 840c3fd77eb0efee_i b3f6cec233130e6a_r*, rekeying
in 6 hours
ipseclab[54]: IKE proposal:
AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
con1{54}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: c206ad8c_i c631ebba_o
con1{54}: AES_GCM_16_256, 0 bytes_i, 0 bytes_o, rekeying in 48 minutes
con1{54}: 10.10.10.0/30 === 10.10.10.0/30
ipseclab[53]: ESTABLISHED 19 seconds ago,
192.168.45.30[ipsec-lab-openwrt]...192.168.45.10[ipsec-lab-pfsense]
ipseclab[53]: IKEv2 SPIs: 133d2066ebf6a358_i d8da41bca97601ca_r*, rekeying
in 6 hours
ipseclab[53]: IKE proposal:
AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
con1{53}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: cb169570_i c35823e7_o
con1{53}: AES_GCM_16_256, 0 bytes_i, 0 bytes_o, rekeying in 51 minutes
con1{53}: 10.10.10.0/30 === 10.10.10.0/30
I will add to the thread later but NFLOG is able to capture MARKS in the
packets. The packets captured by NFLOG has a special section with quite useful
information.
...
Linux Netfilter NFLOG
Family: IPv4 (2)
Version: 0
Resource id: 7
TLV Type: NFULA_PACKET_HDR (1), Length: 8
...
Once again, thanks for the help.
Tiago Stoco.
________________________________
From: Noel Kuntze
Sent: Tuesday, August 31, 2021 2:20 PM
To: Tiago Stoco; Tobias Brunner; users@lists.strongswan.org
Subject: Re: [strongSwan] IPSec route based VPN - VTI interface TX Errors
NoRoute
Hello Tiago,
> And, I have moved the route for the VTI to table 220 because it seems to be
> the right way to config routed based IPSec VPN.
>
> [root@arch-linux ~]# ip rule
> 0: from all lookup local
> 220: from all lookup 220
> 32766: from all lookup main
> 32767: from all lookup default
Don't do that.
220 is managed by strongSwan. Keep them in the main table.
> -A PREROUTING -d 10.10.10.0/30 -c 2352 230776 -j MARK --set-xmark
> 0x2a/0xffffffff
> -A OUTPUT -d 10.10.10.0/30 -c 3605 336028 -j MARK --set-xmark 0x2a/0xffffffff
Disable the connmark plugin.
> I have added a few more NFLOG captures into my iptables and I am a bit
> confused with the results.
>
> A tcpdump capture in the VTI interface with a ping from the remote ( pfSense
> - 10.10.10.1 ) shows :
>
> No Time Source Destination
>
> 1 0.000000 10.10.10.1 10.10.10.2 ICMP 84 Echo (ping) request id=0x9877,
> seq=471/55041, ttl=64 (reply in 2)
> 2 0.000038 10.10.10.2 > 10.10.10.1 ICMP 84 Echo (ping) reply id=0x9877,
> seq=471/55041, ttl=64 (request in 1)
>
> I do not see the IPSec MARK in these packets.
They are only visible in the output of the TRACE target. I suspect they are
note copied into the buffer passed to the applications.
>
> Also, the NAT chain is not having packets passing through it.
>
> [root@arch-linux ~]# snat
> -P PREROUTING ACCEPT -c 0 0
> -P INPUT ACCEPT -c 0 0
> -P OUTPUT ACCEPT -c 0 0
> -P POSTROUTING ACCEPT -c 0 0
> -A PREROUTING -c 0 0 -j NFLOG --nflog-group 9
>
> That is odd cause I am not able to manipulate the packets.
>
> I will run a ping from the local Linux (10.10.10.2) and see how the packets
> are flowing through the iptables chains and will update in another email.
>
The *nat table is only consulted for the first packet of a connection.
Kind regards
Noel
Am 31.08.21 um 17:22 schrieb Tiago Stoco:
> Hi Tobias,
>
> First of all, THANKS for replying and clarifying some settings.
>
> I have completely disabled the bypass-lan plugin since I do not have a use
> for it right now.
>
> [root@arch-linux ~]# cat /etc/strongswan.conf
> ...
> plugins {
> include strongswan.d/charon/*.conf
> bypass-lan {
> load = no
> }
> ...
>
> And, I have moved the route for the VTI to table 220 because it seems to be
> the right way to config routed based IPSec VPN.
>
> [root@arch-linux ~]# ip rule
> 0: from all lookup local
> 220: from all lookup 220
> 32766: from all lookup main
> 32767: from all lookup default
>
> [root@arch-linux ~]# ip r s t 220
> 10.10.10.0/30 via 10.10.10.2 dev ip_vti1 src 10.10.10.2
>
> [root@arch-linux ~]# ip route
> default via 192.168.45.1 dev ens18
> 192.168.45.0/24 dev ens18 proto kernel scope link src 192.168.45.30
>
> I am going to add some more details of my configs because the TX Errors
> NoRoute are still present.
>
> 7: ip_vti1@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1436 qdisc noqueue state
> UNKNOWN group default qlen 1000
> link/ipip 192.168.45.30peer 192.168.45.10promiscuity 0 minmtu 0 maxmtu 0
> vti remote 192.168.45.10 local 192.168.45.30 ikey 0.0.0.42 okey 0.0.0.42
> numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
> inet 10.10.10.2peer 10.10.10.1/32 scope global ip_vti1
> valid_lft forever preferred_lft forever
> inet6 fe80::5efe:c0a8:2d1e/64 scope link
> valid_lft forever preferred_lft forever
>
> I can also see that the IPSec added some rules to MARK packets in my iptables.
>
> -A PREROUTING -d 10.10.10.0/30 -c 2352 230776 -j MARK --set-xmark
> 0x2a/0xffffffff
> -A OUTPUT -d 10.10.10.0/30 -c 3605 336028 -j MARK --set-xmark 0x2a/0xffffffff
>
> The counters confirms that the packets are being marked. I am not sure if I
> should keep the MARK in iptables or remove it allowing routing decisions to
> send the packets to the VTI device that will MARK the packets but according
> to my understanding it should not matter.
>
> [root@arch-linux ~]# ip xfrm policy
> src 0.0.0.0/0 dst 0.0.0.0/0
> socket in priority 0 ptype main
> src 0.0.0.0/0 dst 0.0.0.0/0
> socket out priority 0 ptype main
> src 0.0.0.0/0 dst 0.0.0.0/0
> socket in priority 0 ptype main
> src 0.0.0.0/0 dst 0.0.0.0/0
> socket out priority 0 ptype main
> src ::/0 dst ::/0
> socket in priority 0 ptype main
> src ::/0 dst ::/0
> socket out priority 0 ptype main
> src ::/0 dst ::/0
> socket in priority 0 ptype main
> src ::/0 dst ::/0
> socket out priority 0 ptype main
>
> Above are the policies installed. Again, because it is a routed base VPN
> seems correct.
>
> [root@arch-linux ~]# ip xfrm state
> src 192.168.45.30 dst 192.168.45.10
> proto esp spi 0xc2239b57 reqid 1 mode tunnel
> replay-window 0 flag af-unspec
> mark 0x2a/0xffffffff
> aead rfc4106(gcm(aes))
> 0x264acee3119a4e523af2fbf5905b50c5acc1f7be9079ff23ffa2c6473a9c507fe1ae936b 128
> anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
> src 192.168.45.10 dst 192.168.45.30
> proto esp spi 0xc661b9e5 reqid 1 mode tunnel
> replay-window 32 flag af-unspec
> aead rfc4106(gcm(aes))
> 0x69a86fa6ca9448bece6ffdff77893f0e9ce5ebef604040f681b5cdd2d5976438ed005df1 128
> anti-replay context: seq 0x656, oseq 0x0, bitmap 0xffffffff
>
> I have added a few more NFLOG captures into my iptables and I am a bit
> confused with the results.
>
> A tcpdump capture in the VTI interface with a ping from the remote ( pfSense
> - 10.10.10.1 ) shows :
>
> No Time Source Destination
>
> 1 0.000000 10.10.10.1 10.10.10.2 ICMP 84 Echo (ping) request id=0x9877,
> seq=471/55041, ttl=64 (reply in 2)
> 2 0.000038 10.10.10.2 > 10.10.10.1 ICMP 84 Echo (ping) reply id=0x9877,
> seq=471/55041, ttl=64 (request in 1)
>
> I do not see the IPSec MARK in these packets.
> The reply packets end up in the OUTPUT chain marked but not encrypted as an
> ESP packet. By the way I do not see the replies even being encapsulated at
> all by IPSec.
>
> Also, the NAT chain is not having packets passing through it.
>
> [root@arch-linux ~]# snat
> -P PREROUTING ACCEPT -c 0 0
> -P INPUT ACCEPT -c 0 0
> -P OUTPUT ACCEPT -c 0 0
> -P POSTROUTING ACCEPT -c 0 0
> -A PREROUTING -c 0 0 -j NFLOG --nflog-group 9
>
> That is odd cause I am not able to manipulate the packets.
>
> I will run a ping from the local Linux (10.10.10.2) and see how the packets
> are flowing through the iptables chains and will update in another email.
>
> In the meantime, if someone sees something that I am missing. Please let me
> know.
>
> Many Thanks.
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> *From:* Tobias Brunner <tob...@strongswan.org>
> *Sent:* Tuesday, August 31, 2021 5:51 AM
> *To:* Tiago Stoco <tmsbl...@msn.com>; users@lists.strongswan.org
> <users@lists.strongswan.org>
> *Subject:* Re: [strongSwan] IPSec route based VPN - VTI interface TX Errors
> NoRoute
> Hi Tiago,
>
>> Pings from the Linux system are being seem as errors NoRoute by the tunnel.
>> > ...
>> Shunted Connections:
>> Bypass LAN 10.10.10.0/30: 10.10.10.0/30 === 10.10.10.0/30 PASS
>
> The reason is most likely this passthrough IPsec policy installed by the
> bypass-lan plugin for the subnet that is reachable (according to the
> main routing table) via ip_vti1. For a ping from 10.10.10.2 to
> 10.10.10.1, the VTI interface won't find an IPsec policy to protect the
> packet (the passthrough policy has a higher priority), so it gets dropped.
>
> To avoid that, either install the routes via VTI in table 220 (which is
> ignored by the bypass-lan plugin automatically), exclude the VTI
> interface explicitly via charon.plugins.bypass-lan.interfaces_ignore, or
> just disable the bypass-lan plugin completely if you don't need it.
>
> Regards,
> Tobias
