On Tue, Apr 06, 2021 at 12:30:42PM +0100, Mark Gray wrote: > On 05/04/2021 20:56, Flavio Leitner wrote: > > On Thu, Apr 01, 2021 at 07:48:19PM +0100, Mark Gray wrote: > >> On 01/04/2021 19:29, Ilya Maximets wrote: > >>> On 4/1/21 3:58 PM, Mark Gray wrote: > >>>> This series fixes IPv6 support for Libreswan and introduces > >>>> IPsec system tests for Libreswan. > >>>> > >>>> Mark Gray (5): > >>>> ipsec: IPv6 default route support for Libreswan > >>>> system-common-macros: clean up veth device on test failure > >>>> ipsec: Allow custom file locations > >>>> ipsec: Introduce IPsec system tests for Libreswan > >>>> ipsec: Update ordering of imports > >>>> > >>>> ipsec/ovs-monitor-ipsec.in | 144 ++++++++++-- > >>>> tests/automake.mk | 3 +- > >>>> tests/system-common-macros.at | 2 +- > >>>> tests/system-ipsec.at | 406 +++++++++++++++++++++++++++++++++ > >>>> tests/system-kmod-testsuite.at | 1 + > >>>> 5 files changed, 529 insertions(+), 27 deletions(-) > >>>> create mode 100644 tests/system-ipsec.at > >>>> > >>> > >>> Thanks! Applied to master. > >>> First patch with the fix also backported down to 2.13. > >>> > >>> Best regards, Ilya Maximets. > >>> > >> > >> Thanks Ilya. > >> > >> Flavio was having some issues with the system test patch which appears > >> to have a flake. We have not received his Ack yet. I will follow-up with > >> a fix if it doesn't work for him or you can revert. > > > > It seems left side doesn't load the config fast enough, so the > > replied proposal is none, AES_GCM_C_256-NONE-NONE-DISABLE. Then > > So this gets rejected?
Yes. > > the retransmission happens in 60 seconds by default, but the > > script timeout is 30s. Obviously the timeout always wins failing > > the test. > > > > I used the following patch to test that and then the tunnel > > succeeds on the second attempt retransmitting in 10s. > > > > I got 0 failures in 100 tests. > > > > diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in > > index 89a36fe17..132db4dce 100755 > > --- a/ipsec/ovs-monitor-ipsec.in > > +++ b/ipsec/ovs-monitor-ipsec.in > > @@ -383,6 +383,7 @@ config setup > > conn %%default > > keyingtries=%%forever > > type=transport > > + retransmit-timeout=10 > > auto=route > > ike=aes_gcm256-sha2_256 > > esp=aes_gcm256 > > > > Another thing I tried is to add a simple sleep between the adding > > the ports: > > --- a/tests/system-ipsec.at > > +++ b/tests/system-ipsec.at > > @@ -166,6 +166,7 @@ dnl Set up IPsec tunnel on 'left' host > > OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=geneve \ > > options:remote_ip=10.1.1.2 options:psk=swordfish) > > > > +sleep 1 > > dnl Set up IPsec tunnel on 'right' host > > > > OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=geneve \ > > > > I got 0 failures in 100 tests with the above patch. > > > > What would be a good way to tell if a tunnel port is ready? Perhaps > > we could poll for status similar to what it is done at CHECK_ESP_TRAFFIC > > to make sure both sides are ready. > > You are saying is that one side comes up quick and one does not so it > misses the negotiation window causing the test to timeout while it waits > for IKE to re-transmit. I am surprised that the "slow" side does not > trigger negotiation when it comes up. > > Does it timeout on waiting for the connection to be active? i.e. > > https://github.com/openvswitch/ovs/blob/f8be30acf2eb60d567bb7386b98f5cb58ddb9119/tests/system-ipsec.at#L135 Yes. This is the tunnel state while waiting for it to establish: # for i in /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-monitor-ipsec.*.ctl; do ovs-appctl -t $i tunnels/show; done Interface name: tun v1 (CONFIGURED) Tunnel Type: geneve Local IP: %defaultroute Remote IP: 10.1.1.2 Address Family: IPv4 SKB mark: None Local cert: None Local name: None Local key: None Remote cert: None Remote name: None CA cert: None PSK: swordfish Ofport: 1 CFM state: Disabled Kernel policies installed: Kernel security associations installed: sel src 10.1.1.2/32 dst 10.1.1.1/32 IPsec connections that are active: 000 #2: "tun-in-1":500 STATE_PARENT_I2 (sent IKE_AUTH request); EVENT_RETRANSMIT in 39s; idle; Interface name: tun v1 (CONFIGURED) Tunnel Type: geneve Local IP: %defaultroute Remote IP: 10.1.1.1 Address Family: IPv4 SKB mark: None Local cert: None Local name: None Local key: None Remote cert: None Remote name: None CA cert: None PSK: swordfish Ofport: 1 CFM state: Disabled Kernel policies installed: Kernel security associations installed: sel src 10.1.1.1/32 dst 10.1.1.2/32 IPsec connections that are active: 000 #4: "tun-in-1":500 STATE_PARENT_I2 (sent IKE_AUTH request); EVENT_RETRANSMIT in 39s; idle; Note the EVENT_RETRANSMIT on both. This is the test log. # -*- compilation -*- 140. system-ipsec.at:152: testing IPsec -- Libreswan (ipv4, geneve, defaultroute, psk) ... Linux Libreswan 4.3 (netkey) on 5.11.11-200.fc33.x86_64 ./system-ipsec.at:158: modprobe openvswitch Module vport_lisp not loaded. Module vport_stt not loaded. ./system-ipsec.at:158: ovsdb-tool create conf.db $abs_top_srcdir/vswitchd/vswitch.ovsschema ./system-ipsec.at:158: ovsdb-server --detach --no-chdir --pidfile --log-file --remote=punix:$OVS_RUNDIR/db.sock stderr: 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovsdb-server.log ./system-ipsec.at:158: sed < stderr ' /vlog|INFO|opened log file/d /ovsdb_server|INFO|ovsdb-server (Open vSwitch)/d' ./system-ipsec.at:158: ovs-vsctl --no-wait init ./system-ipsec.at:158: ovs-vswitchd --detach --no-chdir --pidfile --log-file -vvconn -vofproto_dpif -vunixctl stderr: 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.log 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU cores 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/db.sock: connecting... 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/db.sock: connected ./system-ipsec.at:158: sed < stderr ' /ovs_numa|INFO|Discovered /d /vlog|INFO|opened log file/d /vswitchd|INFO|ovs-vswitchd (Open vSwitch)/d /reconnect|INFO|/d /dpif_netlink|INFO|Generic Netlink family .ovs_datapath. does not exist/d /ofproto|INFO|using datapath ID/d /netdev_linux|INFO|.*device has unknown hardware address family/d /ofproto|INFO|datapath ID changed to fedcba9876543210/d /dpdk|INFO|DPDK Disabled - Use other_config:dpdk-init to enable/d /netlink_socket|INFO|netlink: could not enable listening to all nsid/d /probe tc:/d /tc: Using policy/d' ./system-ipsec.at:158: ovs-vsctl -- add-br br0 -- set Bridge br0 protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 fail-mode=secure -- ./system-ipsec.at:159: cp ${abs_top_srcdir}/vswitchd/vswitch.ovsschema vswitch.ovsschema ./system-ipsec.at:159: ovs-ofctl add-flow br0 "actions=normal" Cannot remove namespace file "/var/run/netns/left": No such file or directory ./system-ipsec.at:162: ip netns add left || return 77 net.netfilter.nf_conntrack_helper = 0 net.ipv6.conf.all.accept_dad = 0 net.ipv6.conf.default.accept_dad = 0 ./system-ipsec.at:162: ip link add p0 type veth peer name ovs-p0 || return 77 ./system-ipsec.at:162: ip link set p0 netns left ./system-ipsec.at:162: ip link set dev ovs-p0 up ./system-ipsec.at:162: ovs-vsctl add-port br0 ovs-p0 -- \ set interface ovs-p0 external-ids:iface-id="p0" ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC ip addr add 10.1.1.1/24 dev p0 NS_EXEC_HEREDOC ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC ip link set dev p0 up NS_EXEC_HEREDOC 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/ovsdb.log 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/vswitchd.log 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU cores 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/db.sock: connecting... 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/db.sock: connected Initializing NSS database ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC ipsec pluto --config $ovs_base/left/ipsec.conf \ --ipsecdir $ovs_base/left --nssdir $ovs_base/left/ipsec.d \ --logfile $ovs_base/left/pluto.log --secretsfile $ovs_base/left/secrets \ --rundir $ovs_base/left NS_EXEC_HEREDOC stderr: /usr/libexec/ipsec/pluto: adjusting ipsec.d to /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left /usr/libexec/ipsec/pluto: adjusting nssdir to /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/ipsec.d ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC ovs-monitor-ipsec unix:${OVS_RUNDIR}/left/db.sock\ --pidfile=${OVS_RUNDIR}/left/ovs-monitor-ipsec.pid --ike-daemon=libreswan\ --ipsec-conf=$ovs_base/left/ipsec.conf --ipsec-d=$ovs_base/left/ipsec.d \ --ipsec-secrets=$ovs_base/left/secrets \ --log-file=$ovs_base/left/ovs-monitor-ipsec.log \ --ipsec-ctl=$ovs_base/left/pluto.ctl \ --no-restart-ike-daemon --detach NS_EXEC_HEREDOC stderr: Cannot remove namespace file "/var/run/netns/right": No such file or directory ./system-ipsec.at:163: ip netns add right || return 77 net.netfilter.nf_conntrack_helper = 0 net.ipv6.conf.all.accept_dad = 0 net.ipv6.conf.default.accept_dad = 0 ./system-ipsec.at:163: ip link add p1 type veth peer name ovs-p1 || return 77 ./system-ipsec.at:163: ip link set p1 netns right ./system-ipsec.at:163: ip link set dev ovs-p1 up ./system-ipsec.at:163: ovs-vsctl add-port br0 ovs-p1 -- \ set interface ovs-p1 external-ids:iface-id="p1" ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC ip addr add 10.1.1.2/24 dev p1 NS_EXEC_HEREDOC ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC ip link set dev p1 up NS_EXEC_HEREDOC 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/ovsdb.log 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/vswitchd.log 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA node 0 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU cores 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/db.sock: connecting... 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/db.sock: connected Initializing NSS database ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC ipsec pluto --config $ovs_base/right/ipsec.conf \ --ipsecdir $ovs_base/right --nssdir $ovs_base/right/ipsec.d \ --logfile $ovs_base/right/pluto.log --secretsfile $ovs_base/right/secrets \ --rundir $ovs_base/right NS_EXEC_HEREDOC stderr: /usr/libexec/ipsec/pluto: adjusting ipsec.d to /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right /usr/libexec/ipsec/pluto: adjusting nssdir to /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/ipsec.d ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC ovs-monitor-ipsec unix:${OVS_RUNDIR}/right/db.sock\ --pidfile=${OVS_RUNDIR}/right/ovs-monitor-ipsec.pid --ike-daemon=libreswan\ --ipsec-conf=$ovs_base/right/ipsec.conf --ipsec-d=$ovs_base/right/ipsec.d \ --ipsec-secrets=$ovs_base/right/secrets \ --log-file=$ovs_base/right/ovs-monitor-ipsec.log \ --ipsec-ctl=$ovs_base/right/pluto.ctl \ --no-restart-ike-daemon --detach NS_EXEC_HEREDOC stderr: system-ipsec.at:175: waiting until test `ipsec status --rundir /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left | grep Total... dropped privs to tcpdump tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ovs-p0, link-type EN10MB (Ethernet), snapshot length 262144 bytes dropped privs to tcpdump tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on ovs-p1, link-type EN10MB (Ethernet), snapshot length 262144 bytes system-ipsec.at:175: wait succeeded quickly system-ipsec.at:175: waiting until test `ipsec status --rundir /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right | grep Total... system-ipsec.at:175: wait succeeded immediately system-ipsec.at:175: waiting until test `ipsec status --rundir /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left | grep Total... system-ipsec.at:175: wait failed after 30 seconds ./ovs-macros.at:242: hard failure 0 packets captured 0 packets received by filter 0 packets dropped by kernel 0 packets captured 0 packets received by filter 0 packets dropped by kernel 2021-04-06T12:52:41Z|00001|daemon_unix|WARN|/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.pid: open: No such file or directory ovs-appctl: cannot read pidfile "/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.pid" (No such file or directory) ovs-dpctl: opening datapath (No such device) 140. system-ipsec.at:152: 140. IPsec -- Libreswan (ipv4, geneve, defaultroute, psk) (system-ipsec.at:152): FAILED (ovs-macros.at:242) -- fbl _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
