On Tue, Apr 06, 2021 at 02:53:34PM +0100, Mark Gray wrote: > On 06/04/2021 13:59, Flavio Leitner wrote: > > On Tue, Apr 06, 2021 at 12:30:42PM +0100, Mark Gray wrote: > >> On 05/04/2021 20:56, Flavio Leitner wrote: > >>> On Thu, Apr 01, 2021 at 07:48:19PM +0100, Mark Gray wrote: > >>>> On 01/04/2021 19:29, Ilya Maximets wrote: > >>>>> On 4/1/21 3:58 PM, Mark Gray wrote: > >>>>>> This series fixes IPv6 support for Libreswan and introduces > >>>>>> IPsec system tests for Libreswan. > >>>>>> > >>>>>> Mark Gray (5): > >>>>>> ipsec: IPv6 default route support for Libreswan > >>>>>> system-common-macros: clean up veth device on test failure > >>>>>> ipsec: Allow custom file locations > >>>>>> ipsec: Introduce IPsec system tests for Libreswan > >>>>>> ipsec: Update ordering of imports > >>>>>> > >>>>>> ipsec/ovs-monitor-ipsec.in | 144 ++++++++++-- > >>>>>> tests/automake.mk | 3 +- > >>>>>> tests/system-common-macros.at | 2 +- > >>>>>> tests/system-ipsec.at | 406 +++++++++++++++++++++++++++++++++ > >>>>>> tests/system-kmod-testsuite.at | 1 + > >>>>>> 5 files changed, 529 insertions(+), 27 deletions(-) > >>>>>> create mode 100644 tests/system-ipsec.at > >>>>>> > >>>>> > >>>>> Thanks! Applied to master. > >>>>> First patch with the fix also backported down to 2.13. > >>>>> > >>>>> Best regards, Ilya Maximets. > >>>>> > >>>> > >>>> Thanks Ilya. > >>>> > >>>> Flavio was having some issues with the system test patch which appears > >>>> to have a flake. We have not received his Ack yet. I will follow-up with > >>>> a fix if it doesn't work for him or you can revert. > >>> > >>> It seems left side doesn't load the config fast enough, so the > >>> replied proposal is none, AES_GCM_C_256-NONE-NONE-DISABLE. Then > >> > >> So this gets rejected? > > > > Yes. > > > >>> the retransmission happens in 60 seconds by default, but the > >>> script timeout is 30s. Obviously the timeout always wins failing > >>> the test. > >>> > >>> I used the following patch to test that and then the tunnel > >>> succeeds on the second attempt retransmitting in 10s. > >>> > >>> I got 0 failures in 100 tests. > >>> > >>> diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in > >>> index 89a36fe17..132db4dce 100755 > >>> --- a/ipsec/ovs-monitor-ipsec.in > >>> +++ b/ipsec/ovs-monitor-ipsec.in > >>> @@ -383,6 +383,7 @@ config setup > >>> conn %%default > >>> keyingtries=%%forever > >>> type=transport > >>> + retransmit-timeout=10 > >>> auto=route > >>> ike=aes_gcm256-sha2_256 > >>> esp=aes_gcm256 > >>> > >>> Another thing I tried is to add a simple sleep between the adding > >>> the ports: > >>> --- a/tests/system-ipsec.at > >>> +++ b/tests/system-ipsec.at > >>> @@ -166,6 +166,7 @@ dnl Set up IPsec tunnel on 'left' host > >>> OVS_VSCTL_LEFT(add-port br-ipsec tun -- set Interface tun type=geneve \ > >>> options:remote_ip=10.1.1.2 options:psk=swordfish) > >>> > >>> +sleep 1 > >>> dnl Set up IPsec tunnel on 'right' host > >>> > >>> OVS_VSCTL_RIGHT(add-port br-ipsec tun -- set Interface tun type=geneve \ > >>> > >>> I got 0 failures in 100 tests with the above patch. > >>> > >>> What would be a good way to tell if a tunnel port is ready? Perhaps > >>> we could poll for status similar to what it is done at CHECK_ESP_TRAFFIC > >>> to make sure both sides are ready. > >> > >> You are saying is that one side comes up quick and one does not so it > >> misses the negotiation window causing the test to timeout while it waits > >> for IKE to re-transmit. I am surprised that the "slow" side does not > >> trigger negotiation when it comes up. > >> > >> Does it timeout on waiting for the connection to be active? i.e. > >> > >> https://github.com/openvswitch/ovs/blob/f8be30acf2eb60d567bb7386b98f5cb58ddb9119/tests/system-ipsec.at#L135 > > > > Yes. > > > > This is the tunnel state while waiting for it to establish: > > > > # for i in > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-monitor-ipsec.*.ctl; > > do ovs-appctl -t $i tunnels/show; done > > Interface name: tun v1 (CONFIGURED) > > Tunnel Type: geneve > > Local IP: %defaultroute > > Remote IP: 10.1.1.2 > > Address Family: IPv4 > > SKB mark: None > > Local cert: None > > Local name: None > > Local key: None > > Remote cert: None > > Remote name: None > > CA cert: None > > PSK: swordfish > > Ofport: 1 > > CFM state: Disabled > > Kernel policies installed: > > Kernel security associations installed: > > sel src 10.1.1.2/32 dst 10.1.1.1/32 > > IPsec connections that are active: > > 000 #2: "tun-in-1":500 STATE_PARENT_I2 (sent IKE_AUTH request); > > EVENT_RETRANSMIT in 39s; idle; > > > > Interface name: tun v1 (CONFIGURED) > > Tunnel Type: geneve > > Local IP: %defaultroute > > Remote IP: 10.1.1.1 > > Address Family: IPv4 > > SKB mark: None > > Local cert: None > > Local name: None > > Local key: None > > Remote cert: None > > Remote name: None > > CA cert: None > > PSK: swordfish > > Ofport: 1 > > CFM state: Disabled > > Kernel policies installed: > > Kernel security associations installed: > > sel src 10.1.1.1/32 dst 10.1.1.2/32 > > IPsec connections that are active: > > 000 #4: "tun-in-1":500 STATE_PARENT_I2 (sent IKE_AUTH request); > > EVENT_RETRANSMIT in 39s; idle; > > > > > > Note the EVENT_RETRANSMIT on both. > > > > This is the test log. > > > > > > # -*- compilation -*- > > 140. system-ipsec.at:152: testing IPsec -- Libreswan (ipv4, geneve, > > defaultroute, psk) ... > > Linux Libreswan 4.3 (netkey) on 5.11.11-200.fc33.x86_64 > > ./system-ipsec.at:158: modprobe openvswitch > > Module vport_lisp not loaded. > > Module vport_stt not loaded. > > ./system-ipsec.at:158: ovsdb-tool create conf.db > > $abs_top_srcdir/vswitchd/vswitch.ovsschema > > ./system-ipsec.at:158: ovsdb-server --detach --no-chdir --pidfile > > --log-file --remote=punix:$OVS_RUNDIR/db.sock > > stderr: > > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovsdb-server.log > > ./system-ipsec.at:158: sed < stderr ' > > /vlog|INFO|opened log file/d > > /ovsdb_server|INFO|ovsdb-server (Open vSwitch)/d' > > ./system-ipsec.at:158: ovs-vsctl --no-wait init > > ./system-ipsec.at:158: ovs-vswitchd --detach --no-chdir --pidfile > > --log-file -vvconn -vofproto_dpif -vunixctl > > stderr: > > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.log > > 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA > > node 0 > > 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU > > cores > > 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/db.sock: > > connecting... > > 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/db.sock: > > connected > > ./system-ipsec.at:158: sed < stderr ' > > /ovs_numa|INFO|Discovered /d > > /vlog|INFO|opened log file/d > > /vswitchd|INFO|ovs-vswitchd (Open vSwitch)/d > > /reconnect|INFO|/d > > /dpif_netlink|INFO|Generic Netlink family .ovs_datapath. does not exist/d > > /ofproto|INFO|using datapath ID/d > > /netdev_linux|INFO|.*device has unknown hardware address family/d > > /ofproto|INFO|datapath ID changed to fedcba9876543210/d > > /dpdk|INFO|DPDK Disabled - Use other_config:dpdk-init to enable/d > > /netlink_socket|INFO|netlink: could not enable listening to all nsid/d > > /probe tc:/d > > /tc: Using policy/d' > > ./system-ipsec.at:158: ovs-vsctl -- add-br br0 -- set Bridge br0 > > protocols=OpenFlow10,OpenFlow11,OpenFlow12,OpenFlow13,OpenFlow14,OpenFlow15 > > fail-mode=secure -- > > ./system-ipsec.at:159: cp ${abs_top_srcdir}/vswitchd/vswitch.ovsschema > > vswitch.ovsschema > > ./system-ipsec.at:159: ovs-ofctl add-flow br0 "actions=normal" > > Cannot remove namespace file "/var/run/netns/left": No such file or > > directory > > ./system-ipsec.at:162: ip netns add left || return 77 > > net.netfilter.nf_conntrack_helper = 0 > > net.ipv6.conf.all.accept_dad = 0 > > net.ipv6.conf.default.accept_dad = 0 > > ./system-ipsec.at:162: ip link add p0 type veth peer name ovs-p0 || return > > 77 > > ./system-ipsec.at:162: ip link set p0 netns left > > ./system-ipsec.at:162: ip link set dev ovs-p0 up > > ./system-ipsec.at:162: ovs-vsctl add-port br0 ovs-p0 -- \ > > set interface ovs-p0 external-ids:iface-id="p0" > > ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC > > ip addr add 10.1.1.1/24 dev p0 > > NS_EXEC_HEREDOC > > ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC > > ip link set dev p0 up > > NS_EXEC_HEREDOC > > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/ovsdb.log > > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/vswitchd.log > > 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA > > node 0 > > 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU > > cores > > 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/db.sock: > > connecting... > > 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/db.sock: > > connected > > Initializing NSS database > > > > ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC > > ipsec pluto --config $ovs_base/left/ipsec.conf \ > > --ipsecdir $ovs_base/left --nssdir $ovs_base/left/ipsec.d \ > > --logfile $ovs_base/left/pluto.log --secretsfile > > $ovs_base/left/secrets \ > > --rundir $ovs_base/left > > NS_EXEC_HEREDOC > > stderr: > > /usr/libexec/ipsec/pluto: adjusting ipsec.d to > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left > > /usr/libexec/ipsec/pluto: adjusting nssdir to > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left/ipsec.d > > ./system-ipsec.at:162: ip netns exec left sh << NS_EXEC_HEREDOC > > ovs-monitor-ipsec unix:${OVS_RUNDIR}/left/db.sock\ > > --pidfile=${OVS_RUNDIR}/left/ovs-monitor-ipsec.pid > > --ike-daemon=libreswan\ > > --ipsec-conf=$ovs_base/left/ipsec.conf > > --ipsec-d=$ovs_base/left/ipsec.d \ > > --ipsec-secrets=$ovs_base/left/secrets \ > > --log-file=$ovs_base/left/ovs-monitor-ipsec.log \ > > --ipsec-ctl=$ovs_base/left/pluto.ctl \ > > --no-restart-ike-daemon --detach > > NS_EXEC_HEREDOC > > stderr: > > Cannot remove namespace file "/var/run/netns/right": No such file or > > directory > > ./system-ipsec.at:163: ip netns add right || return 77 > > net.netfilter.nf_conntrack_helper = 0 > > net.ipv6.conf.all.accept_dad = 0 > > net.ipv6.conf.default.accept_dad = 0 > > ./system-ipsec.at:163: ip link add p1 type veth peer name ovs-p1 || return > > 77 > > ./system-ipsec.at:163: ip link set p1 netns right > > ./system-ipsec.at:163: ip link set dev ovs-p1 up > > ./system-ipsec.at:163: ovs-vsctl add-port br0 ovs-p1 -- \ > > set interface ovs-p1 external-ids:iface-id="p1" > > ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC > > ip addr add 10.1.1.2/24 dev p1 > > NS_EXEC_HEREDOC > > ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC > > ip link set dev p1 up > > NS_EXEC_HEREDOC > > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/ovsdb.log > > 2021-04-06T12:52:07Z|00001|vlog|INFO|opened log file > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/vswitchd.log > > 2021-04-06T12:52:07Z|00002|ovs_numa|INFO|Discovered 4 CPU cores on NUMA > > node 0 > > 2021-04-06T12:52:07Z|00003|ovs_numa|INFO|Discovered 1 NUMA nodes and 4 CPU > > cores > > 2021-04-06T12:52:07Z|00004|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/db.sock: > > connecting... > > 2021-04-06T12:52:07Z|00005|reconnect|INFO|unix:/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/db.sock: > > connected > > Initializing NSS database > > > > ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC > > ipsec pluto --config $ovs_base/right/ipsec.conf \ > > --ipsecdir $ovs_base/right --nssdir $ovs_base/right/ipsec.d \ > > --logfile $ovs_base/right/pluto.log --secretsfile > > $ovs_base/right/secrets \ > > --rundir $ovs_base/right > > NS_EXEC_HEREDOC > > stderr: > > /usr/libexec/ipsec/pluto: adjusting ipsec.d to > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right > > /usr/libexec/ipsec/pluto: adjusting nssdir to > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right/ipsec.d > > ./system-ipsec.at:163: ip netns exec right sh << NS_EXEC_HEREDOC > > ovs-monitor-ipsec unix:${OVS_RUNDIR}/right/db.sock\ > > --pidfile=${OVS_RUNDIR}/right/ovs-monitor-ipsec.pid > > --ike-daemon=libreswan\ > > --ipsec-conf=$ovs_base/right/ipsec.conf > > --ipsec-d=$ovs_base/right/ipsec.d \ > > --ipsec-secrets=$ovs_base/right/secrets \ > > --log-file=$ovs_base/right/ovs-monitor-ipsec.log \ > > --ipsec-ctl=$ovs_base/right/pluto.ctl \ > > --no-restart-ike-daemon --detach > > NS_EXEC_HEREDOC > > stderr: > > system-ipsec.at:175: waiting until test `ipsec status --rundir > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left | > > grep Total... > > dropped privs to tcpdump > > tcpdump: verbose output suppressed, use -v[v]... for full protocol decode > > listening on ovs-p0, link-type EN10MB (Ethernet), snapshot length 262144 > > bytes > > dropped privs to tcpdump > > tcpdump: verbose output suppressed, use -v[v]... for full protocol decode > > listening on ovs-p1, link-type EN10MB (Ethernet), snapshot length 262144 > > bytes > > system-ipsec.at:175: wait succeeded quickly > > system-ipsec.at:175: waiting until test `ipsec status --rundir > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/right | > > grep Total... > > system-ipsec.at:175: wait succeeded immediately > > system-ipsec.at:175: waiting until test `ipsec status --rundir > > /root/upstream/ovs/tests/system-kmod-testsuite.dir/140/left | > > grep Total... > > system-ipsec.at:175: wait failed after 30 seconds > > ./ovs-macros.at:242: hard failure > > 0 packets captured > > 0 packets received by filter > > 0 packets dropped by kernel > > 0 packets captured > > 0 packets received by filter > > 0 packets dropped by kernel > > 2021-04-06T12:52:41Z|00001|daemon_unix|WARN|/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.pid: > > open: No such file or directory > > ovs-appctl: cannot read pidfile > > "/root/upstream/ovs/tests/system-kmod-testsuite.dir/140/ovs-vswitchd.pid" > > (No such file or directory) > > ovs-dpctl: opening datapath (No such device) > > 140. system-ipsec.at:152: 140. IPsec -- Libreswan (ipv4, geneve, > > defaultroute, psk) (system-ipsec.at:152): FAILED (ovs-macros.at:242) > > > > Could you post pluto.log from both left and right side?
There you go: http://people.redhat.com/~fleitner/testsuite/ Sending info to direct access if you prefer in the chat. -- fbl _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
