On Tue, Aug 23, 2016 at 10:51:47AM +0200, Simon Horman wrote: > On Mon, Aug 22, 2016 at 02:47:42PM -0700, Joe Stringer wrote: > > On 22 August 2016 at 04:04, Simon Horman <simon.hor...@netronome.com> wrote: > > > On Wed, Aug 10, 2016 at 10:17:30AM -0700, Joe Stringer wrote: > > >> On 10 August 2016 at 03:20, Simon Horman <simon.hor...@netronome.com> > > >> wrote: > > >> > On Tue, Aug 09, 2016 at 08:47:40AM -0700, pravin shelar wrote: > > >> >> On Mon, Aug 8, 2016 at 8:17 AM, Simon Horman > > >> >> <simon.hor...@netronome.com> wrote: > > >> >> > Light testing seems to indicate that it works for GSO skbs > > >> >> > received over both L3 and L2 GRE tunnels by OvS with both > > >> >> > IP-in-MPLS and IP (without MPLS) payloads. > > >> >> > > > >> >> > > >> >> Thanks for testing it. Can you also add those tests to OVS kmod test > > >> >> suite? > > >> >> .. > > >> > > > >> > Sure, I will look into doing that. > > >> > Am I correct in thinking Joe Stringer is the best person to contact if > > >> > I run into trouble there? > > >> > > >> Sure. The basics of running the tests is documented here: > > >> https://github.com/openvswitch/ovs/blob/master/INSTALL.md#datapath-testing > > >> > > >> You should be able to get a good feel for how to add tests by perusing > > >> the commits to tests/system-{traffic,kmod-macros}.at in the OVS source > > >> tree. > > > > > > Thanks Joe, > > > > > > it took me a while but I think that I have something working > > > against the head branch of the OVS tree. I'd value opinions > > > on the direction I have taken. > > > > > > Subject: [PATCH] system-traffic: Exercise GSO > > > > > > Exercise GSO for: unencapsulated; MPLS; GRE; and MPLS in GRE. > > > > > > There is scope to extend this testing to other encapsulation formats > > > if desired. > > > > > > This is motivated by a desire to test GRE and MPLS encapsulation in > > > the context of L3/VPN (MPLS over non-TEB GRE work). That is not > > > tested here but tests for those cases would ideally be based on those in > > > this patch. > > > > This makes sense to me. There's a few corners that could be improved, > > primarily for reproducing sane results on a variety of systems, then a > > couple of style comments. Please do run the tests via both "make > > check-kernel" and "make check-system-userspace" before submitting, > > ideally with at least two varieties of kernel: One where you would > > expect the test to pass, and one where you would expect the tests to > > be skipped.
Both make check-kernel and make check-system-userspace are now working. I have tested against net-next and the 3.16 kernel that ships with Debian stable. > Thanks. I'm glad I ran this by you before expanding the number of tests. > > > * CHECK_MPLS is defined in system-kmod-macros.at, so a corresponding > > version should be provided in system-userspace-macros.at. If the > > criteria for running the test(s) with both userspace and kernel > > datapaths is the same, then this could instead be moved into > > system-common-macros.at. > > Understood. > > > * "datapath - ping over gre tunnel" adds a command to execute in > > at_ns1, but that namespace doesn't exist. > > Oops. I have removed the chunk in question, it seems to be an artifact of my development of the tests. > > * "datapath - http over gre tunnel" is missing MPLS_CHECK. > > Thanks, I'll fix that. On further inspection it seems tome that this check does not use MPLS, rather it is testing GSO for GRE (without MPLS). > > * Is there a way to clear the netstat statistics before running the > > tests which rely on it? I'm getting a failure on one of my systems > > (ubuntu trusty with a 4.7 kernel), but I'm not sure if the counter was > > already high before I ran the test. > > I'll look into that. If not they could be recorded to allow a check > for a non-zero delta. > > Possibly an entirely different mechanism is needed to check for GSO > functioning. But I'm not sure what it would be at this point. > > > * "datapath - http over mpls between two ports" (maybe others too?) > > should shift all openflow rules into a single section using AT_DATA, > > similar to the other tests. This makes it easier to reason about the > > flow table and understand what's going on before reading through the > > rest of the test. > > Sure, will do. > > > * If there is a common set of configuration you do for local stack > > within a namespace to route MPLS traffic, you could consider adding > > another macro into system-common-macros.at. > > Ok, possibly there is if some of the configuration is parametrised: > e.g. over the namespace/netdev to send/receive MPLS using native Linux MPLS > routing. > > > I also see this error on "http over mpls over gre tunnel": > > +sh: 1: cannot create /proc/sys/net/mpls/conf/ns_gre0/input: Directory > > nonexistent > > > > Maybe MPLS + GRE needs a separate check? > > Yes, that is probably the case. > > I believe some versions of the kernel support MPLS routing for > some interfaces but not GRE interfaces. Please find my working patch below. From: Simon Horman <ho...@verge.net.au> Subject: [PATCH] system-traffic: Exercise GSO Exercise GSO for: unencapsulated; MPLS; GRE; and MPLS in GRE. There is scope to extend this testing to other encapsulation formats if desired. This is motivated by a desire to test GRE and MPLS encapsulation in the context of L3/VPN (MPLS over non-TEB GRE work). That is not tested here but tests for those cases would idealy be based on those in this patch. Signed-off-by: Simon Horman <ho...@verge.net.au> --- v2 * As suggested by Joe Stringer - Do not add spurious ns1 / http commands to "datapath - ping over gre tunnel" - Add OVS_CHECK_MPLS_GRE - Use AT_SKIP_IF - Use AT_DATA - Check for zero delta of retransmits rather than absoloute zero - Consolidate MPLS configuration into NS_ADD_MPLS_ROUTE macro - Move macros from system-kmod-macros.at to system-common-marcos.at to allow tests to work for make check-system-userspace to work as well as make check-kernel --- tests/system-common-macros.at | 94 +++++++++++++++++++-- tests/system-traffic.at | 191 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 276 insertions(+), 9 deletions(-) diff --git a/tests/system-common-macros.at b/tests/system-common-macros.at index 4ffc382..41040a8 100644 --- a/tests/system-common-macros.at +++ b/tests/system-common-macros.at @@ -56,7 +56,7 @@ m4_define([ADD_INT], ] ) -# ADD_VETH([port], [namespace], [ovs-br], [ip_addr] [mac_addr [gateway]]) +# ADD_VETH([port], [namespace], [ovs-br], [ip_addr] [mac_addr [gateway [ofport]]]) # # Add a pair of veth ports. 'port' will be added to name space 'namespace', # and "ovs-'port'" will be added to ovs bridge 'ovs-br'. @@ -64,8 +64,8 @@ m4_define([ADD_INT], # The 'port' in 'namespace' will be brought up with static IP address # with 'ip_addr' in CIDR notation. # -# Optionally, one can specify the 'mac_addr' for 'port' and the default -# 'gateway'. +# Optionally, one can specify the 'mac_addr' for 'port', the default +# 'gateway' and the 'ofport' number. # # The existing 'port' or 'ovs-port' will be removed before new ones are added. # @@ -74,8 +74,14 @@ m4_define([ADD_VETH], CONFIGURE_VETH_OFFLOADS([$1]) AT_CHECK([ip link set $1 netns $2]) AT_CHECK([ip link set dev ovs-$1 up]) - AT_CHECK([ovs-vsctl add-port $3 ovs-$1 -- \ - set interface ovs-$1 external-ids:iface-id="$1"]) + if test -n "$7"; then + AT_CHECK([ovs-vsctl add-port $3 ovs-$1 -- \ + set interface ovs-$1 external-ids:iface-id="$1" \ + ofport_request=$7]) + else + AT_CHECK([ovs-vsctl add-port $3 ovs-$1 -- \ + set interface ovs-$1 external-ids:iface-id="$1"]) + fi NS_CHECK_EXEC([$2], [ip addr add $4 dev $1]) NS_CHECK_EXEC([$2], [ip link set dev $1 up]) if test -n "$5"; then @@ -99,7 +105,7 @@ m4_define([ADD_VLAN], ] ) -# ADD_OVS_TUNNEL([type], [bridge], [port], [remote-addr], [overlay-addr]) +# ADD_OVS_TUNNEL([type], [bridge], [port], [remote-addr], [overlay-addr [ofport]]) # # Add an ovs-based tunnel device in the root namespace, with name 'port' and # type 'type'. The tunnel device will be configured as point-to-point with the @@ -107,9 +113,17 @@ m4_define([ADD_VLAN], # # 'port will be configured with the address 'overlay-addr'. # +# Optionally one can specify the 'ofport' number +# m4_define([ADD_OVS_TUNNEL], - [AT_CHECK([ovs-vsctl add-port $2 $3 -- \ - set int $3 type=$1 options:remote_ip=$4]) + [if test -n "$6"; then + AT_CHECK([ovs-vsctl add-port $2 $3 -- \ + set int $3 type=$1 options:remote_ip=$4 \ + ofport_request=$6]) + else + AT_CHECK([ovs-vsctl add-port $2 $3 -- \ + set int $3 type=$1 options:remote_ip=$4]) + fi AT_CHECK([ip addr add dev $2 $5]) AT_CHECK([ip link set dev $2 up]) AT_CHECK([ip link set dev $2 mtu 1450]) @@ -143,6 +157,12 @@ m4_define([ADD_NATIVE_TUNNEL], # m4_define([FORMAT_PING], [grep "transmitted" | sed 's/time.*ms$/time 0ms/']) +# FORMAT_DD([]) +# +# Strip variant pieces from dd output so the output can be reliably compared. +# +m4_define([FORMAT_DD], [sed 's/copied,.*$/copied, .../']) + # FORMAT_CT([ip-addr]) # # Strip content from the piped input which would differ from test to test @@ -168,10 +188,68 @@ m4_define([NETNS_DAEMONIZE], m4_define([OVS_CHECK_VXLAN], [AT_SKIP_IF([! ip link add foo type vxlan help 2>&1 | grep dstport >/dev/null])]) +# OVS_CHECK_MPLS() +# +# Perform requirements checks for running MPLS tests. +# +m4_define([OVS_CHECK_MPLS], + [m4_foreach([mod], [[mpls_router]], + [modprobe mod || echo "Module mod not loaded." + on_exit 'modprobe -r mod' + ]) + dnl Requires Linux v4.7+ and ip route v4.7+ + dnl Prepare + AT_SKIP_IF([! echo 101 > /proc/sys/net/mpls/platform_labels]) + [ip -f mpls route del 100] + dnl Test + AT_SKIP_IF([! ip -f mpls route add 100 dev lo]) + dnl Cleanup + AT_SKIP_IF([! ip -f mpls route del 100]) + AT_SKIP_IF([! echo 0 > /proc/sys/net/mpls/platform_labels]) + ] +) + # OVS_CHECK_GRE() m4_define([OVS_CHECK_GRE], [AT_SKIP_IF([! ip link add foo type gretap help 2>&1 | grep gre >/dev/null])]) +# OVS_CHECK_MPLS_GRE() +# +# Perform requirements checks for running MPLS over GRE tests. +# +m4_define([OVS_CHECK_MPLS_GRE], + [OVS_CHECK_MPLS() + OVS_CHECK_GRE() + m4_foreach([mod], [[ip_gre]], + [modprobe mod || echo "Module mod not loaded." + on_exit 'modprobe -r mod' + ]) + dnl Requires Linux v4.8 + AT_SKIP_IF([! test -e /proc/sys/net/mpls/conf/gretap0/input]) + ] +) + # OVS_CHECK_GENEVE() m4_define([OVS_CHECK_GENEVE], [AT_SKIP_IF([! ip link add foo type geneve help 2>&1 | grep geneve >/dev/null])]) + +# NS_ADD_MPLS_ROUTE([ns], [port], [prefix], [gateway]) +# +# Route IP packets to prefix via gateway on port using MPLS label 100 and +# recieve MPLS packets with label 101 on port locally as IP packets +m4_define([NS_ADD_MPLS_ROUTE], + [dnl Allow MPLS forwarding of packets received on $1 + NS_CHECK_EXEC([$1], [echo 1 > /proc/sys/net/mpls/conf/$2/input]) + + dnl Larger than MPLS label to be routed (101) + NS_CHECK_EXEC([$1], [echo 102 > /proc/sys/net/mpls/platform_labels]) + + dnl Set up route to encapsulate packets to $2 in MPLS + NS_CHECK_EXEC([$1], [ip route add $3 encap mpls 100 via inet $4 dev $2]) + + dnl Set up route to decapsulate MPLS label 101 and deliver locally + NS_CHECK_EXEC([$1], [ip -f mpls route add 101 dev lo]) + + dnl Set loopback interface up for local delivery + NS_CHECK_EXEC([$1], [ip link set up dev lo]) + ]) diff --git a/tests/system-traffic.at b/tests/system-traffic.at index 2f42efa..3f9db51 100644 --- a/tests/system-traffic.at +++ b/tests/system-traffic.at @@ -33,12 +33,27 @@ ADD_NAMESPACES(at_ns0, at_ns1) ADD_VETH(p0, at_ns0, br0, "10.1.1.1/24") ADD_VETH(p1, at_ns1, br0, "10.1.1.2/24") +dnl Disable TSO as to exercise software segmentation +dnl when outputting GSO skbs over GRE from OvS +AT_CHECK([ethtool -K ovs-p0 tso off]) + NS_CHECK_EXEC([at_ns0], [ping -q -c 3 -i 0.3 -w 2 10.1.1.2 | FORMAT_PING], [0], [dnl 3 packets transmitted, 3 received, 0% packet loss, time 0ms ]) +AT_CHECK([yes | dd bs=1k count=32 of=32k.txt], [0], [], stderr) +AT_CHECK([cat stderr | FORMAT_DD], [0], [dnl +32+0 records in +32+0 records out +32768 bytes (33 kB) copied, ... +]) NETNS_DAEMONIZE([at_ns1], [[$PYTHON $srcdir/test-l7.py]], [http0.pid]) -NS_CHECK_EXEC([at_ns0], [wget 10.1.1.2 -t 3 -T 1 --retry-connrefused -v -o wget0.log]) +NS_CHECK_EXEC([at_ns1], [netstat -s | grep retransmited > retransmit_before]) +NS_CHECK_EXEC([at_ns0], [wget 10.1.1.2/32k.txt -t 3 -T 1 --retry-connrefused -v -o wget0.log]) +NS_CHECK_EXEC([at_ns1], [netstat -s | grep retransmited > retransmit_after]) + +dnl Use the absence of retransmitted segments as a proxy for functioning TSO +AT_CHECK([diff -u retransmit_before retransmit_after], [0]) OVS_TRAFFIC_VSWITCHD_STOP AT_CLEANUP @@ -69,6 +84,58 @@ NS_CHECK_EXEC([at_ns0], [ping -s 3200 -q -c 3 -i 0.3 -w 2 10.2.2.2 | FORMAT_PING OVS_TRAFFIC_VSWITCHD_STOP AT_CLEANUP +AT_SETUP([datapath - http over mpls between two ports]) +OVS_CHECK_MPLS() +OVS_TRAFFIC_VSWITCHD_START() + +ADD_NAMESPACES(at_ns0, at_ns1) + +dnl Set up underlay +ADD_VETH(p0, at_ns0, br0, "172.31.1.1/24", [], [], 2) +ADD_VETH(p1, at_ns1, br0, "172.31.1.2/24", [], [], 3) + +dnl Set up MPLS overlay +dnl IP is encapsulated in MPLS when sent from and recieved from ns0 +dnl OvS, sitting between ns0 and ns1 pushes MPLS onto IP recieved from ns1 +dnl befor sending to ns0, and pops MPLS recieved from ns0 and sends the +dnl resulting IP packets to ns1 +NS_CHECK_EXEC([at_ns1], [ip addr add 10.1.1.2/24 dev p1]) + +AT_DATA([flows.txt], [dnl +in_port=3,ip,nw_src=10.1.1.2,actions=push_mpls:0x8847,set_field:101->mpls_label,output:2 +in_port=2,dl_type=0x8847,mpls_label=100,actions=pop_mpls:0x0800,output:3 +actions=normal +]) +AT_CHECK([ovs-ofctl add-flows br0 flows.txt]) + +NS_ADD_MPLS_ROUTE([at_ns0], [p0], [10.1.1.0/24], [172.31.1.2]) + +dnl Allow for overhead of MPLS LSE +NS_CHECK_EXEC([at_ns1], [ip link set mtu 1496 dev p1]) + +dnl Test ping +NS_CHECK_EXEC([at_ns0], [ping -q -c 3 -i 0.3 -w 2 10.1.1.2 | FORMAT_PING], [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +dnl Test http +AT_CHECK([yes | dd bs=1k count=32 of=32k.txt], [0], [], stderr) +AT_CHECK([cat stderr | FORMAT_DD], [0], [dnl +32+0 records in +32+0 records out +32768 bytes (33 kB) copied, ... +]) +NETNS_DAEMONIZE([at_ns1], [[$PYTHON $srcdir/test-l7.py]], [http0.pid]) +NS_CHECK_EXEC([at_ns1], [netstat -s | grep retransmited > retransmit_before]) +NS_CHECK_EXEC([at_ns0], [wget 10.1.1.2/32k.txt -t 3 -T 1 --retry-connrefused -v -o wget0.log]) +NS_CHECK_EXEC([at_ns1], [netstat -s | grep retransmited > retransmit_after]) + +dnl Use the absence of retransmitted segments as a proxy for functioning GSO +AT_CHECK([diff -u retransmit_before retransmit_after], [0]) + +OVS_TRAFFIC_VSWITCHD_STOP +AT_CLEANUP + AT_SETUP([datapath - ping6 between two ports]) OVS_TRAFFIC_VSWITCHD_START() @@ -209,6 +276,128 @@ NS_CHECK_EXEC([at_ns0], [ping -s 3200 -q -c 3 -i 0.3 -w 2 10.1.1.100 | FORMAT_PI OVS_TRAFFIC_VSWITCHD_STOP AT_CLEANUP +AT_SETUP([datapath - http over gre tunnel]) +OVS_CHECK_GRE() + +OVS_TRAFFIC_VSWITCHD_START() +ADD_BR([br-underlay]) + +AT_CHECK([ovs-ofctl add-flow br0 "actions=normal"]) +AT_CHECK([ovs-ofctl add-flow br-underlay "actions=normal"]) + +ADD_NAMESPACES(at_ns0, at_ns1) + +dnl Set up underlay link from host into the at_ns0 namespace using veth pair. +ADD_VETH(p0, at_ns0, br-underlay, "172.31.1.1/24") +AT_CHECK([ip addr add dev br-underlay "172.31.1.100/24"]) +AT_CHECK([ip link set dev br-underlay up]) + +dnl Set up tunnel endpoints on OVS outside the at_ns0 namespace and +dnl with a native linux device inside the namespace. +ADD_OVS_TUNNEL([gre], [br0], [at_gre0], [172.31.1.1], [10.1.1.100/24]) +ADD_NATIVE_TUNNEL([gretap], [ns_gre0], [at_ns0], [172.31.1.100], [10.1.1.1/24]) + +dnl Add veth pair connected to to br0 and at_ns1 namespace +ADD_VETH(p1, at_ns1, br0, "10.1.1.2/24") + +dnl Disable TSO as to exercise software segmentation +dnl when outputting GSO skbs over GRE from OvS +AT_CHECK([ethtool -K ovs-p0 tso off]) + +dnl First check the underlay +NS_CHECK_EXEC([at_ns0], [ping -q -c 3 -i 0.3 -w 2 172.31.1.100 | FORMAT_PING], [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +dnl Next check the overlay +NS_CHECK_EXEC([at_ns0], [ping -q -c 3 -i 0.3 -w 2 10.1.1.2 | FORMAT_PING], [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +dnl Now check HTTP transfer +AT_CHECK([yes | dd bs=1k count=32 of=32k.txt], [0], [], stderr) +AT_CHECK([cat stderr | FORMAT_DD], [0], [dnl +32+0 records in +32+0 records out +32768 bytes (33 kB) copied, ... +]) +NETNS_DAEMONIZE([at_ns1], [[$PYTHON $srcdir/test-l7.py]], [http0.pid]) +NS_CHECK_EXEC([at_ns1], [netstat -s | grep retransmited > retransmit_before]) +NS_CHECK_EXEC([at_ns0], [wget 10.1.1.2/32k.txt -t 3 -T 1 --retry-connrefused -v -o wget0.log]) +NS_CHECK_EXEC([at_ns1], [netstat -s | grep retransmited > retransmit_after]) + +dnl Use the absence of retransmitted segments as a proxy for functioning TSO +AT_CHECK([diff -u retransmit_before retransmit_after], [0]) + +OVS_TRAFFIC_VSWITCHD_STOP +AT_CLEANUP + +AT_SETUP([datapath - http over mpls over gre tunnel]) +OVS_CHECK_MPLS_GRE() + +OVS_TRAFFIC_VSWITCHD_START() +ADD_BR([br-underlay]) + +ADD_NAMESPACES(at_ns0, at_ns1) + +dnl Set up underlay link from host into the at_ns0 namespace using veth pair. +ADD_VETH(p0, at_ns0, br-underlay, "172.31.1.1/24") +AT_CHECK([ip addr add dev br-underlay "172.31.1.100/24"]) +AT_CHECK([ip link set dev br-underlay up]) + +dnl Set up tunnel endpoints on OVS outside the at_ns0 namespace and +dnl with a native linux device inside the namespace. +ADD_OVS_TUNNEL([gre], [br0], [at_gre0], [172.31.1.1], [10.1.1.100/24], 2) +ADD_NATIVE_TUNNEL([gretap], [ns_gre0], [at_ns0], [172.31.1.100], [10.1.1.1/24]) + +dnl Add veth pair connected to to br0 and at_ns1 namespace +ADD_VETH(p1, at_ns1, br0, "10.1.1.2/24", [], [], 3) + +dnl Set up MPLS overlay +dnl IP is encapsulated in MPLS when sent from and recieved from ns0 +dnl OvS, sitting between ns0 and ns1 pushes MPLS onto IP recieved from ns1 +dnl befor sending to ns0, and pops MPLS recieved from ns0 and sends the +dnl resulting IP packets to ns1 +NS_CHECK_EXEC([at_ns1], [ip addr add 10.1.2.2/24 dev p1]) + +AT_DATA([flows.txt], [dnl +in_port=3,ip,nw_src=10.1.2.2,actions=push_mpls:0x8847,set_field:101->mpls_label,output:2 +in_port=2,dl_type=0x8847,mpls_label=100,actions=pop_mpls:0x0800,output:3 +actions=normal +]) +AT_CHECK([ovs-ofctl add-flows br0 flows.txt]) +AT_CHECK([ovs-ofctl add-flow br-underlay "actions=normal"]) + +NS_ADD_MPLS_ROUTE([at_ns0], [ns_gre0], [10.1.2.0/24], [10.1.1.2]) + +dnl First check the underlay +NS_CHECK_EXEC([at_ns0], [ping -q -c 3 -i 0.3 -w 2 172.31.1.100 | FORMAT_PING], [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +dnl Next check the overlay +NS_CHECK_EXEC([at_ns0], [ping -q -c 3 -i 0.3 -w 2 10.1.2.2 | FORMAT_PING], [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + +dnl Now check HTTP transfer +AT_CHECK([yes | dd bs=1k count=32 of=32k.txt], [0], [], stderr) +AT_CHECK([cat stderr | FORMAT_DD], [0], [dnl +32+0 records in +32+0 records out +32768 bytes (33 kB) copied, ... +]) +NETNS_DAEMONIZE([at_ns1], [[$PYTHON $srcdir/test-l7.py]], [http0.pid]) +NS_CHECK_EXEC([at_ns1], [netstat -s | grep retransmited > retransmit_before]) +NS_CHECK_EXEC([at_ns0], [wget 10.1.2.2/32k.txt -t 3 -T 1 --retry-connrefused -v -o wget0.log]) +NS_CHECK_EXEC([at_ns1], [netstat -s | grep retransmited > retransmit_after]) + +dnl Use the absence of retransmitted segments as a proxy for functioning GSO +AT_CHECK([diff -u retransmit_before retransmit_after], [0]) + +OVS_TRAFFIC_VSWITCHD_STOP +AT_CLEANUP + AT_SETUP([datapath - ping over geneve tunnel]) OVS_CHECK_GENEVE() -- 2.1.4