On Wed, Mar 29, 2017 at 03:43:10PM +0300, Roi Dayan wrote:
> This patch series introduces rule offload functionality to dpif-netlink
> via netdev ports new flow offloading API. The user can specify whether to
> enable rule offloading or not via OVS configuration. Netdev providers
> are able to implement netdev flow offload API in order to offload rules.
>
> This patch series also implements one offload scheme for netdev-linux,
> using TC flower classifier, which was chosen because its sort of natural
> to state OVS DP rules for this classifier. However, the code can be
> extended to support other classifiers such as U32, eBPF, etc which support
> offload as well.
>
> The use-case we are currently addressing is the newly sriov switchdev mode
> in the Linux kernel which was introduced in version 4.8 [1][2].
> This series was tested against sriov vfs vports representors of the
> Mellanox 100G ConnectX-4 series exposed by the mlx5 kernel driver.
>
>
> V5->V6:
> - Rebase over master branch, fix compilation issue
> - Add Nicira copyright to tc interface
Hi,
I built rpm packages based on OVS upstream commit 36664f336664. Then I
built a new set of packages with this patchset, v6, applied.
When offloading is enabled, the rules are merged in an unexpected way.
See the results below from the two scenarios.
openvswitch-2.7.90-1.fc24.36664f336664.x86_64.rpm
^^^^^^^^^^^^ upstream ovs commit
[root@wsfd-netdev36 ~]# ovs-ofctl dump-flows ovs_pvp_br0
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=12.501s, table=0, n_packets=1022759, n_bytes=61365540,
idle_age=0, ip,in_port=1,nw_src=16.0.0.1,nw_dst=1.0.0.1 actions=output:2
cookie=0x0, duration=12.501s, table=0, n_packets=1020990, n_bytes=61259400,
idle_age=0, ip,in_port=1,nw_src=16.0.0.2,nw_dst=1.0.0.2 actions=output:2
cookie=0x0, duration=12.500s, table=0, n_packets=1023860, n_bytes=61431600,
idle_age=0, ip,in_port=1,nw_src=16.0.0.3,nw_dst=1.0.0.3 actions=output:2
cookie=0x0, duration=12.500s, table=0, n_packets=1022554, n_bytes=61353240,
idle_age=0, ip,in_port=1,nw_src=16.0.0.4,nw_dst=1.0.0.4 actions=output:2
cookie=0x0, duration=12.500s, table=0, n_packets=1021516, n_bytes=61290960,
idle_age=0, ip,in_port=1,nw_src=16.0.0.5,nw_dst=1.0.0.5 actions=output:2
cookie=0x0, duration=12.500s, table=0, n_packets=1020599, n_bytes=61235940,
idle_age=0, ip,in_port=1,nw_src=16.0.0.6,nw_dst=1.0.0.6 actions=output:2
cookie=0x0, duration=12.500s, table=0, n_packets=1022285, n_bytes=61337100,
idle_age=0, ip,in_port=1,nw_src=16.0.0.7,nw_dst=1.0.0.7 actions=output:2
cookie=0x0, duration=12.500s, table=0, n_packets=1021725, n_bytes=61303500,
idle_age=0, ip,in_port=1,nw_src=16.0.0.8,nw_dst=1.0.0.8 actions=output:2
cookie=0x0, duration=12.500s, table=0, n_packets=1022281, n_bytes=61336860,
idle_age=0, ip,in_port=1,nw_src=16.0.0.9,nw_dst=1.0.0.9 actions=output:2
cookie=0x0, duration=12.500s, table=0, n_packets=1022449, n_bytes=61346940,
idle_age=0, ip,in_port=1,nw_src=16.0.0.10,nw_dst=1.0.0.10 actions=output:2
cookie=0x0, duration=12.208s, table=0, n_packets=2580, n_bytes=154800,
idle_age=0, ip,in_port=2,nw_src=16.0.0.1,nw_dst=1.0.0.1 actions=output:1
cookie=0x0, duration=12.208s, table=0, n_packets=2675, n_bytes=160500,
idle_age=0, ip,in_port=2,nw_src=16.0.0.2,nw_dst=1.0.0.2 actions=output:1
cookie=0x0, duration=12.208s, table=0, n_packets=2520, n_bytes=151200,
idle_age=0, ip,in_port=2,nw_src=16.0.0.3,nw_dst=1.0.0.3 actions=output:1
cookie=0x0, duration=12.208s, table=0, n_packets=2587, n_bytes=155220,
idle_age=0, ip,in_port=2,nw_src=16.0.0.4,nw_dst=1.0.0.4 actions=output:1
cookie=0x0, duration=12.208s, table=0, n_packets=2640, n_bytes=158400,
idle_age=0, ip,in_port=2,nw_src=16.0.0.5,nw_dst=1.0.0.5 actions=output:1
cookie=0x0, duration=12.208s, table=0, n_packets=2593, n_bytes=155580,
idle_age=0, ip,in_port=2,nw_src=16.0.0.6,nw_dst=1.0.0.6 actions=output:1
cookie=0x0, duration=12.208s, table=0, n_packets=2634, n_bytes=158040,
idle_age=0, ip,in_port=2,nw_src=16.0.0.7,nw_dst=1.0.0.7 actions=output:1
cookie=0x0, duration=12.208s, table=0, n_packets=2549, n_bytes=152940,
idle_age=0, ip,in_port=2,nw_src=16.0.0.8,nw_dst=1.0.0.8 actions=output:1
cookie=0x0, duration=12.207s, table=0, n_packets=2627, n_bytes=157620,
idle_age=0, ip,in_port=2,nw_src=16.0.0.9,nw_dst=1.0.0.9 actions=output:1
cookie=0x0, duration=12.207s, table=0, n_packets=2530, n_bytes=151800,
idle_age=0, ip,in_port=2,nw_src=16.0.0.10,nw_dst=1.0.0.10 actions=output:1
[root@wsfd-netdev36 ~]# ovs-dpctl dump-flows
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.7,dst=1.0.0.7,frag=no),
packets:1162916, bytes:69774960, used:0.000s, actions:1
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.10,dst=1.0.0.10,frag=no),
packets:1163402, bytes:69804120, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.6,dst=1.0.0.6,frag=no),
packets:2969, bytes:178140, used:0.011s, actions:2
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.4,dst=1.0.0.4,frag=no),
packets:2940, bytes:176400, used:0.011s, actions:2
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.2,dst=1.0.0.2,frag=no),
packets:3028, bytes:181680, used:0.011s, actions:2
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.9,dst=1.0.0.9,frag=no),
packets:3025, bytes:181500, used:0.011s, actions:2
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.5,dst=1.0.0.5,frag=no),
packets:3008, bytes:180480, used:0.011s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.3,dst=1.0.0.3,frag=no),
packets:1164847, bytes:69890820, used:0.000s, actions:1
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.4,dst=1.0.0.4,frag=no),
packets:1162983, bytes:69778980, used:0.000s, actions:1
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.6,dst=1.0.0.6,frag=no),
packets:1161018, bytes:69661080, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.8,dst=1.0.0.8,frag=no),
packets:2876, bytes:172560, used:0.011s, actions:2
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.1,dst=1.0.0.1,frag=no),
packets:2968, bytes:178080, used:0.011s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.8,dst=1.0.0.8,frag=no),
packets:1161890, bytes:69713400, used:0.000s, actions:1
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.9,dst=1.0.0.9,frag=no),
packets:1163228, bytes:69793680, used:0.000s, actions:1
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.2,dst=1.0.0.2,frag=no),
packets:1161539, bytes:69692340, used:0.000s, actions:1
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.5,dst=1.0.0.5,frag=no),
packets:1161402, bytes:69684120, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.10,dst=1.0.0.10,frag=no),
packets:2880, bytes:172800, used:0.011s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.1,dst=1.0.0.1,frag=no),
packets:1163599, bytes:69815940, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.7,dst=1.0.0.7,frag=no),
packets:2984, bytes:179040, used:0.011s, actions:2
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.3,dst=1.0.0.3,frag=no),
packets:2889, bytes:173340, used:0.011s, actions:2
Now with
openvswitch-2.7.90-1.fc24.36664f336664.v6.x86_64.rpm
^^^^^^^^^^^^^^^ this upstream commit + V6 patchset
[root@wsfd-netdev36 ~]# ovs-ofctl dump-flows ovs_pvp_br0
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=6.876s, table=0, n_packets=171, n_bytes=10260,
idle_age=5, ip,in_port=1,nw_src=16.0.0.1,nw_dst=1.0.0.1 actions=output:2
cookie=0x0, duration=6.877s, table=0, n_packets=139976996, n_bytes=8398290046,
idle_age=0, ip,in_port=1,nw_src=16.0.0.2,nw_dst=1.0.0.2 actions=output:2
cookie=0x0, duration=6.877s, table=0, n_packets=171, n_bytes=10260,
idle_age=5, ip,in_port=1,nw_src=16.0.0.3,nw_dst=1.0.0.3 actions=output:2
cookie=0x0, duration=6.877s, table=0, n_packets=170, n_bytes=10200,
idle_age=5, ip,in_port=1,nw_src=16.0.0.4,nw_dst=1.0.0.4 actions=output:2
cookie=0x0, duration=6.877s, table=0, n_packets=170, n_bytes=10200,
idle_age=5, ip,in_port=1,nw_src=16.0.0.5,nw_dst=1.0.0.5 actions=output:2
cookie=0x0, duration=6.876s, table=0, n_packets=170, n_bytes=10200,
idle_age=5, ip,in_port=1,nw_src=16.0.0.6,nw_dst=1.0.0.6 actions=output:2
cookie=0x0, duration=6.876s, table=0, n_packets=170, n_bytes=10200,
idle_age=5, ip,in_port=1,nw_src=16.0.0.7,nw_dst=1.0.0.7 actions=output:2
cookie=0x0, duration=6.876s, table=0, n_packets=170, n_bytes=10200,
idle_age=5, ip,in_port=1,nw_src=16.0.0.8,nw_dst=1.0.0.8 actions=output:2
cookie=0x0, duration=6.876s, table=0, n_packets=170, n_bytes=10200,
idle_age=5, ip,in_port=1,nw_src=16.0.0.9,nw_dst=1.0.0.9 actions=output:2
cookie=0x0, duration=6.876s, table=0, n_packets=170, n_bytes=10200,
idle_age=5, ip,in_port=1,nw_src=16.0.0.10,nw_dst=1.0.0.10 actions=output:2
cookie=0x0, duration=6.585s, table=0, n_packets=82, n_bytes=4920, idle_age=5,
ip,in_port=2,nw_src=16.0.0.1,nw_dst=1.0.0.1 actions=output:1
cookie=0x0, duration=6.585s, table=0, n_packets=81, n_bytes=4860, idle_age=5,
ip,in_port=2,nw_src=16.0.0.2,nw_dst=1.0.0.2 actions=output:1
cookie=0x0, duration=6.584s, table=0, n_packets=77, n_bytes=4620, idle_age=5,
ip,in_port=2,nw_src=16.0.0.3,nw_dst=1.0.0.3 actions=output:1
cookie=0x0, duration=6.584s, table=0, n_packets=78, n_bytes=4680, idle_age=5,
ip,in_port=2,nw_src=16.0.0.4,nw_dst=1.0.0.4 actions=output:1
cookie=0x0, duration=6.584s, table=0, n_packets=72, n_bytes=4320, idle_age=5,
ip,in_port=2,nw_src=16.0.0.5,nw_dst=1.0.0.5 actions=output:1
cookie=0x0, duration=6.584s, table=0, n_packets=67, n_bytes=4020, idle_age=5,
ip,in_port=2,nw_src=16.0.0.6,nw_dst=1.0.0.6 actions=output:1
cookie=0x0, duration=6.584s, table=0, n_packets=75, n_bytes=4500, idle_age=5,
ip,in_port=2,nw_src=16.0.0.7,nw_dst=1.0.0.7 actions=output:1
cookie=0x0, duration=6.584s, table=0, n_packets=63, n_bytes=3780, idle_age=5,
ip,in_port=2,nw_src=16.0.0.8,nw_dst=1.0.0.8 actions=output:1
cookie=0x0, duration=6.584s, table=0, n_packets=61708007, n_bytes=3702480420,
idle_age=0, ip,in_port=2,nw_src=16.0.0.9,nw_dst=1.0.0.9 actions=output:1
cookie=0x0, duration=6.583s, table=0, n_packets=65, n_bytes=3900, idle_age=5,
ip,in_port=2,nw_src=16.0.0.10,nw_dst=1.0.0.10 actions=output:1
[root@wsfd-netdev36 ~]# ovs-dpctl dump-flows
in_port(1),eth(dst=00:00:00:00:00:01),eth_type(0x0800), packets:86436432,
bytes:5186185920, used:0.430s, actions:2
in_port(2),eth(dst=00:00:00:00:00:01),eth_type(0x0800), packets:196008690,
bytes:11760191686, used:0.430s, actions:1
^--- issue above
listing tc filter rules gives exactly the rules above too, on both
interfaces.
Then:
# ovs-vsctl set Open_vSwitch . other_config:hw-offload=false
# systemctl restart openvswitch.service
[root@wsfd-netdev36 ~]# ovs-ofctl dump-flows ovs_pvp_br0
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=5.710s, table=0, n_packets=399560, n_bytes=23973600,
idle_age=0, ip,in_port=1,nw_src=16.0.0.1,nw_dst=1.0.0.1 actions=output:2
cookie=0x0, duration=5.710s, table=0, n_packets=399256, n_bytes=23955360,
idle_age=0, ip,in_port=1,nw_src=16.0.0.2,nw_dst=1.0.0.2 actions=output:2
cookie=0x0, duration=5.710s, table=0, n_packets=400018, n_bytes=24001080,
idle_age=0, ip,in_port=1,nw_src=16.0.0.3,nw_dst=1.0.0.3 actions=output:2
cookie=0x0, duration=5.709s, table=0, n_packets=399667, n_bytes=23980020,
idle_age=0, ip,in_port=1,nw_src=16.0.0.4,nw_dst=1.0.0.4 actions=output:2
cookie=0x0, duration=5.709s, table=0, n_packets=400011, n_bytes=24000660,
idle_age=0, ip,in_port=1,nw_src=16.0.0.5,nw_dst=1.0.0.5 actions=output:2
cookie=0x0, duration=5.709s, table=0, n_packets=399651, n_bytes=23979060,
idle_age=0, ip,in_port=1,nw_src=16.0.0.6,nw_dst=1.0.0.6 actions=output:2
cookie=0x0, duration=5.709s, table=0, n_packets=400000, n_bytes=24000000,
idle_age=0, ip,in_port=1,nw_src=16.0.0.7,nw_dst=1.0.0.7 actions=output:2
cookie=0x0, duration=5.709s, table=0, n_packets=399198, n_bytes=23951880,
idle_age=0, ip,in_port=1,nw_src=16.0.0.8,nw_dst=1.0.0.8 actions=output:2
cookie=0x0, duration=5.709s, table=0, n_packets=400489, n_bytes=24029340,
idle_age=0, ip,in_port=1,nw_src=16.0.0.9,nw_dst=1.0.0.9 actions=output:2
cookie=0x0, duration=5.709s, table=0, n_packets=399782, n_bytes=23986920,
idle_age=0, ip,in_port=1,nw_src=16.0.0.10,nw_dst=1.0.0.10 actions=output:2
cookie=0x0, duration=5.419s, table=0, n_packets=1081, n_bytes=64860,
idle_age=0, ip,in_port=2,nw_src=16.0.0.1,nw_dst=1.0.0.1 actions=output:1
cookie=0x0, duration=5.419s, table=0, n_packets=1037, n_bytes=62220,
idle_age=0, ip,in_port=2,nw_src=16.0.0.2,nw_dst=1.0.0.2 actions=output:1
cookie=0x0, duration=5.419s, table=0, n_packets=1048, n_bytes=62880,
idle_age=0, ip,in_port=2,nw_src=16.0.0.3,nw_dst=1.0.0.3 actions=output:1
cookie=0x0, duration=5.419s, table=0, n_packets=1072, n_bytes=64320,
idle_age=0, ip,in_port=2,nw_src=16.0.0.4,nw_dst=1.0.0.4 actions=output:1
cookie=0x0, duration=5.418s, table=0, n_packets=1096, n_bytes=65760,
idle_age=0, ip,in_port=2,nw_src=16.0.0.5,nw_dst=1.0.0.5 actions=output:1
cookie=0x0, duration=5.418s, table=0, n_packets=1028, n_bytes=61680,
idle_age=0, ip,in_port=2,nw_src=16.0.0.6,nw_dst=1.0.0.6 actions=output:1
cookie=0x0, duration=5.418s, table=0, n_packets=1042, n_bytes=62520,
idle_age=0, ip,in_port=2,nw_src=16.0.0.7,nw_dst=1.0.0.7 actions=output:1
cookie=0x0, duration=5.418s, table=0, n_packets=1017, n_bytes=61020,
idle_age=0, ip,in_port=2,nw_src=16.0.0.8,nw_dst=1.0.0.8 actions=output:1
cookie=0x0, duration=5.418s, table=0, n_packets=1080, n_bytes=64800,
idle_age=0, ip,in_port=2,nw_src=16.0.0.9,nw_dst=1.0.0.9 actions=output:1
cookie=0x0, duration=5.418s, table=0, n_packets=1055, n_bytes=63300,
idle_age=0, ip,in_port=2,nw_src=16.0.0.10,nw_dst=1.0.0.10 actions=output:1
[root@wsfd-netdev36 ~]# ovs-dpctl dump-flows
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.2,dst=1.0.0.2,frag=no),
packets:608565, bytes:36513900, used:0.000s, actions:1
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.3,dst=1.0.0.3,frag=no),
packets:610002, bytes:36600120, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.5,dst=1.0.0.5,frag=no),
packets:1595, bytes:95700, used:0.013s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.5,dst=1.0.0.5,frag=no),
packets:609962, bytes:36597720, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.10,dst=1.0.0.10,frag=no),
packets:1573, bytes:94380, used:0.013s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.7,dst=1.0.0.7,frag=no),
packets:610297, bytes:36617820, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.9,dst=1.0.0.9,frag=no),
packets:1656, bytes:99360, used:0.013s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.6,dst=1.0.0.6,frag=no),
packets:608943, bytes:36536580, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.3,dst=1.0.0.3,frag=no),
packets:1595, bytes:95700, used:0.038s, actions:2
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.1,dst=1.0.0.1,frag=no),
packets:1627, bytes:97620, used:0.013s, actions:2
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.2,dst=1.0.0.2,frag=no),
packets:1588, bytes:95280, used:0.013s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.8,dst=1.0.0.8,frag=no),
packets:608778, bytes:36526680, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.4,dst=1.0.0.4,frag=no),
packets:1550, bytes:93000, used:0.013s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.4,dst=1.0.0.4,frag=no),
packets:609149, bytes:36548940, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.7,dst=1.0.0.7,frag=no),
packets:1574, bytes:94440, used:0.038s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.9,dst=1.0.0.9,frag=no),
packets:610471, bytes:36628260, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.8,dst=1.0.0.8,frag=no),
packets:1566, bytes:93960, used:0.013s, actions:2
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.1,dst=1.0.0.1,frag=no),
packets:609238, bytes:36554280, used:0.000s, actions:1
recirc_id(0),in_port(2),eth_type(0x0800),ipv4(src=16.0.0.10,dst=1.0.0.10,frag=no),
packets:609208, bytes:36552480, used:0.000s, actions:1
recirc_id(0),in_port(1),eth_type(0x0800),ipv4(src=16.0.0.6,dst=1.0.0.6,frag=no),
packets:1573, bytes:94380, used:0.013s, actions:2
^--- rules are not merged anymore.
devlink inline-mode = transport for all tests above.
I can provide a packet capture if wanted. Replaying it should be enough
to trigger it, though these packets have nothing really especial. They
have all the same src/dst MAC addresses and increasing IP addresses, as
seen above.
I tried checking ovs logs, but couldn't spot anything interesting.
This is also observed from a performance point of view. It doesn't
matter if we add 10 or 10k flows with L3 matching, performance is always
the same, as it's matching using only 2 L2 rules.
Please let me know if you need anything else.
Thanks,
Marcelo
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
