Hi Tom, Can you send an example for an rte_flow rule that you are trying ? I guess since you are using RSS it might affect more the performance what flows are being used.
Kindest regards, Raslan Darawsheh > -----Original Message----- > From: users <users-boun...@dpdk.org> On Behalf Of Tom Barbette > Sent: Tuesday, April 21, 2020 12:00 PM > To: Yan Lei <l....@epfl.ch>; Wisam Monther <wis...@mellanox.com>; > Thomas Monjalon <tho...@monjalon.net> > Cc: users@dpdk.org > Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less than > 4K per sec > > Interesting! No I did not try the flow_perf, it was from our own > application. > > I'm actually taking that number from the installation time of a single > rule, that have RSS action which is probably more costly. So this and > that may bring down the performance. > > Tom > > Le 20/04/2020 à 15:48, Yan Lei a écrit : > > > > Hi Tom, > > > > I guess "SW steering" refers to the "direct verbs/rules" > > > (https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail > s.dpdk.org%2Farchives%2Fdev%2F2019- > February%2F125303.html&data=02%7C01%7Crasland%40mellanox.com > %7C5a9698dca303410e1cf208d7e5d251d7%7Ca652971c7d2e4d9ba6a4d14925 > 6f461b%7C0%7C0%7C637230563788809406&sdata=AFzmKniGXDV5yCnd > 3qQIRQNUQ0YCxS0jXv2b2n6QW0Q%3D&reserved=0). group=0 > > is still the same old (pre dpdk 19.05) slow implementation of flow > > insertion. But just my guess. > > > > How did you measure the flow insertion rate? Did you use the patch they > > mentioned earlier in the thread? With that patch I got 330K with > > sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > > --group=1 --ether --ipv4 --udp --queue --flows-count=1000000. > > > > Cheers, > > Lei > > > > ------------------------------------------------------------------------ > > *From:* Tom Barbette <barbe...@kth.se> > > *Sent:* Monday, April 20, 2020 2:24 PM > > *To:* Wisam Monther; Thomas Monjalon; Yan Lei > > *Cc:* users@dpdk.org > > *Subject:* Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > > than 4K per sec > > Hi, > > > > Le 19/04/2020 à 16:07, Wisam Monther a écrit : > >> Hey Yan, > >> > >> For FW steering there is HW limitation on number of flows that can be > added on it. > >> Which is table 0, and I think it's 65536 > >> > >> But to get Millions of rules use --group=1 which is SW steering. > > What do you mean by SW steering? > > > > Using group 1 we had better performance but only around 10K. I think the > > whole process lacks an update feature instead of delete+create, and the > > ability to batch rules. > > > >> Moreover make sure you have enough memory in the app to have good > insertion rate. > >> > >> If you have enough 1G huge pages then it's ok. > >> If you are working with 2M pages your command should be like this: > >> > >> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 --socket-mem=4096 > -- --ingress --group=1 --ether --ipv4 --udp --queue --flows-count=1000000 > >> > >> BRs, > >> Wisam Jaddo > >> > >>> -----Original Message----- > >>> From: Thomas Monjalon <tho...@monjalon.net> > >>> Sent: Sunday, April 19, 2020 4:58 PM > >>> To: Yan Lei <l....@epfl.ch> > >>> Cc: users@dpdk.org; Wisam Monther <wis...@mellanox.com> > >>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > than 4K > >>> per sec > >>> > >>> +Cc Wisam > >>> > >>> 16/04/2020 17:32, Yan Lei: > >>>> Hi Thomas, > >>>> > >>>> > >>>> I tried the patch (68057 + 68058) on DPDK 19.11/20.02 + ofed 4.7.3. > >>>> > >>>> > >>>> TL;DR > >>>> > >>>> > >>>> 1. I was only able to generate 3K rules per second. > >>>> > >>>> 2. The maximum number of distinct rules the NIC can support seems to > be > >>> 65536. > >>>> > >>>> > >>>> How can I increase the insertion rate? Any firmware/driver config I > need to > >>> tune? Also, is 65536 distinct flows truly a limit of the NIC? The patch > defaults to > >>> generate 4 million distinct flows though... > >>>> > >>>> > >>>> Thanks in advance! > >>>> > >>>> > >>>> > >>>> Initially, running > >>>> > >>>> > >>>> ``` > >>>> > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=1000000 > >>>> > >>>> ``` > >>>> > >>>> > >>>> failed after a few seconds and it gave > >>>> > >>>> > >>>> ``` > >>>> Flow can't be created 1 message: hardware refuses to create flow > >>>> EAL: Error - exiting with code: 1 > >>>> Cause: error in creating flow > >>>> ``` > >>>> > >>>> > >>>> Then I added a small debug patch (attached) and it showed that the > error > >>> happens when creating the 65536th flow rule. > > > > The first table is indeed limited to something around that number. But > > performance are already degrading before that point. Even with OFED 5 > > and the firmware that comes with it. > > > >>>> > >>>> > >>>> ``` > >>>> Flow can't be created 1 message: hardware refuses to create flow > >>>> EAL: Error - exiting with code: 1 > >>>> Cause: error in creating flow,flows generated: 65536 ``` > >>>> > >>>> > >>>> My guess is that the NIC can only accept 65536 concurrent rules. Once I > >>> changed the outer ip mask to 0xffff, the above command runs fine. > >>>> > >>>> > >>>> To see how many rules I can generate per second. I ran (with the outer > >>>> ip mask 0xffff) > >>>> > >>>> > >>>> ``` > >>>> > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=65536 > >>>> > >>>> ``` > >>>> > >>>> > >>>> and it gives > >>>> > >>>> > >>>> ``` > >>>> > >>>> :: Total flow insertion rate -> 3.015922 K/Sec > >>>> :: The time for creating 65536 in flows 21.730005 seconds > >>>> :: EAGAIN counter = 0 > >>>> ``` > >>>> So 3 rules per sec. Which is close to what I observed before. > >>>> > >>>> ``` > >>>> sudo ./flow_perf -l 3-7 -n 4 -w 02:00.0,dv_flow_en=1 -- --ingress > >>>> --ether --ipv4 --udp --queue --flows-count=100000 ``` gives > >>>> > >>>> ``` > >>>> :: Total flow insertion rate -> 0.949381 K/Sec > >>>> :: The time for creating 100000 in flows 105.331842 seconds > >>>> :: EAGAIN counter = 0 > >>>> ``` > >>>> Have no idea why it's only 1k/sec in this case... > >>>> > >>>> Thanks and cheers, > >>>> Lei > >>>> > >>>> > >>>> ________________________________ > >>>> From: users <users-boun...@dpdk.org> on behalf of Yan Lei > >>>> <l....@epfl.ch> > >>>> Sent: Tuesday, April 14, 2020 1:20 PM > >>>> To: Thomas Monjalon > >>>> Cc: users@dpdk.org > >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > >>>> than 4K per sec > >>>> > >>>> Hi Thomas, > >>>> > >>>> Thanks! I will give it a try (using DPDK 19.11 + ofed 4.7.3). > >>>> > >>>> Cheers, > >>>> Lei > >>>> ________________________________ > >>>> From: Thomas Monjalon <tho...@monjalon.net> > >>>> Sent: Tuesday, April 14, 2020 12:12:28 PM > >>>> To: Yan Lei > >>>> Cc: users@dpdk.org > >>>> Subject: Re: [dpdk-users] [mlx5 + DPDK 19.11] Flow insertion rate less > >>>> than 4K per sec > >>>> > >>>> Hi, > >>>> > >>>> 10/04/2020 20:11, Yan Lei: > >>>>> I am doing some study that requires inserting more than 1 million > >>>>> flow rules per second to the NIC. And I runs DPDK 19.11 on a > ConnectX-5 > >>> NIC. > >>>>> > >>>>> But I only managed to create around 3.3K rules per second. > >>>>> Below is the code I used to measure the insertion rate: > >>>> > >>>> Please could you review this new application designed for such > measure? > >>>> > >>>> > https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc > >>>> > >>> > hes.dpdk.org%2Fpatch%2F68058%2F&data=02%7C01%7Cwisamm%40m > >>> ellanox.c > >>>> > >>> > om%7Cc046523203de456919da08d7e469add0%7Ca652971c7d2e4d9ba6a4d14 > 9 > >>> 256f46 > >>>> > >>> > 1b%7C0%7C0%7C637229014854391590&sdata=dxxEmm4DWoMPeNGy > M > >>> FaYgk%2BjSE > >>>> %2FwVKLnYAwQ7QhjKGc%3D&reserved=0 > >>>> > >>>> Any feedback about the above patch is welcome. Feel free to try and > review > >>> it. > >>> > >>> > >>> > >> > > > > > > Tom
