On 9 Apr 2024, at 16:58, Shahaji Bhosle wrote:
> Thanks Eelco, Sorry for the late reply.
> We are not using NVDIA NIC, and are using OVS+DPDK.
> It appears the revalidator threads are complaining because it is not able
> to get all the stats for million flows within its time window.
> Just wondering if there are knobs to relax the revalidator to not complain.
> We have used the below settings
> other_config : {dpdk-extra="--log-level=pmd.*,debug",
> dpdk-init="true", dpdk-lcore-mask="0x4000000000000", dpdk-mem-channels="4",
> dpdk-socket-mem="4096,0", flow-limit="2000000", hw-offload="true",
> max-idle="6000000", max-revalidator="4000", n-handler-threads="4",
> n-offload-threads="3", n-revalidator-threads="4", per-port-memory="true",
> pmd-cpu-mask="0x155555554"}
It would be interesting to profile, and see where statistics gathering can be
further optimized. What I remember from the past, is that 1M none hw-offloaded
OVS-DPDK flows were not a problem.
The following blog explains the revalidator process;
https://developers.redhat.com/articles/2022/10/19/open-vswitch-revalidator-process-explained#error=login_required&state=e0f29d5d-e75c-4e6a-89de-cfd7cdccff7e
There is a tool (also explained in the blog) you can use to see how much time
the revalidator needs.
You can also see the reason for not having the desired number of flows, i.e. is
it because the dynamic flow_limit is adjusted lower, and/or ‘duration <
(other_config:max-revalidator / 2’ see ‘The revalidate_ukey function’.
For the later, you can try to increase the max-revalidator value to a second,
‘ovs-vsctl set Open_vSwitch . other_config:max-revalidator=1000’. I would
personally not go over 1seconds, as it affects the statists shown in dump-flows.
Good luck,
Eelco
> Thanks, Shahaji
>
>
> On Thu, Mar 28, 2024 at 1:54 PM Eelco Chaudron <echau...@redhat.com> wrote:
>
>>
>>
>> On 28 Mar 2024, at 16:31, Shahaji Bhosle via dev wrote:
>>
>>> Hi,
>>> I am looking for tuning params for scaling OVS-DPDK/TC to million flows.
>> It
>>> seems like flow dump(revalidator) is not able to collect the stats after
>>> about 150K flows per second, thus it is trying to delete the flows.
>>> Any guidance where million flows are offloaded to hardware would be
>> great.
>>> Thanks, Shahaji
>>
>> This is probably because it takes OVS a while to get all the stats from
>> the kernel/driver. What kind of NIC are you using? If it’s Nvidia, which
>> mode, dmfs or smfs. I’ve seen dmfs being rather slow causing a similar
>> effect as you describe. Try switching to smfs and see if the problem goes
>> away.
>>
>> I also assume you use a recent version of OVS, if not make sure your
>> version includes the following:
>>
>> e1e5eac5b016 ("tc: Add TCA_KIND flower to delete and get operation to
>> avoid rtnl_lock().")
>>
>>
>> Cheers,
>>
>>
>> Eelco
>>
>>
>
> --
> This electronic communication and the information and any files transmitted
> with it, or attached to it, are confidential and are intended solely for
> the use of the individual or entity to whom it is addressed and may contain
> information that is confidential, legally privileged, protected by privacy
> laws, or otherwise restricted from disclosure to anyone else. If you are
> not the intended recipient or the person responsible for delivering the
> e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
_______________________________________________
dev mailing list
d...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
