On Mon, Jun 01, 2020 at 07:27:09PM -0400, Shahaji Bhosle via dev wrote: > Hi Ben/Ilya, > Hope you guys are doing well and staying safe. I have been chasing a weird > problem with small drops and I think that is causing lots of TCP > retransmission. > > Setup details > iPerf3(1k-5K > Servers)<--DPDK2:OvS+DPDK(VxLAN:BOND)[DPDK0+DPDK1)<====2x25G<==== > [DPDK0+DPDK1)(VxLAN:BOND)OVS+DPDKDPDK2<---iPerf3(Clients) > > All the Drops are ring drops on BONDed functions on the server side. I > have 4 CPUs each with 3PMD threads, DPDK0, DPDK1 and DPDK2 all running with > 4 Rx rings each. > > What is interesting is when I give each Rx rings its own CPU the drops go > away. Or if I set cother_config:emc-insert-inv-prob=1 the drops go away. > But I need to scale up the number of flows so trying to run this with EMC > disabled. > > I can tell that the rings are not getting serviced for 30-40usec because of > some kind context switch or interrupts on these cores. I have tried to do > the usual isolation, nohz_full rcu_nocbs etc. Move all the interrupts away > from these cores etc. But nothing helps. I mean it improves, but the drops > still happen.
When you disable the EMC (or reduce its efficiency) the per packet cost increases, then it becomes more sensitive to variations. If you share a CPU with multiple queues, you decrease the amount of time available to process the queue. In either case, there will be less room to tolerate variations. Well, you might want to use 'perf' and monitor for the scheduling events and then based on the stack trace see what is causing it and try to prevent it. For example: # perf record -e sched:sched_switch -a -g sleep 1 For instance, you might see that another NIC used for management has IRQs assigned to one isolated CPU. You can move it to another CPU to reduce the noise, etc... Another suggestion is look at PMD thread idle statistics because it will tell you how much "extra" room you have left. As it approaches to 0, more fine tuned your setup needs to be to avoid drops. HTH, -- fbl _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
