Hi Klement, thanks for your elaboration and I would try to see this issue with
some counters.
Best wishes
Pan
发件人: Klement Sekera
发送时间: 2022年3月17日 17:13
收件人: Pan Yueyang
抄送: [email protected]
主题: Re: [vpp-dev] Meanings of different vector rates and questions about NAT44
handoff queue size
A lookup is performed based on 6 tuple (fib, saddr, daddr, sport, dport, proto)
- value contains thread index + pool index. If on the wrong thread, then a
handoff is performed. See nat44_ed_classify.c.
Mentioned extreme case with buffer shortage can occur if total number of
buffers available to VPP is close to or less than the number of possible
buffers stuck in frame queues. If you dedicated lots of memory to VPP, then
this won’t occur.
I don’t have an answer for increasing congestion drop, but I would read through
the infra code which handles frame queues. A while ago (>1 year) the algorithm
wasn’t optimal - as in, the frames would enter the queue as they were, causing
stalls and congestion drop if the consuming thread was slower than the
producing thread. Producing thread would pick a couple of packets from NIC at a
time (say 1 or 2) and create lots of small frames for consuming thread. But
because consuming thread would only process 1 or 2 packets at a time, the
performance would be bad because vectorisation savings would not kick in. My
fix at that time (https://gerrit.fd.io/r/c/vpp/+/28980) was to add coalescing
to infra so that small frames would be merged into bigger ones. This fixed my
test scenario where one worker would hand off 100% traffic to another worker.
This patch was never merged as Damjan did a rework of that part of code and if
I remember correctly, it was no longer needed as the new algorithm didn’t
suffer from that issue. Maybe it’s worth it to take a look at what happens (you
would need to add some debug counters or maybe prints (careful with those - IO
messes up performance and can change system dynamic, my solution was to print
stats once per second or so) of some sorts probably, I don’t think there is a
tool for that).
Regards,
Klement
On 17 Mar 2022, at 09:51, Pan Yueyang <[email protected]> wrote:
Hi Klement, thanks for your prompt reply. I am not clear about how VPP makes
this split choice. In which file does it make such a split and what are the
factors that influence this split decision?
Also, I still can see a case in which you will have more congestion drop with a
larger queue size. I did not see the VPP complaining about the buffer
allocation so I think the sizes are valid (I test 32 to 2048). I tested with
the same settings except for NAT_FQ_NELTS and with the same traffic. I think
the latency of processing a packet should be similar in my case. Thus the
processing speed of the part after the handoff should be similar (similar mean
and variance). Then with a larger queue, I expect the congestion drop should be
no more than with a small queue because it could hold more variance. If the
incoming speed is basically larger than the processing speed itself, then I ]
expect with the larger queue, the congestion drop will not increase heavily at
least (with the assumption that the handoff rate is almost constant). Thus I
still don’t understand why the congestion drop would increase after some queue
size even if the packets pile up in the handoff queue. Could you please
elaborate more on this?
Best Wishes
Pan
发件人: Klement Sekera <[email protected]>
发送时间: 2022年3月17日 14:59:58
收件人: Pan Yueyang
抄送: [email protected]
主题: Re: [vpp-dev] Meanings of different vector rates and questions about NAT44
handoff queue size
Hey,
I can provide insight into the NAT bit.
On 17 Mar 2022, at 06:06, Yueyang Pan via lists.fd.io
<[email protected]> wrote:
Also I noticed size of handoff queue is very much important to the performance
of NAT44 but I was wondering why the congestion drop in NAT handoff would
increase after the handoff queue size (NAT_FQ_NELTS) is larger than a certain
value (in my case 512). Does anyone also experience this case and have any
ideas?
Handoff queue size is in frames, so a size of 64 can hold up to 64*256 packets,
but there is rarely a full frame there, because these frames are the result of
an incoming frame being split in between workers. Say you have 2 workers and a
full frame of 256 packets hits worker #1, which sifts through the packets and
decides that 156 packets are for #1 and 100 are for #2. So in this case, 156
packets continue processing on worker #1 and there is a new frame created with
100 packets and put into a queue to be processed on worker #2.
If you are hitting congestion drop it means your system is very close to or
simply cannot handle such packet rates. Increasing frame queue size might help
with some slight invariances, but ultimately makes the situation worse as
packets begin to pile up within vpp waiting to be process. If you continued
increasing this size, eventually you’d hit a situation where NIC driver would
have problems allocating new vlib_buffers, because all buffers would be stuck
in queues waiting.
Regards,
Klement
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21057): https://lists.fd.io/g/vpp-dev/message/21057
Mute This Topic: https://lists.fd.io/mt/89839411/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
