Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

Thanks for the clarification about the TCP changes b/w the 2 releases

I will use your patch, hopefully I will catch the issue about where the
drops are. I will try to debug further.
I will revert back if required.


Regards.

On Wed, Mar 16, 2022 at 11:12 AM Florin Coras 
wrote:

> Hi Vijay,
>
> On Mar 15, 2022, at 9:58 PM, Vijay Kumar  wrote:
>
> Hi florin,
>
> Thanks a lot for helping me out. I will try your patch and update you with
> the result.
>
>
> Thanks!
>
>
>
> A general observation
> ==
> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD
> counter is incremented.
>
>
> That’s what I’d expect as well.
>
>
> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph
> node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is
> called towards the end i.e. after calling tcp_send_synack() but in 21.06
> code, the tcp46_listen_trace_frame() is called at the very beginning
> of tcp46_listen_inline() graph node.
>
> I think moving the pkt trace macro to the end of the function is good
> (placing it close to the line that calls tcp_send_synack), otherwise it can
> mislead us and will not help debugging.
>
>
> Most of the changes were code refactoring/cleanup and buffer handling
> optimizations, not protocol related. TCP tracing, at least as it is now,
> doesn’t provide info about the errors hit, instead it reports the
> connections hit and the reporting of errors is done through node and
> session counters (see show session verbose 2). Obviously that didn’t work
> properly for the listen node.
>
> Regards,
> Florin
>
>
>
> On Wed, Mar 16, 2022 at 10:22 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> That’s probably because packets are hitting an actual error and it seems
>> the listen node is not reporting anything but syns received. Here’s a patch
>> that might help [1]. It might not cherry-pick cleanly on 21.06.
>>
>> Regards,
>> Florin
>>
>> [1] https://gerrit.fd.io/r/c/vpp/+/35654
>>
>> On Mar 15, 2022, at 7:54 PM, Vijay Kumar  wrote:
>>
>> Hi Florin,
>>
>> I have the output of show node counters (show errors) taken on both 20.05
>> and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or
>> tcp4-output etc.
>>
>> Please let me know why SYN rcvd counter itself is not incremented but in
>> the earlier reply I already pasted the show trace ouput where we saw the
>> SYN pkt was landed on tcp4-litsen node.
>>
>> VPP 20.05
>> ===
>> pp# show node counters
>>CountNode  Reason
>>  3   memif-input  not ip packet
>>  10259 an_ppe_wfectrl wfectrl packets received
>>  10259 an_ppe_wfectrl wfectrl replies sent
>>  1 an_ppe_wfectrl wfectrl packet
>> processing failed
>>   8123 an_ppe_wfectrl session stat request
>> received
>>  1 an_ppe_wfectrl service construct
>> config request received
>>  1 an_ppe_wfectrl service construct
>> config request success
>>  1 an_ppe_wfectrl service config request
>> received
>>  1 an_ppe_wfectrl service config request
>> success
>>   1686 an_ppe_wfectrl dpi stats request
>> received
>>   1686 an_ppe_wfectrl dpi stats request
>> success
>> 70 an_ppe_wfectrl nat stats request
>> received
>> 70 an_ppe_wfectrl nat stats request
>> success
>> 70 an_ppe_wfectrl vpp stats request
>> received
>> 70 an_ppe_wfectrl vpp stats request
>> success
>>  1 an_ppe_wfectrl udp tunnel resource
>> block add request received
>>  1 an_ppe_wfectrl udp tunnel resource
>> block add request success
>>  1 an_ppe_wfectrl l3rm resource block
>> update request received
>>  1 an_ppe_wfectrl l3rm resource block
>> update request success
>>  1 an_ppe_wfectrl ue registration request
>> received
>> 17 an_ppe_wfectrl ike msg received from
>> ikemgr
>> 17 an_ppe_wfectrl ike msg send to network
>> success
>>  3 an_ppe_wfectrl ipsec sa install msg
>> received from ikemgr
>>  3 an_ppe_wfectrl ipsec sa install msg
>> processed successfully
>>  1  an_ppe_vppctrl_input  Success in insert in
>> ueidentifier
>>  1  an_ppe_vppctrl_input  Success  in insert in
>> uecontext
>>  2  an_ppe_vppctrl_input  Downlink message sent
>> to UE successfully
>>  1  an_ppe_vppctrl_input  

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

The patch helped me to find the exact point of failure in
tcp46_listen_inline() graph node.
When I tested again, I get the below error and it maps to this error code "
TCP_ERROR_CREATE_SESSION_FAIL"
This counter is incremented when the call to function
session_stream_accept() returns failure.

Is there any potential reason why allocation fails at this place?


show errors output
===
4 arp-reply ARP replies
sent  error
14   ip4-udp-lookup  No
error  error
 5tcp4-listen Sessions couldn't
be allocated   error
 5  esp4-decrypt-tunESP pkts
received  error
 5  ipsec4-tun-input  good packets
receivederror
 1 ip4-input   ip4 ttl
<= 1error
 1   ip4-icmp-error  hop limit exceeded
response sent  error
  3346   ethernet-inputunknown
vlanerror


On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar  wrote:

> Hi Florin,
>
> Thanks for the clarification about the TCP changes b/w the 2 releases
>
> I will use your patch, hopefully I will catch the issue about where the
> drops are. I will try to debug further.
> I will revert back if required.
>
>
> Regards.
>
> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar  wrote:
>>
>> Hi florin,
>>
>> Thanks a lot for helping me out. I will try your patch and update you
>> with the result.
>>
>>
>> Thanks!
>>
>>
>>
>> A general observation
>> ==
>> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD
>> counter is incremented.
>>
>>
>> That’s what I’d expect as well.
>>
>>
>> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph
>> node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is
>> called towards the end i.e. after calling tcp_send_synack() but in 21.06
>> code, the tcp46_listen_trace_frame() is called at the very beginning
>> of tcp46_listen_inline() graph node.
>>
>> I think moving the pkt trace macro to the end of the function is good
>> (placing it close to the line that calls tcp_send_synack), otherwise it can
>> mislead us and will not help debugging.
>>
>>
>> Most of the changes were code refactoring/cleanup and buffer handling
>> optimizations, not protocol related. TCP tracing, at least as it is now,
>> doesn’t provide info about the errors hit, instead it reports the
>> connections hit and the reporting of errors is done through node and
>> session counters (see show session verbose 2). Obviously that didn’t work
>> properly for the listen node.
>>
>> Regards,
>> Florin
>>
>>
>>
>> On Wed, Mar 16, 2022 at 10:22 AM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> That’s probably because packets are hitting an actual error and it seems
>>> the listen node is not reporting anything but syns received. Here’s a patch
>>> that might help [1]. It might not cherry-pick cleanly on 21.06.
>>>
>>> Regards,
>>> Florin
>>>
>>> [1] https://gerrit.fd.io/r/c/vpp/+/35654
>>>
>>> On Mar 15, 2022, at 7:54 PM, Vijay Kumar  wrote:
>>>
>>> Hi Florin,
>>>
>>> I have the output of show node counters (show errors) taken on both
>>> 20.05 and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or
>>> tcp4-output etc.
>>>
>>> Please let me know why SYN rcvd counter itself is not incremented but in
>>> the earlier reply I already pasted the show trace ouput where we saw the
>>> SYN pkt was landed on tcp4-litsen node.
>>>
>>> VPP 20.05
>>> ===
>>> pp# show node counters
>>>CountNode  Reason
>>>  3   memif-input  not ip packet
>>>  10259 an_ppe_wfectrl wfectrl packets
>>> received
>>>  10259 an_ppe_wfectrl wfectrl replies sent
>>>  1 an_ppe_wfectrl wfectrl packet
>>> processing failed
>>>   8123 an_ppe_wfectrl session stat request
>>> received
>>>  1 an_ppe_wfectrl service construct
>>> config request received
>>>  1 an_ppe_wfectrl service construct
>>> config request success
>>>  1 an_ppe_wfectrl service config request
>>> received
>>>  1 an_ppe_wfectrl service config request
>>> success
>>>   1686 an_ppe_wfectrl dpi stats request
>>> received
>>>   1686 an_ppe_wfectrl dpi stats request
>>> success
>>> 70 an_ppe_wfectrl nat

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

That’s a sign that either fifo allocations failed (not enough memory in the 
fifo segment) or that the app refused the session (app_worker_accept_notify 
returns non zero). 

Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in the 
attach options passed to vnet_application_attach. Some vpp versions ago we 
switched to using the first fifo segment as connects segment and all listeners 
allocate their first segments based on size provided with this option. If not 
provided, listeners fail to allocate segments. 

Regards,
Florin

> On Mar 16, 2022, at 3:49 AM, Vijay Kumar  wrote:
> 
> Hi Florin,
> 
> The patch helped me to find the exact point of failure in 
> tcp46_listen_inline() graph node.
> When I tested again, I get the below error and it maps to this error code 
> "TCP_ERROR_CREATE_SESSION_FAIL"
> This counter is incremented when the call to function session_stream_accept() 
> returns failure.
> 
> Is there any potential reason why allocation fails at this place?
> 
> 
> show errors output
> ===
> 4 arp-reply ARP replies 
> sent  error  
> 14   ip4-udp-lookup  No error 
>  error  
>  5tcp4-listen Sessions couldn't 
> be allocated   error  
>  5  esp4-decrypt-tunESP pkts 
> received  error  
>  5  ipsec4-tun-input  good packets 
> receivederror  
>  1 ip4-input   ip4 ttl <= 
> 1error  
>  1   ip4-icmp-error  hop limit exceeded 
> response sent  error  
>   3346   ethernet-inputunknown 
> vlanerror  
> 
> 
> On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar  > wrote:
> Hi Florin,
> 
> Thanks for the clarification about the TCP changes b/w the 2 releases
> 
> I will use your patch, hopefully I will catch the issue about where the drops 
> are. I will try to debug further.
> I will revert back if required.
> 
> 
> Regards.
> 
> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras  > wrote:
> Hi Vijay, 
> 
>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar > > wrote:
>> 
>> Hi florin,
>> 
>> Thanks a lot for helping me out. I will try your patch and update you with 
>> the result.
> 
> Thanks! 
> 
>> 
>> 
>> A general observation
>> ==
>> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD 
>> counter is incremented.
> 
> That’s what I’d expect as well. 
> 
>> 
>> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph 
>> node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is 
>> called towards the end i.e. after calling tcp_send_synack() but in 21.06 
>> code, the tcp46_listen_trace_frame() is called at the very beginning of 
>> tcp46_listen_inline() graph node.
>> 
>> I think moving the pkt trace macro to the end of the function is good 
>> (placing it close to the line that calls tcp_send_synack), otherwise it can 
>> mislead us and will not help debugging.
>> 
> 
> Most of the changes were code refactoring/cleanup and buffer handling 
> optimizations, not protocol related. TCP tracing, at least as it is now, 
> doesn’t provide info about the errors hit, instead it reports the connections 
> hit and the reporting of errors is done through node and session counters 
> (see show session verbose 2). Obviously that didn’t work properly for the 
> listen node. 
> 
> Regards,
> Florin
> 
>> 
>> 
>> On Wed, Mar 16, 2022 at 10:22 AM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> That’s probably because packets are hitting an actual error and it seems the 
>> listen node is not reporting anything but syns received. Here’s a patch that 
>> might help [1]. It might not cherry-pick cleanly on 21.06. 
>> 
>> Regards,
>> Florin
>> 
>> [1] https://gerrit.fd.io/r/c/vpp/+/35654 
>> 
>> 
>>> On Mar 15, 2022, at 7:54 PM, Vijay Kumar >> > wrote:
>>> 
>>> Hi Florin,
>>> 
>>> I have the output of show node counters (show errors) taken on both 20.05 
>>> and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or 
>>> tcp4-output etc. 
>>> 
>>> Please let me know why SYN rcvd counter itself is not incremented but in 
>>> the earlier reply I already pasted the show trace ouput where we saw the 
>>> SYN pkt was landed on tcp4-litsen node.
>>> 
>>> VPP 20.05
>>> ===
>>> pp# show node counters 
>>>CountNode  Reason
>>>  3   memif-input  not ip packet
>>>  102

[vpp-dev] vlib_plugin_registration size mismatch

[Matthew Smith via lists.fd.io]

Hi,

I have been testing against a build from yesterday's master branch (commit
id b0f0f8c8dd9d694bfc13652f89b8b577e9c1c708) and have been seeing these
messages when VPP starts up:

vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
bufmon_plugin.so
vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
dispatch_trace_plugin.so
vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
oddbuf_plugin.so
vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
perfmon_plugin.so
vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
tracedump_plugin.so
vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
unittest_plugin.so

I noticed the problem because I was trying to do some testing on the
tracedump plugin and I could not get the tests to work. But many other
plugins (vrrp, nat, acl) work fine and do not have similar messages logged.

Does anyone have any ideas on what may cause this? Is anyone else seeing
this issue on a build from the current master branch?

Thanks,
-Matt

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21039): https://lists.fd.io/g/vpp-dev/message/21039
Mute This Topic: https://lists.fd.io/mt/89826377/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vlib_plugin_registration size mismatch

[Florin Coras]

Hi Matt, 

Just tried running after a make build and no such message in show log for me. 
Did you try a make wipe? 

Regards,
Florin

> On Mar 16, 2022, at 9:50 AM, Matthew Smith via lists.fd.io 
>  wrote:
> 
> Hi,
> 
> I have been testing against a build from yesterday's master branch (commit id 
> b0f0f8c8dd9d694bfc13652f89b8b577e9c1c708) and have been seeing these messages 
> when VPP starts up:
> 
> vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> bufmon_plugin.so
> vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> dispatch_trace_plugin.so
> vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> oddbuf_plugin.so
> vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> perfmon_plugin.so
> vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> tracedump_plugin.so
> vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> unittest_plugin.so
> 
> I noticed the problem because I was trying to do some testing on the 
> tracedump plugin and I could not get the tests to work. But many other 
> plugins (vrrp, nat, acl) work fine and do not have similar messages logged.
> 
> Does anyone have any ideas on what may cause this? Is anyone else seeing this 
> issue on a build from the current master branch?
> 
> Thanks,
> -Matt
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21040): https://lists.fd.io/g/vpp-dev/message/21040
Mute This Topic: https://lists.fd.io/mt/89826377/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vlib_plugin_registration size mismatch

[Matthew Smith via lists.fd.io]

Hi Florin!

Apparently all those plugins are in vpp-plugin-devtools and I had an old
build of that package installed. Oops!

Thanks,
-Matt


On Wed, Mar 16, 2022 at 12:00 PM Florin Coras 
wrote:

> Hi Matt,
>
> Just tried running after a make build and no such message in show log for
> me. Did you try a make wipe?
>
> Regards,
> Florin
>
> > On Mar 16, 2022, at 9:50 AM, Matthew Smith via lists.fd.io  netgate@lists.fd.io> wrote:
> >
> > Hi,
> >
> > I have been testing against a build from yesterday's master branch
> (commit id b0f0f8c8dd9d694bfc13652f89b8b577e9c1c708) and have been seeing
> these messages when VPP starts up:
> >
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
> bufmon_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
> dispatch_trace_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
> oddbuf_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
> perfmon_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
> tracedump_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin
> unittest_plugin.so
> >
> > I noticed the problem because I was trying to do some testing on the
> tracedump plugin and I could not get the tests to work. But many other
> plugins (vrrp, nat, acl) work fine and do not have similar messages logged.
> >
> > Does anyone have any ideas on what may cause this? Is anyone else seeing
> this issue on a build from the current master branch?
> >
> > Thanks,
> > -Matt
> >
> >
> > 
> >
>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21041): https://lists.fd.io/g/vpp-dev/message/21041
Mute This Topic: https://lists.fd.io/mt/89826377/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

My application code has not changed b/w 20.05 and 21.06. The below is the
code snippet of my application that binds the TCP  listen IP/Port
The option that you mentioned "*APP_OPTIONS_ADD_SEGMENT_SIZE*" is not set
in my application code but "*APP_OPTIONS_SEGMENT_SIZE*" is set.

My application code pasted below worked fine in VPP 20.05 but not in 21.06
Is the missing option "*APP_OPTIONS_ADD_SEGMENT_SIZ*E" important to be set
in 21.06 VPP?


static int nas_server_attach (u32 *nasAppIndex)
{
an_ppe_nas_main_t *pm = &an_ppe_nas_main;
u64 options[APP_OPTIONS_N_OPTIONS];
vnet_app_attach_args_t _a, *a = &_a;
u32 segment_size = 512 << 20;

clib_memset (a, 0, sizeof (*a));
clib_memset (options, 0, sizeof (options));

if (pm->private_segment_size)
segment_size = pm->private_segment_size;
a->name = format (0, "nas-tcp-server");
a->api_client_index = APP_INVALID_INDEX;
a->session_cb_vft = &nas_server_session_cb_vft;
a->options = options;
a->options[APP_OPTIONS_SEGMENT_SIZE] = segment_size;
a->options[APP_OPTIONS_RX_FIFO_SIZE] = pm->fifo_size;
a->options[APP_OPTIONS_TX_FIFO_SIZE] = pm->fifo_size;
a->options[APP_OPTIONS_PRIVATE_SEGMENT_COUNT] =
pm->private_segment_count;
a->options[APP_OPTIONS_PREALLOC_FIFO_PAIRS] =
pm->prealloc_fifos ? pm->prealloc_fifos : 0;

a->options[APP_OPTIONS_FLAGS] = APP_OPTIONS_FLAGS_IS_BUILTIN;

if (vnet_application_attach (a))
{
NAS_DBG("Failed to attach ");
return -1;
}
*nasAppIndex = a->app_index;
pm->app_index = a->app_index;

vec_free (a->name);
return 0;
}



Regards,
Vijay

On Wed, Mar 16, 2022 at 10:19 PM Florin Coras 
wrote:

> Hi Vijay,
>
> That’s a sign that either fifo allocations failed (not enough memory in
> the fifo segment) or that the app refused the session
> (app_worker_accept_notify returns non zero).
>
> Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in the
> attach options passed to vnet_application_attach. Some vpp versions ago we
> switched to using the first fifo segment as connects segment and all
> listeners allocate their first segments based on size provided with this
> option. If not provided, listeners fail to allocate segments.
>
> Regards,
> Florin
>
> On Mar 16, 2022, at 3:49 AM, Vijay Kumar  wrote:
>
> Hi Florin,
>
> The patch helped me to find the exact point of failure in
> tcp46_listen_inline() graph node.
> When I tested again, I get the below error and it maps to this error code "
> TCP_ERROR_CREATE_SESSION_FAIL"
> This counter is incremented when the call to function
> session_stream_accept() returns failure.
>
> Is there any potential reason why allocation fails at this place?
>
>
> show errors output
> ===
> 4 arp-reply ARP
> replies sent  error
> 14   ip4-udp-lookup  No
> error  error
>  5tcp4-listen Sessions
> couldn't be allocated   error
>  5  esp4-decrypt-tunESP pkts
> received  error
>  5  ipsec4-tun-input  good packets
> receivederror
>  1 ip4-input   ip4 ttl
> <= 1error
>  1   ip4-icmp-error  hop limit
> exceeded response sent  error
>   3346   ethernet-inputunknown
> vlanerror
>
>
> On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar  wrote:
>
>> Hi Florin,
>>
>> Thanks for the clarification about the TCP changes b/w the 2 releases
>>
>> I will use your patch, hopefully I will catch the issue about where the
>> drops are. I will try to debug further.
>> I will revert back if required.
>>
>>
>> Regards.
>>
>> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar  wrote:
>>>
>>> Hi florin,
>>>
>>> Thanks a lot for helping me out. I will try your patch and update you
>>> with the result.
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>> A general observation
>>> ==
>>> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD
>>> counter is incremented.
>>>
>>>
>>> That’s what I’d expect as well.
>>>
>>>
>>> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph
>>> node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is
>>> called towards the end i.e. after calling tcp_send_synack() but in 21.06
>>> code, the tcp46_listen_trace_frame() is called at the very beginning
>>> of tcp46_listen_inline() graph node.
>>>
>>> I think moving the pkt trace macro to the end of the function is good
>>> (placing it close to the line that calls tcp_send_synack), otherwise it ca

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

Yes, APP_OPTIONS_ADD_SEGMENT_SIZE will be needed if any listeners are used. It 
was there before but was not needed if only one listener was configured.

Regards, 
Florin

> On Mar 16, 2022, at 11:15 AM, Vijay Kumar  wrote:
> 
> Hi Florin,
> 
> My application code has not changed b/w 20.05 and 21.06. The below is the 
> code snippet of my application that binds the TCP  listen IP/Port
> The option that you mentioned "APP_OPTIONS_ADD_SEGMENT_SIZE" is not set in my 
> application code but "APP_OPTIONS_SEGMENT_SIZE" is set.
> 
> My application code pasted below worked fine in VPP 20.05 but not in 21.06
> Is the missing option "APP_OPTIONS_ADD_SEGMENT_SIZE" important to be set in 
> 21.06 VPP?
> 
> 
> static int nas_server_attach (u32 *nasAppIndex)
> {
> an_ppe_nas_main_t *pm = &an_ppe_nas_main;
> u64 options[APP_OPTIONS_N_OPTIONS];
> vnet_app_attach_args_t _a, *a = &_a;
> u32 segment_size = 512 << 20;
> 
> clib_memset (a, 0, sizeof (*a));
> clib_memset (options, 0, sizeof (options));
> 
> if (pm->private_segment_size)
> segment_size = pm->private_segment_size;
> a->name = format (0, "nas-tcp-server");
> a->api_client_index = APP_INVALID_INDEX;
> a->session_cb_vft = &nas_server_session_cb_vft;
> a->options = options;
> a->options[APP_OPTIONS_SEGMENT_SIZE] = segment_size;
> a->options[APP_OPTIONS_RX_FIFO_SIZE] = pm->fifo_size;
> a->options[APP_OPTIONS_TX_FIFO_SIZE] = pm->fifo_size;
> a->options[APP_OPTIONS_PRIVATE_SEGMENT_COUNT] = pm->private_segment_count;
> a->options[APP_OPTIONS_PREALLOC_FIFO_PAIRS] =
> pm->prealloc_fifos ? pm->prealloc_fifos : 0;
> 
> a->options[APP_OPTIONS_FLAGS] = APP_OPTIONS_FLAGS_IS_BUILTIN;
> 
> if (vnet_application_attach (a))
> {
> NAS_DBG("Failed to attach ");
> return -1;
> }
> *nasAppIndex = a->app_index;
> pm->app_index = a->app_index;
>   
> vec_free (a->name);
> return 0;
> }
> 
> 
> 
> Regards,
> Vijay
> 
> On Wed, Mar 16, 2022 at 10:19 PM Florin Coras  > wrote:
> Hi Vijay, 
> 
> That’s a sign that either fifo allocations failed (not enough memory in the 
> fifo segment) or that the app refused the session (app_worker_accept_notify 
> returns non zero). 
> 
> Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in the 
> attach options passed to vnet_application_attach. Some vpp versions ago we 
> switched to using the first fifo segment as connects segment and all 
> listeners allocate their first segments based on size provided with this 
> option. If not provided, listeners fail to allocate segments. 
> 
> Regards,
> Florin
> 
>> On Mar 16, 2022, at 3:49 AM, Vijay Kumar > > wrote:
>> 
>> Hi Florin,
>> 
>> The patch helped me to find the exact point of failure in 
>> tcp46_listen_inline() graph node.
>> When I tested again, I get the below error and it maps to this error code 
>> "TCP_ERROR_CREATE_SESSION_FAIL"
>> This counter is incremented when the call to function 
>> session_stream_accept() returns failure.
>> 
>> Is there any potential reason why allocation fails at this place?
>> 
>> 
>> show errors output
>> ===
>> 4 arp-reply ARP replies 
>> sent  error  
>> 14   ip4-udp-lookup  No 
>> error  error  
>>  5tcp4-listen Sessions couldn't 
>> be allocated   error  
>>  5  esp4-decrypt-tunESP pkts 
>> received  error  
>>  5  ipsec4-tun-input  good packets 
>> receivederror  
>>  1 ip4-input   ip4 ttl 
>> <= 1error  
>>  1   ip4-icmp-error  hop limit exceeded 
>> response sent  error  
>>   3346   ethernet-inputunknown 
>> vlanerror  
>> 
>> 
>> On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar > > wrote:
>> Hi Florin,
>> 
>> Thanks for the clarification about the TCP changes b/w the 2 releases
>> 
>> I will use your patch, hopefully I will catch the issue about where the 
>> drops are. I will try to debug further.
>> I will revert back if required.
>> 
>> 
>> Regards.
>> 
>> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar >> > wrote:
>>> 
>>> Hi florin,
>>> 
>>> Thanks a lot for helping me out. I will try your patch and update you with 
>>> the result.
>> 
>> Thanks! 
>> 
>>> 
>>> 
>>> A general observation
>>> ==
>>> In my setup, I think the SYN pkt is dropped much before

Re: [vpp-dev] vlib_plugin_registration size mismatch

[Florin Coras]

No problems, glad it’s solved :-)

Cheers,
Florin

> On Mar 16, 2022, at 10:36 AM, Matthew Smith  wrote:
> 
> Hi Florin!
> 
> Apparently all those plugins are in vpp-plugin-devtools and I had an old 
> build of that package installed. Oops!
> 
> Thanks,
> -Matt
> 
> 
> On Wed, Mar 16, 2022 at 12:00 PM Florin Coras  > wrote:
> Hi Matt, 
> 
> Just tried running after a make build and no such message in show log for me. 
> Did you try a make wipe? 
> 
> Regards,
> Florin
> 
> > On Mar 16, 2022, at 9:50 AM, Matthew Smith via lists.fd.io 
> >   > > wrote:
> > 
> > Hi,
> > 
> > I have been testing against a build from yesterday's master branch (commit 
> > id b0f0f8c8dd9d694bfc13652f89b8b577e9c1c708) and have been seeing these 
> > messages when VPP starts up:
> > 
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> > bufmon_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> > dispatch_trace_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> > oddbuf_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> > perfmon_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> > tracedump_plugin.so
> > vpp: plugin/load: vlib_plugin_registration size mismatch in plugin 
> > unittest_plugin.so
> > 
> > I noticed the problem because I was trying to do some testing on the 
> > tracedump plugin and I could not get the tests to work. But many other 
> > plugins (vrrp, nat, acl) work fine and do not have similar messages logged.
> > 
> > Does anyone have any ideas on what may cause this? Is anyone else seeing 
> > this issue on a build from the current master branch?
> > 
> > Thanks,
> > -Matt
> > 
> > 
> > 
> > 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21044): https://lists.fd.io/g/vpp-dev/message/21044
Mute This Topic: https://lists.fd.io/mt/89826377/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] vpp hangs with bfd configuration along with mpls (inner and outer ctxt)

[Sastry Sista]

Hi Neale,
Could you please help me on this. I am using fdio 21.06. Some one reported the 
same issue and there is no clarity further on this.

With Regards
Sastry

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21045): https://lists.fd.io/g/vpp-dev/message/21045
Mute This Topic: https://lists.fd.io/mt/89811914/21656
Mute #mpls:https://lists.fd.io/g/vpp-dev/mutehashtag/mpls
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Meanings of different vector rates and questions about NAT44 handoff queue size

[Yueyang Pan via lists.fd.io]

Hi, I am experimenting with the NAT44 plugin in VPP v21.01. I found the 10-sec 
internal node vector rate is very much different with vector rate in? Is there 
anyone who could explain the differences?
The result of “show run” looks like:

---
Thread 22 vpp_wk_21 (lcore 44)
Time 35.6, 10 sec internal node vector rate 11.41 loops/sec 11568016.03
  vector rates in 4.3465e5, out 3.2152e5, drop 4.7701e3, punt 0.e0

The internal node vector rate differs from vector rates hugely. Why is the case?

Also I noticed size of handoff queue is very much important to the performance 
of NAT44 but I was wondering why the congestion drop in NAT handoff would 
increase after the handoff queue size (NAT_FQ_NELTS) is larger than a certain 
value (in my case 512). Does anyone also experience this case and have any 
ideas?
   Best wishes
   Pan


从 Windows 版邮件发送


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21046): https://lists.fd.io/g/vpp-dev/message/21046
Mute This Topic: https://lists.fd.io/mt/89839411/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-