Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

That's fine.  You can submit the changes to the vpp latest branch while I
maintain the patch in my private branch





On Mon, 21 Mar 2022, 03:50 Florin Coras,  wrote:

> Hi Vijay,
>
> Glad it helped!
>
> The patch does not cleanly cherry-pick through gerrit and 21.06 is pretty
> old at this point, i.e., not sure if we’re planning any maintenance release
> at this point. Unfortunately that means your only options are either to
> switch to 22.02, I already cherry-picked the patch, or to maintain the fix
> in a private branch.
>
> Regards,
> Florin
>
> On Mar 20, 2022, at 4:27 AM, Vijay Kumar  wrote:
>
> Hi Florin,
>
> Thanks for the wonderful suggestion.
>
> Upon adding the below option the tcp_listen works fine. The syn-ack is
> being sent from vpp to peer and the TCP handshake is completing
> successfully.
>
> a->options[APP_OPTIONS_ADD_SEGMENT_SIZE] = segment_size;
>
> As you mentioned, earlier there was only one listener but now we added to
> more ALG applications that are also listening on TCP. Hence this
> option becomes mandatory for me.
>
> Florin, can u pls commit the code changes that you shared in this patch.
> The 21.06 code was buggy and it was not reporting any error in the
> tcp46_listen_inline() node.
>
> https://gerrit.fd.io/r/c/vpp/+/35654
>
>
> Thanks.
>
>
> On Thu, Mar 17, 2022 at 12:07 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> Yes, APP_OPTIONS_ADD_SEGMENT_SIZE will be needed if any listeners are
>> used. It was there before but was not needed if only one listener was
>> configured.
>>
>> Regards,
>> Florin
>>
>> On Mar 16, 2022, at 11:15 AM, Vijay Kumar  wrote:
>>
>> Hi Florin,
>>
>> My application code has not changed b/w 20.05 and 21.06. The below is the
>> code snippet of my application that binds the TCP  listen IP/Port
>> The option that you mentioned "*APP_OPTIONS_ADD_SEGMENT_SIZE*" is not
>> set in my application code but "*APP_OPTIONS_SEGMENT_SIZE*" is set.
>>
>> My application code pasted below worked fine in VPP 20.05 but not in 21.06
>> Is the missing option "*APP_OPTIONS_ADD_SEGMENT_SIZ*E" important to be
>> set in 21.06 VPP?
>>
>>
>> static int nas_server_attach (u32 *nasAppIndex)
>> {
>> an_ppe_nas_main_t *pm = _ppe_nas_main;
>> u64 options[APP_OPTIONS_N_OPTIONS];
>> vnet_app_attach_args_t _a, *a = &_a;
>> u32 segment_size = 512 << 20;
>>
>> clib_memset (a, 0, sizeof (*a));
>> clib_memset (options, 0, sizeof (options));
>>
>> if (pm->private_segment_size)
>> segment_size = pm->private_segment_size;
>> a->name = format (0, "nas-tcp-server");
>> a->api_client_index = APP_INVALID_INDEX;
>> a->session_cb_vft = _server_session_cb_vft;
>> a->options = options;
>> a->options[APP_OPTIONS_SEGMENT_SIZE] = segment_size;
>> a->options[APP_OPTIONS_RX_FIFO_SIZE] = pm->fifo_size;
>> a->options[APP_OPTIONS_TX_FIFO_SIZE] = pm->fifo_size;
>> a->options[APP_OPTIONS_PRIVATE_SEGMENT_COUNT] =
>> pm->private_segment_count;
>> a->options[APP_OPTIONS_PREALLOC_FIFO_PAIRS] =
>> pm->prealloc_fifos ? pm->prealloc_fifos : 0;
>>
>> a->options[APP_OPTIONS_FLAGS] = APP_OPTIONS_FLAGS_IS_BUILTIN;
>>
>> if (vnet_application_attach (a))
>> {
>> NAS_DBG("Failed to attach ");
>> return -1;
>> }
>> *nasAppIndex = a->app_index;
>> pm->app_index = a->app_index;
>>
>> vec_free (a->name);
>> return 0;
>> }
>>
>>
>>
>> Regards,
>> Vijay
>>
>> On Wed, Mar 16, 2022 at 10:19 PM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> That’s a sign that either fifo allocations failed (not enough memory in
>>> the fifo segment) or that the app refused the session
>>> (app_worker_accept_notify returns non zero).
>>>
>>> Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in
>>> the attach options passed to vnet_application_attach. Some vpp versions ago
>>> we switched to using the first fifo segment as connects segment and all
>>> listeners allocate their first segments based on size provided with this
>>> option. If not provided, listeners fail to allocate segments.
>>>
>>> Regards,
>>> Florin
>>>
>>> On Mar 16, 2022, at 3:49 AM, Vijay Kumar  wrote:
>>>
>>> Hi Florin,
>>>
>>> The patch helped me to find the exact point of failure in
>>> tcp46_listen_inline() graph node.
>>> When I tested again, I get the below error and it maps to this error
>>> code "TCP_ERROR_CREATE_SESSION_FAIL"
>>> This counter is incremented when the call to function
>>> session_stream_accept() returns failure.
>>>
>>> Is there any potential reason why allocation fails at this place?
>>>
>>>
>>> show errors output
>>> ===
>>> 4 arp-reply ARP
>>> replies sent  error
>>> 14   ip4-udp-lookup  No
>>> error  error
>>>  5tcp4-listen Sessions
>>> couldn't be allocated   error
>>>  5  

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

Glad it helped!

The patch does not cleanly cherry-pick through gerrit and 21.06 is pretty old 
at this point, i.e., not sure if we’re planning any maintenance release at this 
point. Unfortunately that means your only options are either to switch to 
22.02, I already cherry-picked the patch, or to maintain the fix in a private 
branch.

Regards,
Florin

> On Mar 20, 2022, at 4:27 AM, Vijay Kumar  wrote:
> 
> Hi Florin,
> 
> Thanks for the wonderful suggestion.
> 
> Upon adding the below option the tcp_listen works fine. The syn-ack is being 
> sent from vpp to peer and the TCP handshake is completing successfully.
> 
> a->options[APP_OPTIONS_ADD_SEGMENT_SIZE] = segment_size;
> 
> As you mentioned, earlier there was only one listener but now we added to 
> more ALG applications that are also listening on TCP. Hence this 
> option becomes mandatory for me.
> 
> Florin, can u pls commit the code changes that you shared in this patch. The 
> 21.06 code was buggy and it was not reporting any error in the 
> tcp46_listen_inline() node.
> 
> https://gerrit.fd.io/r/c/vpp/+/35654 
> 
> 
> Thanks.
> 
> 
> On Thu, Mar 17, 2022 at 12:07 AM Florin Coras  > wrote:
> Hi Vijay, 
> 
> Yes, APP_OPTIONS_ADD_SEGMENT_SIZE will be needed if any listeners are used. 
> It was there before but was not needed if only one listener was configured.
> 
> Regards, 
> Florin
> 
>> On Mar 16, 2022, at 11:15 AM, Vijay Kumar > > wrote:
>> 
>> Hi Florin,
>> 
>> My application code has not changed b/w 20.05 and 21.06. The below is the 
>> code snippet of my application that binds the TCP  listen IP/Port
>> The option that you mentioned "APP_OPTIONS_ADD_SEGMENT_SIZE" is not set in 
>> my application code but "APP_OPTIONS_SEGMENT_SIZE" is set.
>> 
>> My application code pasted below worked fine in VPP 20.05 but not in 21.06
>> Is the missing option "APP_OPTIONS_ADD_SEGMENT_SIZE" important to be set in 
>> 21.06 VPP?
>> 
>> 
>> static int nas_server_attach (u32 *nasAppIndex)
>> {
>> an_ppe_nas_main_t *pm = _ppe_nas_main;
>> u64 options[APP_OPTIONS_N_OPTIONS];
>> vnet_app_attach_args_t _a, *a = &_a;
>> u32 segment_size = 512 << 20;
>> 
>> clib_memset (a, 0, sizeof (*a));
>> clib_memset (options, 0, sizeof (options));
>> 
>> if (pm->private_segment_size)
>> segment_size = pm->private_segment_size;
>> a->name = format (0, "nas-tcp-server");
>> a->api_client_index = APP_INVALID_INDEX;
>> a->session_cb_vft = _server_session_cb_vft;
>> a->options = options;
>> a->options[APP_OPTIONS_SEGMENT_SIZE] = segment_size;
>> a->options[APP_OPTIONS_RX_FIFO_SIZE] = pm->fifo_size;
>> a->options[APP_OPTIONS_TX_FIFO_SIZE] = pm->fifo_size;
>> a->options[APP_OPTIONS_PRIVATE_SEGMENT_COUNT] = 
>> pm->private_segment_count;
>> a->options[APP_OPTIONS_PREALLOC_FIFO_PAIRS] =
>> pm->prealloc_fifos ? pm->prealloc_fifos : 0;
>> 
>> a->options[APP_OPTIONS_FLAGS] = APP_OPTIONS_FLAGS_IS_BUILTIN;
>> 
>> if (vnet_application_attach (a))
>> {
>> NAS_DBG("Failed to attach ");
>> return -1;
>> }
>> *nasAppIndex = a->app_index;
>> pm->app_index = a->app_index;
>>   
>> vec_free (a->name);
>> return 0;
>> }
>> 
>> 
>> 
>> Regards,
>> Vijay
>> 
>> On Wed, Mar 16, 2022 at 10:19 PM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> That’s a sign that either fifo allocations failed (not enough memory in the 
>> fifo segment) or that the app refused the session (app_worker_accept_notify 
>> returns non zero). 
>> 
>> Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in the 
>> attach options passed to vnet_application_attach. Some vpp versions ago we 
>> switched to using the first fifo segment as connects segment and all 
>> listeners allocate their first segments based on size provided with this 
>> option. If not provided, listeners fail to allocate segments. 
>> 
>> Regards,
>> Florin
>> 
>>> On Mar 16, 2022, at 3:49 AM, Vijay Kumar >> > wrote:
>>> 
>>> Hi Florin,
>>> 
>>> The patch helped me to find the exact point of failure in 
>>> tcp46_listen_inline() graph node.
>>> When I tested again, I get the below error and it maps to this error code 
>>> "TCP_ERROR_CREATE_SESSION_FAIL"
>>> This counter is incremented when the call to function 
>>> session_stream_accept() returns failure.
>>> 
>>> Is there any potential reason why allocation fails at this place?
>>> 
>>> 
>>> show errors output
>>> ===
>>> 4 arp-reply ARP replies 
>>> sent  error  
>>> 14   ip4-udp-lookup  No 
>>> error  error  
>>>  5tcp4-listen Sessions couldn't 
>>> be allocated   error  
>>>  5  

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

Thanks for the wonderful suggestion.

Upon adding the below option the tcp_listen works fine. The syn-ack is
being sent from vpp to peer and the TCP handshake is completing
successfully.

a->options[APP_OPTIONS_ADD_SEGMENT_SIZE] = segment_size;

As you mentioned, earlier there was only one listener but now we added to
more ALG applications that are also listening on TCP. Hence this
option becomes mandatory for me.

Florin, can u pls commit the code changes that you shared in this patch.
The 21.06 code was buggy and it was not reporting any error in the
tcp46_listen_inline() node.

https://gerrit.fd.io/r/c/vpp/+/35654


Thanks.


On Thu, Mar 17, 2022 at 12:07 AM Florin Coras 
wrote:

> Hi Vijay,
>
> Yes, APP_OPTIONS_ADD_SEGMENT_SIZE will be needed if any listeners are
> used. It was there before but was not needed if only one listener was
> configured.
>
> Regards,
> Florin
>
> On Mar 16, 2022, at 11:15 AM, Vijay Kumar  wrote:
>
> Hi Florin,
>
> My application code has not changed b/w 20.05 and 21.06. The below is the
> code snippet of my application that binds the TCP  listen IP/Port
> The option that you mentioned "*APP_OPTIONS_ADD_SEGMENT_SIZE*" is not set
> in my application code but "*APP_OPTIONS_SEGMENT_SIZE*" is set.
>
> My application code pasted below worked fine in VPP 20.05 but not in 21.06
> Is the missing option "*APP_OPTIONS_ADD_SEGMENT_SIZ*E" important to be
> set in 21.06 VPP?
>
>
> static int nas_server_attach (u32 *nasAppIndex)
> {
> an_ppe_nas_main_t *pm = _ppe_nas_main;
> u64 options[APP_OPTIONS_N_OPTIONS];
> vnet_app_attach_args_t _a, *a = &_a;
> u32 segment_size = 512 << 20;
>
> clib_memset (a, 0, sizeof (*a));
> clib_memset (options, 0, sizeof (options));
>
> if (pm->private_segment_size)
> segment_size = pm->private_segment_size;
> a->name = format (0, "nas-tcp-server");
> a->api_client_index = APP_INVALID_INDEX;
> a->session_cb_vft = _server_session_cb_vft;
> a->options = options;
> a->options[APP_OPTIONS_SEGMENT_SIZE] = segment_size;
> a->options[APP_OPTIONS_RX_FIFO_SIZE] = pm->fifo_size;
> a->options[APP_OPTIONS_TX_FIFO_SIZE] = pm->fifo_size;
> a->options[APP_OPTIONS_PRIVATE_SEGMENT_COUNT] =
> pm->private_segment_count;
> a->options[APP_OPTIONS_PREALLOC_FIFO_PAIRS] =
> pm->prealloc_fifos ? pm->prealloc_fifos : 0;
>
> a->options[APP_OPTIONS_FLAGS] = APP_OPTIONS_FLAGS_IS_BUILTIN;
>
> if (vnet_application_attach (a))
> {
> NAS_DBG("Failed to attach ");
> return -1;
> }
> *nasAppIndex = a->app_index;
> pm->app_index = a->app_index;
>
> vec_free (a->name);
> return 0;
> }
>
>
>
> Regards,
> Vijay
>
> On Wed, Mar 16, 2022 at 10:19 PM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> That’s a sign that either fifo allocations failed (not enough memory in
>> the fifo segment) or that the app refused the session
>> (app_worker_accept_notify returns non zero).
>>
>> Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in the
>> attach options passed to vnet_application_attach. Some vpp versions ago we
>> switched to using the first fifo segment as connects segment and all
>> listeners allocate their first segments based on size provided with this
>> option. If not provided, listeners fail to allocate segments.
>>
>> Regards,
>> Florin
>>
>> On Mar 16, 2022, at 3:49 AM, Vijay Kumar  wrote:
>>
>> Hi Florin,
>>
>> The patch helped me to find the exact point of failure in
>> tcp46_listen_inline() graph node.
>> When I tested again, I get the below error and it maps to this error code
>> "TCP_ERROR_CREATE_SESSION_FAIL"
>> This counter is incremented when the call to function
>> session_stream_accept() returns failure.
>>
>> Is there any potential reason why allocation fails at this place?
>>
>>
>> show errors output
>> ===
>> 4 arp-reply ARP
>> replies sent  error
>> 14   ip4-udp-lookup  No
>> error  error
>>  5tcp4-listen Sessions
>> couldn't be allocated   error
>>  5  esp4-decrypt-tunESP pkts
>> received  error
>>  5  ipsec4-tun-input  good
>> packets receivederror
>>  1 ip4-input   ip4
>> ttl <= 1error
>>  1   ip4-icmp-error  hop limit
>> exceeded response sent  error
>>   3346   ethernet-input
>>  unknown vlanerror
>>
>>
>> On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar 
>> wrote:
>>
>>> Hi Florin,
>>>
>>> Thanks for the clarification about the TCP changes b/w the 2 releases
>>>
>>> I will use your patch, hopefully I will catch the issue about where the
>>> drops 

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

Yes, APP_OPTIONS_ADD_SEGMENT_SIZE will be needed if any listeners are used. It 
was there before but was not needed if only one listener was configured.

Regards, 
Florin

> On Mar 16, 2022, at 11:15 AM, Vijay Kumar  wrote:
> 
> Hi Florin,
> 
> My application code has not changed b/w 20.05 and 21.06. The below is the 
> code snippet of my application that binds the TCP  listen IP/Port
> The option that you mentioned "APP_OPTIONS_ADD_SEGMENT_SIZE" is not set in my 
> application code but "APP_OPTIONS_SEGMENT_SIZE" is set.
> 
> My application code pasted below worked fine in VPP 20.05 but not in 21.06
> Is the missing option "APP_OPTIONS_ADD_SEGMENT_SIZE" important to be set in 
> 21.06 VPP?
> 
> 
> static int nas_server_attach (u32 *nasAppIndex)
> {
> an_ppe_nas_main_t *pm = _ppe_nas_main;
> u64 options[APP_OPTIONS_N_OPTIONS];
> vnet_app_attach_args_t _a, *a = &_a;
> u32 segment_size = 512 << 20;
> 
> clib_memset (a, 0, sizeof (*a));
> clib_memset (options, 0, sizeof (options));
> 
> if (pm->private_segment_size)
> segment_size = pm->private_segment_size;
> a->name = format (0, "nas-tcp-server");
> a->api_client_index = APP_INVALID_INDEX;
> a->session_cb_vft = _server_session_cb_vft;
> a->options = options;
> a->options[APP_OPTIONS_SEGMENT_SIZE] = segment_size;
> a->options[APP_OPTIONS_RX_FIFO_SIZE] = pm->fifo_size;
> a->options[APP_OPTIONS_TX_FIFO_SIZE] = pm->fifo_size;
> a->options[APP_OPTIONS_PRIVATE_SEGMENT_COUNT] = pm->private_segment_count;
> a->options[APP_OPTIONS_PREALLOC_FIFO_PAIRS] =
> pm->prealloc_fifos ? pm->prealloc_fifos : 0;
> 
> a->options[APP_OPTIONS_FLAGS] = APP_OPTIONS_FLAGS_IS_BUILTIN;
> 
> if (vnet_application_attach (a))
> {
> NAS_DBG("Failed to attach ");
> return -1;
> }
> *nasAppIndex = a->app_index;
> pm->app_index = a->app_index;
>   
> vec_free (a->name);
> return 0;
> }
> 
> 
> 
> Regards,
> Vijay
> 
> On Wed, Mar 16, 2022 at 10:19 PM Florin Coras  > wrote:
> Hi Vijay, 
> 
> That’s a sign that either fifo allocations failed (not enough memory in the 
> fifo segment) or that the app refused the session (app_worker_accept_notify 
> returns non zero). 
> 
> Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in the 
> attach options passed to vnet_application_attach. Some vpp versions ago we 
> switched to using the first fifo segment as connects segment and all 
> listeners allocate their first segments based on size provided with this 
> option. If not provided, listeners fail to allocate segments. 
> 
> Regards,
> Florin
> 
>> On Mar 16, 2022, at 3:49 AM, Vijay Kumar > > wrote:
>> 
>> Hi Florin,
>> 
>> The patch helped me to find the exact point of failure in 
>> tcp46_listen_inline() graph node.
>> When I tested again, I get the below error and it maps to this error code 
>> "TCP_ERROR_CREATE_SESSION_FAIL"
>> This counter is incremented when the call to function 
>> session_stream_accept() returns failure.
>> 
>> Is there any potential reason why allocation fails at this place?
>> 
>> 
>> show errors output
>> ===
>> 4 arp-reply ARP replies 
>> sent  error  
>> 14   ip4-udp-lookup  No 
>> error  error  
>>  5tcp4-listen Sessions couldn't 
>> be allocated   error  
>>  5  esp4-decrypt-tunESP pkts 
>> received  error  
>>  5  ipsec4-tun-input  good packets 
>> receivederror  
>>  1 ip4-input   ip4 ttl 
>> <= 1error  
>>  1   ip4-icmp-error  hop limit exceeded 
>> response sent  error  
>>   3346   ethernet-inputunknown 
>> vlanerror  
>> 
>> 
>> On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar > > wrote:
>> Hi Florin,
>> 
>> Thanks for the clarification about the TCP changes b/w the 2 releases
>> 
>> I will use your patch, hopefully I will catch the issue about where the 
>> drops are. I will try to debug further.
>> I will revert back if required.
>> 
>> 
>> Regards.
>> 
>> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar >> > wrote:
>>> 
>>> Hi florin,
>>> 
>>> Thanks a lot for helping me out. I will try your patch and update you with 
>>> the result.
>> 
>> Thanks! 
>> 
>>> 
>>> 
>>> A general observation
>>> ==
>>> In my setup, I think the SYN pkt is dropped much before the 

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

My application code has not changed b/w 20.05 and 21.06. The below is the
code snippet of my application that binds the TCP  listen IP/Port
The option that you mentioned "*APP_OPTIONS_ADD_SEGMENT_SIZE*" is not set
in my application code but "*APP_OPTIONS_SEGMENT_SIZE*" is set.

My application code pasted below worked fine in VPP 20.05 but not in 21.06
Is the missing option "*APP_OPTIONS_ADD_SEGMENT_SIZ*E" important to be set
in 21.06 VPP?


static int nas_server_attach (u32 *nasAppIndex)
{
an_ppe_nas_main_t *pm = _ppe_nas_main;
u64 options[APP_OPTIONS_N_OPTIONS];
vnet_app_attach_args_t _a, *a = &_a;
u32 segment_size = 512 << 20;

clib_memset (a, 0, sizeof (*a));
clib_memset (options, 0, sizeof (options));

if (pm->private_segment_size)
segment_size = pm->private_segment_size;
a->name = format (0, "nas-tcp-server");
a->api_client_index = APP_INVALID_INDEX;
a->session_cb_vft = _server_session_cb_vft;
a->options = options;
a->options[APP_OPTIONS_SEGMENT_SIZE] = segment_size;
a->options[APP_OPTIONS_RX_FIFO_SIZE] = pm->fifo_size;
a->options[APP_OPTIONS_TX_FIFO_SIZE] = pm->fifo_size;
a->options[APP_OPTIONS_PRIVATE_SEGMENT_COUNT] =
pm->private_segment_count;
a->options[APP_OPTIONS_PREALLOC_FIFO_PAIRS] =
pm->prealloc_fifos ? pm->prealloc_fifos : 0;

a->options[APP_OPTIONS_FLAGS] = APP_OPTIONS_FLAGS_IS_BUILTIN;

if (vnet_application_attach (a))
{
NAS_DBG("Failed to attach ");
return -1;
}
*nasAppIndex = a->app_index;
pm->app_index = a->app_index;

vec_free (a->name);
return 0;
}



Regards,
Vijay

On Wed, Mar 16, 2022 at 10:19 PM Florin Coras 
wrote:

> Hi Vijay,
>
> That’s a sign that either fifo allocations failed (not enough memory in
> the fifo segment) or that the app refused the session
> (app_worker_accept_notify returns non zero).
>
> Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in the
> attach options passed to vnet_application_attach. Some vpp versions ago we
> switched to using the first fifo segment as connects segment and all
> listeners allocate their first segments based on size provided with this
> option. If not provided, listeners fail to allocate segments.
>
> Regards,
> Florin
>
> On Mar 16, 2022, at 3:49 AM, Vijay Kumar  wrote:
>
> Hi Florin,
>
> The patch helped me to find the exact point of failure in
> tcp46_listen_inline() graph node.
> When I tested again, I get the below error and it maps to this error code "
> TCP_ERROR_CREATE_SESSION_FAIL"
> This counter is incremented when the call to function
> session_stream_accept() returns failure.
>
> Is there any potential reason why allocation fails at this place?
>
>
> show errors output
> ===
> 4 arp-reply ARP
> replies sent  error
> 14   ip4-udp-lookup  No
> error  error
>  5tcp4-listen Sessions
> couldn't be allocated   error
>  5  esp4-decrypt-tunESP pkts
> received  error
>  5  ipsec4-tun-input  good packets
> receivederror
>  1 ip4-input   ip4 ttl
> <= 1error
>  1   ip4-icmp-error  hop limit
> exceeded response sent  error
>   3346   ethernet-inputunknown
> vlanerror
>
>
> On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar  wrote:
>
>> Hi Florin,
>>
>> Thanks for the clarification about the TCP changes b/w the 2 releases
>>
>> I will use your patch, hopefully I will catch the issue about where the
>> drops are. I will try to debug further.
>> I will revert back if required.
>>
>>
>> Regards.
>>
>> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar  wrote:
>>>
>>> Hi florin,
>>>
>>> Thanks a lot for helping me out. I will try your patch and update you
>>> with the result.
>>>
>>>
>>> Thanks!
>>>
>>>
>>>
>>> A general observation
>>> ==
>>> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD
>>> counter is incremented.
>>>
>>>
>>> That’s what I’d expect as well.
>>>
>>>
>>> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph
>>> node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is
>>> called towards the end i.e. after calling tcp_send_synack() but in 21.06
>>> code, the tcp46_listen_trace_frame() is called at the very beginning
>>> of tcp46_listen_inline() graph node.
>>>
>>> I think moving the pkt trace macro to the end of the function is good
>>> (placing it close to the line that calls tcp_send_synack), otherwise it can
>>> 

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

That’s a sign that either fifo allocations failed (not enough memory in the 
fifo segment) or that the app refused the session (app_worker_accept_notify 
returns non zero). 

Here’s a guess, your app does not set APP_OPTIONS_ADD_SEGMENT_SIZE in the 
attach options passed to vnet_application_attach. Some vpp versions ago we 
switched to using the first fifo segment as connects segment and all listeners 
allocate their first segments based on size provided with this option. If not 
provided, listeners fail to allocate segments. 

Regards,
Florin

> On Mar 16, 2022, at 3:49 AM, Vijay Kumar  wrote:
> 
> Hi Florin,
> 
> The patch helped me to find the exact point of failure in 
> tcp46_listen_inline() graph node.
> When I tested again, I get the below error and it maps to this error code 
> "TCP_ERROR_CREATE_SESSION_FAIL"
> This counter is incremented when the call to function session_stream_accept() 
> returns failure.
> 
> Is there any potential reason why allocation fails at this place?
> 
> 
> show errors output
> ===
> 4 arp-reply ARP replies 
> sent  error  
> 14   ip4-udp-lookup  No error 
>  error  
>  5tcp4-listen Sessions couldn't 
> be allocated   error  
>  5  esp4-decrypt-tunESP pkts 
> received  error  
>  5  ipsec4-tun-input  good packets 
> receivederror  
>  1 ip4-input   ip4 ttl <= 
> 1error  
>  1   ip4-icmp-error  hop limit exceeded 
> response sent  error  
>   3346   ethernet-inputunknown 
> vlanerror  
> 
> 
> On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar  > wrote:
> Hi Florin,
> 
> Thanks for the clarification about the TCP changes b/w the 2 releases
> 
> I will use your patch, hopefully I will catch the issue about where the drops 
> are. I will try to debug further.
> I will revert back if required.
> 
> 
> Regards.
> 
> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras  > wrote:
> Hi Vijay, 
> 
>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar > > wrote:
>> 
>> Hi florin,
>> 
>> Thanks a lot for helping me out. I will try your patch and update you with 
>> the result.
> 
> Thanks! 
> 
>> 
>> 
>> A general observation
>> ==
>> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD 
>> counter is incremented.
> 
> That’s what I’d expect as well. 
> 
>> 
>> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph 
>> node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is 
>> called towards the end i.e. after calling tcp_send_synack() but in 21.06 
>> code, the tcp46_listen_trace_frame() is called at the very beginning of 
>> tcp46_listen_inline() graph node.
>> 
>> I think moving the pkt trace macro to the end of the function is good 
>> (placing it close to the line that calls tcp_send_synack), otherwise it can 
>> mislead us and will not help debugging.
>> 
> 
> Most of the changes were code refactoring/cleanup and buffer handling 
> optimizations, not protocol related. TCP tracing, at least as it is now, 
> doesn’t provide info about the errors hit, instead it reports the connections 
> hit and the reporting of errors is done through node and session counters 
> (see show session verbose 2). Obviously that didn’t work properly for the 
> listen node. 
> 
> Regards,
> Florin
> 
>> 
>> 
>> On Wed, Mar 16, 2022 at 10:22 AM Florin Coras > > wrote:
>> Hi Vijay, 
>> 
>> That’s probably because packets are hitting an actual error and it seems the 
>> listen node is not reporting anything but syns received. Here’s a patch that 
>> might help [1]. It might not cherry-pick cleanly on 21.06. 
>> 
>> Regards,
>> Florin
>> 
>> [1] https://gerrit.fd.io/r/c/vpp/+/35654 
>> 
>> 
>>> On Mar 15, 2022, at 7:54 PM, Vijay Kumar >> > wrote:
>>> 
>>> Hi Florin,
>>> 
>>> I have the output of show node counters (show errors) taken on both 20.05 
>>> and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or 
>>> tcp4-output etc. 
>>> 
>>> Please let me know why SYN rcvd counter itself is not incremented but in 
>>> the earlier reply I already pasted the show trace ouput where we saw the 
>>> SYN pkt was landed on tcp4-litsen node.
>>> 
>>> VPP 20.05
>>> ===
>>> pp# show node counters 
>>>CountNode  Reason
>>>  3   memif-input  not ip packet
>>>  

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

The patch helped me to find the exact point of failure in
tcp46_listen_inline() graph node.
When I tested again, I get the below error and it maps to this error code "
TCP_ERROR_CREATE_SESSION_FAIL"
This counter is incremented when the call to function
session_stream_accept() returns failure.

Is there any potential reason why allocation fails at this place?


show errors output
===
4 arp-reply ARP replies
sent  error
14   ip4-udp-lookup  No
error  error
 5tcp4-listen Sessions couldn't
be allocated   error
 5  esp4-decrypt-tunESP pkts
received  error
 5  ipsec4-tun-input  good packets
receivederror
 1 ip4-input   ip4 ttl
<= 1error
 1   ip4-icmp-error  hop limit exceeded
response sent  error
  3346   ethernet-inputunknown
vlanerror


On Wed, Mar 16, 2022 at 1:31 PM Vijay Kumar  wrote:

> Hi Florin,
>
> Thanks for the clarification about the TCP changes b/w the 2 releases
>
> I will use your patch, hopefully I will catch the issue about where the
> drops are. I will try to debug further.
> I will revert back if required.
>
>
> Regards.
>
> On Wed, Mar 16, 2022 at 11:12 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> On Mar 15, 2022, at 9:58 PM, Vijay Kumar  wrote:
>>
>> Hi florin,
>>
>> Thanks a lot for helping me out. I will try your patch and update you
>> with the result.
>>
>>
>> Thanks!
>>
>>
>>
>> A general observation
>> ==
>> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD
>> counter is incremented.
>>
>>
>> That’s what I’d expect as well.
>>
>>
>> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph
>> node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is
>> called towards the end i.e. after calling tcp_send_synack() but in 21.06
>> code, the tcp46_listen_trace_frame() is called at the very beginning
>> of tcp46_listen_inline() graph node.
>>
>> I think moving the pkt trace macro to the end of the function is good
>> (placing it close to the line that calls tcp_send_synack), otherwise it can
>> mislead us and will not help debugging.
>>
>>
>> Most of the changes were code refactoring/cleanup and buffer handling
>> optimizations, not protocol related. TCP tracing, at least as it is now,
>> doesn’t provide info about the errors hit, instead it reports the
>> connections hit and the reporting of errors is done through node and
>> session counters (see show session verbose 2). Obviously that didn’t work
>> properly for the listen node.
>>
>> Regards,
>> Florin
>>
>>
>>
>> On Wed, Mar 16, 2022 at 10:22 AM Florin Coras 
>> wrote:
>>
>>> Hi Vijay,
>>>
>>> That’s probably because packets are hitting an actual error and it seems
>>> the listen node is not reporting anything but syns received. Here’s a patch
>>> that might help [1]. It might not cherry-pick cleanly on 21.06.
>>>
>>> Regards,
>>> Florin
>>>
>>> [1] https://gerrit.fd.io/r/c/vpp/+/35654
>>>
>>> On Mar 15, 2022, at 7:54 PM, Vijay Kumar  wrote:
>>>
>>> Hi Florin,
>>>
>>> I have the output of show node counters (show errors) taken on both
>>> 20.05 and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or
>>> tcp4-output etc.
>>>
>>> Please let me know why SYN rcvd counter itself is not incremented but in
>>> the earlier reply I already pasted the show trace ouput where we saw the
>>> SYN pkt was landed on tcp4-litsen node.
>>>
>>> VPP 20.05
>>> ===
>>> pp# show node counters
>>>CountNode  Reason
>>>  3   memif-input  not ip packet
>>>  10259 an_ppe_wfectrl wfectrl packets
>>> received
>>>  10259 an_ppe_wfectrl wfectrl replies sent
>>>  1 an_ppe_wfectrl wfectrl packet
>>> processing failed
>>>   8123 an_ppe_wfectrl session stat request
>>> received
>>>  1 an_ppe_wfectrl service construct
>>> config request received
>>>  1 an_ppe_wfectrl service construct
>>> config request success
>>>  1 an_ppe_wfectrl service config request
>>> received
>>>  1 an_ppe_wfectrl service config request
>>> success
>>>   1686 an_ppe_wfectrl dpi stats request
>>> received
>>>   1686 an_ppe_wfectrl dpi stats request
>>> success
>>> 70 an_ppe_wfectrl 

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

Thanks for the clarification about the TCP changes b/w the 2 releases

I will use your patch, hopefully I will catch the issue about where the
drops are. I will try to debug further.
I will revert back if required.


Regards.

On Wed, Mar 16, 2022 at 11:12 AM Florin Coras 
wrote:

> Hi Vijay,
>
> On Mar 15, 2022, at 9:58 PM, Vijay Kumar  wrote:
>
> Hi florin,
>
> Thanks a lot for helping me out. I will try your patch and update you with
> the result.
>
>
> Thanks!
>
>
>
> A general observation
> ==
> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD
> counter is incremented.
>
>
> That’s what I’d expect as well.
>
>
> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph
> node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is
> called towards the end i.e. after calling tcp_send_synack() but in 21.06
> code, the tcp46_listen_trace_frame() is called at the very beginning
> of tcp46_listen_inline() graph node.
>
> I think moving the pkt trace macro to the end of the function is good
> (placing it close to the line that calls tcp_send_synack), otherwise it can
> mislead us and will not help debugging.
>
>
> Most of the changes were code refactoring/cleanup and buffer handling
> optimizations, not protocol related. TCP tracing, at least as it is now,
> doesn’t provide info about the errors hit, instead it reports the
> connections hit and the reporting of errors is done through node and
> session counters (see show session verbose 2). Obviously that didn’t work
> properly for the listen node.
>
> Regards,
> Florin
>
>
>
> On Wed, Mar 16, 2022 at 10:22 AM Florin Coras 
> wrote:
>
>> Hi Vijay,
>>
>> That’s probably because packets are hitting an actual error and it seems
>> the listen node is not reporting anything but syns received. Here’s a patch
>> that might help [1]. It might not cherry-pick cleanly on 21.06.
>>
>> Regards,
>> Florin
>>
>> [1] https://gerrit.fd.io/r/c/vpp/+/35654
>>
>> On Mar 15, 2022, at 7:54 PM, Vijay Kumar  wrote:
>>
>> Hi Florin,
>>
>> I have the output of show node counters (show errors) taken on both 20.05
>> and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or
>> tcp4-output etc.
>>
>> Please let me know why SYN rcvd counter itself is not incremented but in
>> the earlier reply I already pasted the show trace ouput where we saw the
>> SYN pkt was landed on tcp4-litsen node.
>>
>> VPP 20.05
>> ===
>> pp# show node counters
>>CountNode  Reason
>>  3   memif-input  not ip packet
>>  10259 an_ppe_wfectrl wfectrl packets received
>>  10259 an_ppe_wfectrl wfectrl replies sent
>>  1 an_ppe_wfectrl wfectrl packet
>> processing failed
>>   8123 an_ppe_wfectrl session stat request
>> received
>>  1 an_ppe_wfectrl service construct
>> config request received
>>  1 an_ppe_wfectrl service construct
>> config request success
>>  1 an_ppe_wfectrl service config request
>> received
>>  1 an_ppe_wfectrl service config request
>> success
>>   1686 an_ppe_wfectrl dpi stats request
>> received
>>   1686 an_ppe_wfectrl dpi stats request
>> success
>> 70 an_ppe_wfectrl nat stats request
>> received
>> 70 an_ppe_wfectrl nat stats request
>> success
>> 70 an_ppe_wfectrl vpp stats request
>> received
>> 70 an_ppe_wfectrl vpp stats request
>> success
>>  1 an_ppe_wfectrl udp tunnel resource
>> block add request received
>>  1 an_ppe_wfectrl udp tunnel resource
>> block add request success
>>  1 an_ppe_wfectrl l3rm resource block
>> update request received
>>  1 an_ppe_wfectrl l3rm resource block
>> update request success
>>  1 an_ppe_wfectrl ue registration request
>> received
>> 17 an_ppe_wfectrl ike msg received from
>> ikemgr
>> 17 an_ppe_wfectrl ike msg send to network
>> success
>>  3 an_ppe_wfectrl ipsec sa install msg
>> received from ikemgr
>>  3 an_ppe_wfectrl ipsec sa install msg
>> processed successfully
>>  1  an_ppe_vppctrl_input  Success in insert in
>> ueidentifier
>>  1  an_ppe_vppctrl_input  Success  in insert in
>> uecontext
>>  2  an_ppe_vppctrl_input  Downlink message sent
>> to UE successfully
>>  1  an_ppe_vppctrl_input  

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

> On Mar 15, 2022, at 9:58 PM, Vijay Kumar  wrote:
> 
> Hi florin,
> 
> Thanks a lot for helping me out. I will try your patch and update you with 
> the result.

Thanks! 

> 
> 
> A general observation
> ==
> In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD counter 
> is incremented.

That’s what I’d expect as well. 

> 
> I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph node 
> tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is called 
> towards the end i.e. after calling tcp_send_synack() but in 21.06 code, the 
> tcp46_listen_trace_frame() is called at the very beginning of 
> tcp46_listen_inline() graph node.
> 
> I think moving the pkt trace macro to the end of the function is good 
> (placing it close to the line that calls tcp_send_synack), otherwise it can 
> mislead us and will not help debugging.
> 

Most of the changes were code refactoring/cleanup and buffer handling 
optimizations, not protocol related. TCP tracing, at least as it is now, 
doesn’t provide info about the errors hit, instead it reports the connections 
hit and the reporting of errors is done through node and session counters (see 
show session verbose 2). Obviously that didn’t work properly for the listen 
node. 

Regards,
Florin

> 
> 
> On Wed, Mar 16, 2022 at 10:22 AM Florin Coras  > wrote:
> Hi Vijay, 
> 
> That’s probably because packets are hitting an actual error and it seems the 
> listen node is not reporting anything but syns received. Here’s a patch that 
> might help [1]. It might not cherry-pick cleanly on 21.06. 
> 
> Regards,
> Florin
> 
> [1] https://gerrit.fd.io/r/c/vpp/+/35654 
> 
> 
>> On Mar 15, 2022, at 7:54 PM, Vijay Kumar > > wrote:
>> 
>> Hi Florin,
>> 
>> I have the output of show node counters (show errors) taken on both 20.05 
>> and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or 
>> tcp4-output etc. 
>> 
>> Please let me know why SYN rcvd counter itself is not incremented but in the 
>> earlier reply I already pasted the show trace ouput where we saw the SYN pkt 
>> was landed on tcp4-litsen node.
>> 
>> VPP 20.05
>> ===
>> pp# show node counters 
>>CountNode  Reason
>>  3   memif-input  not ip packet
>>  10259 an_ppe_wfectrl wfectrl packets received
>>  10259 an_ppe_wfectrl wfectrl replies sent
>>  1 an_ppe_wfectrl wfectrl packet processing 
>> failed
>>   8123 an_ppe_wfectrl session stat request 
>> received
>>  1 an_ppe_wfectrl service construct config 
>> request received
>>  1 an_ppe_wfectrl service construct config 
>> request success
>>  1 an_ppe_wfectrl service config request 
>> received
>>  1 an_ppe_wfectrl service config request 
>> success
>>   1686 an_ppe_wfectrl dpi stats request received
>>   1686 an_ppe_wfectrl dpi stats request success
>> 70 an_ppe_wfectrl nat stats request received
>> 70 an_ppe_wfectrl nat stats request success
>> 70 an_ppe_wfectrl vpp stats request received
>> 70 an_ppe_wfectrl vpp stats request success
>>  1 an_ppe_wfectrl udp tunnel resource block 
>> add request received
>>  1 an_ppe_wfectrl udp tunnel resource block 
>> add request success
>>  1 an_ppe_wfectrl l3rm resource block update 
>> request received
>>  1 an_ppe_wfectrl l3rm resource block update 
>> request success
>>  1 an_ppe_wfectrl ue registration request 
>> received
>> 17 an_ppe_wfectrl ike msg received from 
>> ikemgr
>> 17 an_ppe_wfectrl ike msg send to network 
>> success
>>  3 an_ppe_wfectrl ipsec sa install msg 
>> received from ikemgr
>>  3 an_ppe_wfectrl ipsec sa install msg 
>> processed successfully
>>  1  an_ppe_vppctrl_input  Success in insert in 
>> ueidentifier
>>  1  an_ppe_vppctrl_input  Success  in insert in 
>> uecontext
>>  2  an_ppe_vppctrl_input  Downlink message sent to 
>> UE successfully
>>  1  an_ppe_vppctrl_input  TCP ESTABLISHMENT message 
>> received
>>  1  an_ppe_vppctrl_input  UPLINK NAS message received
>>  1  an_ppe_vppctrl_input  NAS TCP 

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi florin,

Thanks a lot for helping me out. I will try your patch and update you with
the result.


A general observation
==
In my setup, I think the SYN pkt is dropped much before the SYNS_RCVD
counter is incremented.

I have seen a lot of changes b/w 20.05 and 21.06 code, like in the graph
node tcp46_listen_inline() of VPP 20.05 the SYN  pkt trace function is
called towards the end i.e. after calling tcp_send_synack() but in 21.06
code, the tcp46_listen_trace_frame() is called at the very beginning
of tcp46_listen_inline() graph node.

I think moving the pkt trace macro to the end of the function is good
(placing it close to the line that calls tcp_send_synack), otherwise it can
mislead us and will not help debugging.



On Wed, Mar 16, 2022 at 10:22 AM Florin Coras 
wrote:

> Hi Vijay,
>
> That’s probably because packets are hitting an actual error and it seems
> the listen node is not reporting anything but syns received. Here’s a patch
> that might help [1]. It might not cherry-pick cleanly on 21.06.
>
> Regards,
> Florin
>
> [1] https://gerrit.fd.io/r/c/vpp/+/35654
>
> On Mar 15, 2022, at 7:54 PM, Vijay Kumar  wrote:
>
> Hi Florin,
>
> I have the output of show node counters (show errors) taken on both 20.05
> and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or
> tcp4-output etc.
>
> Please let me know why SYN rcvd counter itself is not incremented but in
> the earlier reply I already pasted the show trace ouput where we saw the
> SYN pkt was landed on tcp4-litsen node.
>
> VPP 20.05
> ===
> pp# show node counters
>CountNode  Reason
>  3   memif-input  not ip packet
>  10259 an_ppe_wfectrl wfectrl packets received
>  10259 an_ppe_wfectrl wfectrl replies sent
>  1 an_ppe_wfectrl wfectrl packet
> processing failed
>   8123 an_ppe_wfectrl session stat request
> received
>  1 an_ppe_wfectrl service construct config
> request received
>  1 an_ppe_wfectrl service construct config
> request success
>  1 an_ppe_wfectrl service config request
> received
>  1 an_ppe_wfectrl service config request
> success
>   1686 an_ppe_wfectrl dpi stats request
> received
>   1686 an_ppe_wfectrl dpi stats request success
> 70 an_ppe_wfectrl nat stats request
> received
> 70 an_ppe_wfectrl nat stats request success
> 70 an_ppe_wfectrl vpp stats request
> received
> 70 an_ppe_wfectrl vpp stats request success
>  1 an_ppe_wfectrl udp tunnel resource
> block add request received
>  1 an_ppe_wfectrl udp tunnel resource
> block add request success
>  1 an_ppe_wfectrl l3rm resource block
> update request received
>  1 an_ppe_wfectrl l3rm resource block
> update request success
>  1 an_ppe_wfectrl ue registration request
> received
> 17 an_ppe_wfectrl ike msg received from
> ikemgr
> 17 an_ppe_wfectrl ike msg send to network
> success
>  3 an_ppe_wfectrl ipsec sa install msg
> received from ikemgr
>  3 an_ppe_wfectrl ipsec sa install msg
> processed successfully
>  1  an_ppe_vppctrl_input  Success in insert in
> ueidentifier
>  1  an_ppe_vppctrl_input  Success  in insert in
> uecontext
>  2  an_ppe_vppctrl_input  Downlink message sent to
> UE successfully
>  1  an_ppe_vppctrl_input  TCP ESTABLISHMENT
> message received
>  1  an_ppe_vppctrl_input  UPLINK NAS message
> received
>  1  an_ppe_vppctrl_input  NAS TCP CLOSE message
> received
>  7   an_ppe_router_input  packets received in rtr
>  7   an_ppe_router_input  packets dropped to host.
> no tacptcp session found
> 17  an-ppe-isakmp4-output Total IKEV4 packets
> dispatched to network
> 17 an-ppe-isakmpmgr-input packets processed by
> isakmpmgr input plugin
> 17 an-ppe-isakmpmgr-input Received IKE packets
> 17 an-ppe-isakmpmgr-input Successfully sent ike
> message to Session Manager
>  5 an-ppe-isakmpmgr-input Received ike exchange
> AUTH packet
>  4 an-ppe-isakmpmgr-input Received ike exchange
> CREATE CHILD SA packet
>  1

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

That’s probably because packets are hitting an actual error and it seems the 
listen node is not reporting anything but syns received. Here’s a patch that 
might help [1]. It might not cherry-pick cleanly on 21.06. 

Regards,
Florin

[1] https://gerrit.fd.io/r/c/vpp/+/35654

> On Mar 15, 2022, at 7:54 PM, Vijay Kumar  wrote:
> 
> Hi Florin,
> 
> I have the output of show node counters (show errors) taken on both 20.05 and 
> 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or tcp4-output 
> etc. 
> 
> Please let me know why SYN rcvd counter itself is not incremented but in the 
> earlier reply I already pasted the show trace ouput where we saw the SYN pkt 
> was landed on tcp4-litsen node.
> 
> VPP 20.05
> ===
> pp# show node counters 
>CountNode  Reason
>  3   memif-input  not ip packet
>  10259 an_ppe_wfectrl wfectrl packets received
>  10259 an_ppe_wfectrl wfectrl replies sent
>  1 an_ppe_wfectrl wfectrl packet processing 
> failed
>   8123 an_ppe_wfectrl session stat request 
> received
>  1 an_ppe_wfectrl service construct config 
> request received
>  1 an_ppe_wfectrl service construct config 
> request success
>  1 an_ppe_wfectrl service config request 
> received
>  1 an_ppe_wfectrl service config request 
> success
>   1686 an_ppe_wfectrl dpi stats request received
>   1686 an_ppe_wfectrl dpi stats request success
> 70 an_ppe_wfectrl nat stats request received
> 70 an_ppe_wfectrl nat stats request success
> 70 an_ppe_wfectrl vpp stats request received
> 70 an_ppe_wfectrl vpp stats request success
>  1 an_ppe_wfectrl udp tunnel resource block 
> add request received
>  1 an_ppe_wfectrl udp tunnel resource block 
> add request success
>  1 an_ppe_wfectrl l3rm resource block update 
> request received
>  1 an_ppe_wfectrl l3rm resource block update 
> request success
>  1 an_ppe_wfectrl ue registration request 
> received
> 17 an_ppe_wfectrl ike msg received from ikemgr
> 17 an_ppe_wfectrl ike msg send to network 
> success
>  3 an_ppe_wfectrl ipsec sa install msg 
> received from ikemgr
>  3 an_ppe_wfectrl ipsec sa install msg 
> processed successfully
>  1  an_ppe_vppctrl_input  Success in insert in 
> ueidentifier
>  1  an_ppe_vppctrl_input  Success  in insert in 
> uecontext
>  2  an_ppe_vppctrl_input  Downlink message sent to UE 
> successfully
>  1  an_ppe_vppctrl_input  TCP ESTABLISHMENT message 
> received
>  1  an_ppe_vppctrl_input  UPLINK NAS message received
>  1  an_ppe_vppctrl_input  NAS TCP CLOSE message 
> received
>  7   an_ppe_router_input  packets received in rtr
>  7   an_ppe_router_input  packets dropped to host. no 
> tacptcp session found
> 17  an-ppe-isakmp4-output Total IKEV4 packets 
> dispatched to network
> 17 an-ppe-isakmpmgr-input packets processed by 
> isakmpmgr input plugin
> 17 an-ppe-isakmpmgr-input Received IKE packets
> 17 an-ppe-isakmpmgr-input Successfully sent ike 
> message to Session Manager
>  5 an-ppe-isakmpmgr-input Received ike exchange AUTH 
> packet
>  4 an-ppe-isakmpmgr-input Received ike exchange 
> CREATE CHILD SA packet
>  1 an-ppe-isakmpmgr-input Received ike exchange SA 
> INIT packet
>  7 an-ppe-isakmpmgr-input Received ike exchange 
> INFORMATIONAL packet
>  5arp-reply   ARP replies sent
>  4  session-queue Packets transmitted
> 17 ip4-udp-lookup No error
>  1   tcp4-listen  SYNs received
>  2tcp4-rcv-processPure ACKs received
>  1tcp4-establishedPackets pushed into rx fifo
>  2tcp4-establishedPure ACKs received
>  1tcp4-establishedFINs received
>  5   tcp4-output  Packets sent
>  

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

I have the output of show node counters (show errors) taken on both 20.05
and 21.06 vpp. In 21.06 I don't see any counters in tcp4-listen or
tcp4-output etc.

Please let me know why SYN rcvd counter itself is not incremented but in
the earlier reply I already pasted the show trace ouput where we saw the
SYN pkt was landed on tcp4-litsen node.

VPP 20.05
===
pp# show node counters
   CountNode  Reason
 3   memif-input  not ip packet
 10259 an_ppe_wfectrl wfectrl packets received
 10259 an_ppe_wfectrl wfectrl replies sent
 1 an_ppe_wfectrl wfectrl packet processing
failed
  8123 an_ppe_wfectrl session stat request
received
 1 an_ppe_wfectrl service construct config
request received
 1 an_ppe_wfectrl service construct config
request success
 1 an_ppe_wfectrl service config request
received
 1 an_ppe_wfectrl service config request
success
  1686 an_ppe_wfectrl dpi stats request received
  1686 an_ppe_wfectrl dpi stats request success
70 an_ppe_wfectrl nat stats request received
70 an_ppe_wfectrl nat stats request success
70 an_ppe_wfectrl vpp stats request received
70 an_ppe_wfectrl vpp stats request success
 1 an_ppe_wfectrl udp tunnel resource block
add request received
 1 an_ppe_wfectrl udp tunnel resource block
add request success
 1 an_ppe_wfectrl l3rm resource block
update request received
 1 an_ppe_wfectrl l3rm resource block
update request success
 1 an_ppe_wfectrl ue registration request
received
17 an_ppe_wfectrl ike msg received from
ikemgr
17 an_ppe_wfectrl ike msg send to network
success
 3 an_ppe_wfectrl ipsec sa install msg
received from ikemgr
 3 an_ppe_wfectrl ipsec sa install msg
processed successfully
 1  an_ppe_vppctrl_input  Success in insert in
ueidentifier
 1  an_ppe_vppctrl_input  Success  in insert in
uecontext
 2  an_ppe_vppctrl_input  Downlink message sent to
UE successfully
 1  an_ppe_vppctrl_input  TCP ESTABLISHMENT message
received
 1  an_ppe_vppctrl_input  UPLINK NAS message
received
 1  an_ppe_vppctrl_input  NAS TCP CLOSE message
received
 7   an_ppe_router_input  packets received in rtr
 7   an_ppe_router_input  packets dropped to host.
no tacptcp session found
17  an-ppe-isakmp4-output Total IKEV4 packets
dispatched to network
17 an-ppe-isakmpmgr-input packets processed by
isakmpmgr input plugin
17 an-ppe-isakmpmgr-input Received IKE packets
17 an-ppe-isakmpmgr-input Successfully sent ike
message to Session Manager
 5 an-ppe-isakmpmgr-input Received ike exchange
AUTH packet
 4 an-ppe-isakmpmgr-input Received ike exchange
CREATE CHILD SA packet
 1 an-ppe-isakmpmgr-input Received ike exchange SA
INIT packet
 7 an-ppe-isakmpmgr-input Received ike exchange
INFORMATIONAL packet
 5arp-reply   ARP replies sent
 4  session-queue Packets transmitted
17 ip4-udp-lookup No error
 1   tcp4-listen  SYNs received
 2tcp4-rcv-processPure ACKs received
 1tcp4-establishedPackets pushed into rx
fifo
 2tcp4-establishedPure ACKs received
 1tcp4-establishedFINs received
 5   tcp4-output  Packets sent
 7esp4-decrypt-tunESP pkts received
 5esp4-encrypt-tunESP pkts received
 5esp4-encrypt-tuntotal control pkts
received
 7ipsec4-tun-inputgood packets received
 2ip4-local   unknown ip protocol
 5 ethernet-input unknown vlan
vpp#
vpp#





VPP 21.06 output
===
vpp#
vpp# show version
vpp v21.06.0-3~g2a4af2b5d-dirty built by an-vijay_kumar on 

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

You won’t see the syn/syn-acks in traces because of the way they are generated. 
Nonetheless, you can verify that the syns were received with “show error” and 
check that the syn-acks were actually generated with a pcap trace rxtx 

Previously tracing worked because buffers were re-used and incoming packets 
were directly sent to output. More recently we’ve started dispatching syn-acks 
through the session layer in order to minimize size of tx bursts per dispatch. 

Regards,
Florin

> On Mar 15, 2022, at 6:08 PM, Vijay Kumar  wrote:
> 
> Hi Florin,
> 
> Thanks for the quick response.
> 
> It means it is expected to see the SYN pkts getting dropped after it is 
> terminated but I was not able to see any SYN-ACK going out of VPP. 
> 
> In both, "show trace" cmd output and also the pcap taken at dspatch graph 
> node level, I was not able to see the SYN-ACK going out.  The route to the 
> SYN-ACK destination (which is the original source that started SYN) is also 
> present in the ip fib output. The configuration is the same that was working 
> fine for me in vp 20.05. 
> 
> Is there anything that I can look at for the SYN-ACK not sending issue?
> 
> 
> 
> On Wed, 16 Mar 2022, 00:40 Florin Coras,  > wrote:
> Hi Vijay, 
> 
> I see an_ppe_router_input forwards syns to tcp-input and those packets are 
> delivered to tcp-listen, i.e., you’ve created a listener that’s matched by 
> the incoming traffic. 
> 
> The thing to keep in mind is that tcp terminates incoming flows and it does 
> not reuse buffers. That is, the syn hits the listen node and a syn-ack is 
> programmed for this new connection that still needs to complete the 
> handshake. The original syn packet is discarded and therefore you see it as a 
> drop. 
> 
> Regards, 
> Florin 
> 
>> On Mar 15, 2022, at 3:50 AM, Vijay Kumar > > wrote:
>> 
>> The is the output of show trace and show interface
>> 
>> 
>> Packet 36
>> 
>> 00:03:26:875694: dpdk-input
>>   VirtualFuncEthernet0/7/0 rx queue 0
>>   buffer 0x4c1b21: current data 0, length 138, buffer-pool 0, ref-count 1, 
>> totlen-nifb 0, trace handle 0x123
>>ext-hdr-valid 
>>l4-cksum-computed l4-cksum-correct 
>>   PKT MBUF: port 0, nb_segs 1, pkt_len 138
>> buf_len 2176, data_len 138, ol_flags 0x182, data_off 128, phys_addr 
>> 0xc266c8c0
>> packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>> rss 0xde0c21da fdir.hi 0x0 fdir.lo 0xde0c21da
>> Packet Offload Flags
>>   PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
>>   PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
>>   PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
>> Packet Types
>>   RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
>>   RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without 
>> extension headers
>>   RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet
>>   IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
>>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>> fragment id 0x, flags DONT_FRAGMENT
>> 00:03:26:875702: ethernet-input
>>   frame: flags 0x3, hw-if-index 3, sw-if-index 3
>>   IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
>> 00:03:26:875708: ip4-input
>>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>> fragment id 0x, flags DONT_FRAGMENT
>> 00:03:26:875714: ip4-lookup
>>   fib 2 dpo-idx 27 flow hash: 0x
>>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>> fragment id 0x, flags DONT_FRAGMENT
>> 00:03:26:875715: ip4-local
>> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>>   tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>>   fragment id 0x, flags DONT_FRAGMENT
>> 00:03:26:875717: ipsec4-tun-input
>>   IPSec: remote:20.20.99.215 spi:4213 (0x1075) sa:1 tun:0 seq 1 sa 
>> -2029505104
>> 00:03:26:875719: esp4-decrypt-tun
>>   esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 1 sa-seq 1 sa-seq-hi 0
>> 00:03:26:875801: ip4-input-no-checksum
>>   TCP: 44.44.44.44 -> 30.30.30.30
>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
>> fragment id 0x692e, flags DONT_FRAGMENT
>>   TCP: 45723 -> 1234
>> seq. 0x119d35d0 ack 0x
>> flags 0x02 SYN, tcp header: 40 bytes
>> window 64240, checksum 0xd6be
>> 00:03:26:875803: ip4-lookup
>>   fib 2 dpo-idx 25 flow hash: 0x
>>   TCP: 44.44.44.44 -> 30.30.30.30
>> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
>> fragment id 0x692e, flags DONT_FRAGMENT
>>   TCP: 45723 -> 1234
>> seq. 0x119d35d0 ack 0x
>> flags 0x02 SYN, tcp header: 40 bytes
>> window 64240, 

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

Hi Florin,

Thanks for the quick response.

It means it is expected to see the SYN pkts getting dropped after it is
terminated but I was not able to see any SYN-ACK going out of VPP.

In both, "show trace" cmd output and also the pcap taken at dspatch
graph node level, I was not able to see the SYN-ACK going out.  The route
to the SYN-ACK destination (which is the original source that started SYN)
is also present in the ip fib output. The configuration is the same that
was working fine for me in vp 20.05.

Is there anything that I can look at for the SYN-ACK not sending issue?



On Wed, 16 Mar 2022, 00:40 Florin Coras,  wrote:

> Hi Vijay,
>
> I see an_ppe_router_input forwards syns to tcp-input and those packets are
> delivered to tcp-listen, i.e., you’ve created a listener that’s matched by
> the incoming traffic.
>
> The thing to keep in mind is that tcp terminates incoming flows and it
> does not reuse buffers. That is, the syn hits the listen node and a syn-ack
> is programmed for this new connection that still needs to complete the
> handshake. The original syn packet is discarded and therefore you see it as
> a drop.
>
> Regards,
> Florin
>
> On Mar 15, 2022, at 3:50 AM, Vijay Kumar  wrote:
>
> The is the output of show trace and show interface
> 
>
> Packet 36
>
> 00:03:26:875694: dpdk-input
>   VirtualFuncEthernet0/7/0 rx queue 0
>   buffer 0x4c1b21: current data 0, length 138, buffer-pool 0, ref-count 1,
> totlen-nifb 0, trace handle 0x123
>ext-hdr-valid
>l4-cksum-computed l4-cksum-correct
>   PKT MBUF: port 0, nb_segs 1, pkt_len 138
> buf_len 2176, data_len 138, ol_flags 0x182, data_off 128, phys_addr
> 0xc266c8c0
> packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
> rss 0xde0c21da fdir.hi 0x0 fdir.lo 0xde0c21da
> Packet Offload Flags
>   PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
>   PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
>   PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
> Packet Types
>   RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
>   RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without
> extension headers
>   RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet
>   IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
> fragment id 0x, flags DONT_FRAGMENT
> 00:03:26:875702: ethernet-input
>   frame: flags 0x3, hw-if-index 3, sw-if-index 3
>   IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
> 00:03:26:875708: ip4-input
>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
> fragment id 0x, flags DONT_FRAGMENT
> 00:03:26:875714: ip4-lookup
>   fib 2 dpo-idx 27 flow hash: 0x
>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
> fragment id 0x, flags DONT_FRAGMENT
> 00:03:26:875715: ip4-local
> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>   tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>   fragment id 0x, flags DONT_FRAGMENT
> 00:03:26:875717: ipsec4-tun-input
>   IPSec: remote:20.20.99.215 spi:4213 (0x1075) sa:1 tun:0 seq 1 sa
> -2029505104
> 00:03:26:875719: esp4-decrypt-tun
>   esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 1 sa-seq 1 sa-seq-hi 0
> 00:03:26:875801: ip4-input-no-checksum
>   TCP: 44.44.44.44 -> 30.30.30.30
> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
> fragment id 0x692e, flags DONT_FRAGMENT
>   TCP: 45723 -> 1234
> seq. 0x119d35d0 ack 0x
> flags 0x02 SYN, tcp header: 40 bytes
> window 64240, checksum 0xd6be
> 00:03:26:875803: ip4-lookup
>   fib 2 dpo-idx 25 flow hash: 0x
>   TCP: 44.44.44.44 -> 30.30.30.30
> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
> fragment id 0x692e, flags DONT_FRAGMENT
>   TCP: 45723 -> 1234
> seq. 0x119d35d0 ack 0x
> flags 0x02 SYN, tcp header: 40 bytes
> window 64240, checksum 0xd6be
> 00:03:26:875803: ip4-local
> TCP: 44.44.44.44 -> 30.30.30.30
>   tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
>   fragment id 0x692e, flags DONT_FRAGMENT
> TCP: 45723 -> 1234
>   seq. 0x119d35d0 ack 0x
>   flags 0x02 SYN, tcp header: 40 bytes
>   window 64240, checksum 0xd6be
> 00:03:26:875805: an_ppe_router_input
>   AN_PPE_ROUTER_INPUT: sw_if_index 1, next index 3
>
> 00:03:26:875819: tcp4-input
>   [0:0][T] :::0->:::0 state CLOSED
>   TCP: 45723 -> 1234
> seq. 0x119d35d0 ack 0x
> flags 0x02 SYN, tcp header: 40 bytes
> window 64240, checksum 0xd6be
> 00:03:26:875826: tcp4-listen
>   1234 -> 45723 (LISTEN)
>
>
> Interface details
> 

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Florin Coras]

Hi Vijay, 

I see an_ppe_router_input forwards syns to tcp-input and those packets are 
delivered to tcp-listen, i.e., you’ve created a listener that’s matched by the 
incoming traffic. 

The thing to keep in mind is that tcp terminates incoming flows and it does not 
reuse buffers. That is, the syn hits the listen node and a syn-ack is 
programmed for this new connection that still needs to complete the handshake. 
The original syn packet is discarded and therefore you see it as a drop. 

Regards, 
Florin 

> On Mar 15, 2022, at 3:50 AM, Vijay Kumar  wrote:
> 
> The is the output of show trace and show interface
> 
> 
> Packet 36
> 
> 00:03:26:875694: dpdk-input
>   VirtualFuncEthernet0/7/0 rx queue 0
>   buffer 0x4c1b21: current data 0, length 138, buffer-pool 0, ref-count 1, 
> totlen-nifb 0, trace handle 0x123
>ext-hdr-valid 
>l4-cksum-computed l4-cksum-correct 
>   PKT MBUF: port 0, nb_segs 1, pkt_len 138
> buf_len 2176, data_len 138, ol_flags 0x182, data_off 128, phys_addr 
> 0xc266c8c0
> packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
> rss 0xde0c21da fdir.hi 0x0 fdir.lo 0xde0c21da
> Packet Offload Flags
>   PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
>   PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
>   PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
> Packet Types
>   RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
>   RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without 
> extension headers
>   RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet
>   IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
> fragment id 0x, flags DONT_FRAGMENT
> 00:03:26:875702: ethernet-input
>   frame: flags 0x3, hw-if-index 3, sw-if-index 3
>   IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
> 00:03:26:875708: ip4-input
>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
> fragment id 0x, flags DONT_FRAGMENT
> 00:03:26:875714: ip4-lookup
>   fib 2 dpo-idx 27 flow hash: 0x
>   IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
> tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
> fragment id 0x, flags DONT_FRAGMENT
> 00:03:26:875715: ip4-local
> IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
>   tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
>   fragment id 0x, flags DONT_FRAGMENT
> 00:03:26:875717: ipsec4-tun-input
>   IPSec: remote:20.20.99.215 spi:4213 (0x1075) sa:1 tun:0 seq 1 sa 
> -2029505104
> 00:03:26:875719: esp4-decrypt-tun
>   esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 1 sa-seq 1 sa-seq-hi 0
> 00:03:26:875801: ip4-input-no-checksum
>   TCP: 44.44.44.44 -> 30.30.30.30
> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
> fragment id 0x692e, flags DONT_FRAGMENT
>   TCP: 45723 -> 1234
> seq. 0x119d35d0 ack 0x
> flags 0x02 SYN, tcp header: 40 bytes
> window 64240, checksum 0xd6be
> 00:03:26:875803: ip4-lookup
>   fib 2 dpo-idx 25 flow hash: 0x
>   TCP: 44.44.44.44 -> 30.30.30.30
> tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
> fragment id 0x692e, flags DONT_FRAGMENT
>   TCP: 45723 -> 1234
> seq. 0x119d35d0 ack 0x
> flags 0x02 SYN, tcp header: 40 bytes
> window 64240, checksum 0xd6be
> 00:03:26:875803: ip4-local
> TCP: 44.44.44.44 -> 30.30.30.30
>   tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
>   fragment id 0x692e, flags DONT_FRAGMENT
> TCP: 45723 -> 1234
>   seq. 0x119d35d0 ack 0x
>   flags 0x02 SYN, tcp header: 40 bytes
>   window 64240, checksum 0xd6be
> 00:03:26:875805: an_ppe_router_input
>   AN_PPE_ROUTER_INPUT: sw_if_index 1, next index 3
> 
> 00:03:26:875819: tcp4-input
>   [0:0][T] :::0->:::0 state CLOSED
>   TCP: 45723 -> 1234
> seq. 0x119d35d0 ack 0x
> flags 0x02 SYN, tcp header: 40 bytes
> window 64240, checksum 0xd6be
> 00:03:26:875826: tcp4-listen
>   1234 -> 45723 (LISTEN)
> 
> 
> Interface details
> 
> vpp# show interface 
>   Name   IdxState  MTU (L3/IP4/IP6/MPLS) 
> Counter  Count 
> VirtualFuncEthernet0/7/0  3  up  1500/0/0/0 rx 
> packets  3922
> rx bytes  
> 239610
> tx 
> packets17
> tx bytes  
>   6704
> drops 
>

Re: [vpp-dev] VPP 21.06 - seeing TCP packet drops

[Vijay Kumar]

The is the output of show trace and show interface


Packet 36

00:03:26:875694: dpdk-input
  VirtualFuncEthernet0/7/0 rx queue 0
  buffer 0x4c1b21: current data 0, length 138, buffer-pool 0, ref-count 1,
totlen-nifb 0, trace handle 0x123
   ext-hdr-valid
   l4-cksum-computed l4-cksum-correct
  PKT MBUF: port 0, nb_segs 1, pkt_len 138
buf_len 2176, data_len 138, ol_flags 0x182, data_off 128, phys_addr
0xc266c8c0
packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0xde0c21da fdir.hi 0x0 fdir.lo 0xde0c21da
Packet Offload Flags
  PKT_RX_RSS_HASH (0x0002) RX packet with RSS hash result
  PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
  PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
Packet Types
  RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
  RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without
extension headers
  RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet
  IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
  IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
fragment id 0x, flags DONT_FRAGMENT
00:03:26:875702: ethernet-input
  frame: flags 0x3, hw-if-index 3, sw-if-index 3
  IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:97:04:19 802.1q vlan 1556
00:03:26:875708: ip4-input
  IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
fragment id 0x, flags DONT_FRAGMENT
00:03:26:875714: ip4-lookup
  fib 2 dpo-idx 27 flow hash: 0x
  IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
fragment id 0x, flags DONT_FRAGMENT
00:03:26:875715: ip4-local
IPSEC_ESP: 20.20.99.215 -> 80.80.80.80
  tos 0x00, ttl 64, length 120, checksum 0x21c9 dscp CS0 ecn NON_ECN
  fragment id 0x, flags DONT_FRAGMENT
00:03:26:875717: ipsec4-tun-input
  IPSec: remote:20.20.99.215 spi:4213 (0x1075) sa:1 tun:0 seq 1 sa
-2029505104
00:03:26:875719: esp4-decrypt-tun
  esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 1 sa-seq 1 sa-seq-hi 0
00:03:26:875801: ip4-input-no-checksum
  TCP: 44.44.44.44 -> 30.30.30.30
tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
fragment id 0x692e, flags DONT_FRAGMENT
  TCP: 45723 -> 1234
seq. 0x119d35d0 ack 0x
flags 0x02 SYN, tcp header: 40 bytes
window 64240, checksum 0xd6be
00:03:26:875803: ip4-lookup
  fib 2 dpo-idx 25 flow hash: 0x
  TCP: 44.44.44.44 -> 30.30.30.30
tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
fragment id 0x692e, flags DONT_FRAGMENT
  TCP: 45723 -> 1234
seq. 0x119d35d0 ack 0x
flags 0x02 SYN, tcp header: 40 bytes
window 64240, checksum 0xd6be
00:03:26:875803: ip4-local
TCP: 44.44.44.44 -> 30.30.30.30
  tos 0x00, ttl 64, length 60, checksum 0x3cfa dscp CS0 ecn NON_ECN
  fragment id 0x692e, flags DONT_FRAGMENT
TCP: 45723 -> 1234
  seq. 0x119d35d0 ack 0x
  flags 0x02 SYN, tcp header: 40 bytes
  window 64240, checksum 0xd6be
00:03:26:875805: an_ppe_router_input
  AN_PPE_ROUTER_INPUT: sw_if_index 1, next index 3

00:03:26:875819: tcp4-input
  [0:0][T] :::0->:::0 state CLOSED
  TCP: 45723 -> 1234
seq. 0x119d35d0 ack 0x
flags 0x02 SYN, tcp header: 40 bytes
window 64240, checksum 0xd6be
00:03:26:875826: tcp4-listen
  1234 -> 45723 (LISTEN)


Interface details

vpp# show interface
  Name   IdxState  MTU (L3/IP4/IP6/MPLS)
Counter  Count
VirtualFuncEthernet0/7/0  3  up  1500/0/0/0 rx
packets  3922
rx
bytes  239610
tx
packets17
tx
bytes6704
drops
3914
VirtualFuncEthernet0/7/0.1557 13 up  1500/0/0/0
VirtualFuncEthernet0/7/0.1556 14 up  1500/0/0/0 rx
packets22
rx
bytes5130
tx
packets17
tx
bytes6704
drops
  14
ip4
  19
VirtualFuncEthernet0/8/0  4 down 9000/0/0/0
gre1  2  up