Re: [vpp-dev]: Crash in Timer wheel infra

[Rajith PR via lists.fd.io]

Hi Andreas/Dave,

I did some experiments to debug the crash.

Firstly, I added some profiling code in vlib/main.c. The code basically is
added to know the timer_wheel slips that can cause such havocs(as mentioned
by Andreas). There are slippages as you can see from the data collected
from the core file.

Total slips = 21489 out of total 98472987 runs.

10 usec for the process timer wheel is something we may not be able to
achieve as we have a process node in which our solution runs. We would like
to increase 10 usec to 100 usec and observe the behaviour.  I tried
increasing the interval from 10 usec to 100 usec but then the process nodes
were scheduled very slow. What is the correct way to increase the interval?

Profiling code added,

   tw_start_time = vlib_time_now
 (vm);

 
   if (tw_start_time > tw_last_start_time) {
   interval = tw_start_time - tw_last_start_time;
   if (interval > PROCESS_TW_TIMER_INTERVAL) {

  tw_slips++;

  }
   tw_total_run++;

  }
   tw_last_start_time = tw_start_time;

   nm->data_from_advancing_timing_wheel =

TW 
(tw_timer_expire_timers_vec
)

((TWT 
(tw_timer_wheel) *) nm->timing_wheel, vlib_time_now
 (vm),
  nm->data_from_advancing_timing_wheel);

Secondly, during the debugging we got another crash (line 1904 of
vlib/main.c) below.
>From gdb we found that vec_len of nm->data_from_advancing_timing_wheel is
1. But nm->data_from_advancing_timing_wheel[0] = ~0.

1896 
  if (PREDICT_FALSE
1897

  (_vec_len

(nm->data_from_advancing_timing_wheel) > 0))1898

{1899 
  uword
 i
;1900

1901 
  for (i  =
0; i  < _vec_len

(nm->data_from_advancing_timing_wheel);1902

   i
++)1903

{1904

  u32  d
= nm->data_from_advancing_timing_wheel[i
];1905

  u32  di
 =
vlib_timing_wheel_data_get_index

(d);


Thanks,
Rajith

On Wed, Sep 2, 2020 at 8:15 PM Dave Barach (dbarach) 
wrote:

> It looks like vpp is crashing while expiring timers from the main thread
> process timer wheel. That’s not been reported before.
>
>
>
> You might want to dust off .../extras/deprecated/vlib/unix/cj.[ch], and
> make a circular log of timer pool_put operations to work out what’s
> happening.
>
>
>
> D.
>
>
>
> *From:* vpp-dev@lists.fd.io  *On Behalf Of *Rajith
> PR via lists.fd.io
> *Sent:* Wednesday, September 2, 2020 9:42 AM
> *To:* Dave Barach (dbarach) 
> *Cc:* vpp-dev 
> *Subject:* Re: [vpp-dev]: Crash in Timer wheel infra
>
>
>
> Thanks Dave for the quick analysis. Are there some Debug CLIs that I can
> run to analyse?
>
> We are not using the VPP timers as we have our own timer library. In VPP,
> we have added a couple of VPP nodes(process, internal and input). Could
> these be causing the problem?
>
>
>
> Thanks,
>
> Rajith
>
>
>
> On Wed, Sep 2, 2020 at 6:43 PM Dave Barach (dbarach) 
> wrote:
>
> Given the amount of soak-time / perf/scale / stress tes

[vpp-dev] how to configure ACL to match VXLAN info

[Davi Scofield]

Dear All,

 I have two questions on how to configure an VXLAN ACL on vpp:
1. how to match a special VXLAN session, include the soure IP,  the soure UDP 
port and the VNI field.
2. how to match the inner ethernet header and the inner IP header.
does anyone can share some configuretion or idea?  Thanks.


Best regards.
Davi

 -=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17349): https://lists.fd.io/g/vpp-dev/message/17349
Mute This Topic: https://lists.fd.io/mt/76726142/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] nat44 ed mode assertion failed when mulithread enabled in startup.conf

[dfliuc]

Dear VPP Folks,

I found vpp assertion failed when nat44  forward is enabled. I'm using vpp 
version on master branch.

I configured a simple nat scenario with nat44.
My startup.conf :

unix {
interactive
nodaemon
cli-listen  /run/vpp/cli.sock
cli-no-pager
poll-sleep-usec 100
}
api-trace {
on
}
api-segment {
gid vpp
}
socksvr {
default
}
cpu {
main-core 1
corelist-workers 2,3
}
dpdk {
uio-driver vfio-pci
vdev crypto_aesni_mb0,max_nb_queue_pairs=2,socket_id=0
dev :02:05.0 { name G0}
}
plugins {
plugin ikev2_plugin.so { disable }
}
nat {
endpoint-dependent
}
My config is shown below:

set int state G0 up
set dhcp client intfc G0
set interface nat44 out G0 output-feature
nat44 add interface address G0
*nat44 forwarding enable*

assertion error is below:
DBGvpp# 1: /root/work/vpp-master/src/plugins/nat/out2in_ed.c:451 
(create_bypass_for_fwd) assertion `thread_index == ed_value_get_thread_index 
(&value)' fails

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff3bfff700 (LWP 51834)]
0x7546c387 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:55
55        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install 
krb5-libs-1.15.1-37.el7_7.2.x86_64 libcom_err-1.42.9-16.el7.x86_64 
mbedtls-2.7.16-2.el7.x86_64 pkcs11-helper-1.11-3.el7.x86_64
(gdb) bt
#0  0x7546c387 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x7546da78 in __GI_abort () at abort.c:90
#2  0x0040858e in os_panic () at 
/root/work/vpp-master/src/vpp/vnet/main.c:371
#3  0x7620daa6 in debugger () at 
/root/work/vpp-master/src/vppinfra/error.c:84
#4  0x7620de91 in _clib_error (how_to_die=2, function_name=0x0, 
line_number=0,
fmt=0x7fffaff07040 "%s:%d (%s) assertion `%s' fails") at 
/root/work/vpp-master/src/vppinfra/error.c:143
#5  0x7fffafdb058d in create_bypass_for_fwd (sm=0x7fffaff68ec0 , 
b=0x10026d68c0,
ip=0x10026d69ce, rx_fib_index=0, thread_index=1) at 
/root/work/vpp-master/src/plugins/nat/out2in_ed.c:451
#6  0x7fffafdb0a54 in create_bypass_for_fwd_worker (sm=0x7fffaff68ec0 
, b=0x10026d68c0,
ip=0x10026d69ce, rx_fib_index=0) at 
/root/work/vpp-master/src/plugins/nat/out2in_ed.c:521
#7  0x7fffafdb2ca2 in nat44_ed_out2in_slow_path_node_fn_inline 
(vm=0x7fffb7312cc0, node=0x7fffb76618c0,
frame=0x7fffb75a07c0) at /root/work/vpp-master/src/plugins/nat/out2in_ed.c:1205
#8  0x7fffafdb36c1 in nat44_ed_out2in_slowpath_node_fn_hsw 
(vm=0x7fffb7312cc0, node=0x7fffb76618c0,
frame=0x7fffb75a07c0) at /root/work/vpp-master/src/plugins/nat/out2in_ed.c:1407
#9  0x77edec68 in dispatch_node (vm=0x7fffb7312cc0, 
node=0x7fffb76618c0, type=VLIB_NODE_TYPE_INTERNAL,
dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x7fffb75a07c0, 
last_time_stamp=2247957037122854)
at /root/work/vpp-master/src/vlib/main.c:1194
#10 0x77edf372 in dispatch_pending_node (vm=0x7fffb7312cc0, 
pending_frame_index=1,
last_time_stamp=2247957037122854) at /root/work/vpp-master/src/vlib/main.c:1353
#11 0x77ee13be in vlib_main_or_worker_loop (vm=0x7fffb7312cc0, 
is_main=0)
at /root/work/vpp-master/src/vlib/main.c:1848
#12 0x77ee1e40 in vlib_worker_loop (vm=0x7fffb7312cc0) at 
/root/work/vpp-master/src/vlib/main.c:1982
#13 0x77f22e62 in vlib_worker_thread_fn (arg=0x7fffb4124e00)
at /root/work/vpp-master/src/vlib/threads.c:1861
#14 0x7622c870 in clib_calljmp () at 
/root/work/vpp-master/src/vppinfra/longjmp.S:123
#15 0x7fff3bffebb0 in ?? ()
#16 0x77f1ca52 in vlib_worker_thread_bootstrap_fn (arg=0x7fffb4124e00)
at /root/work/vpp-master/src/vlib/threads.c:585
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
(gdb)

without " nat44 forwarding enable ", no above assertion failed.
or
without  "corelist-workers 2,3" in startup.conf, no above assertion failed
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17348): https://lists.fd.io/g/vpp-dev/message/17348
Mute This Topic: https://lists.fd.io/mt/76724560/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Reminder: VPP 20.09 RC1 is tomorrow 09 Sep 2020

[Andrew Yourtchenko]

Hi all,

Just a friendly reminder that tomorrow 18:00 UTC is our milestone RC1, at which 
point we are making a new LTS branch stable/2009 off master and building the 
RC1 artifacts.

Committers - please avoid merging risky commits to master until my email that 
RC1 milestone is completed.

--a /* your friendly 20.09 release manager */-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17347): https://lists.fd.io/g/vpp-dev/message/17347
Mute This Topic: https://lists.fd.io/mt/76715991/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] ip full reassembly - is_custom_app field broken ?

[Satya Murthy]

Ok. I think, required changes would be more complex / involved than I initially 
imagined.
Can you please make these changes.

--
Thanks & Regards,
Murthy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17346): https://lists.fd.io/g/vpp-dev/message/17346
Mute This Topic: https://lists.fd.io/mt/76705450/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] ip full reassembly - is_custom_app field broken ?

[Klement Sekera via lists.fd.io]

That’s a very good point. Looking at vnet_buffer, there seems to be some space 
left in it for this case …

  190   struct  

  191   {   
 
  192 /* input variables */ 
 
  193 struct

  194 { 
 
  195   u32 next_index;   /* index of next node - used by custom 
apps */
  196   u32 error_next_index; /* index of next node if error - 
used by custom apps */
  197 };

  198 /* handoff variables */   
 
  199 struct

  200 { 
 
  201   u16 owner_thread_index; 
 
  202 };

  203   };  

we could put is_custom flag next to owner_thread_index; and if it is set then 
the handoff code would send to ip4-full-reass-custom instead of ip4-full-reass. 
We will need a fq_custom_index as well to be able to do that.

this would also be needed for ip6-full and ip4-sv flavours … 

would you like to take this or should I write the code?

Thanks,
Klement

> On 8 Sep 2020, at 18:28, Satya Murthy  wrote:
> 
> Thanks Klement for the quick response.
> 
> I can make the changes you suggested. But, one major doubt I have is on the 
> HANDOFF scenario.
> 
> Let's say, as part of custom-reasm-node, if the packet is decided to be 
> handed off to another thread, then the next_node is ALWAYS getting set as 
> "ip4-full-reassembly-handoff". 
> Shouldn't it go to custom-reasm-node ? This also needs to be decided based on 
> "is_custom_app" , right ?
> 
> Please let me know your inputs.
> 
> -- 
> Thanks & Regards,
> Murthy 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17345): https://lists.fd.io/g/vpp-dev/message/17345
Mute This Topic: https://lists.fd.io/mt/76705450/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] ip full reassembly - is_custom_app field broken ?

[Satya Murthy]

Thanks Klement for the quick response.

I can make the changes you suggested. But, one major doubt I have is on the 
HANDOFF scenario.

Let's say, as part of custom-reasm-node, if the packet is decided to be handed 
off to another thread, then the next_node is ALWAYS getting set as 
"ip4-full-reassembly-handoff".
Shouldn't it go to custom-reasm-node ? This also needs to be decided based on 
"is_custom_app" , right ?

Please let me know your inputs.

--
Thanks & Regards,
Murthy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17344): https://lists.fd.io/g/vpp-dev/message/17344
Mute This Topic: https://lists.fd.io/mt/76705450/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Emergency Maintenance

[Vanessa Valderrama]

We are aware of latency issues on jenkins.fd.io. Jenkins requires a
restart to resolve this issue.

Thank you,
Vanessa
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17343): https://lists.fd.io/g/vpp-dev/message/17343
Mute This Topic: https://lists.fd.io/mt/76711640/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] ip full reassembly - is_custom_app field broken ?

[Klement Sekera via lists.fd.io]

Hi Satya,

so this is obviously unfinished. Would you mind writing the code for that? You 
just need to steal a piece of code from ip4_sv_reass.c to do that ;-) and then 
steer your packets at custom node.

   986 /* *INDENT-OFF* */ 
   987 VLIB_REGISTER_NODE (ip4_sv_reass_custom_node) = {
   
   988 .name = "ip4-sv-reassembly-custom-next", 
   989 .vector_size = sizeof (u32), 
   990 .format_trace = format_ip4_sv_reass_trace,   
   991 .n_errors = ARRAY_LEN (ip4_sv_reass_error_strings),  
   
   992 .error_strings = ip4_sv_reass_error_strings, 
   993 .n_next_nodes = IP4_SV_REASSEMBLY_N_NEXT,
   
   994 .next_nodes =
   
   995 {
   
   996 [IP4_SV_REASSEMBLY_NEXT_INPUT] = "ip4-input",
   
   997 [IP4_SV_REASSEMBLY_NEXT_DROP] = "ip4-drop",  
   
   998 [IP4_SV_REASSEMBLY_NEXT_HANDOFF] = 
"ip4-sv-reassembly-handoff", 
   999  
   
  1000 }, 
  1001 };   

  1002 /* *INDENT-ON* */
   
  1003  
   
  1004 VLIB_NODE_FN (ip4_sv_reass_custom_node) (vlib_main_t * vm, 
  1005  vlib_node_runtime_t * node, 

  1006  vlib_frame_t * frame) 
  1007 {  
  1008   return ip4_sv_reass_inline (vm, node, frame, false /* is_feature */ ,  
   
  1009   false /* is_output_feature */ ,
   
  1010   true /* is_custom */ );
   
  1011 }

Thanks,
Klement

> On 8 Sep 2020, at 11:56, Satya Murthy  wrote:
> 
> [Edited Message Follows]
> 
> Hi,
> 
> Looking at the ip4 full reassembly graph nodes for the purpose of punting the 
> fragments and getting the reassembled packets from our custom graph node.
> However, from the code it seems that,  is_custom_app flag is effectively 
> disabled.
> 
> I see that the node function is always getting called with is_custom_app = 
> false.
> 
> VLIB_NODE_FN (ip4_full_reass_node) (vlib_main_t * vm,
> vlib_node_runtime_t * node,
> vlib_frame_t * frame)
> {
>   return ip4_full_reass_inline (vm, node, frame, false /* is_feature */ ,
> false /* is_custom_app */ );    always called 
> with  FALSE
> }
> 
> We also observed that sv_reassembly has custom_app functionality. But, we 
> need fully reassembled packet, and hence looking at full_reassembly 
> functionality.
> 
> Is there a way to use this is_custom_app flag functionality  in 
> full_reassembly ? 
> 
> -- 
> Thanks & Regards,
> Murthy 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17342): https://lists.fd.io/g/vpp-dev/message/17342
Mute This Topic: https://lists.fd.io/mt/76705450/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Query on ip4-local / ip4-unicast Feature Arc sequence

[Klement Sekera via lists.fd.io]

Hi Satya,

it’s also necessary to enable full reassembly feature along with your feature 
by calling ip4_full_reass_enable_disable_with_refcnt().

You can take a look at NAT code which already does that - 
snat_interface_add_del().

Regards,
Klement

> On 4 Sep 2020, at 14:48, Satya Murthy  wrote:
> 
>  
> Hi ,
>  
> We wanted to have our custom graph node to receive all the IP packets that 
> are destined to our local interfaces.
> However, we want the packets after reassmbly is done by the vnet reassembly 
> code, if needed.
>  
> Hence, we added a custom_feature on the ip4-unicast feature-arc as below.
>  
> VNET_FEATURE_INIT (custom_feature, static) =
> {
> .arc_name = "ip4-unicast",
> .node_name = "custom_feature",
> .runs_after = VNET_FEATURES ("ip4-full-reassembly-feature"),
> };
>  
> Also, this feature is enabled on the interface in question.
>  
> After this code, show features verbose is showing as below.
>  
> [16] ip4-unicast:
>   [ 0]: ip4-rx-urpf-loose
>   [ 1]: ip4-rx-urpf-strict
>   [ 2]: svs-ip4
> ..
>   [39]: ip4-full-reassembly-feature
>   [40]: custom_feature
> ..
>  
> But, when the two fragments of a packet are received, they are given to 
> custom_feature node without reassembly.
>  
> show trace is showing as below.
>  
> 00:03:35:565905: ethernet-input
>   frame: flags 0x3, hw-if-index 1, sw-if-index 1
>   IP4: fa:16:3e:e0:70:43 -> fa:16:3e:eb:fb:74
> 00:03:35:565942: ip4-input-no-checksum
>   UDP: 10.10.5.2 -> 10.10.5.1
> tos 0x00, ttl 64, length 1500, checksum 0x36fa dscp CS0 ecn NON_ECN
> fragment id 0x0001, flags MORE_FRAGMENTS
>   UDP: 53 -> 53
> length 2008, checksum 0x0585
> 00:03:35:565968: custom_feature_node
>   custome_feature_node: sw_if_index 1, next_worker 5, buffer 0x4c7cc8
>  
> I wanted to put our custom_feature in ip4-local feature arc, as it seems more 
> logical.
> However, i dont see reassembly feature in that feature arc.
>  
> Is there any work around to achieve this functionality (or)
> anything wrong that I am doing here.
>  
> Appreciate any inputs regarding this.
>  
> -- 
> Thanks & Regards,
> Murthy 

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17341): https://lists.fd.io/g/vpp-dev/message/17341
Mute This Topic: https://lists.fd.io/mt/76628685/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] ip full reassembly - is_custom_app field broken ?

[Satya Murthy]

[Edited Message Follows]

Hi,

Looking at the ip4 full reassembly graph nodes for the purpose of punting the 
fragments and getting the reassembled packets from our custom graph node.
However, from the code it seems that,  is_custom_app flag is effectively 
disabled.

I see that the node function is always getting called with is_custom_app = 
false.

VLIB_NODE_FN (ip4_full_reass_node) (vlib_main_t * vm,
vlib_node_runtime_t * node,
vlib_frame_t * frame)
{
return ip4_full_reass_inline (vm, node, frame, false /* is_feature */ ,
false /* is_custom_app */ );    always called with  FALSE
}

We also observed that sv_reassembly has custom_app functionality. But, we need 
fully reassembled packet, and hence looking at full_reassembly functionality.

Is there a way to use this is_custom_app flag functionality  in full_reassembly 
?

--
Thanks & Regards,
Murthy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17340): https://lists.fd.io/g/vpp-dev/message/17340
Mute This Topic: https://lists.fd.io/mt/76705450/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] ip full reassembly - is_custom_app field broken ?

[Satya Murthy]

Hi,

Looking at the ip4 full reassembly graph nodes for the purpose of punting the 
fragments and getting the reassembled packets from our custom graph node.
However, from the code it seems that,  is_custom_app flag is effectively 
disabled.

I see that the node function is always getting called with is_custom_app = 
false.

VLIB_NODE_FN (ip4_full_reass_node) (vlib_main_t * vm,
vlib_node_runtime_t * node,
vlib_frame_t * frame)
{
return ip4_full_reass_inline (vm, node, frame, false /* is_feature */ ,
false /* is_custom_app */ );    always called with  FALSE
}

Is there a way to use this is_custom_app flag functionality ?

--
Thanks & Regards,
Murthy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#17340): https://lists.fd.io/g/vpp-dev/message/17340
Mute This Topic: https://lists.fd.io/mt/76705450/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub  [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-