Thanks Dave, will check it out. -Rajith
On Tue, Nov 17, 2020 at 8:40 PM <v...@barachs.net> wrote: > Let’s be clear: you’re seeing a crash in a *modified fork* of vpp-19.08. > I’ve never seen such a crash myself, nor has one such been reported by > anyone else to my knowledge. > > > > That having been written, all signs point to the volatile int ** vector > vl_api_queue_cursizes having had an accident: > > > > static void > > memclnt_queue_callback (vlib_main_t * vm) > > { > > <snip> > > for (i = 0; i < vec_len (vl_api_queue_cursizes); i++) > > { > > if (*vl_api_queue_cursizes[i]) > > { > > vm->queue_signal_pending = 1; > > vm->api_queue_nonempty = 1; > > vlib_process_signal_event (vm, vl_api_clnt_node.index, > > /* event_type */ QUEUE_SIGNAL_EVENT, > > /* event_data */ 0); > > break; > > } > > } > > <snip> > > } > > > > Try a debug image. Try capturing “i”, and the value > vl_api_queue_cursizes[i] before dereferencing as a pointer. Add a couple of > global variables with names which won’t collide with anything else: > > > > void int oingo_save_i; > > void oingo_save_cursizep; > > > > In the loop, set: > > oingo_save_i = i; > > oingo_save_cursizep = vl_api_queue_cursizes[i]; > > > > if(*vl_api_queue_cursizes[i]) > > <etc> > > > > Capture a coredump. It should be obvious why the reference blows up. If > you can, change your custom signal handler so that the faulting virtual > address is as obvious as possible. > > > > Beyond that, you’re on your own. > > > > HTH... Dave > > > > *From:* vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> *On Behalf Of *Rajith > PR via lists.fd.io > *Sent:* Tuesday, November 17, 2020 7:03 AM > *To:* vpp-dev <vpp-dev@lists.fd.io> > *Subject:* [vpp-dev]: Crash in memclnt_queue_callback(). > > > > Hi All, > > > > We are seeing a random crash in *VPP-19.08*. The crash is occurring in > memclnt_queue_callback > and it is in code that we are not using. Any pointers to fix the crash > would be helpful. > > > > *Complete Call Stack:* > > > > Thread 1 (Thread 0x7fe728f43d00 (LWP 189)): > > #0 0x00007fe728049492 in __GI___waitpid (pid=732, > stat_loc=stat_loc@entry=0x7fe6f9ebeed8, options=options@entry=0) > > at ../sysdeps/unix/sysv/linux/waitpid.c:30 > > #1 0x00007fe727fb4177 in do_system (line=<optimized out>) at > ../sysdeps/posix/system.c:149 > > #2 0x00007fe728ad6457 in bd_signal_handler_cb (signo=11) at > /development/librtbrickinfra/bd/src/bd.c:770 > > #3 0x00007fe71c90fbf7 in rtb_bd_signal_handler (signo=11) at > /development/libvpp/src/vlib/unix/main.c:80 > > #4 0x00007fe71c90ff92 in unix_signal_handler (signum=11, si=0x7fe6f9ebf7b0, > uc=0x7fe6f9ebf680) > > at /development/libvpp/src/vlib/unix/main.c:180 > > #5 <signal handler called> > > #6 memclnt_queue_callback (vm=0x7fe71cb49e80 <vlib_global_main>) at > /development/libvpp/src/vlibmemory/memory_api.c:96 > > #7 0x00007fe71c8a9258 in vlib_main_or_worker_loop (vm=0x7fe71cb49e80 > <vlib_global_main>, is_main=1) > > at /development/libvpp/src/vlib/main.c:1799 > > #8 0x00007fe71c8a9f9d in vlib_main_loop (vm=0x7fe71cb49e80 > <vlib_global_main>) at /development/libvpp/src/vlib/main.c:1982 > > #9 0x00007fe71c8aac7b in vlib_main (vm=0x7fe71cb49e80 <vlib_global_main>, > input=0x7fe6f9ebffb0) at /development/libvpp/src/vlib/main.c:2209 > > #10 0x00007fe71c911745 in thread0 (arg=140630595772032) at > /development/libvpp/src/vlib/unix/main.c:666 > > #11 0x00007fe71c568560 in clib_calljmp () from > /usr/local/lib/libvppinfra.so.1.0.1 > > #12 0x00007ffe85672480 in ?? () > > #13 0x00007fe71c911cbb in vlib_unix_main (argc=42, argv=0x563be4aaa5a0) at > /development/libvpp/src/vlib/unix/main.c:736 > > #14 0x00007fe71e0bc9eb in rtb_vpp_core_init (argc=42, argv=0x563be4aaa5a0) at > /development/libvpp/src/vpp/vnet/main.c:483 > > #15 0x00007fe71e18fba2 in rtb_vpp_main () at > /development/libvpp/src/vpp/rtbrick/rtb_vpp_main.c:113 > > #16 0x00007fe728ad5e46 in bd_load_daemon_lib (dmn_lib_cfg=0x7fe728cf2820 > <bd_json_global+21408>) > > ---Type <return> to continue, or q <return> to quit--- > > at /development/librtbrickinfra/bd/src/bd.c:627 > > #17 0x00007fe728ad5ef1 in bd_load_all_daemon_libs () at > /development/librtbrickinfra/bd/src/bd.c:646 > > #18 0x00007fe728ad7362 in bd_start_process () at > /development/librtbrickinfra/bd/src/bd.c:1128 > > #19 0x00007fe72583c860 in bds_bd_init () at > /development/librtbrickinfra/libbds/code/bds/src/bds.c:657 > > #20 0x00007fe7258c8a30 in pubsub_bd_init_expiry (data=0x0) at > /development/librtbrickinfra/libbds/code/pubsub/src/pubsub_helper.c:1444 > > #21 0x00007fe7285d6640 in timer_dispatch (item=0x563be68209b0, > p=QB_LOOP_HIGH) at /development/librtbrickinfra/libqb/lib/loop_timerlist.c:56 > > #22 0x00007fe7285d25d6 in qb_loop_run_level (level=0x563be47a17a0) at > /development/librtbrickinfra/libqb/lib/loop.c:43 > > #23 0x00007fe7285d2d4b in qb_loop_run (lp=0x563be47a1730) at > /development/librtbrickinfra/libqb/lib/loop.c:210 > > #24 0x00007fe7285e461e in lib_qb_service_start_event_loop () at > /development/librtbrickinfra/libqb/lib/wrapper/lib_qb_service.c:257 > > #25 0x0000563be3d9f153 in main () > > (gdb) > > > > *Code Snippet:* > > > > 94 for (i = 0; i < vec_len (vl_api_queue_cursizes); i++) > > 95 { > 96 if (*vl_api_queue_cursizes[i]) > <---------------------------------Crashed here > 97 { > 98 vm->queue_signal_pending = 1; > 99 vm->api_queue_nonempty = 1; > 100 vlib_process_signal_event (vm, vl_api_clnt_node.index, > 101 /* event_type */ QUEUE_SIGNAL_EVENT, > 102 /* event_data */ 0); > 103 break; > 104 } > 105 } > > > > Thanks, > > Rajith > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18060): https://lists.fd.io/g/vpp-dev/message/18060 Mute This Topic: https://lists.fd.io/mt/78314224/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-