You can press the “cherrypick” button as easily as Florin... Hint... From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Raj Kumar Sent: Monday, August 17, 2020 5:09 PM To: Ayush Gautam <raj.gauta...@gmail.com> Cc: Florin Coras <fcoras.li...@gmail.com>; vpp-dev <vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] VPP 2005 is crashing on stopping the VCL applications #vpp-hoststack
Hi Florin, Can we please have the fix[1] on "stable/2005" branch. [1] https://gerrit.fd.io/r/c/vpp/+/28173 Thanks, -Raj On Wed, Aug 5, 2020 at 7:30 PM Raj Kumar via lists.fd.io<http://lists.fd.io> <raj.gautam25=gmail....@lists.fd.io<mailto:gmail....@lists.fd.io>> wrote: Hi Florin, Yes , this[1] fixed the issue. thanks, -Raj On Wed, Aug 5, 2020 at 1:57 AM Florin Coras <fcoras.li...@gmail.com<mailto:fcoras.li...@gmail.com>> wrote: Hi Raj, Does this [1] fix the issue? Regards, Florin [1] https://gerrit.fd.io/r/c/vpp/+/28173 On Aug 4, 2020, at 8:24 AM, Raj Kumar <raj.gauta...@gmail.com<mailto:raj.gauta...@gmail.com>> wrote: Hi Florin, I tried vppcom_epoll_wait() on 2 different servers by using a simple application ( with only 1 worker thread) . But, still vppcom_epoll_wait() is not being timed out if I do not use use-mq-eventfd . Here are the server details - server 1 - Red hat 7.5 , vpp release - 20.01 server 2 - Centos 8.1 , vpp release - 20.05 thanks, -Raj On Tue, Aug 4, 2020 at 10:24 AM Florin Coras <fcoras.li...@gmail.com<mailto:fcoras.li...@gmail.com>> wrote: Hi Raj, Interesting. For some reason, the message queue’s underlying pthread_cond_timedwait does not work in your setup. Not sure what to make of that, unless maybe you’re trying to epoll wait from multiple threads that share the same message queue. That is not supported since each thread must have its own message queue, i.e., all threads that call epoll should be registered as workers. Alternatively, some form of locking or vls, instead of vcl, should be used. The downside to switching the message queue to eventfd notifications, instead of mutext/condvar, is that waits are inefficient, i.e., they act pretty much like spinlocks. Do keep that in mind. Regards, Florin On Aug 4, 2020, at 6:37 AM, Raj Kumar <raj.gauta...@gmail.com<mailto:raj.gauta...@gmail.com>> wrote: Hi Florin, After adding use-mq-eventfd in VCL configuration, it is working as expected. Thanks! for your help. vcl { rx-fifo-size 4000000 tx-fifo-size 4000000 app-scope-local app-scope-global use-mq-eventfd api-socket-name /tmp/vpp-api.sock } thanks, -Raj On Tue, Aug 4, 2020 at 12:08 AM Florin Coras <fcoras.li...@gmail.com<mailto:fcoras.li...@gmail.com>> wrote: Hi Raj, Glad to hear that issue is solved. What vcl config are you running? Did you configure use-mq-eventd? Regards, Florin On Aug 3, 2020, at 8:33 PM, Raj Kumar <raj.gauta...@gmail.com<mailto:raj.gauta...@gmail.com>> wrote: Hi Florin, This issue is resolved now. In my application, on receiving the kill signal main thread was sending phread_cancel() to the child thread because of that child thread was not exiting gracefully. I have one question; it seems that vppcom_epoll_wait(epfd, rcvEvents, MAX_RETURN_EVENTS, 60000.0) is not returning after timed out if the timeout value is a non zero value. It timed out only if the timeout value is 0. The issue that I am facing is that if there is no traffic at all ( receiver is just listening on the connections ) then the worker thread is not exiting as it is blocked by vppcom_epoll_wait(). Thanks, -Raj On Wed, Jul 29, 2020 at 11:23 PM Florin Coras <fcoras.li...@gmail.com<mailto:fcoras.li...@gmail.com>> wrote: Hi Raj, In that case it should work. Just from the trace lower it’s hard to figure out what exactly happened. Also, keep in mind that vcl is not thread safe, so make sure you’re not trying to share sessions or allow two workers to interact with the message queue(s) at the same time. Regards, Florin On Jul 29, 2020, at 8:17 PM, Raj Kumar <raj.gauta...@gmail.com<mailto:raj.gauta...@gmail.com>> wrote: Hi Florin, I am using kill <pid> to stop the application. But , the application has a kill signal handler and after receiving the signal it is exiting gracefully. About vppcom_app_exit, I think this function is registered with atexit() inside vppcom_app_create() so it should call when the application exits. Even, I also tried this vppcom_app_exit() explicitly before exiting the application but still I am seeing the same issue. My application is a multithreaded application. Can you please suggest some cleanup functions ( vppcom functions) that I should call before exiting a thread and the main application for a proper cleanup. I also tried vppcom_app_destroy() before exiting the main application but still I am seeing the same issue. thanks, -Raj On Wed, Jul 29, 2020 at 5:34 PM Florin Coras <fcoras.li...@gmail.com<mailto:fcoras.li...@gmail.com>> wrote: Hi Raj, Does stopping include a call to vppcom_app_exit or killing the applications? If the latter, the apps might be killed with some mutexes/spinlocks held. For now, we only support the former. Regards, Florin > On Jul 29, 2020, at 1:49 PM, Raj Kumar > <raj.gauta...@gmail.com<mailto:raj.gauta...@gmail.com>> wrote: > > Hi, > In my UDP application , I am using VPP host stack to receive packets and > memIf to transmit packets. There are a total 6 application connected to VPP. > if I stop the application(s) then VPP is crashing. In vpp configuration , 4 > worker threads are configured. If there is no worker thread configured then > I do not see this crash. > Here is the VPP task trace - > (gdb) bt > #0 0x00007ffff51818df in raise () from /lib64/libc.so.6 > #1 0x00007ffff516bcf5 in abort () from /lib64/libc.so.6 > #2 0x000055555555c123 in os_panic () at > /usr/src/debug/vpp-20.05-9~g0bf9c294c_dirty.x86_64/src/vpp/vnet/main.c:366 > #3 0x00007ffff6b466bb in vlib_worker_thread_barrier_sync_int > (vm=0x7ffff6d78200 <vlib_global_main>, func_name=<optimized out>) > at > /usr/src/debug/vpp-20.05-9~g0bf9c294c_dirty.x86_64/src/vlib/threads.c:1529 > #4 0x00007ffff7bc5ef0 in vl_msg_api_handler_with_vm_node > (am=am@entry=0x7ffff7dd2ea0 <api_global_main>, > vlib_rp=vlib_rp@entry=0x7fee7c001000, the_msg=0x7fee7c02bbd8, > vm=vm@entry=0x7ffff6d78200 <vlib_global_main>, > node=node@entry=0x7fffb6295000, is_private=is_private@entry=1 '\001') > at > /usr/src/debug/vpp-20.05-9~g0bf9c294c_dirty.x86_64/src/vlibapi/api_shared.c:596 > #5 0x00007ffff7bb000f in void_mem_api_handle_msg_i (is_private=1 '\001', > node=0x7fffb6295000, vm=0x7ffff6d78200 <vlib_global_main>, > vlib_rp=0x7fee7c001000, am=0x7ffff7dd2ea0 <api_global_main>) > at > /usr/src/debug/vpp-20.05-9~g0bf9c294c_dirty.x86_64/src/vlibmemory/memory_api.c:698 > #6 vl_mem_api_handle_msg_private (vm=vm@entry=0x7ffff6d78200 > <vlib_global_main>, node=node@entry=0x7fffb6295000, reg_index=<optimized out>) > at > /usr/src/debug/vpp-20.05-9~g0bf9c294c_dirty.x86_64/src/vlibmemory/memory_api.c:762 > #7 0x00007ffff7bbe346 in vl_api_clnt_process (vm=<optimized out>, > node=0x7fffb6295000, f=<optimized out>) > at > /usr/src/debug/vpp-20.05-9~g0bf9c294c_dirty.x86_64/src/vlibmemory/vlib_api.c:370 > #8 0x00007ffff6b161d6 in vlib_process_bootstrap (_a=<optimized out>) > at /usr/src/debug/vpp-20.05-9~g0bf9c294c_dirty.x86_64/src/vlib/main.c:1502 > #9 0x00007ffff602ac0c in clib_calljmp () from /lib64/libvppinfra.so.20.05 > #10 0x00007fffb5e93dd0 in ?? () > #11 0x00007ffff6b19821 in dispatch_process (vm=0x7ffff6d78200 > <vlib_global_main>, p=0x7fffb6295000, last_time_stamp=15931923011231136, > f=0x0) at > /usr/src/debug/vpp-20.05-9~g0bf9c294c_dirty.x86_64/src/vppinfra/types.h:133 > #12 0x7f0f660000009024 in ?? () > > > Thanks, > -Raj >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17251): https://lists.fd.io/g/vpp-dev/message/17251 Mute This Topic: https://lists.fd.io/mt/75873900/21656 Mute #vpp-hoststack: https://lists.fd.io/g/fdio+vpp-dev/mutehashtag/vpp-hoststack Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
