Done patch: https://gerrit.fd.io/r/13070
1. make use of glic backtrace in execinfo.h. the old clib_backtrace is changed to clib_backtrace_no_good the old one is using __builtin_frame_address which can not retrieve stack trace in release version (don’t know why) 2. install SIGABRT in signal handler, but have to remove it when backtrace is done reason is we want to capture stack trace caused by SIGABRT. vPP ASSERT always call os_exit then abort() we definitely want to know the trace of this situation. It is a little tricky to avoid SIGABRT infinite loop 3. A text log file will be generated as well as the output to stderr. we'd like to have a simple file to trace the crash this file generation can be removed if vPP thinks it is not suitable 4. always load symbols by calling clib_elf_main_init () in main(). Otherwise, only addresses can be displayed. In the end, this is what we got: …… load_one_plugin:189: Loaded plugin: srv6ad_plugin.so (Dynamic SRv6 proxy) load_one_plugin:189: Loaded plugin: srv6am_plugin.so (Masquerading SRv6 proxy) load_one_plugin:189: Loaded plugin: srv6as_plugin.so (Static SRv6 proxy) load_one_plugin:189: Loaded plugin: stn_plugin.so (VPP Steals the NIC for Container integration) load_one_plugin:189: Loaded plugin: tlsmbedtls_plugin.so (mbedtls based TLS Engine) load_one_plugin:189: Loaded plugin: tlsopenssl_plugin.so (openssl based TLS Engine) Writing crashdump to ./crashdump-2018-06-15-04-55-45.log ... #0 0x00007f872383d0bc generate_crash_backtrace + 0xfe #1 0x00007f872383d366 unix_signal_handler + 0x1b3 #2 0x00007f872206f390 0x7f872206f390 #3 0x00007f87222c89a8 hash_header_bytes + 0x2a #4 0x00007f87222c8a04 hash_header + 0x27 #5 0x00007f87222ca554 _hash_get + 0x2b #6 0x00007f87222ed17a mheap_get_trace + 0x18e #7 0x00007f87222ea9db mheap_get_aligned + 0x236 #8 0x00007f872233fe6a clib_mem_alloc_aligned_at_offset + 0xa5 #9 0x00007f872234020a vec_resize_allocate_memory + 0x6f #10 0x00007f87222bbf4a _vec_resize_inline + 0x136 #11 0x00007f87222bcf9a do_percent + 0xcb6 #12 0x00007f87222bd407 va_format + 0x113 #13 0x00007f87222bd5a0 format + 0xb8 #14 0x00007f872259614a shm_name_from_svm_map_region_args + 0x32d Aborted (core dumped) Makefile:446: recipe for target 'run' failed From: Damjan Marion <dmar...@me.com> Sent: Thursday, June 14, 2018 8:31 PM To: Dave Barach <dbar...@cisco.com>; Kingwel Xie <kingwel....@ericsson.com> Cc: vpp-dev <vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Integration with libbacktrace +1, send to stderr so systemd can pick it up Thanks for your contribution... On 14 Jun 2018, at 14:21, Dave Barach via Lists.Fd.Io <dbarach=cisco....@lists.fd.io<mailto:dbarach=cisco....@lists.fd.io>> wrote: Sure. Personally, I'd send the backtrace to syslog vs. creating yet-another logfile. D. -----Original Message----- From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Kingwel Xie Sent: Thursday, June 14, 2018 2:54 AM To: Kingwel Xie <kingwel....@ericsson.com<mailto:kingwel....@ericsson.com>>; Damjan Marion <dmar...@me.com<mailto:dmar...@me.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Integration with libbacktrace Hi, Finally I removed libbacktrace dependency, but to use glibc 'backtrace' which can be found in execinfo.h. The generated crashdump will look like below. I can create a patch if you think it is valuable. Regards, Kingwel ----------------------------------------------------------------------------------------------- DBGvpp# DBGvpp# test crash Thread 1 "kw_main" received signal SIGSEGV, Segmentation fault. 0x0000000000407ace in test_crash_command_fn (vm=0x7ffff7b89ac0 <vlib_global_main>, input=0x7fffbceabee0, cmd=0x7fffbcdda1a0) at /home/vppshare/kingwel/vpp/build-data/../src/vpp/vnet/main.c:387 /home/vppshare/kingwel/vpp/src/vpp/vnet/main.c:387:8300:beg:0x407ace (gdb) c Continuing. Writing crashdump to ./crashdump-2018-06-14-06-51-53.log ... #0 0x00007ffff794a428 unix_signal_handler + 0x2e7 #1 0x00007ffff6168390 0x7ffff6168390 #2 0x0000000000407ace test_crash_command_fn + 0x53 #3 0x00007ffff789bd47 vlib_cli_dispatch_sub_commands + 0xc27 #4 0x00007ffff789bc55 vlib_cli_dispatch_sub_commands + 0xb35 #5 0x00007ffff789c02c vlib_cli_input + 0xc0 #6 0x00007ffff7940ac0 unix_cli_process_input + 0x2dc #7 0x00007ffff79415e2 unix_cli_process + 0x94 #8 0x00007ffff78db84d vlib_process_bootstrap + 0x66 #9 0x00007ffff63ca8a8 0x7ffff63ca8a8 Thread 1 "kw_main" received signal SIGABRT, Aborted. 0x00007ffff5bbe428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007ffff5bbe428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #1 0x00007ffff5bc002a in __GI_abort () at abort.c:89 #2 0x00000000004079c9 in os_exit (code=1) at /home/vppshare/kingwel/vpp/build-data/../src/vpp/vnet/main.c:334 #3 0x00007ffff794a528 in unix_signal_handler (signum=11, si=0x7fffbceab6f0, uc=0x7fffbceab5c0) at /home/vppshare/kingwel/vpp/build-data/../src/vlib/unix/main.c:181 #4 <signal handler called> #5 0x0000000000407ace in test_crash_command_fn (vm=0x7ffff7b89ac0 <vlib_global_main>, input=0x7fffbceabee0, cmd=0x7fffbcdda1a0) at /home/vppshare/kingwel/vpp/build-data/../src/vpp/vnet/main.c:387 #6 0x00007ffff789bd47 in vlib_cli_dispatch_sub_commands (vm=0x7ffff7b89ac0 <vlib_global_main>, cm=0x7ffff7b89ca0 <vlib_global_main+480>, input=0x7fffbceabee0, parent_command_index=143) at /home/vppshare/kingwel/vpp/build-data/../src/vlib/cli.c:589 #7 0x00007ffff789bc55 in vlib_cli_dispatch_sub_commands (vm=0x7ffff7b89ac0 <vlib_global_main>, cm=0x7ffff7b89ca0 <vlib_global_main+480>, input=0x7fffbceabee0, parent_command_index=0) at /home/vppshare/kingwel/vpp/build-data/../src/vlib/cli.c:567 #8 0x00007ffff789c02c in vlib_cli_input (vm=0x7ffff7b89ac0 <vlib_global_main>, input=0x7fffbceabee0, function=0x7ffff793afe8 <unix_vlib_cli_output>, function_arg=0) at /home/vppshare/kingwel/vpp/build-data/../src/vlib/cli.c:663 #9 0x00007ffff7940ac0 in unix_cli_process_input (cm=0x7ffff7b89880 <unix_cli_main>, cli_file_index=0) at /home/vppshare/kingwel/vpp/build-data/../src/vlib/unix/cli.c:2419 #10 0x00007ffff79415e2 in unix_cli_process (vm=0x7ffff7b89ac0 <vlib_global_main>, rt=0x7fffbce9b000, f=0x0) at /home/vppshare/kingwel/vpp/build-data/../src/vlib/unix/cli.c:2531 #11 0x00007ffff78db84d in vlib_process_bootstrap (_a=140736265931088) at /home/vppshare/kingwel/vpp/build-data/../src/vlib/main.c:1231 #12 0x00007ffff63ca8a8 in clib_calljmp () at /home/vppshare/kingwel/vpp/build-data/../src/vppinfra/longjmp.S:110 #13 0x00007fffb7234920 in ?? () #14 0x00007ffff78db978 in vlib_process_startup (vm=0x28, p=0x1, f=0x0) at /home/vppshare/kingwel/vpp/build-data/../src/vlib/main.c:1253 -----Original Message----- From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> On Behalf Of Kingwel Xie Sent: Monday, June 11, 2018 7:48 PM To: Damjan Marion <dmar...@me.com<mailto:dmar...@me.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Integration with libbacktrace No idea. Hope someone can point it out. The libbacktrace can locate the line number of the code, otherwise we'd like to use glibc backtrace which can generate like: mheap_put+0x230 clib_mem_free+0x190 It would take extra time to convert the offset to the line number by using addr2line. It is a bit heavy to integrate something like libbacktrace. This is why I'm wondering. Probably we can take a light-weight approach to add glib backtrace shown above. No dependency :-) Regards, Kingwel -----Original Message----- From: Damjan Marion <dmar...@me.com<mailto:dmar...@me.com>> Sent: Monday, June 11, 2018 7:27 PM To: Kingwel Xie <kingwel....@ericsson.com<mailto:kingwel....@ericsson.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] Integration with libbacktrace What is the difference between libunwind and libbacktrace? I can see that libunwind is available as package on ubuntu, and libbacktrace is not... On 11 Jun 2018, at 11:41, Kingwel Xie <kingwel....@ericsson.com<mailto:kingwel....@ericsson.com>> wrote: Hi all, I’m wondering if it can be accepted by the community. Every time when vPP crashes for some reason, core dump file can be used for further analysis. However, core file is usually big and even unfortunately isn’t generated. A trick we used before is to generate a crash dump file to indicate some basic information. This file is a text file like below, not intended to replace core dump, but useful for a quick analysis. To make it more friendly, we use libbacktrace instead of glic backtrace function which can only indicate the function name plus offset. Therefore, integration of libbacktrace is now in our code branch. Can this be pushed upstream? Libbacktrace can be found at: https://github.com/ianlancetaylor/libbacktrace Regards, Kingwel ---------------------------------------------------------------------- ------------------------------------------------ DBGvpp# write crashdump to ../crashdump-2018-06-11-05-14-31.log /home/vppshare/rich/vpp/build-root/install-vpp_debug-native/vpp/bin/vp p: libbacktrace: DWARF underflow in .debug_abbrev at 402002 7f8907da336b unix_signal_handler /home/vppshare/rich/vpp/build-data/../src/vlib/unix/main.c:204 7f89065f238f ?? ??:0 7f8905e32428 gsignal ??:0 7f8905e34029 abort ??:0 407f7d os_panic /home/vppshare/rich/vpp/build-data/../src/vpp/vnet/main.c:310 7f890686d890 mheap_put /home/vppshare/rich/vpp/build-data/../src/vppinfra/mheap.c:815 7f89068c1531 clib_mem_free /home/vppshare/rich/vpp/build-data/../src/vppinfra/mem.h:186 7f89068c1845 vec_resize_allocate_memory /home/vppshare/rich/vpp/build-data/../src/vppinfra/vec.c:96 7f890683ea09 _vec_resize_inline /home/vppshare/rich/vpp/build-data/../src/vppinfra/vec.h:145 7f890683fa59 do_percent /home/vppshare/rich/vpp/build-data/../src/vppinfra/format.c:341 7f890683fec6 va_format /home/vppshare/rich/vpp/build-data/../src/vppinfra/format.c:404 7f890682bfc3 elog_string /home/vppshare/rich/vpp/build-data/../src/vppinfra/elog.c:541 7f8907ff0dc7 elog_id_for_msg_name /home/vppshare/rich/vpp/build-data/../src/vlibapi/api_shared.c:408 7f8907ff1250 vl_msg_api_handler_with_vm_node /home/vppshare/rich/vpp/build-data/../src/vlibapi/api_shared.c:540 7f8907ffbcec void_mem_api_handle_msg_i /home/vppshare/rich/vpp/build-data/../src/vlibmemory/memory_api.c:675 7f8907ffbd5b vl_mem_api_handle_msg_main /home/vppshare/rich/vpp/build-data/../src/vlibmemory/memory_api.c:685 7f89080168fe vl_api_clnt_process /home/vppshare/rich/vpp/build-data/../src/vlibmemory/vlib_api.c:380 7f8907d346e0 vlib_process_bootstrap /home/vppshare/rich/vpp/build-data/../src/vlib/main.c:1280 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9622): https://lists.fd.io/g/vpp-dev/message/9622 Mute This Topic: https://lists.fd.io/mt/21987007/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
