Re: [vpp-dev]: Unable to run VPP with ASAN enabled
Hi Rajith, > The problem seems to be due to external libraries that we have linked with > VPP. These external libraries have not been compiled with ASAN. > I could see that when those external libraries were suppressed through the > MyASAN.supp file, VPP started running with ASAN enabled. This is surprising because the default policy of ASan is to allow access to any memory by default - IOW ASan will not detect access memory errors *unless* this memory address was *explicitly* marked as unaccessible. ASan interpose malloc() etc. symbols so libc allocations are automatically marked as accessible/unaccessible by the ASan runtime, and VPP does the same for its own memory allocator. Of course there might still be false positive but in my experience they are usually rare. Best ben -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19486): https://lists.fd.io/g/vpp-dev/message/19486 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
Hi Ben, The problem seems to be due to external libraries that we have linked with VPP. These external libraries have not been compiled with ASAN. I could see that when those external libraries were suppressed through the MyASAN.supp file, VPP started running with ASAN enabled. Thanks, Rajith On Wed, May 26, 2021 at 2:25 PM Benoit Ganne (bganne) wrote: > Hi Rajith, > > > I was able to proceed further after setting LD_PRELOAD to the asan > > library. After this i get SIGSEGV crash in asan. These dont seem to be > > related to our code, as without ASAN they have been perfectly working. > > I suspect the opposite - ASan detects errors we do not detect in > release or debug mode, esp. out-of-bound access and use-after-free. Look > carefully at /home/supervisor/libvpp/src/vpp/rtbrick/rtb_vpp_ifp.c:287 > > Best > ben > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19484): https://lists.fd.io/g/vpp-dev/message/19484 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
Hi Rajith, > I was able to proceed further after setting LD_PRELOAD to the asan > library. After this i get SIGSEGV crash in asan. These dont seem to be > related to our code, as without ASAN they have been perfectly working. I suspect the opposite - ASan detects errors we do not detect in release or debug mode, esp. out-of-bound access and use-after-free. Look carefully at /home/supervisor/libvpp/src/vpp/rtbrick/rtb_vpp_ifp.c:287 Best ben -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19471): https://lists.fd.io/g/vpp-dev/message/19471 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
HI Ben, We have linked our shared library that has the main function with VPP's .so. The main function does few things and falls to the VPP's main function loop. After changing the compiler t*o gcc-8 I dont get the unresolved symbol error*. However, i get the f*ollowing ASAN error:* ==7111==*ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD*. Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) q I was able to proceed further after *setting LD_PRELOAD to the asan library.* After this i get *SIGSEGV* crash in asan. These dont seem to be related to our code, as without ASAN they have been perfectly working. *set environment LD_PRELOAD /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so* Program received signal SIGSEGV, Segmentation fault. 0x76e39841 in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so (gdb) bt #0 0x76e39841 in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #1 0x76f059b1 in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #2 0x76f1e9d3 in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #3 0x76f06d39 in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #4 0x76e35836 in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #5 0x76e366b9 in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #6 0x76e383a0 in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #7 0x76f00d5b in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #8 0x76eb383c in ?? () from /usr/lib/gcc/x86_64-linux-gnu/8/libasan.so #9 0x7fffe823727d in rtb_vpp_ifp_mapping_object_get (key_type=1 '\001', ifp_mapping=0x7fffc69a4680) at /home/supervisor/libvpp/src/vpp/rtbrick/rtb_vpp_ifp.c:287 #10 0x7fffe823a07c in rtb_vpp_hw_interface_add_delete_cb (vnm=0x7fffe7c10dc0 , hw_if_index=0, is_create=1) at /home/supervisor/libvpp/src/vpp/rtbrick/rtb_vpp_ifp.c:888 #11 0x7fffe57eff3a in call_elf_section_interface_callbacks (vnm=0x7fffe7c10dc0 , if_index=0, flags=1, elts=0x7fffe7c10e60 ) at /home/supervisor/libvpp/src/vnet/interface.c:251 #12 0x7fffe57f0137 in call_hw_interface_add_del_callbacks (vnm=0x7fffe7c10dc0 , hw_if_index=0, is_create=1) at /home/supervisor/libvpp/src/vnet/interface.c:282 #13 0x7fffe57f0228 in vnet_hw_interface_set_flags_helper (vnm=0x7fffe7c10dc0 , hw_if_index=0, flags=VNET_HW_INTERFACE_FLAG_NONE, helper_flags=VNET_INTERFACE_SET_FLAGS_HELPER_IS_CREATE) at /home/supervisor/libvpp/src/vnet/interface.c:316 #14 0x7fffe57f767e in vnet_register_interface (vnm=0x7fffe7c10dc0 , dev_class_index=35, dev_instance=0, hw_class_index=30, hw_instance=0) at /home/supervisor/libvpp/src/vnet/interface.c:998 #15 0x7fffe58787c0 in vnet_main_init (vm=0x7fffe484a7c0 ) at /home/supervisor/libvpp/src/vnet/misc.c:81 #16 0x7fffe444f659 in call_init_exit_functions_internal (vm=0x7fffe484a7c0 , headp=0x7fffe484ae18 , call_once=1, do_sort=1) at /home/supervisor/libvpp/src/vlib/init.c:350 #17 0x7fffe444f6d4 in vlib_call_init_exit_functions (vm=0x7fffe484a7c0 , headp=0x7fffe484ae18 , call_once=1) at /home/supervisor/libvpp/src/vlib/init.c:364 #18 0x7fffe444f738 in vlib_call_all_init_functions (vm=0x7fffe484a7c0 ) at /home/supervisor/libvpp/src/vlib/init.c:386 #19 0x7fffe44a7966 in vlib_main (vm=0x7fffe484a7c0 , input=0x7fffc69a4f60) at /home/supervisor/libvpp/src/vlib/main.c:2183 #20 0x7fffe45a121f in thread0 (arg=140737027286976) at /home/supervisor/libvpp/src/vlib/unix/main.c:681 #21 0x7fffe3fa5e00 in clib_calljmp () at /home/supervisor/libvpp/src/vppinfra/longjmp.S:123 #22 0x7fffce50 in ?? () #23 0x7fffe45a1cd9 in vlib_unix_main (argc=51, argv=0x61416a40) at /home/supervisor/libvpp/src/vlib/unix/main.c:754 #24 0x7fffe7fb7a72 in rtb_vpp_core_init (argc=51, argv=0x61416a40) at /home/supervisor/libvpp/src/vpp/vnet/main.c:496 #25 0x7fffe8214dd0 in rtb_vpp_main () at /home/supervisor/libvpp/src/vpp/rtbrick/rtb_vpp_main.c:119 #26 0x76ba4902 in bd_load_daemon_lib (dmn_lib_cfg=0x5575b400 ) at /development/rtbrick-infrastructure/code/bd/src/bdinfra/bd.c:611 #27 0x76ba5a34 in bd_load_all_daemon_libs () at /development/rtbrick-infrastructure/code/bd/src/bdinfra/bd.c:630 #28 0x76ba634c in bd_start_process () at /development/rtbrick-infrastructure/code/bd/src/bdinfra/bd.c:1097 #29 0x747ea28c in bds_bd_init () at /development/rtbrick-infrastructure/code/bds/bds_core/src/bds.c:659 #30 0x7485ca22 in pubsub_bd_init_expiry (data=) at /development/rtbrick-infrastructure/code/bds/pubsub/src/pubsub_helper.c:1452 #31 0x766c178b in timer_dispatch (item=0x61a0001ccf20, p=) at /development/rtbrick-infrastructure/code/qb/lib/loop_timerlist.c:56 #32 0x766bf11f in
Re: [vpp-dev]: Unable to run VPP with ASAN enabled
How are you starting VPP? If this is through 'make test' then chances are the culprit is the interaction of asan, clang and python [1]. The easy way to fix is to rebuild with gcc instead of clang, eg. ~# make rebuild VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON CC=gcc-9 Best ben [1] https://gerrit.fd.io/r/c/vpp/+/27268/3/src/vpp-api/python/vpp_papi/vpp_ffi.py.in#1 > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of Rajith PR via > lists.fd.io > Sent: mardi 25 mai 2021 09:51 > To: vpp-dev > Subject: [vpp-dev]: Unable to run VPP with ASAN enabled > > Hi All, > > I am not able to run VPP with ASAN. Though we have been using VPP for > sometime this is the first time we enabled ASAN in the build. > I have followed the steps as mentioned in the sanitizer doc, can someone > please let me know what is missed here. > > Run Time Error(Missing symbol): > > /usr/local/lib/librtbvpp.so:/home/supervisor/libvpp/build-root/install- > vpp_debug-native/vpp/lib/libvppinfra.so.1.0.1: undefined symbol: > __asan_option_detect_stack_use_after_return > > > VPP Version : 20.09 > > > Build Command : make rebuild VPP_EXTRA_CMAKE_ARGS=- > DVPP_ENABLE_SANITIZE_ADDR=ON > > > Build Summary : > > VPP version : 1.0.1-3032~g4b28254fc-dirty > VPP library version : 1.0.1 > GIT toplevel dir: /home/supervisor/libvpp > Build type : debug > C flags : -fsanitize=address -DCLIB_SANITIZE_ADDR -Wno- > address-of-packed-member -g -fPIC -Werror -Wall -march=corei7 - > mtune=corei7-avx -O0 -DCLIB_DEBUG -fstack-protector -DFORTIFY_SOURCE=2 - > fno-common > Linker flags (apps) : -fsanitize=address > Linker flags (libs) : -fsanitize=address > Host processor : x86_64 > Target processor: x86_64 > Prefix path : > /opt/vpp/external/x86_64;/home/supervisor/libvpp/build-root/install- > vpp_debug-native/external > Install prefix : /home/supervisor/libvpp/build-root/install- > vpp_debug-native/vpp > > > > Reference : https://fd.io/docs/vpp/master/troubleshooting/sanitizer.html > > > Thanks, > Rajith -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19452): https://lists.fd.io/g/vpp-dev/message/19452 Mute This Topic: https://lists.fd.io/mt/83071228/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
