Hi Chetan, Something like the following should exercise using iperf and homegrown test apps most of the components of the host stack, i.e., vcl, session layer and the transports, including tcp:
make test-debug VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON CC=gcc-10 TEST=vcl CACHE_OUTPUT=0 Having said that, it seems some issue has creeped in since last time I’ve tested (granted it’s been a long time). That is, most of the tests seems to fail but I’ve only checked without Benoit’s patch. Regards, Florin > On Sep 7, 2021, at 9:57 PM, chetan bhasin <chetan.bhasin...@gmail.com> wrote: > > Hi Florin , > > This issue is coming when we are testing with iperf traffic with our plugins > that involve VPP TCP Host stack (VPP 21.06). > > What's the best way to test VPP TCP host stack with Sanitizer without > involving our out of tree plugins ? > > > Thanks, > Chetan > > On Tue, Sep 7, 2021 at 7:47 PM Florin Coras <fcoras.li...@gmail.com > <mailto:fcoras.li...@gmail.com>> wrote: > Hi Chetan, > > Out of curiosity, is this while running some make test (like the iperf3 one) > or with actual traffic through vpp? > > Regards, > Florin > >> On Sep 7, 2021, at 12:17 AM, chetan bhasin <chetan.bhasin...@gmail.com >> <mailto:chetan.bhasin...@gmail.com>> wrote: >> >> Hi Ben, >> >> After applying the patch , now its crashing at below place >> >> static void >> session_flush_pending_tx_buffers (session_worker_t * wrk, >> vlib_node_runtime_t * node) >> { >> vlib_buffer_enqueue_to_next (wrk->vm, node, wrk->pending_tx_buffers, >> wrk->pending_tx_nexts, >> vec_len (wrk->pending_tx_nexts)); >> vec_reset_length (wrk->pending_tx_buffers); >> vec_reset_length (wrk->pending_tx_nexts); >> } >> Program received signal SIGSEGV, Segmentation fault. >> [Switching to Thread 0x7ffb527f9700 (LWP 27762)] >> 0x00007ffff71db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, >> unsigned long*, unsigned long*) () from /lib64/libasan.so.5 >> (gdb) bt >> #0 0x00007ffff71db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned >> long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5 >> #1 0x00007ffff72c2a11 in >> __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) >> () from /lib64/libasan.so.5 >> #2 0x00007ffff72dcdc2 in >> __sanitizer::ThreadRegistry::FindThreadContextLocked(bool >> (*)(__sanitizer::ThreadContextBase*, void*), void*) () from >> /lib64/libasan.so.5 >> #3 0x00007ffff72c3e5a in __asan::FindThreadByStackAddress(unsigned long) () >> from /lib64/libasan.so.5 >> #4 0x00007ffff71d5fb6 in __asan::GetStackAddressInformation(unsigned long, >> unsigned long, __asan::StackAddressDescription*) () from /lib64/libasan.so.5 >> #5 0x00007ffff71d73f9 in >> __asan::AddressDescription::AddressDescription(unsigned long, unsigned long, >> bool) () from /lib64/libasan.so.5 >> #6 0x00007ffff71d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, >> unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned >> long) () from /lib64/libasan.so.5 >> #7 0x00007ffff72bdc2a in __asan::ReportGenericError(unsigned long, unsigned >> long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) >> () from /lib64/libasan.so.5 >> #8 0x00007ffff72be927 in __asan_report_load4 () from /lib64/libasan.so.5 >> #9 0x00007ffff58e5185 in session_flush_pending_tx_buffers >> (wrk=0x7fff89101e80, node=0x7fff891a6e80) >> at src/vnet/session/session_node.c:1658 >> #10 0x00007ffff58e8531 in session_queue_node_fn (vm=0x7fff72bf2c40, >> node=0x7fff891a6e80, frame=0x0) >> at src/vnet/session/session_node.c:1812 >> #11 0x00007ffff3e04121 in dispatch_node (vm=0x7fff72bf2c40, >> node=0x7fff891a6e80, type=VLIB_NODE_TYPE_INPUT, >> dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, >> last_time_stamp=2510336347228480) >> at src/vlib/main.c:1024 >> #12 0x00007ffff3e13394 in vlib_main_or_worker_loop (vm=0x7fff72bf2c40, >> is_main=0) at src/vlib/main.c:2949 >> #13 0x00007ffff3e14fb8 in vlib_worker_loop (vm=0x7fff72bf2c40) at >> src/vlib/main.c:3114 >> #14 0x00007ffff3eac8f1 in vlib_worker_thread_fn (arg=0x7fff700ef480) at >> src/vlib/threads.c:1560 >> #15 0x00007ffff34d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123 >> #16 0x00007ffb527f8230 in ?? () >> #17 0x00007ffff3ea0141 in vlib_worker_thread_bootstrap_fn >> (arg=0x7fff700ef480) at src/vlib/threads.c:431 >> #18 0x00007fff6d41971b in eal_thread_loop () from >> /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so >> #19 0x00007ffff38abea5 in start_thread () from /lib64/libpthread.so.0 >> #20 0x00007ffff2e328cd in clone () from /lib64/libc.so.6 >> (gdb) >> >> >> On Mon, Sep 6, 2021 at 6:10 PM Benoit Ganne (bganne) <bga...@cisco.com >> <mailto:bga...@cisco.com>> wrote: >> Yes we are aware of it, still working on the correct fix though. >> In the meantime you can try to apply https://gerrit.fd.io/r/c/vpp/+/32765 >> <https://gerrit.fd.io/r/c/vpp/+/32765> which should workaround that for now. >> >> Best >> ben >> >> > -----Original Message----- >> > From: chetan bhasin <chetan.bhasin...@gmail.com >> > <mailto:chetan.bhasin...@gmail.com>> >> > Sent: lundi 6 septembre 2021 14:21 >> > To: Benoit Ganne (bganne) <bga...@cisco.com <mailto:bga...@cisco.com>> >> > Cc: vpp-dev <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> >> > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled >> > >> > Hi, >> > >> > The below crash is coming as we involved VPP TCP host stack. >> > >> > >> > Program received signal SIGSEGV, Segmentation fault. >> > [Switching to Thread 0x7ffb527f9700 (LWP 2013)] >> > 0x00007ffff71db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, >> > unsigned long*, unsigned long*) () from /lib64/libasan.so.5 >> > (gdb) bt >> > #0 0x00007ffff71db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned >> > long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5 >> > #1 0x00007ffff72c2a11 in >> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) >> > () from /lib64/libasan.so.5 >> > #2 0x00007ffff72dcdc2 in >> > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool >> > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from >> > /lib64/libasan.so.5 >> > #3 0x00007ffff72c3e5a in __asan::FindThreadByStackAddress(unsigned long) >> > () from /lib64/libasan.so.5 >> > #4 0x00007ffff71d5fb6 in __asan::GetStackAddressInformation(unsigned >> > long, unsigned long, __asan::StackAddressDescription*) () from >> > /lib64/libasan.so.5 >> > #5 0x00007ffff71d73f9 in >> > __asan::AddressDescription::AddressDescription(unsigned long, unsigned >> > long, bool) () from /lib64/libasan.so.5 >> > #6 0x00007ffff71d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, >> > unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned >> > long) () from /lib64/libasan.so.5 >> > #7 0x00007ffff72bdc2a in __asan::ReportGenericError(unsigned long, >> > unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned >> > int, bool) () from /lib64/libasan.so.5 >> > #8 0x00007ffff72bf194 in __asan_report_load_n () from /lib64/libasan.so.5 >> > #9 0x00007ffff3f45463 in clib_mask_compare_u16_x64 (v=2, >> > a=0x7fff89fd6150, n_elts=2) at src/vppinfra/vector_funcs.h:24 >> > #10 0x00007ffff3f4571e in clib_mask_compare_u16 (v=2, a=0x7fff89fd6150, >> > mask=0x7ffb51fe2310, n_elts=2) >> > at src/vppinfra/vector_funcs.h:79 >> > #11 0x00007ffff3f45b81 in enqueue_one (vm=0x7fff7314ec40, >> > node=0x7fff89fe1e00, used_elt_bmp=0x7ffb51fe2440, next_index=2, >> > buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, n_buffers=2, n_left=2, >> > tmp=0x7ffb51fe2480) at src/vlib/buffer_funcs.c:30 >> > #12 0x00007ffff3f6bdae in vlib_buffer_enqueue_to_next_fn_skx >> > (vm=0x7fff7314ec40, node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, >> > nexts=0x7fff89fd6150, count=2) >> > at src/vlib/buffer_funcs.c:110 >> > #13 0x00007ffff58cd2b6 in vlib_buffer_enqueue_to_next (vm=0x7fff7314ec40, >> > node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, >> > count=2) >> > at src/vlib/buffer_node.h:344 >> > #14 0x00007ffff58e4cf1 in session_flush_pending_tx_buffers >> > (wrk=0x7fff8912b780, node=0x7fff89fe1e00) >> > at src/vnet/session/session_node.c:1654 >> > #15 0x00007ffff58e844f in session_queue_node_fn (vm=0x7fff7314ec40, >> > node=0x7fff89fe1e00, frame=0x0) >> > at src/vnet/session/session_node.c:1812 >> > #16 0x00007ffff3e0402f in dispatch_node (vm=0x7fff7314ec40, >> > node=0x7fff89fe1e00, type=VLIB_NODE_TYPE_INPUT, >> > dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, >> > last_time_stamp=2463904994699296) >> > at src/vlib/main.c:1024 >> > #17 0x00007ffff3e132a2 in vlib_main_or_worker_loop (vm=0x7fff7314ec40, >> > is_main=0) at src/vlib/main.c:2949 >> > #18 0x00007ffff3e14ec6 in vlib_worker_loop (vm=0x7fff7314ec40) at >> > src/vlib/main.c:3114 >> > #19 0x00007ffff3eac7ff in vlib_worker_thread_fn (arg=0x7fff7050ef40) at >> > src/vlib/threads.c:1560 >> > #20 0x00007ffff34d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123 >> > #21 0x00007ffb527f8230 in ?? () >> > #22 0x00007ffff3ea004f in vlib_worker_thread_bootstrap_fn >> > (arg=0x7fff7050ef40) at src/vlib/threads.c:431 >> > #23 0x00007fff6d41971b in eal_thread_loop () from >> > /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so >> > #24 0x00007ffff38abea5 in start_thread () from /lib64/libpthread.so.0 >> > #25 0x00007ffff2e328cd in clone () from /lib64/libc.so.6 >> > (gdb) :q >> > >> > On Mon, Sep 6, 2021 at 1:52 PM chetan bhasin <chetan.bhasin...@gmail.com >> > <mailto:chetan.bhasin...@gmail.com> >> > <mailto:chetan.bhasin...@gmail.com <mailto:chetan.bhasin...@gmail.com>> > >> > wrote: >> > >> > >> > Hi Ben, >> > >> > Thanks for the direction. Looks like it will fix both the issues as >> > mentioned above. I will update you with the results after applying the >> > patch. >> > >> > Is there any ASAN related patch inside the TCP host stack code. I >> > will be sharing the issue shortly with you. >> > >> > Thanks, >> > Chetan >> > >> > On Mon, Sep 6, 2021 at 1:22 PM Benoit Ganne (bganne) >> > <bga...@cisco.com <mailto:bga...@cisco.com> <mailto:bga...@cisco.com >> > <mailto:bga...@cisco.com>> > wrote: >> > >> > >> > It should be fixed in master by >> > https://gerrit.fd.io/r/c/vpp/+/32643 <https://gerrit.fd.io/r/c/vpp/+/32643> >> > >> > ben >> > >> > > -----Original Message----- >> > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> >> > <mailto:vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> >> > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> >> > <mailto:vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> > On Behalf Of >> > chetan >> > bhasin >> > > Sent: lundi 6 septembre 2021 09:36 >> > > To: vpp-dev <vpp-dev@lists.fd.io >> > <mailto:vpp-dev@lists.fd.io> <mailto:vpp- <mailto:vpp-> >> > d...@lists.fd.io <mailto:d...@lists.fd.io>> > >> > > Subject: [vpp-dev] VPP 2106 with Sanitizer enabled >> > > >> > > Hi >> > > >> > > >> > > We are facing two errors with vpp2106 and Address Sanitizer >> > enabled. >> > > >> > > make V=1 -j4 build VPP_EXTRA_CMAKE_ARGS=- >> > DVPP_ENABLE_SANITIZE_ADDR=ON >> > > >> > > >> > > Work-around - After adding the two api’s string_key_sum and >> > > strnlen_s_inline to ASAN suppression, vpp2106 comes up >> > fine. >> > > >> > > >> > > Error 1: >> > > ---------- >> > > Program received signal SIGSEGV, Segmentation fault. >> > > 0x00007ffff71db5c1 in >> > __asan::FakeStack::AddrIsInFakeStack(unsigned long, >> > > unsigned long*, unsigned long*) () >> > > from /lib64/libasan.so.5 >> > > (gdb) bt >> > > #0 0x00007ffff71db5c1 in >> > __asan::FakeStack::AddrIsInFakeStack(unsigned >> > > long, unsigned long*, unsigned long*) () >> > > from /lib64/libasan.so.5 >> > > #1 0x00007ffff72c2a11 in >> > > >> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) >> > > () >> > > from /lib64/libasan.so.5 >> > > #2 0x00007ffff72dcdc2 in >> > > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool >> > > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from >> > > /lib64/libasan.so.5 >> > > #3 0x00007ffff72c3e5a in >> > __asan::FindThreadByStackAddress(unsigned long) >> > > () from /lib64/libasan.so.5 >> > > #4 0x00007ffff71d5fb6 in >> > __asan::GetStackAddressInformation(unsigned >> > > long, unsigned long, __asan::StackAddressDescription*) () >> > from >> > > /lib64/libasan.so.5 >> > > #5 0x00007ffff71d73f9 in >> > > __asan::AddressDescription::AddressDescription(unsigned >> > long, unsigned >> > > long, bool) () >> > > from /lib64/libasan.so.5 >> > > #6 0x00007ffff71d9e51 in >> > __asan::ErrorGeneric::ErrorGeneric(unsigned int, >> > > unsigned long, unsigned long, unsigned long, unsigned long, >> > bool, unsigned >> > > long) () from /lib64/libasan.so.5 >> > > #7 0x00007ffff72bdc2a in >> > __asan::ReportGenericError(unsigned long, >> > > unsigned long, unsigned long, unsigned long, bool, unsigned >> > long, unsigned >> > > int, bool) () from /lib64/libasan.so.5 >> > > #8 0x00007ffff720ef9c in __interceptor_strlen.part.0 () >> > from >> > > /lib64/libasan.so.5 >> > > #9 0x00007ffff34ce2ec in string_key_sum (h=0x7fff6ff6e970, >> > > key=140735097062688) >> > > at src/vppinfra/hash.c:947 >> > > #10 0x00007ffff34caf15 in key_sum (h=0x7fff6ff6e970, >> > key=140735097062688) >> > > at src/vppinfra/hash.c:333 >> > > #11 0x00007ffff34cbf76 in lookup (v=0x7fff6ff6e9b8, >> > key=140735097062688, >> > > op=GET, new_value=0x0, old_value=0x0) >> > > at src/vppinfra/hash.c:557 >> > > #12 0x00007ffff34cc59d in _hash_get_pair (v=0x7fff6ff6e9b8, >> > > key=140735097062688) >> > > at src/vppinfra/hash.c:653 >> > > #13 0x000000000042e885 in lookup_hash_index >> > (name=0x7fff7177c520 >> > > "/mem/stat segment") >> > > at src/vpp/stats/stat_segment.c:69 >> > > #14 0x0000000000431790 in stat_segment_new_entry >> > (name=0x7fff7177c520 >> > > "/mem/stat segment", >> > > t=STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE) >> > > at src/vpp/stats/stat_segment.c:402 >> > > #15 0x00000000004401fb in vlib_stats_register_mem_heap >> > > (heap=0x7ffb67e00000) >> > > at src/vpp/stats/stat_segment_provider.c:96 >> > > #16 0x00000000004327fa in vlib_map_stat_segment_init () >> > > at src/vpp/stats/stat_segment.c:493 >> > > ---Type <return> to continue, or q <return> to quit--- >> > > #17 0x00007ffff3e15d19 in vlib_main (vm=0x7fff6eeff680, >> > > input=0x7fff6a0a9f70) >> > > at src/vlib/main.c:3272 >> > > #18 0x00007ffff3f0d924 in thread0 (arg=140735054608000) >> > > at src/vlib/unix/main.c:671 >> > > #19 0x00007ffff34d9504 in clib_calljmp () >> > > at src/vppinfra/longjmp.S:123 >> > > #20 0x00007fffffffc940 in ?? () >> > > #21 0x00007ffff3f0e67e in vlib_unix_main (argc=282, >> > argv=0x61d00001a480) >> > > at src/vlib/unix/main.c:751 >> > > #22 0x000000000040b482 in main (argc=282, >> > argv=0x61d00001a480) >> > > at src/vpp/vnet/main.c:336 >> > > (gdb) q >> > > >> > > >> > > >> > > >> > > Error 2: >> > > ----------- >> > > Program received signal SIGSEGV, Segmentation fault. >> > > 0x00007ffff71db5c1 in >> > __asan::FakeStack::AddrIsInFakeStack(unsigned long, >> > > unsigned long*, unsigned long*) () >> > > from /lib64/libasan.so.5 >> > > (gdb) bt >> > > #0 0x00007ffff71db5c1 in >> > __asan::FakeStack::AddrIsInFakeStack(unsigned >> > > long, unsigned long*, unsigned long*) () >> > > from /lib64/libasan.so.5 >> > > #1 0x00007ffff72c2a11 in >> > > >> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) >> > > () >> > > from /lib64/libasan.so.5 >> > > #2 0x00007ffff72dcdc2 in >> > > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool >> > > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from >> > > /lib64/libasan.so.5 >> > > #3 0x00007ffff72c3e5a in >> > __asan::FindThreadByStackAddress(unsigned long) >> > > () from /lib64/libasan.so.5 >> > > #4 0x00007ffff71d5fb6 in >> > __asan::GetStackAddressInformation(unsigned >> > > long, unsigned long, __asan::StackAddressDescription*) () >> > from >> > > /lib64/libasan.so.5 >> > > #5 0x00007ffff71d73f9 in >> > > __asan::AddressDescription::AddressDescription(unsigned >> > long, unsigned >> > > long, bool) () >> > > from /lib64/libasan.so.5 >> > > #6 0x00007ffff71d9e51 in >> > __asan::ErrorGeneric::ErrorGeneric(unsigned int, >> > > unsigned long, unsigned long, unsigned long, unsigned long, >> > bool, unsigned >> > > long) () from /lib64/libasan.so.5 >> > > #7 0x00007ffff72bdc2a in >> > __asan::ReportGenericError(unsigned long, >> > > unsigned long, unsigned long, unsigned long, bool, unsigned >> > long, unsigned >> > > int, bool) () from /lib64/libasan.so.5 >> > > #8 0x00007ffff721252c in __interceptor_strnlen.part.0 () >> > from >> > > /lib64/libasan.so.5 >> > > #9 0x00007ffff3595a0c in strnlen_s_inline >> > (s=0x7fff7177c520 >> > "/mem/stat >> > > segment", maxsize=128) >> > > at src/vppinfra/string.h:800 >> > > #10 0x00007ffff3595f63 in strcpy_s_inline >> > (dest=0x7fff6a0a9ab0 "", >> > > dmax=128, src=0x7fff7177c520 "/mem/stat segment") >> > > at src/vppinfra/string.h:960 >> > > #11 0x00007ffff3597e1c in strcpy_s (dest=0x7fff6a0a9ab0 "", >> > dmax=128, >> > > src=0x7fff7177c520 "/mem/stat segment") >> > > at src/vppinfra/string.c:274 >> > > #12 0x0000000000431820 in stat_segment_new_entry >> > (name=0x7fff7177c520 >> > > "/mem/stat segment", >> > > t=STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE) >> > > at src/vpp/stats/stat_segment.c:408 >> > > #13 0x00000000004401fb in vlib_stats_register_mem_heap >> > > (heap=0x7ffb67e00000) >> > > at src/vpp/stats/stat_segment_provider.c:96 >> > > #14 0x00000000004327fa in vlib_map_stat_segment_init () >> > > at src/vpp/stats/stat_segment.c:493 >> > > #15 0x00007ffff3e15d19 in vlib_main (vm=0x7fff6eeff680, >> > > input=0x7fff6a0a9f70) >> > > at src/vlib/main.c:3272 >> > > #16 0x00007ffff3f0d924 in thread0 (arg=140735054608000) >> > > at src/vlib/unix/main.c:671 >> > > ---Type <return> to continue, or q <return> to quit--- >> > > #17 0x00007ffff34d9504 in clib_calljmp () >> > > at src/vppinfra/longjmp.S:123 >> > > #18 0x00007fffffffc940 in ?? () >> > > #19 0x00007ffff3f0e67e in vlib_unix_main (argc=282, >> > argv=0x61d00001a480) >> > > at src/vlib/unix/main.c:751 >> > > #20 0x000000000040b482 in main (argc=282, >> > argv=0x61d00001a480) >> > > at src/vpp/vnet/main.c:336 >> > > (gdb) >> > > >> > > >> > > Thanks, >> > > Chetan >> > >> > >> >> >> >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20092): https://lists.fd.io/g/vpp-dev/message/20092 Mute This Topic: https://lists.fd.io/mt/85407456/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
