Hi Florin ,
This issue is coming when we are testing with iperf traffic with our
plugins that involve VPP TCP Host stack (VPP 21.06).
What's the best way to test VPP TCP host stack with Sanitizer without
involving our out of tree plugins ?
Thanks,
Chetan
On Tue, Sep 7, 2021 at 7:47 PM Florin Coras <fcoras.li...@gmail.com> wrote:
> Hi Chetan,
>
> Out of curiosity, is this while running some make test (like the iperf3
> one) or with actual traffic through vpp?
>
> Regards,
> Florin
>
> On Sep 7, 2021, at 12:17 AM, chetan bhasin <chetan.bhasin...@gmail.com>
> wrote:
>
> Hi Ben,
>
> After applying the patch , now its crashing at below place
>
> static void
> session_flush_pending_tx_buffers (session_worker_t * wrk,
> vlib_node_runtime_t * node)
> {
> vlib_buffer_enqueue_to_next (wrk->vm, node, wrk->pending_tx_buffers,
> wrk->pending_tx_nexts,
> vec_len (wrk->pending_tx_nexts));
> vec_reset_length (wrk->pending_tx_buffers);
> * vec_reset_length (wrk->pending_tx_nexts);*
> }
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffb527f9700 (LWP 27762)]
> 0x00007ffff71db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long,
> unsigned long*, unsigned long*) () from /lib64/libasan.so.5
> (gdb) bt
> #0 0x00007ffff71db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned
> long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5
> #1 0x00007ffff72c2a11 in
> __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*)
> () from /lib64/libasan.so.5
> #2 0x00007ffff72dcdc2 in
> __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
> (*)(__sanitizer::ThreadContextBase*, void*), void*) () from
> /lib64/libasan.so.5
> #3 0x00007ffff72c3e5a in __asan::FindThreadByStackAddress(unsigned long)
> () from /lib64/libasan.so.5
> #4 0x00007ffff71d5fb6 in __asan::GetStackAddressInformation(unsigned
> long, unsigned long, __asan::StackAddressDescription*) () from
> /lib64/libasan.so.5
> #5 0x00007ffff71d73f9 in
> __asan::AddressDescription::AddressDescription(unsigned long, unsigned
> long, bool) () from /lib64/libasan.so.5
> #6 0x00007ffff71d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int,
> unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned
> long) () from /lib64/libasan.so.5
> #7 0x00007ffff72bdc2a in __asan::ReportGenericError(unsigned long,
> unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned
> int, bool) () from /lib64/libasan.so.5
> #8 0x00007ffff72be927 in __asan_report_load4 () from /lib64/libasan.so.5
> *#9 0x00007ffff58e5185 in session_flush_pending_tx_buffers
> (wrk=0x7fff89101e80, node=0x7fff891a6e80)*
> * at src/vnet/session/session_node.c:1658*
> #10 0x00007ffff58e8531 in session_queue_node_fn (vm=0x7fff72bf2c40,
> node=0x7fff891a6e80, frame=0x0)
> at src/vnet/session/session_node.c:1812
> #11 0x00007ffff3e04121 in dispatch_node (vm=0x7fff72bf2c40,
> node=0x7fff891a6e80, type=VLIB_NODE_TYPE_INPUT,
> dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0,
> last_time_stamp=2510336347228480)
> at src/vlib/main.c:1024
> #12 0x00007ffff3e13394 in vlib_main_or_worker_loop (vm=0x7fff72bf2c40,
> is_main=0) at src/vlib/main.c:2949
> #13 0x00007ffff3e14fb8 in vlib_worker_loop (vm=0x7fff72bf2c40) at
> src/vlib/main.c:3114
> #14 0x00007ffff3eac8f1 in vlib_worker_thread_fn (arg=0x7fff700ef480) at
> src/vlib/threads.c:1560
> #15 0x00007ffff34d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123
> #16 0x00007ffb527f8230 in ?? ()
> #17 0x00007ffff3ea0141 in vlib_worker_thread_bootstrap_fn
> (arg=0x7fff700ef480) at src/vlib/threads.c:431
> #18 0x00007fff6d41971b in eal_thread_loop () from
> /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so
> #19 0x00007ffff38abea5 in start_thread () from /lib64/libpthread.so.0
> #20 0x00007ffff2e328cd in clone () from /lib64/libc.so.6
> (gdb)
>
>
> On Mon, Sep 6, 2021 at 6:10 PM Benoit Ganne (bganne) <bga...@cisco.com>
> wrote:
>
>> Yes we are aware of it, still working on the correct fix though.
>> In the meantime you can try to apply https://gerrit.fd.io/r/c/vpp/+/32765
>> which should workaround that for now.
>>
>> Best
>> ben
>>
>> > -----Original Message-----
>> > From: chetan bhasin <chetan.bhasin...@gmail.com>
>> > Sent: lundi 6 septembre 2021 14:21
>> > To: Benoit Ganne (bganne) <bga...@cisco.com>
>> > Cc: vpp-dev <vpp-dev@lists.fd.io>
>> > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled
>> >
>> > Hi,
>> >
>> > The below crash is coming as we involved VPP TCP host stack.
>> >
>> >
>> > Program received signal SIGSEGV, Segmentation fault.
>> > [Switching to Thread 0x7ffb527f9700 (LWP 2013)]
>> > 0x00007ffff71db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned
>> long,
>> > unsigned long*, unsigned long*) () from /lib64/libasan.so.5
>> > (gdb) bt
>> > #0 0x00007ffff71db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned
>> > long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5
>> > #1 0x00007ffff72c2a11 in
>> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*,
>> void*)
>> > () from /lib64/libasan.so.5
>> > #2 0x00007ffff72dcdc2 in
>> > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
>> > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from
>> > /lib64/libasan.so.5
>> > #3 0x00007ffff72c3e5a in __asan::FindThreadByStackAddress(unsigned
>> long)
>> > () from /lib64/libasan.so.5
>> > #4 0x00007ffff71d5fb6 in __asan::GetStackAddressInformation(unsigned
>> > long, unsigned long, __asan::StackAddressDescription*) () from
>> > /lib64/libasan.so.5
>> > #5 0x00007ffff71d73f9 in
>> > __asan::AddressDescription::AddressDescription(unsigned long, unsigned
>> > long, bool) () from /lib64/libasan.so.5
>> > #6 0x00007ffff71d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned
>> int,
>> > unsigned long, unsigned long, unsigned long, unsigned long, bool,
>> unsigned
>> > long) () from /lib64/libasan.so.5
>> > #7 0x00007ffff72bdc2a in __asan::ReportGenericError(unsigned long,
>> > unsigned long, unsigned long, unsigned long, bool, unsigned long,
>> unsigned
>> > int, bool) () from /lib64/libasan.so.5
>> > #8 0x00007ffff72bf194 in __asan_report_load_n () from
>> /lib64/libasan.so.5
>> > #9 0x00007ffff3f45463 in clib_mask_compare_u16_x64 (v=2,
>> > a=0x7fff89fd6150, n_elts=2) at src/vppinfra/vector_funcs.h:24
>> > #10 0x00007ffff3f4571e in clib_mask_compare_u16 (v=2, a=0x7fff89fd6150,
>> > mask=0x7ffb51fe2310, n_elts=2)
>> > at src/vppinfra/vector_funcs.h:79
>> > #11 0x00007ffff3f45b81 in enqueue_one (vm=0x7fff7314ec40,
>> > node=0x7fff89fe1e00, used_elt_bmp=0x7ffb51fe2440, next_index=2,
>> > buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, n_buffers=2, n_left=2,
>> > tmp=0x7ffb51fe2480) at src/vlib/buffer_funcs.c:30
>> > #12 0x00007ffff3f6bdae in vlib_buffer_enqueue_to_next_fn_skx
>> > (vm=0x7fff7314ec40, node=0x7fff89fe1e00, buffers=0x7fff7045a9e0,
>> > nexts=0x7fff89fd6150, count=2)
>> > at src/vlib/buffer_funcs.c:110
>> > #13 0x00007ffff58cd2b6 in vlib_buffer_enqueue_to_next
>> (vm=0x7fff7314ec40,
>> > node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150,
>> > count=2)
>> > at src/vlib/buffer_node.h:344
>> > #14 0x00007ffff58e4cf1 in session_flush_pending_tx_buffers
>> > (wrk=0x7fff8912b780, node=0x7fff89fe1e00)
>> > at src/vnet/session/session_node.c:1654
>> > #15 0x00007ffff58e844f in session_queue_node_fn (vm=0x7fff7314ec40,
>> > node=0x7fff89fe1e00, frame=0x0)
>> > at src/vnet/session/session_node.c:1812
>> > #16 0x00007ffff3e0402f in dispatch_node (vm=0x7fff7314ec40,
>> > node=0x7fff89fe1e00, type=VLIB_NODE_TYPE_INPUT,
>> > dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0,
>> > last_time_stamp=2463904994699296)
>> > at src/vlib/main.c:1024
>> > #17 0x00007ffff3e132a2 in vlib_main_or_worker_loop (vm=0x7fff7314ec40,
>> > is_main=0) at src/vlib/main.c:2949
>> > #18 0x00007ffff3e14ec6 in vlib_worker_loop (vm=0x7fff7314ec40) at
>> > src/vlib/main.c:3114
>> > #19 0x00007ffff3eac7ff in vlib_worker_thread_fn (arg=0x7fff7050ef40) at
>> > src/vlib/threads.c:1560
>> > #20 0x00007ffff34d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123
>> > #21 0x00007ffb527f8230 in ?? ()
>> > #22 0x00007ffff3ea004f in vlib_worker_thread_bootstrap_fn
>> > (arg=0x7fff7050ef40) at src/vlib/threads.c:431
>> > #23 0x00007fff6d41971b in eal_thread_loop () from
>> > /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so
>> > #24 0x00007ffff38abea5 in start_thread () from /lib64/libpthread.so.0
>> > #25 0x00007ffff2e328cd in clone () from /lib64/libc.so.6
>> > (gdb) :q
>> >
>> > On Mon, Sep 6, 2021 at 1:52 PM chetan bhasin <
>> chetan.bhasin...@gmail.com
>> > <mailto:chetan.bhasin...@gmail.com> > wrote:
>> >
>> >
>> > Hi Ben,
>> >
>> > Thanks for the direction. Looks like it will fix both the issues
>> as
>> > mentioned above. I will update you with the results after applying the
>> > patch.
>> >
>> > Is there any ASAN related patch inside the TCP host stack code. I
>> > will be sharing the issue shortly with you.
>> >
>> > Thanks,
>> > Chetan
>> >
>> > On Mon, Sep 6, 2021 at 1:22 PM Benoit Ganne (bganne)
>> > <bga...@cisco.com <mailto:bga...@cisco.com> > wrote:
>> >
>> >
>> > It should be fixed in master by
>> > https://gerrit.fd.io/r/c/vpp/+/32643
>> >
>> > ben
>> >
>> > > -----Original Message-----
>> > > From: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>
>> > <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io> > On Behalf Of chetan
>> > bhasin
>> > > Sent: lundi 6 septembre 2021 09:36
>> > > To: vpp-dev <vpp-dev@lists.fd.io <mailto:vpp-
>> > d...@lists.fd.io> >
>> > > Subject: [vpp-dev] VPP 2106 with Sanitizer enabled
>> > >
>> > > Hi
>> > >
>> > >
>> > > We are facing two errors with vpp2106 and Address
>> Sanitizer
>> > enabled.
>> > >
>> > > make V=1 -j4 build VPP_EXTRA_CMAKE_ARGS=-
>> > DVPP_ENABLE_SANITIZE_ADDR=ON
>> > >
>> > >
>> > > Work-around - After adding the two api’s string_key_sum
>> and
>> > > strnlen_s_inline to ASAN suppression, vpp2106 comes up
>> > fine.
>> > >
>> > >
>> > > Error 1:
>> > > ----------
>> > > Program received signal SIGSEGV, Segmentation fault.
>> > > 0x00007ffff71db5c1 in
>> > __asan::FakeStack::AddrIsInFakeStack(unsigned long,
>> > > unsigned long*, unsigned long*) ()
>> > > from /lib64/libasan.so.5
>> > > (gdb) bt
>> > > #0 0x00007ffff71db5c1 in
>> > __asan::FakeStack::AddrIsInFakeStack(unsigned
>> > > long, unsigned long*, unsigned long*) ()
>> > > from /lib64/libasan.so.5
>> > > #1 0x00007ffff72c2a11 in
>> > >
>> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*,
>> void*)
>> > > ()
>> > > from /lib64/libasan.so.5
>> > > #2 0x00007ffff72dcdc2 in
>> > >
>> __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
>> > > (*)(__sanitizer::ThreadContextBase*, void*), void*) ()
>> from
>> > > /lib64/libasan.so.5
>> > > #3 0x00007ffff72c3e5a in
>> > __asan::FindThreadByStackAddress(unsigned long)
>> > > () from /lib64/libasan.so.5
>> > > #4 0x00007ffff71d5fb6 in
>> > __asan::GetStackAddressInformation(unsigned
>> > > long, unsigned long, __asan::StackAddressDescription*)
>> ()
>> > from
>> > > /lib64/libasan.so.5
>> > > #5 0x00007ffff71d73f9 in
>> > > __asan::AddressDescription::AddressDescription(unsigned
>> > long, unsigned
>> > > long, bool) ()
>> > > from /lib64/libasan.so.5
>> > > #6 0x00007ffff71d9e51 in
>> > __asan::ErrorGeneric::ErrorGeneric(unsigned int,
>> > > unsigned long, unsigned long, unsigned long, unsigned
>> long,
>> > bool, unsigned
>> > > long) () from /lib64/libasan.so.5
>> > > #7 0x00007ffff72bdc2a in
>> > __asan::ReportGenericError(unsigned long,
>> > > unsigned long, unsigned long, unsigned long, bool,
>> unsigned
>> > long, unsigned
>> > > int, bool) () from /lib64/libasan.so.5
>> > > #8 0x00007ffff720ef9c in __interceptor_strlen.part.0 ()
>> > from
>> > > /lib64/libasan.so.5
>> > > #9 0x00007ffff34ce2ec in string_key_sum
>> (h=0x7fff6ff6e970,
>> > > key=140735097062688)
>> > > at src/vppinfra/hash.c:947
>> > > #10 0x00007ffff34caf15 in key_sum (h=0x7fff6ff6e970,
>> > key=140735097062688)
>> > > at src/vppinfra/hash.c:333
>> > > #11 0x00007ffff34cbf76 in lookup (v=0x7fff6ff6e9b8,
>> > key=140735097062688,
>> > > op=GET, new_value=0x0, old_value=0x0)
>> > > at src/vppinfra/hash.c:557
>> > > #12 0x00007ffff34cc59d in _hash_get_pair
>> (v=0x7fff6ff6e9b8,
>> > > key=140735097062688)
>> > > at src/vppinfra/hash.c:653
>> > > #13 0x000000000042e885 in lookup_hash_index
>> > (name=0x7fff7177c520
>> > > "/mem/stat segment")
>> > > at src/vpp/stats/stat_segment.c:69
>> > > #14 0x0000000000431790 in stat_segment_new_entry
>> > (name=0x7fff7177c520
>> > > "/mem/stat segment",
>> > > t=STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE)
>> > > at src/vpp/stats/stat_segment.c:402
>> > > #15 0x00000000004401fb in vlib_stats_register_mem_heap
>> > > (heap=0x7ffb67e00000)
>> > > at src/vpp/stats/stat_segment_provider.c:96
>> > > #16 0x00000000004327fa in vlib_map_stat_segment_init ()
>> > > at src/vpp/stats/stat_segment.c:493
>> > > ---Type <return> to continue, or q <return> to quit---
>> > > #17 0x00007ffff3e15d19 in vlib_main (vm=0x7fff6eeff680,
>> > > input=0x7fff6a0a9f70)
>> > > at src/vlib/main.c:3272
>> > > #18 0x00007ffff3f0d924 in thread0 (arg=140735054608000)
>> > > at src/vlib/unix/main.c:671
>> > > #19 0x00007ffff34d9504 in clib_calljmp ()
>> > > at src/vppinfra/longjmp.S:123
>> > > #20 0x00007fffffffc940 in ?? ()
>> > > #21 0x00007ffff3f0e67e in vlib_unix_main (argc=282,
>> > argv=0x61d00001a480)
>> > > at src/vlib/unix/main.c:751
>> > > #22 0x000000000040b482 in main (argc=282,
>> > argv=0x61d00001a480)
>> > > at src/vpp/vnet/main.c:336
>> > > (gdb) q
>> > >
>> > >
>> > >
>> > >
>> > > Error 2:
>> > > -----------
>> > > Program received signal SIGSEGV, Segmentation fault.
>> > > 0x00007ffff71db5c1 in
>> > __asan::FakeStack::AddrIsInFakeStack(unsigned long,
>> > > unsigned long*, unsigned long*) ()
>> > > from /lib64/libasan.so.5
>> > > (gdb) bt
>> > > #0 0x00007ffff71db5c1 in
>> > __asan::FakeStack::AddrIsInFakeStack(unsigned
>> > > long, unsigned long*, unsigned long*) ()
>> > > from /lib64/libasan.so.5
>> > > #1 0x00007ffff72c2a11 in
>> > >
>> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*,
>> void*)
>> > > ()
>> > > from /lib64/libasan.so.5
>> > > #2 0x00007ffff72dcdc2 in
>> > >
>> __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
>> > > (*)(__sanitizer::ThreadContextBase*, void*), void*) ()
>> from
>> > > /lib64/libasan.so.5
>> > > #3 0x00007ffff72c3e5a in
>> > __asan::FindThreadByStackAddress(unsigned long)
>> > > () from /lib64/libasan.so.5
>> > > #4 0x00007ffff71d5fb6 in
>> > __asan::GetStackAddressInformation(unsigned
>> > > long, unsigned long, __asan::StackAddressDescription*)
>> ()
>> > from
>> > > /lib64/libasan.so.5
>> > > #5 0x00007ffff71d73f9 in
>> > > __asan::AddressDescription::AddressDescription(unsigned
>> > long, unsigned
>> > > long, bool) ()
>> > > from /lib64/libasan.so.5
>> > > #6 0x00007ffff71d9e51 in
>> > __asan::ErrorGeneric::ErrorGeneric(unsigned int,
>> > > unsigned long, unsigned long, unsigned long, unsigned
>> long,
>> > bool, unsigned
>> > > long) () from /lib64/libasan.so.5
>> > > #7 0x00007ffff72bdc2a in
>> > __asan::ReportGenericError(unsigned long,
>> > > unsigned long, unsigned long, unsigned long, bool,
>> unsigned
>> > long, unsigned
>> > > int, bool) () from /lib64/libasan.so.5
>> > > #8 0x00007ffff721252c in __interceptor_strnlen.part.0
>> ()
>> > from
>> > > /lib64/libasan.so.5
>> > > #9 0x00007ffff3595a0c in strnlen_s_inline
>> (s=0x7fff7177c520
>> > "/mem/stat
>> > > segment", maxsize=128)
>> > > at src/vppinfra/string.h:800
>> > > #10 0x00007ffff3595f63 in strcpy_s_inline
>> > (dest=0x7fff6a0a9ab0 "",
>> > > dmax=128, src=0x7fff7177c520 "/mem/stat segment")
>> > > at src/vppinfra/string.h:960
>> > > #11 0x00007ffff3597e1c in strcpy_s (dest=0x7fff6a0a9ab0
>> "",
>> > dmax=128,
>> > > src=0x7fff7177c520 "/mem/stat segment")
>> > > at src/vppinfra/string.c:274
>> > > #12 0x0000000000431820 in stat_segment_new_entry
>> > (name=0x7fff7177c520
>> > > "/mem/stat segment",
>> > > t=STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE)
>> > > at src/vpp/stats/stat_segment.c:408
>> > > #13 0x00000000004401fb in vlib_stats_register_mem_heap
>> > > (heap=0x7ffb67e00000)
>> > > at src/vpp/stats/stat_segment_provider.c:96
>> > > #14 0x00000000004327fa in vlib_map_stat_segment_init ()
>> > > at src/vpp/stats/stat_segment.c:493
>> > > #15 0x00007ffff3e15d19 in vlib_main (vm=0x7fff6eeff680,
>> > > input=0x7fff6a0a9f70)
>> > > at src/vlib/main.c:3272
>> > > #16 0x00007ffff3f0d924 in thread0 (arg=140735054608000)
>> > > at src/vlib/unix/main.c:671
>> > > ---Type <return> to continue, or q <return> to quit---
>> > > #17 0x00007ffff34d9504 in clib_calljmp ()
>> > > at src/vppinfra/longjmp.S:123
>> > > #18 0x00007fffffffc940 in ?? ()
>> > > #19 0x00007ffff3f0e67e in vlib_unix_main (argc=282,
>> > argv=0x61d00001a480)
>> > > at src/vlib/unix/main.c:751
>> > > #20 0x000000000040b482 in main (argc=282,
>> > argv=0x61d00001a480)
>> > > at src/vpp/vnet/main.c:336
>> > > (gdb)
>> > >
>> > >
>> > > Thanks,
>> > > Chetan
>> >
>> >
>>
>>
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20091): https://lists.fd.io/g/vpp-dev/message/20091
Mute This Topic: https://lists.fd.io/mt/85407456/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
