Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Hi Benoit,
Thanks a lot!
We are not observing any crash in the VPP TCP host stack after applying the
second patch provided by you with SANITIZER enabled build.
Thanks,
Chetan
On Wed, Sep 8, 2021 at 9:41 PM chetan bhasin via lists.fd.io
wrote:
> Thanks Benoit.
>
> Actually Same out of tree plugins are working fine with SANITIZER Enabled
> on vpp 2101.
>
> Will update the result with the patch provided by you and test which
> Florin suggested .
>
> Thanks & Regards,
> Chetan
>
> On Wed, Sep 8, 2021 at 8:13 PM Benoit Ganne (bganne)
> wrote:
>
>> Here is maybe a better fix: https://gerrit.fd.io/r/c/vpp/+/33693
>> My previous changeset was only hiding the underlying issue which is
>> arrays passed to vlib_buffer_enqueue_to_next() must be big enough to allow
>> for some overflow (because of the use of clib_mask_compare_u16() and
>> clib_mask_compare_u32() that overflow for optimization reasons).
>> This is usually fine for your typical VPP node which allocates
>> buffer[VLIB_FRAME_SIZE] and nexts[VLIB_FRAME_SIZE] but not when using
>> vectors.
>> This patch should actually fix the issue (and tests should pass), but
>> there might be better ways.
>> However I do not think the last issue reported by Chetan will be fixed
>> with that one, looks like a new one.
>> Chetan, Florin's request is good: do you reproduce it only with your
>> plugin (in which case that could be an issue in your plugin) or also with
>> stock VPP?
>>
>> ben
>>
>> > -Original Message-
>> > From: Florin Coras
>> > Sent: mercredi 8 septembre 2021 08:39
>> > To: chetan bhasin
>> > Cc: Benoit Ganne (bganne) ; vpp-dev > > d...@lists.fd.io>
>> > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled
>> >
>> > Hi Chetan,
>> >
>> > Something like the following should exercise using iperf and homegrown
>> > test apps most of the components of the host stack, i.e., vcl, session
>> > layer and the transports, including tcp:
>> >
>> > make test-debug VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON
>> CC=gcc-
>> > 10 TEST=vcl CACHE_OUTPUT=0
>> >
>> > Having said that, it seems some issue has creeped in since last time
>> I’ve
>> > tested (granted it’s been a long time). That is, most of the tests seems
>> > to fail but I’ve only checked without Benoit’s patch.
>> >
>> > Regards,
>> > Florin
>> >
>> >
>> > On Sep 7, 2021, at 9:57 PM, chetan bhasin
>> > mailto:chetan.bhasin...@gmail.com> >
>> wrote:
>> >
>> > Hi Florin ,
>> >
>> > This issue is coming when we are testing with iperf traffic with
>> our
>> > plugins that involve VPP TCP Host stack (VPP 21.06).
>> >
>> > What's the best way to test VPP TCP host stack with Sanitizer
>> > without involving our out of tree plugins ?
>> >
>> >
>> > Thanks,
>> > Chetan
>> >
>> > On Tue, Sep 7, 2021 at 7:47 PM Florin Coras <
>> fcoras.li...@gmail.com
>> > <mailto:fcoras.li...@gmail.com> > wrote:
>> >
>> >
>> > Hi Chetan,
>> >
>> > Out of curiosity, is this while running some make test
>> (like
>> > the iperf3 one) or with actual traffic through vpp?
>> >
>> > Regards,
>> > Florin
>> >
>> >
>> >
>> > On Sep 7, 2021, at 12:17 AM, chetan bhasin
>> > mailto:chetan.bhasin...@gmail.com> >
>> wrote:
>> >
>> > Hi Ben,
>> >
>> > After applying the patch , now its crashing at
>> below
>> > place
>> >
>> >
>> > static void
>> > session_flush_pending_tx_buffers (session_worker_t * wrk,
>> > vlib_node_runtime_t * node)
>> > {
>> > vlib_buffer_enqueue_to_next (wrk->vm, node,
>> wrk->pending_tx_buffers,
>> >
>> > wrk->pending_tx_nexts,
>> > vec_len (wrk->pending_tx_nexts));
>> > vec_reset_length (wrk->pending_tx_buffers);
>> > vec_reset_length (wrk->pending_tx_nexts);
>> > }
>> > Program received signal SIGSEGV, Segmentation
>> fault.
>> > [Switching to Thread 0x7ffb527f9700 (LWP 27762)]
>> >
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Thanks Benoit.
Actually Same out of tree plugins are working fine with SANITIZER Enabled
on vpp 2101.
Will update the result with the patch provided by you and test which
Florin suggested .
Thanks & Regards,
Chetan
On Wed, Sep 8, 2021 at 8:13 PM Benoit Ganne (bganne)
wrote:
> Here is maybe a better fix: https://gerrit.fd.io/r/c/vpp/+/33693
> My previous changeset was only hiding the underlying issue which is arrays
> passed to vlib_buffer_enqueue_to_next() must be big enough to allow for
> some overflow (because of the use of clib_mask_compare_u16() and
> clib_mask_compare_u32() that overflow for optimization reasons).
> This is usually fine for your typical VPP node which allocates
> buffer[VLIB_FRAME_SIZE] and nexts[VLIB_FRAME_SIZE] but not when using
> vectors.
> This patch should actually fix the issue (and tests should pass), but
> there might be better ways.
> However I do not think the last issue reported by Chetan will be fixed
> with that one, looks like a new one.
> Chetan, Florin's request is good: do you reproduce it only with your
> plugin (in which case that could be an issue in your plugin) or also with
> stock VPP?
>
> ben
>
> > -Original Message-
> > From: Florin Coras
> > Sent: mercredi 8 septembre 2021 08:39
> > To: chetan bhasin
> > Cc: Benoit Ganne (bganne) ; vpp-dev > d...@lists.fd.io>
> > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled
> >
> > Hi Chetan,
> >
> > Something like the following should exercise using iperf and homegrown
> > test apps most of the components of the host stack, i.e., vcl, session
> > layer and the transports, including tcp:
> >
> > make test-debug VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON
> CC=gcc-
> > 10 TEST=vcl CACHE_OUTPUT=0
> >
> > Having said that, it seems some issue has creeped in since last time I’ve
> > tested (granted it’s been a long time). That is, most of the tests seems
> > to fail but I’ve only checked without Benoit’s patch.
> >
> > Regards,
> > Florin
> >
> >
> > On Sep 7, 2021, at 9:57 PM, chetan bhasin
> > mailto:chetan.bhasin...@gmail.com> > wrote:
> >
> > Hi Florin ,
> >
> > This issue is coming when we are testing with iperf traffic with
> our
> > plugins that involve VPP TCP Host stack (VPP 21.06).
> >
> > What's the best way to test VPP TCP host stack with Sanitizer
> > without involving our out of tree plugins ?
> >
> >
> > Thanks,
> > Chetan
> >
> > On Tue, Sep 7, 2021 at 7:47 PM Florin Coras <
> fcoras.li...@gmail.com
> > <mailto:fcoras.li...@gmail.com> > wrote:
> >
> >
> > Hi Chetan,
> >
> > Out of curiosity, is this while running some make test
> (like
> > the iperf3 one) or with actual traffic through vpp?
> >
> > Regards,
> > Florin
> >
> >
> >
> > On Sep 7, 2021, at 12:17 AM, chetan bhasin
> > mailto:chetan.bhasin...@gmail.com> > wrote:
> >
> > Hi Ben,
> >
> > After applying the patch , now its crashing at
> below
> > place
> >
> >
> > static void
> > session_flush_pending_tx_buffers (session_worker_t * wrk,
> > vlib_node_runtime_t * node)
> > {
> > vlib_buffer_enqueue_to_next (wrk->vm, node,
> wrk->pending_tx_buffers,
> >
> > wrk->pending_tx_nexts,
> > vec_len (wrk->pending_tx_nexts));
> > vec_reset_length (wrk->pending_tx_buffers);
> > vec_reset_length (wrk->pending_tx_nexts);
> > }
> > Program received signal SIGSEGV, Segmentation
> fault.
> > [Switching to Thread 0x7ffb527f9700 (LWP 27762)]
> > 0x771db5c1 in
> > __asan::FakeStack::AddrIsInFakeStack(unsigned long, unsigned long*,
> > unsigned long*) () from /lib64/libasan.so.5
> > (gdb) bt
> > #0 0x771db5c1 in
> > __asan::FakeStack::AddrIsInFakeStack(unsigned long, unsigned long*,
> > unsigned long*) () from /lib64/libasan.so.5
> > #1 0x772c2a11 in
> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*,
> void*)
> > () from /lib64/libasan.so.5
> > #2 0x772dcdc2 in
> > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
> > (*)(__sanitizer::ThreadContextBase*, void*), voi
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Here is maybe a better fix: https://gerrit.fd.io/r/c/vpp/+/33693
My previous changeset was only hiding the underlying issue which is arrays
passed to vlib_buffer_enqueue_to_next() must be big enough to allow for some
overflow (because of the use of clib_mask_compare_u16() and
clib_mask_compare_u32() that overflow for optimization reasons).
This is usually fine for your typical VPP node which allocates
buffer[VLIB_FRAME_SIZE] and nexts[VLIB_FRAME_SIZE] but not when using vectors.
This patch should actually fix the issue (and tests should pass), but there
might be better ways.
However I do not think the last issue reported by Chetan will be fixed with
that one, looks like a new one.
Chetan, Florin's request is good: do you reproduce it only with your plugin (in
which case that could be an issue in your plugin) or also with stock VPP?
ben
> -Original Message-
> From: Florin Coras
> Sent: mercredi 8 septembre 2021 08:39
> To: chetan bhasin
> Cc: Benoit Ganne (bganne) ; vpp-dev d...@lists.fd.io>
> Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled
>
> Hi Chetan,
>
> Something like the following should exercise using iperf and homegrown
> test apps most of the components of the host stack, i.e., vcl, session
> layer and the transports, including tcp:
>
> make test-debug VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON CC=gcc-
> 10 TEST=vcl CACHE_OUTPUT=0
>
> Having said that, it seems some issue has creeped in since last time I’ve
> tested (granted it’s been a long time). That is, most of the tests seems
> to fail but I’ve only checked without Benoit’s patch.
>
> Regards,
> Florin
>
>
> On Sep 7, 2021, at 9:57 PM, chetan bhasin
> mailto:chetan.bhasin...@gmail.com> > wrote:
>
> Hi Florin ,
>
> This issue is coming when we are testing with iperf traffic with our
> plugins that involve VPP TCP Host stack (VPP 21.06).
>
> What's the best way to test VPP TCP host stack with Sanitizer
> without involving our out of tree plugins ?
>
>
> Thanks,
> Chetan
>
> On Tue, Sep 7, 2021 at 7:47 PM Florin Coras <mailto:fcoras.li...@gmail.com> > wrote:
>
>
> Hi Chetan,
>
> Out of curiosity, is this while running some make test (like
> the iperf3 one) or with actual traffic through vpp?
>
> Regards,
> Florin
>
>
>
> On Sep 7, 2021, at 12:17 AM, chetan bhasin
> mailto:chetan.bhasin...@gmail.com> > wrote:
>
> Hi Ben,
>
> After applying the patch , now its crashing at below
> place
>
>
> static void
> session_flush_pending_tx_buffers (session_worker_t * wrk,
> vlib_node_runtime_t * node)
> {
> vlib_buffer_enqueue_to_next (wrk->vm, node, wrk->pending_tx_buffers,
>
> wrk->pending_tx_nexts,
> vec_len (wrk->pending_tx_nexts));
> vec_reset_length (wrk->pending_tx_buffers);
> vec_reset_length (wrk->pending_tx_nexts);
> }
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffb527f9700 (LWP 27762)]
> 0x771db5c1 in
> __asan::FakeStack::AddrIsInFakeStack(unsigned long, unsigned long*,
> unsigned long*) () from /lib64/libasan.so.5
> (gdb) bt
> #0 0x771db5c1 in
> __asan::FakeStack::AddrIsInFakeStack(unsigned long, unsigned long*,
> unsigned long*) () from /lib64/libasan.so.5
> #1 0x772c2a11 in
> __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*)
> () from /lib64/libasan.so.5
> #2 0x772dcdc2 in
> __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
> (*)(__sanitizer::ThreadContextBase*, void*), void*) () from
> /lib64/libasan.so.5
> #3 0x772c3e5a in
> __asan::FindThreadByStackAddress(unsigned long) () from
> /lib64/libasan.so.5
> #4 0x771d5fb6 in
> __asan::GetStackAddressInformation(unsigned long, unsigned long,
> __asan::StackAddressDescription*) () from /lib64/libasan.so.5
> #5 0x771d73f9 in
> __asan::AddressDescription::AddressDescription(unsigned long, unsigned
> long, bool) () from /lib64/libasan.so.5
> #6 0x771d9e51 in
> __asan::ErrorGeneric::ErrorGeneric(unsigned int, unsigned long, unsigned
> long, unsigned long, unsigned long, bool, unsigned long) () from
> /lib64/libasan.so.5
> #7 0x772bdc2a in
> __asan::ReportGenericError(unsigne
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so >> #19 0x738abea5 in start_thread () from /lib64/libpthread.so.0 >> #20 0x72e328cd in clone () from /lib64/libc.so.6 >> (gdb) >> >> >> On Mon, Sep 6, 2021 at 6:10 PM Benoit Ganne (bganne) > <mailto:bga...@cisco.com>> wrote: >> Yes we are aware of it, still working on the correct fix though. >> In the meantime you can try to apply https://gerrit.fd.io/r/c/vpp/+/32765 >> <https://gerrit.fd.io/r/c/vpp/+/32765> which should workaround that for now. >> >> Best >> ben >> >> > -Original Message- >> > From: chetan bhasin > > <mailto:chetan.bhasin...@gmail.com>> >> > Sent: lundi 6 septembre 2021 14:21 >> > To: Benoit Ganne (bganne) mailto:bga...@cisco.com>> >> > Cc: vpp-dev mailto:vpp-dev@lists.fd.io>> >> > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled >> > >> > Hi, >> > >> > The below crash is coming as we involved VPP TCP host stack. >> > >> > >> > Program received signal SIGSEGV, Segmentation fault. >> > [Switching to Thread 0x7ffb527f9700 (LWP 2013)] >> > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, >> > unsigned long*, unsigned long*) () from /lib64/libasan.so.5 >> > (gdb) bt >> > #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned >> > long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5 >> > #1 0x772c2a11 in >> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) >> > () from /lib64/libasan.so.5 >> > #2 0x772dcdc2 in >> > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool >> > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from >> > /lib64/libasan.so.5 >> > #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long) >> > () from /lib64/libasan.so.5 >> > #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned >> > long, unsigned long, __asan::StackAddressDescription*) () from >> > /lib64/libasan.so.5 >> > #5 0x771d73f9 in >> > __asan::AddressDescription::AddressDescription(unsigned long, unsigned >> > long, bool) () from /lib64/libasan.so.5 >> > #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, >> > unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned >> > long) () from /lib64/libasan.so.5 >> > #7 0x772bdc2a in __asan::ReportGenericError(unsigned long, >> > unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned >> > int, bool) () from /lib64/libasan.so.5 >> > #8 0x772bf194 in __asan_report_load_n () from /lib64/libasan.so.5 >> > #9 0x73f45463 in clib_mask_compare_u16_x64 (v=2, >> > a=0x7fff89fd6150, n_elts=2) at src/vppinfra/vector_funcs.h:24 >> > #10 0x73f4571e in clib_mask_compare_u16 (v=2, a=0x7fff89fd6150, >> > mask=0x7ffb51fe2310, n_elts=2) >> > at src/vppinfra/vector_funcs.h:79 >> > #11 0x73f45b81 in enqueue_one (vm=0x7fff7314ec40, >> > node=0x7fff89fe1e00, used_elt_bmp=0x7ffb51fe2440, next_index=2, >> > buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, n_buffers=2, n_left=2, >> > tmp=0x7ffb51fe2480) at src/vlib/buffer_funcs.c:30 >> > #12 0x73f6bdae in vlib_buffer_enqueue_to_next_fn_skx >> > (vm=0x7fff7314ec40, node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, >> > nexts=0x7fff89fd6150, count=2) >> > at src/vlib/buffer_funcs.c:110 >> > #13 0x758cd2b6 in vlib_buffer_enqueue_to_next (vm=0x7fff7314ec40, >> > node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, >> > count=2) >> > at src/vlib/buffer_node.h:344 >> > #14 0x758e4cf1 in session_flush_pending_tx_buffers >> > (wrk=0x7fff8912b780, node=0x7fff89fe1e00) >> > at src/vnet/session/session_node.c:1654 >> > #15 0x758e844f in session_queue_node_fn (vm=0x7fff7314ec40, >> > node=0x7fff89fe1e00, frame=0x0) >> > at src/vnet/session/session_node.c:1812 >> > #16 0x73e0402f in dispatch_node (vm=0x7fff7314ec40, >> > node=0x7fff89fe1e00, type=VLIB_NODE_TYPE_INPUT, >> > dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, >> > last_time_stamp=2463904994699296) >> > at src/vlib/main.c:1024 >> > #17 0x73e132a2 in vlib_main_or_worker_loop (vm=0x7fff7314ec40, >> >
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Hi Florin ,
This issue is coming when we are testing with iperf traffic with our
plugins that involve VPP TCP Host stack (VPP 21.06).
What's the best way to test VPP TCP host stack with Sanitizer without
involving our out of tree plugins ?
Thanks,
Chetan
On Tue, Sep 7, 2021 at 7:47 PM Florin Coras wrote:
> Hi Chetan,
>
> Out of curiosity, is this while running some make test (like the iperf3
> one) or with actual traffic through vpp?
>
> Regards,
> Florin
>
> On Sep 7, 2021, at 12:17 AM, chetan bhasin
> wrote:
>
> Hi Ben,
>
> After applying the patch , now its crashing at below place
>
> static void
> session_flush_pending_tx_buffers (session_worker_t * wrk,
> vlib_node_runtime_t * node)
> {
> vlib_buffer_enqueue_to_next (wrk->vm, node, wrk->pending_tx_buffers,
> wrk->pending_tx_nexts,
> vec_len (wrk->pending_tx_nexts));
> vec_reset_length (wrk->pending_tx_buffers);
> * vec_reset_length (wrk->pending_tx_nexts);*
> }
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffb527f9700 (LWP 27762)]
> 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long,
> unsigned long*, unsigned long*) () from /lib64/libasan.so.5
> (gdb) bt
> #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned
> long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5
> #1 0x772c2a11 in
> __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*)
> () from /lib64/libasan.so.5
> #2 0x772dcdc2 in
> __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
> (*)(__sanitizer::ThreadContextBase*, void*), void*) () from
> /lib64/libasan.so.5
> #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long)
> () from /lib64/libasan.so.5
> #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned
> long, unsigned long, __asan::StackAddressDescription*) () from
> /lib64/libasan.so.5
> #5 0x771d73f9 in
> __asan::AddressDescription::AddressDescription(unsigned long, unsigned
> long, bool) () from /lib64/libasan.so.5
> #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int,
> unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned
> long) () from /lib64/libasan.so.5
> #7 0x772bdc2a in __asan::ReportGenericError(unsigned long,
> unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned
> int, bool) () from /lib64/libasan.so.5
> #8 0x772be927 in __asan_report_load4 () from /lib64/libasan.so.5
> *#9 0x758e5185 in session_flush_pending_tx_buffers
> (wrk=0x7fff89101e80, node=0x7fff891a6e80)*
> *at src/vnet/session/session_node.c:1658*
> #10 0x758e8531 in session_queue_node_fn (vm=0x7fff72bf2c40,
> node=0x7fff891a6e80, frame=0x0)
> at src/vnet/session/session_node.c:1812
> #11 0x73e04121 in dispatch_node (vm=0x7fff72bf2c40,
> node=0x7fff891a6e80, type=VLIB_NODE_TYPE_INPUT,
> dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0,
> last_time_stamp=2510336347228480)
> at src/vlib/main.c:1024
> #12 0x73e13394 in vlib_main_or_worker_loop (vm=0x7fff72bf2c40,
> is_main=0) at src/vlib/main.c:2949
> #13 0x73e14fb8 in vlib_worker_loop (vm=0x7fff72bf2c40) at
> src/vlib/main.c:3114
> #14 0x73eac8f1 in vlib_worker_thread_fn (arg=0x7fff700ef480) at
> src/vlib/threads.c:1560
> #15 0x734d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123
> #16 0x7ffb527f8230 in ?? ()
> #17 0x73ea0141 in vlib_worker_thread_bootstrap_fn
> (arg=0x7fff700ef480) at src/vlib/threads.c:431
> #18 0x7fff6d41971b in eal_thread_loop () from
> /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so
> #19 0x738abea5 in start_thread () from /lib64/libpthread.so.0
> #20 0x72e328cd in clone () from /lib64/libc.so.6
> (gdb)
>
>
> On Mon, Sep 6, 2021 at 6:10 PM Benoit Ganne (bganne)
> wrote:
>
>> Yes we are aware of it, still working on the correct fix though.
>> In the meantime you can try to apply https://gerrit.fd.io/r/c/vpp/+/32765
>> which should workaround that for now.
>>
>> Best
>> ben
>>
>> > -Original Message-
>> > From: chetan bhasin
>> > Sent: lundi 6 septembre 2021 14:21
>> > To: Benoit Ganne (bganne)
>> > Cc: vpp-dev
>> > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled
>> >
>> > Hi,
>> >
>> > The below crash is coming as we involved VPP TCP host stack.
>> >
>> >
>> > Program received signal SIGSEGV, Segmentation fault.
>> > [Switching to Thread 0x7ffb527f9700 (LWP 2013)]
>> > 0x7f
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Hi Chetan,
Out of curiosity, is this while running some make test (like the iperf3 one) or
with actual traffic through vpp?
Regards,
Florin
> On Sep 7, 2021, at 12:17 AM, chetan bhasin wrote:
>
> Hi Ben,
>
> After applying the patch , now its crashing at below place
>
> static void
> session_flush_pending_tx_buffers (session_worker_t * wrk,
> vlib_node_runtime_t * node)
> {
> vlib_buffer_enqueue_to_next (wrk->vm, node, wrk->pending_tx_buffers,
> wrk->pending_tx_nexts,
> vec_len (wrk->pending_tx_nexts));
> vec_reset_length (wrk->pending_tx_buffers);
> vec_reset_length (wrk->pending_tx_nexts);
> }
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffb527f9700 (LWP 27762)]
> 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long,
> unsigned long*, unsigned long*) () from /lib64/libasan.so.5
> (gdb) bt
> #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long,
> unsigned long*, unsigned long*) () from /lib64/libasan.so.5
> #1 0x772c2a11 in
> __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) ()
> from /lib64/libasan.so.5
> #2 0x772dcdc2 in
> __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
> (*)(__sanitizer::ThreadContextBase*, void*), void*) () from
> /lib64/libasan.so.5
> #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long) ()
> from /lib64/libasan.so.5
> #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned long,
> unsigned long, __asan::StackAddressDescription*) () from /lib64/libasan.so.5
> #5 0x771d73f9 in
> __asan::AddressDescription::AddressDescription(unsigned long, unsigned long,
> bool) () from /lib64/libasan.so.5
> #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int,
> unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned
> long) () from /lib64/libasan.so.5
> #7 0x772bdc2a in __asan::ReportGenericError(unsigned long, unsigned
> long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool)
> () from /lib64/libasan.so.5
> #8 0x772be927 in __asan_report_load4 () from /lib64/libasan.so.5
> #9 0x758e5185 in session_flush_pending_tx_buffers
> (wrk=0x7fff89101e80, node=0x7fff891a6e80)
> at src/vnet/session/session_node.c:1658
> #10 0x758e8531 in session_queue_node_fn (vm=0x7fff72bf2c40,
> node=0x7fff891a6e80, frame=0x0)
> at src/vnet/session/session_node.c:1812
> #11 0x73e04121 in dispatch_node (vm=0x7fff72bf2c40,
> node=0x7fff891a6e80, type=VLIB_NODE_TYPE_INPUT,
> dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0,
> last_time_stamp=2510336347228480)
> at src/vlib/main.c:1024
> #12 0x73e13394 in vlib_main_or_worker_loop (vm=0x7fff72bf2c40,
> is_main=0) at src/vlib/main.c:2949
> #13 0x73e14fb8 in vlib_worker_loop (vm=0x7fff72bf2c40) at
> src/vlib/main.c:3114
> #14 0x73eac8f1 in vlib_worker_thread_fn (arg=0x7fff700ef480) at
> src/vlib/threads.c:1560
> #15 0x734d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123
> #16 0x7ffb527f8230 in ?? ()
> #17 0x73ea0141 in vlib_worker_thread_bootstrap_fn
> (arg=0x7fff700ef480) at src/vlib/threads.c:431
> #18 0x7fff6d41971b in eal_thread_loop () from
> /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so
> #19 0x738abea5 in start_thread () from /lib64/libpthread.so.0
> #20 0x72e328cd in clone () from /lib64/libc.so.6
> (gdb)
>
>
> On Mon, Sep 6, 2021 at 6:10 PM Benoit Ganne (bganne) <mailto:bga...@cisco.com>> wrote:
> Yes we are aware of it, still working on the correct fix though.
> In the meantime you can try to apply https://gerrit.fd.io/r/c/vpp/+/32765
> <https://gerrit.fd.io/r/c/vpp/+/32765> which should workaround that for now.
>
> Best
> ben
>
> > -----Original Message-
> > From: chetan bhasin > <mailto:chetan.bhasin...@gmail.com>>
> > Sent: lundi 6 septembre 2021 14:21
> > To: Benoit Ganne (bganne) mailto:bga...@cisco.com>>
> > Cc: vpp-dev mailto:vpp-dev@lists.fd.io>>
> > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled
> >
> > Hi,
> >
> > The below crash is coming as we involved VPP TCP host stack.
> >
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0x7ffb527f9700 (LWP 2013)]
> > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long,
> > unsigned long*, unsigned long*) () from /lib64/libasan.so.
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Hi Ben,
After applying the patch , now its crashing at below place
static void
session_flush_pending_tx_buffers (session_worker_t * wrk,
vlib_node_runtime_t * node)
{
vlib_buffer_enqueue_to_next (wrk->vm, node, wrk->pending_tx_buffers,
wrk->pending_tx_nexts,
vec_len (wrk->pending_tx_nexts));
vec_reset_length (wrk->pending_tx_buffers);
* vec_reset_length (wrk->pending_tx_nexts);*
}
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffb527f9700 (LWP 27762)]
0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long,
unsigned long*, unsigned long*) () from /lib64/libasan.so.5
(gdb) bt
#0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned
long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5
#1 0x772c2a11 in
__asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*)
() from /lib64/libasan.so.5
#2 0x772dcdc2 in
__sanitizer::ThreadRegistry::FindThreadContextLocked(bool
(*)(__sanitizer::ThreadContextBase*, void*), void*) () from
/lib64/libasan.so.5
#3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long)
() from /lib64/libasan.so.5
#4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned long,
unsigned long, __asan::StackAddressDescription*) () from /lib64/libasan.so.5
#5 0x771d73f9 in
__asan::AddressDescription::AddressDescription(unsigned long, unsigned
long, bool) () from /lib64/libasan.so.5
#6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int,
unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned
long) () from /lib64/libasan.so.5
#7 0x772bdc2a in __asan::ReportGenericError(unsigned long,
unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned
int, bool) () from /lib64/libasan.so.5
#8 0x772be927 in __asan_report_load4 () from /lib64/libasan.so.5
*#9 0x758e5185 in session_flush_pending_tx_buffers
(wrk=0x7fff89101e80, node=0x7fff891a6e80)*
*at src/vnet/session/session_node.c:1658*
#10 0x758e8531 in session_queue_node_fn (vm=0x7fff72bf2c40,
node=0x7fff891a6e80, frame=0x0)
at src/vnet/session/session_node.c:1812
#11 0x73e04121 in dispatch_node (vm=0x7fff72bf2c40,
node=0x7fff891a6e80, type=VLIB_NODE_TYPE_INPUT,
dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0,
last_time_stamp=2510336347228480)
at src/vlib/main.c:1024
#12 0x73e13394 in vlib_main_or_worker_loop (vm=0x7fff72bf2c40,
is_main=0) at src/vlib/main.c:2949
#13 0x73e14fb8 in vlib_worker_loop (vm=0x7fff72bf2c40) at
src/vlib/main.c:3114
#14 0x73eac8f1 in vlib_worker_thread_fn (arg=0x7fff700ef480) at
src/vlib/threads.c:1560
#15 0x734d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123
#16 0x7ffb527f8230 in ?? ()
#17 0x73ea0141 in vlib_worker_thread_bootstrap_fn
(arg=0x7fff700ef480) at src/vlib/threads.c:431
#18 0x7fff6d41971b in eal_thread_loop () from
/opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so
#19 0x738abea5 in start_thread () from /lib64/libpthread.so.0
#20 0x72e328cd in clone () from /lib64/libc.so.6
(gdb)
On Mon, Sep 6, 2021 at 6:10 PM Benoit Ganne (bganne)
wrote:
> Yes we are aware of it, still working on the correct fix though.
> In the meantime you can try to apply https://gerrit.fd.io/r/c/vpp/+/32765
> which should workaround that for now.
>
> Best
> ben
>
> > -Original Message-
> > From: chetan bhasin
> > Sent: lundi 6 septembre 2021 14:21
> > To: Benoit Ganne (bganne)
> > Cc: vpp-dev
> > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled
> >
> > Hi,
> >
> > The below crash is coming as we involved VPP TCP host stack.
> >
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0x7ffb527f9700 (LWP 2013)]
> > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long,
> > unsigned long*, unsigned long*) () from /lib64/libasan.so.5
> > (gdb) bt
> > #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned
> > long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5
> > #1 0x772c2a11 in
> > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*,
> void*)
> > () from /lib64/libasan.so.5
> > #2 0x772dcdc2 in
> > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool
> > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from
> > /lib64/libasan.so.5
> > #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long)
> > () from /lib64/libasan.so.5
> > #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned
> > long, unsigned long, __asan::StackAddressDescription*) () from
> > /lib64/libasan.so.5
> > #5 0x771d73f9 i
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Thanks a ton Ben! On Mon, Sep 6, 2021 at 6:10 PM Benoit Ganne (bganne) wrote: > Yes we are aware of it, still working on the correct fix though. > In the meantime you can try to apply https://gerrit.fd.io/r/c/vpp/+/32765 > which should workaround that for now. > > Best > ben > > > -Original Message- > > From: chetan bhasin > > Sent: lundi 6 septembre 2021 14:21 > > To: Benoit Ganne (bganne) > > Cc: vpp-dev > > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled > > > > Hi, > > > > The below crash is coming as we involved VPP TCP host stack. > > > > > > Program received signal SIGSEGV, Segmentation fault. > > [Switching to Thread 0x7ffb527f9700 (LWP 2013)] > > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, > > unsigned long*, unsigned long*) () from /lib64/libasan.so.5 > > (gdb) bt > > #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned > > long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5 > > #1 0x772c2a11 in > > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, > void*) > > () from /lib64/libasan.so.5 > > #2 0x772dcdc2 in > > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool > > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from > > /lib64/libasan.so.5 > > #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long) > > () from /lib64/libasan.so.5 > > #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned > > long, unsigned long, __asan::StackAddressDescription*) () from > > /lib64/libasan.so.5 > > #5 0x771d73f9 in > > __asan::AddressDescription::AddressDescription(unsigned long, unsigned > > long, bool) () from /lib64/libasan.so.5 > > #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned > int, > > unsigned long, unsigned long, unsigned long, unsigned long, bool, > unsigned > > long) () from /lib64/libasan.so.5 > > #7 0x772bdc2a in __asan::ReportGenericError(unsigned long, > > unsigned long, unsigned long, unsigned long, bool, unsigned long, > unsigned > > int, bool) () from /lib64/libasan.so.5 > > #8 0x772bf194 in __asan_report_load_n () from > /lib64/libasan.so.5 > > #9 0x73f45463 in clib_mask_compare_u16_x64 (v=2, > > a=0x7fff89fd6150, n_elts=2) at src/vppinfra/vector_funcs.h:24 > > #10 0x73f4571e in clib_mask_compare_u16 (v=2, a=0x7fff89fd6150, > > mask=0x7ffb51fe2310, n_elts=2) > > at src/vppinfra/vector_funcs.h:79 > > #11 0x73f45b81 in enqueue_one (vm=0x7fff7314ec40, > > node=0x7fff89fe1e00, used_elt_bmp=0x7ffb51fe2440, next_index=2, > > buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, n_buffers=2, n_left=2, > > tmp=0x7ffb51fe2480) at src/vlib/buffer_funcs.c:30 > > #12 0x73f6bdae in vlib_buffer_enqueue_to_next_fn_skx > > (vm=0x7fff7314ec40, node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, > > nexts=0x7fff89fd6150, count=2) > > at src/vlib/buffer_funcs.c:110 > > #13 0x758cd2b6 in vlib_buffer_enqueue_to_next (vm=0x7fff7314ec40, > > node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, > > count=2) > > at src/vlib/buffer_node.h:344 > > #14 0x758e4cf1 in session_flush_pending_tx_buffers > > (wrk=0x7fff8912b780, node=0x7fff89fe1e00) > > at src/vnet/session/session_node.c:1654 > > #15 0x758e844f in session_queue_node_fn (vm=0x7fff7314ec40, > > node=0x7fff89fe1e00, frame=0x0) > > at src/vnet/session/session_node.c:1812 > > #16 0x73e0402f in dispatch_node (vm=0x7fff7314ec40, > > node=0x7fff89fe1e00, type=VLIB_NODE_TYPE_INPUT, > > dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, > > last_time_stamp=2463904994699296) > > at src/vlib/main.c:1024 > > #17 0x73e132a2 in vlib_main_or_worker_loop (vm=0x7fff7314ec40, > > is_main=0) at src/vlib/main.c:2949 > > #18 0x73e14ec6 in vlib_worker_loop (vm=0x7fff7314ec40) at > > src/vlib/main.c:3114 > > #19 0x73eac7ff in vlib_worker_thread_fn (arg=0x7fff7050ef40) at > > src/vlib/threads.c:1560 > > #20 0x734d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123 > > #21 0x7ffb527f8230 in ?? () > > #22 0x73ea004f in vlib_worker_thread_bootstrap_fn > > (arg=0x7fff7050ef40) at src/vlib/threads.c:431 > > #23 0x7fff6d41971b in eal_thread_loop () from > > /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so > > #24 0x738abea5 in start_thread () from /lib64/libpthread.so.0 > &g
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Yes we are aware of it, still working on the correct fix though. In the meantime you can try to apply https://gerrit.fd.io/r/c/vpp/+/32765 which should workaround that for now. Best ben > -Original Message- > From: chetan bhasin > Sent: lundi 6 septembre 2021 14:21 > To: Benoit Ganne (bganne) > Cc: vpp-dev > Subject: Re: [vpp-dev] VPP 2106 with Sanitizer enabled > > Hi, > > The below crash is coming as we involved VPP TCP host stack. > > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7ffb527f9700 (LWP 2013)] > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, > unsigned long*, unsigned long*) () from /lib64/libasan.so.5 > (gdb) bt > #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned > long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5 > #1 0x772c2a11 in > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) > () from /lib64/libasan.so.5 > #2 0x772dcdc2 in > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from > /lib64/libasan.so.5 > #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long) > () from /lib64/libasan.so.5 > #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned > long, unsigned long, __asan::StackAddressDescription*) () from > /lib64/libasan.so.5 > #5 0x771d73f9 in > __asan::AddressDescription::AddressDescription(unsigned long, unsigned > long, bool) () from /lib64/libasan.so.5 > #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, > unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned > long) () from /lib64/libasan.so.5 > #7 0x772bdc2a in __asan::ReportGenericError(unsigned long, > unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned > int, bool) () from /lib64/libasan.so.5 > #8 0x772bf194 in __asan_report_load_n () from /lib64/libasan.so.5 > #9 0x73f45463 in clib_mask_compare_u16_x64 (v=2, > a=0x7fff89fd6150, n_elts=2) at src/vppinfra/vector_funcs.h:24 > #10 0x73f4571e in clib_mask_compare_u16 (v=2, a=0x7fff89fd6150, > mask=0x7ffb51fe2310, n_elts=2) > at src/vppinfra/vector_funcs.h:79 > #11 0x73f45b81 in enqueue_one (vm=0x7fff7314ec40, > node=0x7fff89fe1e00, used_elt_bmp=0x7ffb51fe2440, next_index=2, > buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, n_buffers=2, n_left=2, > tmp=0x7ffb51fe2480) at src/vlib/buffer_funcs.c:30 > #12 0x73f6bdae in vlib_buffer_enqueue_to_next_fn_skx > (vm=0x7fff7314ec40, node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, > nexts=0x7fff89fd6150, count=2) > at src/vlib/buffer_funcs.c:110 > #13 0x758cd2b6 in vlib_buffer_enqueue_to_next (vm=0x7fff7314ec40, > node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, > count=2) > at src/vlib/buffer_node.h:344 > #14 0x758e4cf1 in session_flush_pending_tx_buffers > (wrk=0x7fff8912b780, node=0x7fff89fe1e00) > at src/vnet/session/session_node.c:1654 > #15 0x758e844f in session_queue_node_fn (vm=0x7fff7314ec40, > node=0x7fff89fe1e00, frame=0x0) > at src/vnet/session/session_node.c:1812 > #16 0x73e0402f in dispatch_node (vm=0x7fff7314ec40, > node=0x7fff89fe1e00, type=VLIB_NODE_TYPE_INPUT, > dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, > last_time_stamp=2463904994699296) > at src/vlib/main.c:1024 > #17 0x73e132a2 in vlib_main_or_worker_loop (vm=0x7fff7314ec40, > is_main=0) at src/vlib/main.c:2949 > #18 0x73e14ec6 in vlib_worker_loop (vm=0x7fff7314ec40) at > src/vlib/main.c:3114 > #19 0x73eac7ff in vlib_worker_thread_fn (arg=0x7fff7050ef40) at > src/vlib/threads.c:1560 > #20 0x734d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123 > #21 0x7ffb527f8230 in ?? () > #22 0x73ea004f in vlib_worker_thread_bootstrap_fn > (arg=0x7fff7050ef40) at src/vlib/threads.c:431 > #23 0x7fff6d41971b in eal_thread_loop () from > /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so > #24 0x738abea5 in start_thread () from /lib64/libpthread.so.0 > #25 0x72e328cd in clone () from /lib64/libc.so.6 > (gdb) :q > > On Mon, Sep 6, 2021 at 1:52 PM chetan bhasin <mailto:chetan.bhasin...@gmail.com> > wrote: > > > Hi Ben, > > Thanks for the direction. Looks like it will fix both the issues as > mentioned above. I will update you with the results after applying the > patch. > > Is there any ASAN related patch inside the TCP host stack code. I > will be sharing the issue shortly with you. > > Thanks, >
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Hi, The below crash is coming as we involved VPP TCP host stack. Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffb527f9700 (LWP 2013)] 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5 (gdb) bt #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, unsigned long*, unsigned long*) () from /lib64/libasan.so.5 #1 0x772c2a11 in __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) () from /lib64/libasan.so.5 #2 0x772dcdc2 in __sanitizer::ThreadRegistry::FindThreadContextLocked(bool (*)(__sanitizer::ThreadContextBase*, void*), void*) () from /lib64/libasan.so.5 #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long) () from /lib64/libasan.so.5 #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned long, unsigned long, __asan::StackAddressDescription*) () from /lib64/libasan.so.5 #5 0x771d73f9 in __asan::AddressDescription::AddressDescription(unsigned long, unsigned long, bool) () from /lib64/libasan.so.5 #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long) () from /lib64/libasan.so.5 #7 0x772bdc2a in __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) () from /lib64/libasan.so.5 #8 0x772bf194 in __asan_report_load_n () from /lib64/libasan.so.5 #9 0x73f45463 in clib_mask_compare_u16_x64 (v=2, a=0x7fff89fd6150, n_elts=2) at src/vppinfra/vector_funcs.h:24 #10 0x73f4571e in clib_mask_compare_u16 (v=2, a=0x7fff89fd6150, mask=0x7ffb51fe2310, n_elts=2) at src/vppinfra/vector_funcs.h:79 #11 0x73f45b81 in enqueue_one (vm=0x7fff7314ec40, node=0x7fff89fe1e00, used_elt_bmp=0x7ffb51fe2440, next_index=2, buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, n_buffers=2, n_left=2, tmp=0x7ffb51fe2480) at src/vlib/buffer_funcs.c:30 #12 0x73f6bdae in vlib_buffer_enqueue_to_next_fn_skx (vm=0x7fff7314ec40, node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, count=2) at src/vlib/buffer_funcs.c:110 #13 0x758cd2b6 in vlib_buffer_enqueue_to_next (vm=0x7fff7314ec40, node=0x7fff89fe1e00, buffers=0x7fff7045a9e0, nexts=0x7fff89fd6150, count=2) at src/vlib/buffer_node.h:344 #14 0x758e4cf1 in session_flush_pending_tx_buffers (wrk=0x7fff8912b780, node=0x7fff89fe1e00) at src/vnet/session/session_node.c:1654 #15 0x758e844f in session_queue_node_fn (vm=0x7fff7314ec40, node=0x7fff89fe1e00, frame=0x0) at src/vnet/session/session_node.c:1812 #16 0x73e0402f in dispatch_node (vm=0x7fff7314ec40, node=0x7fff89fe1e00, type=VLIB_NODE_TYPE_INPUT, dispatch_state=VLIB_NODE_STATE_POLLING, frame=0x0, last_time_stamp=2463904994699296) at src/vlib/main.c:1024 #17 0x73e132a2 in vlib_main_or_worker_loop (vm=0x7fff7314ec40, is_main=0) at src/vlib/main.c:2949 #18 0x73e14ec6 in vlib_worker_loop (vm=0x7fff7314ec40) at src/vlib/main.c:3114 #19 0x73eac7ff in vlib_worker_thread_fn (arg=0x7fff7050ef40) at src/vlib/threads.c:1560 #20 0x734d9504 in clib_calljmp () at src/vppinfra/longjmp.S:123 #21 0x7ffb527f8230 in ?? () #22 0x73ea004f in vlib_worker_thread_bootstrap_fn (arg=0x7fff7050ef40) at src/vlib/threads.c:431 #23 0x7fff6d41971b in eal_thread_loop () from /opt/opwv/integra/99.9/tools/vpp_2106_asan/bin/../lib/dpdk_plugin.so #24 0x738abea5 in start_thread () from /lib64/libpthread.so.0 #25 0x72e328cd in clone () from /lib64/libc.so.6 (gdb) :q On Mon, Sep 6, 2021 at 1:52 PM chetan bhasin wrote: > Hi Ben, > > Thanks for the direction. Looks like it will fix both the issues as > mentioned above. I will update you with the results after applying the > patch. > > Is there any ASAN related patch inside the TCP host stack code. I will be > sharing the issue shortly with you. > > Thanks, > Chetan > > On Mon, Sep 6, 2021 at 1:22 PM Benoit Ganne (bganne) > wrote: > >> It should be fixed in master by https://gerrit.fd.io/r/c/vpp/+/32643 >> >> ben >> >> > -Original Message- >> > From: vpp-dev@lists.fd.io On Behalf Of chetan >> bhasin >> > Sent: lundi 6 septembre 2021 09:36 >> > To: vpp-dev >> > Subject: [vpp-dev] VPP 2106 with Sanitizer enabled >> > >> > Hi >> > >> > >> > We are facing two errors with vpp2106 and Address Sanitizer enabled. >> > >> > make V=1 -j4 build VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON >> > >> > >> > Work-around - After adding the two api’s string_key_sum and >> > strnlen_s_inline to ASAN suppression, vpp2106 comes up fine. >> > >> > >> > Error 1: >> > -- >> > Program received signal SIGSEGV, Segmentation fault. >> > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned >> long, >> > unsigned long*, unsigned long*) () >> >
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
Hi Ben, Thanks for the direction. Looks like it will fix both the issues as mentioned above. I will update you with the results after applying the patch. Is there any ASAN related patch inside the TCP host stack code. I will be sharing the issue shortly with you. Thanks, Chetan On Mon, Sep 6, 2021 at 1:22 PM Benoit Ganne (bganne) wrote: > It should be fixed in master by https://gerrit.fd.io/r/c/vpp/+/32643 > > ben > > > -Original Message- > > From: vpp-dev@lists.fd.io On Behalf Of chetan > bhasin > > Sent: lundi 6 septembre 2021 09:36 > > To: vpp-dev > > Subject: [vpp-dev] VPP 2106 with Sanitizer enabled > > > > Hi > > > > > > We are facing two errors with vpp2106 and Address Sanitizer enabled. > > > > make V=1 -j4 build VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON > > > > > > Work-around - After adding the two api’s string_key_sum and > > strnlen_s_inline to ASAN suppression, vpp2106 comes up fine. > > > > > > Error 1: > > -- > > Program received signal SIGSEGV, Segmentation fault. > > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, > > unsigned long*, unsigned long*) () > >from /lib64/libasan.so.5 > > (gdb) bt > > #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned > > long, unsigned long*, unsigned long*) () > >from /lib64/libasan.so.5 > > #1 0x772c2a11 in > > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, > void*) > > () > >from /lib64/libasan.so.5 > > #2 0x772dcdc2 in > > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool > > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from > > /lib64/libasan.so.5 > > #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long) > > () from /lib64/libasan.so.5 > > #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned > > long, unsigned long, __asan::StackAddressDescription*) () from > > /lib64/libasan.so.5 > > #5 0x771d73f9 in > > __asan::AddressDescription::AddressDescription(unsigned long, unsigned > > long, bool) () > >from /lib64/libasan.so.5 > > #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned > int, > > unsigned long, unsigned long, unsigned long, unsigned long, bool, > unsigned > > long) () from /lib64/libasan.so.5 > > #7 0x772bdc2a in __asan::ReportGenericError(unsigned long, > > unsigned long, unsigned long, unsigned long, bool, unsigned long, > unsigned > > int, bool) () from /lib64/libasan.so.5 > > #8 0x7720ef9c in __interceptor_strlen.part.0 () from > > /lib64/libasan.so.5 > > #9 0x734ce2ec in string_key_sum (h=0x7fff6ff6e970, > > key=140735097062688) > > at src/vppinfra/hash.c:947 > > #10 0x734caf15 in key_sum (h=0x7fff6ff6e970, key=140735097062688) > > at src/vppinfra/hash.c:333 > > #11 0x734cbf76 in lookup (v=0x7fff6ff6e9b8, key=140735097062688, > > op=GET, new_value=0x0, old_value=0x0) > > at src/vppinfra/hash.c:557 > > #12 0x734cc59d in _hash_get_pair (v=0x7fff6ff6e9b8, > > key=140735097062688) > > at src/vppinfra/hash.c:653 > > #13 0x0042e885 in lookup_hash_index (name=0x7fff7177c520 > > "/mem/stat segment") > > at src/vpp/stats/stat_segment.c:69 > > #14 0x00431790 in stat_segment_new_entry (name=0x7fff7177c520 > > "/mem/stat segment", > > t=STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE) > > at src/vpp/stats/stat_segment.c:402 > > #15 0x004401fb in vlib_stats_register_mem_heap > > (heap=0x7ffb67e0) > > at src/vpp/stats/stat_segment_provider.c:96 > > #16 0x004327fa in vlib_map_stat_segment_init () > > at src/vpp/stats/stat_segment.c:493 > > ---Type to continue, or q to quit--- > > #17 0x73e15d19 in vlib_main (vm=0x7fff6eeff680, > > input=0x7fff6a0a9f70) > > at src/vlib/main.c:3272 > > #18 0x73f0d924 in thread0 (arg=140735054608000) > > at src/vlib/unix/main.c:671 > > #19 0x734d9504 in clib_calljmp () > > at src/vppinfra/longjmp.S:123 > > #20 0x7fffc940 in ?? () > > #21 0x73f0e67e in vlib_unix_main (argc=282, argv=0x61d1a480) > > at src/vlib/unix/main.c:751 > > #22 0x0040b482 in main (argc=282, argv=0x61d1a480) > > at src/vpp/vnet/main.c:336 > > (gdb) q > > > > > > > > > > Error 2: > > --- > > Program received signal SIGSEGV, Segmentation fault. > > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, > > unsigned long*, unsigned long*) () > >from /lib64/libasan.so.5 > > (gdb) bt > > #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned > > long, unsigned long*, unsigned long*) () > >from /lib64/libasan.so.5 > > #1 0x772c2a11 in > > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, > void*) > > () > >from /lib64/libasan.so.5 > > #2 0x772dcdc2 in > > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool > > (*)(__sanitizer::ThreadContextBase*, void*),
Re: [vpp-dev] VPP 2106 with Sanitizer enabled
It should be fixed in master by https://gerrit.fd.io/r/c/vpp/+/32643 ben > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of chetan bhasin > Sent: lundi 6 septembre 2021 09:36 > To: vpp-dev > Subject: [vpp-dev] VPP 2106 with Sanitizer enabled > > Hi > > > We are facing two errors with vpp2106 and Address Sanitizer enabled. > > make V=1 -j4 build VPP_EXTRA_CMAKE_ARGS=-DVPP_ENABLE_SANITIZE_ADDR=ON > > > Work-around - After adding the two api’s string_key_sum and > strnlen_s_inline to ASAN suppression, vpp2106 comes up fine. > > > Error 1: > -- > Program received signal SIGSEGV, Segmentation fault. > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, > unsigned long*, unsigned long*) () >from /lib64/libasan.so.5 > (gdb) bt > #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned > long, unsigned long*, unsigned long*) () >from /lib64/libasan.so.5 > #1 0x772c2a11 in > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) > () >from /lib64/libasan.so.5 > #2 0x772dcdc2 in > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from > /lib64/libasan.so.5 > #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long) > () from /lib64/libasan.so.5 > #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned > long, unsigned long, __asan::StackAddressDescription*) () from > /lib64/libasan.so.5 > #5 0x771d73f9 in > __asan::AddressDescription::AddressDescription(unsigned long, unsigned > long, bool) () >from /lib64/libasan.so.5 > #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, > unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned > long) () from /lib64/libasan.so.5 > #7 0x772bdc2a in __asan::ReportGenericError(unsigned long, > unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned > int, bool) () from /lib64/libasan.so.5 > #8 0x7720ef9c in __interceptor_strlen.part.0 () from > /lib64/libasan.so.5 > #9 0x734ce2ec in string_key_sum (h=0x7fff6ff6e970, > key=140735097062688) > at src/vppinfra/hash.c:947 > #10 0x734caf15 in key_sum (h=0x7fff6ff6e970, key=140735097062688) > at src/vppinfra/hash.c:333 > #11 0x734cbf76 in lookup (v=0x7fff6ff6e9b8, key=140735097062688, > op=GET, new_value=0x0, old_value=0x0) > at src/vppinfra/hash.c:557 > #12 0x734cc59d in _hash_get_pair (v=0x7fff6ff6e9b8, > key=140735097062688) > at src/vppinfra/hash.c:653 > #13 0x0042e885 in lookup_hash_index (name=0x7fff7177c520 > "/mem/stat segment") > at src/vpp/stats/stat_segment.c:69 > #14 0x00431790 in stat_segment_new_entry (name=0x7fff7177c520 > "/mem/stat segment", > t=STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE) > at src/vpp/stats/stat_segment.c:402 > #15 0x004401fb in vlib_stats_register_mem_heap > (heap=0x7ffb67e0) > at src/vpp/stats/stat_segment_provider.c:96 > #16 0x004327fa in vlib_map_stat_segment_init () > at src/vpp/stats/stat_segment.c:493 > ---Type to continue, or q to quit--- > #17 0x73e15d19 in vlib_main (vm=0x7fff6eeff680, > input=0x7fff6a0a9f70) > at src/vlib/main.c:3272 > #18 0x73f0d924 in thread0 (arg=140735054608000) > at src/vlib/unix/main.c:671 > #19 0x734d9504 in clib_calljmp () > at src/vppinfra/longjmp.S:123 > #20 0x7fffc940 in ?? () > #21 0x73f0e67e in vlib_unix_main (argc=282, argv=0x61d1a480) > at src/vlib/unix/main.c:751 > #22 0x0040b482 in main (argc=282, argv=0x61d1a480) > at src/vpp/vnet/main.c:336 > (gdb) q > > > > > Error 2: > --- > Program received signal SIGSEGV, Segmentation fault. > 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned long, > unsigned long*, unsigned long*) () >from /lib64/libasan.so.5 > (gdb) bt > #0 0x771db5c1 in __asan::FakeStack::AddrIsInFakeStack(unsigned > long, unsigned long*, unsigned long*) () >from /lib64/libasan.so.5 > #1 0x772c2a11 in > __asan::ThreadStackContainsAddress(__sanitizer::ThreadContextBase*, void*) > () >from /lib64/libasan.so.5 > #2 0x772dcdc2 in > __sanitizer::ThreadRegistry::FindThreadContextLocked(bool > (*)(__sanitizer::ThreadContextBase*, void*), void*) () from > /lib64/libasan.so.5 > #3 0x772c3e5a in __asan::FindThreadByStackAddress(unsigned long) > () from /lib64/libasan.so.5 > #4 0x771d5fb6 in __asan::GetStackAddressInformation(unsigned > long, unsigned long, __asan::StackAddressDescription*) () from > /lib64/libasan.so.5 > #5 0x771d73f9 in > __asan::AddressDescription::AddressDescription(unsigned long, unsigned > long, bool) () >from /lib64/libasan.so.5 > #6 0x771d9e51 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, > unsigned long, unsigned long, unsigned
