Re: [vpp-dev] vpp crash issue with large number of acl session
Your first connection limit is 20million, now it is 300 million - so I am not
sure what your requirements are, nor what you are optimizing/testing for.
The memory consumption of bihash depends on how the connections are hashed into
secondary buckets.
If you know that precisely - then you can calculate the memory requirements
precisely.
If you want worst case - count one connection per bucket, but that will be
probably quite a lot.
Barring the exact knowledge of your traffic - treat the connection count limit
as a traffic sign “warning” before a brick wall of “available memory” when you
drive a car at high speed.
Keep halving the connection count limit until you stop to run our of memory if
your memory is fixed, or keep doubling memory until you get to your needed
connection count without running out of memory if you need to adhere to certain
connection count.
There are also options to play with bihash parameters - but given that they
have performance implications, the reader curious enough ought to be able to
trace these from the code.
--a
> On 31 Jul 2021, at 08:22, NetHappy wrote:
>
> Hi,
>
> I have updated the config with 16G and session count 3
> memory {
> main-heap-size 16G
> }
> acl-plugin {
> connection count max 3
> }
> But i am still seeing same crash i reported earlier. But now occurring less.
> Is there any calculation as to how much sessions should be attempted for 16GB
> or memory ? What's the max suggested values for session count for 64 GB
> system?
>
> Here is the crash dump.
> #8 os_out_of_memory () at /root/code/vpp/src/vppinfra/unix-misc.c:221
> #9 0x7ff9f0fc56de in clib_mem_alloc_aligned_at_offset
> (os_out_of_memory_on_failure=1, align_offset=0, align=64, size=98368)
> at /root/code/vpp/src/vppinfra/mem.h:243
> #10 clib_mem_alloc_aligned (align=64, size=98368) at
> /root/code/vpp/src/vppinfra/mem.h:263
> #11 alloc_aligned_16_8 (nbytes=98304, h=0x7ff7f05fd998 ) at
> /root/code/vpp/src/vppinfra/bihash_template.c:55
> #12 value_alloc_16_8 (h=0x7ff7f05fd998 , log2_pages= out>)
> at /root/code/vpp/src/vppinfra/bihash_template.c:462
> #13 0x7ff9f147eefd in split_and_rehash_16_8 (h=h@entry=0x7ff7f05fd998
> , old_values=old_values@entry=0x7ff92e595240,
> old_log2_pages=old_log2_pages@entry=9,
> new_log2_pages=new_log2_pages@entry=10)
> at /root/code/vpp/src/vppinfra/bihash_template.c:581
> #14 0x7ff9f148096f in clib_bihash_add_del_inline_with_hash_16_8 (arg=0x0,
> is_stale_cb=0x0, is_add=1, hash=,
> add_v=0x7fc01011cb48, h=0x7ff7f05fd998 ) at
> /root/code/vpp/src/vppinfra/bihash_template.c:893
> #15 clib_bihash_add_del_inline_16_8 (arg=0x0, is_stale_cb=0x0, is_add=1,
> add_v=0x7fc01011cb48, h=0x7ff7f05fd998 )
> at /root/code/vpp/src/vppinfra/bihash_template.c:985
> #16 clib_bihash_add_del_16_8 (h=h@entry=0x7ff7f05fd998 ,
> add_v=add_v@entry=0x7fc01011cb48, is_add=is_add@entry=1)
> at /root/code/vpp/src/vppinfra/bihash_template.c:992
> #17 0x7ff7f01618ae in acl_fa_add_session (current_policy_epoch= out>, p5tuple=0x7ff7f43f70f8, now=27359859845375580,
> sw_if_index=, is_ip6=0, is_input=0, am=) at
> /root/code/vpp/src/plugins/acl/session_inlines.h:577
> #18 acl_fa_inner_node_fn (reclassify_sessions=1, node_trace_on=0,
> with_stateful_datapath=1, is_l2_path=1, is_input=0, is_ip6=0,
> frame=0x7ff96e946c00, node=0x7ff96b0b4d00, vm=0x7ff804f20d80) at
> /root/code/vpp/src/plugins/acl/dataplane_node.c:1144
>
> Thanks,
> Mahamuda
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19905): https://lists.fd.io/g/vpp-dev/message/19905
Mute This Topic: https://lists.fd.io/mt/77321080/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp crash issue with large number of acl session
Hi,
I have updated the config with 16G and session count 3
memory {
main-heap-size 16G
}
acl-plugin {
connection count max 3
}
But i am still seeing same crash i reported earlier. But now occurring less.
Is there any calculation as to how much sessions should be attempted for 16GB
or memory ? What's the max suggested values for session count for 64 GB system?
Here is the crash dump.
#8 os_out_of_memory () at /root/code/vpp/src/vppinfra/unix-misc.c:221
#9 0x7ff9f0fc56de in clib_mem_alloc_aligned_at_offset
(os_out_of_memory_on_failure=1, align_offset=0, align=64, size=98368)
at /root/code/vpp/src/vppinfra/mem.h:243
#10 clib_mem_alloc_aligned (align=64, size=98368) at
/root/code/vpp/src/vppinfra/mem.h:263
#11 alloc_aligned_16_8 (nbytes=98304, h=0x7ff7f05fd998 ) at
/root/code/vpp/src/vppinfra/bihash_template.c:55
#12 value_alloc_16_8 (h=0x7ff7f05fd998 , log2_pages=)
at /root/code/vpp/src/vppinfra/bihash_template.c:462
#13 0x7ff9f147eefd in split_and_rehash_16_8 (h=h@entry=0x7ff7f05fd998
, old_values=old_values@entry=0x7ff92e595240,
old_log2_pages=old_log2_pages@entry=9, new_log2_pages=new_log2_pages@entry=10)
at /root/code/vpp/src/vppinfra/bihash_template.c:581
#14 0x7ff9f148096f in clib_bihash_add_del_inline_with_hash_16_8 (arg=0x0,
is_stale_cb=0x0, is_add=1, hash=,
add_v=0x7fc01011cb48, h=0x7ff7f05fd998 ) at
/root/code/vpp/src/vppinfra/bihash_template.c:893
#15 clib_bihash_add_del_inline_16_8 (arg=0x0, is_stale_cb=0x0, is_add=1,
add_v=0x7fc01011cb48, h=0x7ff7f05fd998 )
at /root/code/vpp/src/vppinfra/bihash_template.c:985
#16 clib_bihash_add_del_16_8 (h=h@entry=0x7ff7f05fd998 ,
add_v=add_v@entry=0x7fc01011cb48, is_add=is_add@entry=1)
at /root/code/vpp/src/vppinfra/bihash_template.c:992
#17 0x7ff7f01618ae in acl_fa_add_session (current_policy_epoch=, p5tuple=0x7ff7f43f70f8, now=27359859845375580,
sw_if_index=, is_ip6=0, is_input=0, am=) at
/root/code/vpp/src/plugins/acl/session_inlines.h:577
#18 acl_fa_inner_node_fn (reclassify_sessions=1, node_trace_on=0,
with_stateful_datapath=1, is_l2_path=1, is_input=0, is_ip6=0,
frame=0x7ff96e946c00, node=0x7ff96b0b4d00, vm=0x7ff804f20d80) at
/root/code/vpp/src/plugins/acl/dataplane_node.c:1144
Thanks,
Mahamuda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19882): https://lists.fd.io/g/vpp-dev/message/19882
Mute This Topic: https://lists.fd.io/mt/77321080/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp crash issue with large number of acl session
Hi Andrew, Thanks for your reply. I would modify the configuration as per your suggestions. Thanks, Mahamuda -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19464): https://lists.fd.io/g/vpp-dev/message/19464 Mute This Topic: https://lists.fd.io/mt/77321080/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp crash issue with large number of acl session
to be precise: “memory { main-heap-size 16G }” or something along these lines
is what might be useful for your config.
--a
> On 25 May 2021, at 14:40, Andrew 👽 Yourtchenko wrote:
>
> There is no way to gracefully fail an insertion into a bihash - it either
> succeeds or you get a crash.
>
> The max session limit attempts to guard against that, by failing the addition
> using the heuristic number. You dialed this number up to 20 mil, so no more
> safe guard - which is what you observe.
>
> But you are still running at a default heap size. “heapsize” parameter allows
> to change that.
>
> As there is no hard and fast formula how much memory the bihash will consume,
> it’s up to you to tune the above two variables such that there is happiness.
>
> --a
>
>> On 23 May 2021, at 08:14, NetHappy wrote:
>>
>> Hi An drew,
>>
>> Myself modified the code to suit some requirement. The plain vpp acl module
>> also crashes in similar situation.
>> I am expecting the vpp should crash while trying to add more sessions. The
>> system has 64 GB RAM and vpp was using 3.44% of the memory while crashing.
>> So the system is not out of memory but its not configured to use the
>> available memory.
>>
>> The relevant crash bt is this
>>
>> #5 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
>> #6 0x7f2fba309859 in __GI_abort () at abort.c:79
>> #7 0x7f2fba4e0bba in os_panic () at
>> /root/code/vpp/src/vppinfra/unix-misc.c:221
>> #8 os_out_of_memory () at /root/code/vpp/src/vppinfra/unix-misc.c:221
>> #9 0x7f2fba721701 in clib_mem_alloc_aligned_at_offset
>> (os_out_of_memory_on_failure=1, align_offset=0, align=64, size=98368) at
>> /root/code/vpp/src/vppinfra/mem.h:243
>> #10 clib_mem_alloc_aligned (align=64, size=98368) at
>> /root/code/vpp/src/vppinfra/mem.h:263
>> #11 alloc_aligned_16_8 (nbytes=12288, h=0x7f2f79d59998 ) at
>> /root/code/vpp/src/vppinfra/bihash_template.c:77
>> #12 value_alloc_16_8 (h=0x7f2f79d59998 , log2_pages=> out>) at /root/code/vpp/src/vppinfra/bihash_template.c:462
>> #13 0x7f2fbabdaefd in split_and_rehash_16_8 (h=h@entry=0x7f2f79d59998
>> , old_values=old_values@entry=0x7f2faf1d9a00,
>>old_log2_pages=old_log2_pages@entry=5,
>> new_log2_pages=new_log2_pages@entry=7) at
>> /root/code/vpp/src/vppinfra/bihash_template.c:581
>> #14 0x7f2fbabdcbd0 in clib_bihash_add_del_inline_with_hash_16_8
>> (arg=0x0, is_stale_cb=0x0, is_add=1, hash=,
>> add_v=0x7f19a6991b48,
>>h=0x7f2f79d59998 ) at
>> /root/code/vpp/src/vppinfra/bihash_template.c:901
>> #15 clib_bihash_add_del_inline_16_8 (arg=0x0, is_stale_cb=0x0, is_add=1,
>> add_v=0x7f19a6991b48, h=0x7f2f79d59998 )
>>at /root/code/vpp/src/vppinfra/bihash_template.c:985
>> #16 clib_bihash_add_del_16_8 (h=h@entry=0x7f2f79d59998 ,
>> add_v=add_v@entry=0x7f19a6991b48, is_add=is_add@entry=1)
>>at /root/code/vpp/src/vppinfra/bihash_template.c:992
>> #17 0x7f2f798bc8b5 in acl_fa_add_session
>> (current_policy_epoch=, p5tuple=0x7f2f7cd5bc58,
>> now=7927760728493450, sw_if_index=, is_ip6=0,
>>is_input=0, am=) at
>> /root/code/vpp/src/plugins/acl/session_inlines.h:577
>> #18 acl_fa_inner_node_fn (reclassify_sessions=1, node_trace_on=0,
>> with_stateful_datapath=1, is_l2_path=1, is_input=0, is_ip6=0,
>> frame=0x7f2f8c5f6440,
>>node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0) at
>> /root/code/vpp/src/plugins/acl/dataplane_node.c:1137
>> #19 acl_fa_outer_node_fn (do_stateful_datapath=1, is_l2_path=1, is_input=0,
>> is_ip6=0, frame=0x7f2f8c5f6440, node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0)
>>at /root/code/vpp/src/plugins/acl/dataplane_node.c:1428
>> #20 acl_fa_node_fn (is_l2_path=1, is_input=0, is_ip6=0,
>> frame=0x7f2f8c5f6440, node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0)
>>at /root/code/vpp/src/plugins/acl/dataplane_node.c:1456
>> #21 acl_out_l2_ip4_node_fn_hsw (vm=0x7f2f8d6f3dc0, node=0x7f2f8d6de980,
>> frame=0x7f2f8c5f6440) at /root/code/vpp/src/plugins/acl/dataplane_node.c:1569
>> #22 0x7f2fba595fa6 in dispatch_node (last_time_stamp=,
>> frame=0x7f2f8c5f6440, dispatch_state=VLIB_NODE_STATE_POLLING,
>> type=VLIB_NODE_TYPE_INTERNAL,
>>node=0x7f2f8d6de980, vm=0x7f2f8d6de980) at
>> /root/code/vpp/src/vlib/main.c:1039
>> #23 dispatch_pending_node (vm=vm@entry=0x7f2f8d6f3dc0,
>> pending_frame_index=pending_frame_index@entry=7, last_time_stamp=> out>)
>>at /root/code/vpp/src/vlib/main.c:1198
>> #24 0x7f2fba5978ff in vlib_main_or_worker_loop (is_main=0,
>> vm=0x7f2f8d6f3dc0) at /root/code/vpp/src/vlib/main.c:1662
>> #25 vlib_worker_loop (vm=vm@entry=0x7f2f8d6f3dc0) at
>> /root/code/vpp/src/vlib/main.c:1796
>> #26 0x7f2fba5c4bf8 in vlib_worker_thread_fn (arg=) at
>> /root/code/vpp/src/vlib/threads.c:1872
>> #27 0x7f2fba529cb0 in clib_calljmp () from
>> /lib/x86_64-linux-gnu/libvppinfra.so.21.06
>> #28 0x7f2f72896d10 in ?? ()
>> #29 0x7f2f76976c5d in eal_thread_loop.cold () from
>> /usr/lib/x86_64-linux-gnu/vpp_plugins/dpdk_plugin.
Re: [vpp-dev] vpp crash issue with large number of acl session
There is no way to gracefully fail an insertion into a bihash - it either succeeds or you get a crash. The max session limit attempts to guard against that, by failing the addition using the heuristic number. You dialed this number up to 20 mil, so no more safe guard - which is what you observe. But you are still running at a default heap size. “heapsize” parameter allows to change that. As there is no hard and fast formula how much memory the bihash will consume, it’s up to you to tune the above two variables such that there is happiness. --a > On 23 May 2021, at 08:14, NetHappy wrote: > > Hi An drew, > > Myself modified the code to suit some requirement. The plain vpp acl module > also crashes in similar situation. > I am expecting the vpp should crash while trying to add more sessions. The > system has 64 GB RAM and vpp was using 3.44% of the memory while crashing. > So the system is not out of memory but its not configured to use the > available memory. > > The relevant crash bt is this > > #5 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 > #6 0x7f2fba309859 in __GI_abort () at abort.c:79 > #7 0x7f2fba4e0bba in os_panic () at > /root/code/vpp/src/vppinfra/unix-misc.c:221 > #8 os_out_of_memory () at /root/code/vpp/src/vppinfra/unix-misc.c:221 > #9 0x7f2fba721701 in clib_mem_alloc_aligned_at_offset > (os_out_of_memory_on_failure=1, align_offset=0, align=64, size=98368) at > /root/code/vpp/src/vppinfra/mem.h:243 > #10 clib_mem_alloc_aligned (align=64, size=98368) at > /root/code/vpp/src/vppinfra/mem.h:263 > #11 alloc_aligned_16_8 (nbytes=12288, h=0x7f2f79d59998 ) at > /root/code/vpp/src/vppinfra/bihash_template.c:77 > #12 value_alloc_16_8 (h=0x7f2f79d59998 , log2_pages= out>) at /root/code/vpp/src/vppinfra/bihash_template.c:462 > #13 0x7f2fbabdaefd in split_and_rehash_16_8 (h=h@entry=0x7f2f79d59998 > , old_values=old_values@entry=0x7f2faf1d9a00, > old_log2_pages=old_log2_pages@entry=5, > new_log2_pages=new_log2_pages@entry=7) at > /root/code/vpp/src/vppinfra/bihash_template.c:581 > #14 0x7f2fbabdcbd0 in clib_bihash_add_del_inline_with_hash_16_8 (arg=0x0, > is_stale_cb=0x0, is_add=1, hash=, add_v=0x7f19a6991b48, > h=0x7f2f79d59998 ) at > /root/code/vpp/src/vppinfra/bihash_template.c:901 > #15 clib_bihash_add_del_inline_16_8 (arg=0x0, is_stale_cb=0x0, is_add=1, > add_v=0x7f19a6991b48, h=0x7f2f79d59998 ) > at /root/code/vpp/src/vppinfra/bihash_template.c:985 > #16 clib_bihash_add_del_16_8 (h=h@entry=0x7f2f79d59998 , > add_v=add_v@entry=0x7f19a6991b48, is_add=is_add@entry=1) > at /root/code/vpp/src/vppinfra/bihash_template.c:992 > #17 0x7f2f798bc8b5 in acl_fa_add_session (current_policy_epoch= out>, p5tuple=0x7f2f7cd5bc58, now=7927760728493450, sw_if_index= out>, is_ip6=0, > is_input=0, am=) at > /root/code/vpp/src/plugins/acl/session_inlines.h:577 > #18 acl_fa_inner_node_fn (reclassify_sessions=1, node_trace_on=0, > with_stateful_datapath=1, is_l2_path=1, is_input=0, is_ip6=0, > frame=0x7f2f8c5f6440, > node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0) at > /root/code/vpp/src/plugins/acl/dataplane_node.c:1137 > #19 acl_fa_outer_node_fn (do_stateful_datapath=1, is_l2_path=1, is_input=0, > is_ip6=0, frame=0x7f2f8c5f6440, node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0) > at /root/code/vpp/src/plugins/acl/dataplane_node.c:1428 > #20 acl_fa_node_fn (is_l2_path=1, is_input=0, is_ip6=0, frame=0x7f2f8c5f6440, > node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0) > at /root/code/vpp/src/plugins/acl/dataplane_node.c:1456 > #21 acl_out_l2_ip4_node_fn_hsw (vm=0x7f2f8d6f3dc0, node=0x7f2f8d6de980, > frame=0x7f2f8c5f6440) at /root/code/vpp/src/plugins/acl/dataplane_node.c:1569 > #22 0x7f2fba595fa6 in dispatch_node (last_time_stamp=, > frame=0x7f2f8c5f6440, dispatch_state=VLIB_NODE_STATE_POLLING, > type=VLIB_NODE_TYPE_INTERNAL, > node=0x7f2f8d6de980, vm=0x7f2f8d6de980) at > /root/code/vpp/src/vlib/main.c:1039 > #23 dispatch_pending_node (vm=vm@entry=0x7f2f8d6f3dc0, > pending_frame_index=pending_frame_index@entry=7, last_time_stamp= out>) > at /root/code/vpp/src/vlib/main.c:1198 > #24 0x7f2fba5978ff in vlib_main_or_worker_loop (is_main=0, > vm=0x7f2f8d6f3dc0) at /root/code/vpp/src/vlib/main.c:1662 > #25 vlib_worker_loop (vm=vm@entry=0x7f2f8d6f3dc0) at > /root/code/vpp/src/vlib/main.c:1796 > #26 0x7f2fba5c4bf8 in vlib_worker_thread_fn (arg=) at > /root/code/vpp/src/vlib/threads.c:1872 > #27 0x7f2fba529cb0 in clib_calljmp () from > /lib/x86_64-linux-gnu/libvppinfra.so.21.06 > #28 0x7f2f72896d10 in ?? () > #29 0x7f2f76976c5d in eal_thread_loop.cold () from > /usr/lib/x86_64-linux-gnu/vpp_plugins/dpdk_plugin.so > #30 0x in ?? () > > In the frame 18 before calling acl_fa_can_add_session, why > acl_fa_can_add_session is not failing ? What Is best way to set configuration > so that vpp able to use more RAM ? > > Thanks, > Mahamuda > > -=-=-=-=-=-=-=-=-=-=-=- Links: You
Re: [vpp-dev] vpp crash issue with large number of acl session
Hi An drew, Myself modified the code to suit some requirement. The plain vpp acl module also crashes in similar situation. I am expecting the vpp should crash while trying to add more sessions. The system has 64 GB RAM and vpp was using 3.44% of the memory while crashing. So the system is not out of memory but its not configured to use the available memory. The relevant crash bt is this #5 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #6 0x7f2fba309859 in __GI_abort () at abort.c:79 #7 0x7f2fba4e0bba in os_panic () at /root/code/vpp/src/vppinfra/unix-misc.c:221 #8 os_out_of_memory () at /root/code/vpp/src/vppinfra/unix-misc.c:221 #9 0x7f2fba721701 in clib_mem_alloc_aligned_at_offset (os_out_of_memory_on_failure=1, align_offset=0, align=64, size=98368) at /root/code/vpp/src/vppinfra/mem.h:243 #10 clib_mem_alloc_aligned (align=64, size=98368) at /root/code/vpp/src/vppinfra/mem.h:263 #11 alloc_aligned_16_8 (nbytes=12288, h=0x7f2f79d59998 ) at /root/code/vpp/src/vppinfra/bihash_template.c:77 #12 value_alloc_16_8 (h=0x7f2f79d59998 , log2_pages=) at /root/code/vpp/src/vppinfra/bihash_template.c:462 #13 0x7f2fbabdaefd in split_and_rehash_16_8 (h=h@entry=0x7f2f79d59998 , old_values=old_values@entry=0x7f2faf1d9a00, old_log2_pages=old_log2_pages@entry=5, new_log2_pages=new_log2_pages@entry=7) at /root/code/vpp/src/vppinfra/bihash_template.c:581 #14 0x7f2fbabdcbd0 in clib_bihash_add_del_inline_with_hash_16_8 (arg=0x0, is_stale_cb=0x0, is_add=1, hash=, add_v=0x7f19a6991b48, h=0x7f2f79d59998 ) at /root/code/vpp/src/vppinfra/bihash_template.c:901 #15 clib_bihash_add_del_inline_16_8 (arg=0x0, is_stale_cb=0x0, is_add=1, add_v=0x7f19a6991b48, h=0x7f2f79d59998 ) at /root/code/vpp/src/vppinfra/bihash_template.c:985 #16 clib_bihash_add_del_16_8 (h=h@entry=0x7f2f79d59998 , add_v=add_v@entry=0x7f19a6991b48, is_add=is_add@entry=1) at /root/code/vpp/src/vppinfra/bihash_template.c:992 #17 0x7f2f798bc8b5 in acl_fa_add_session (current_policy_epoch=, p5tuple=0x7f2f7cd5bc58, now=7927760728493450, sw_if_index=, is_ip6=0, is_input=0, am=) at /root/code/vpp/src/plugins/acl/session_inlines.h:577 #18 acl_fa_inner_node_fn (reclassify_sessions=1, node_trace_on=0, with_stateful_datapath=1, is_l2_path=1, is_input=0, is_ip6=0, frame=0x7f2f8c5f6440, node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0) at /root/code/vpp/src/plugins/acl/dataplane_node.c:1137 #19 acl_fa_outer_node_fn (do_stateful_datapath=1, is_l2_path=1, is_input=0, is_ip6=0, frame=0x7f2f8c5f6440, node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0) at /root/code/vpp/src/plugins/acl/dataplane_node.c:1428 #20 acl_fa_node_fn (is_l2_path=1, is_input=0, is_ip6=0, frame=0x7f2f8c5f6440, node=0x7f2f8d6de980, vm=0x7f2f8d6f3dc0) at /root/code/vpp/src/plugins/acl/dataplane_node.c:1456 #21 acl_out_l2_ip4_node_fn_hsw (vm=0x7f2f8d6f3dc0, node=0x7f2f8d6de980, frame=0x7f2f8c5f6440) at /root/code/vpp/src/plugins/acl/dataplane_node.c:1569 #22 0x7f2fba595fa6 in dispatch_node (last_time_stamp=, frame=0x7f2f8c5f6440, dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INTERNAL, node=0x7f2f8d6de980, vm=0x7f2f8d6de980) at /root/code/vpp/src/vlib/main.c:1039 #23 dispatch_pending_node (vm=vm@entry=0x7f2f8d6f3dc0, pending_frame_index=pending_frame_index@entry=7, last_time_stamp=) at /root/code/vpp/src/vlib/main.c:1198 #24 0x7f2fba5978ff in vlib_main_or_worker_loop (is_main=0, vm=0x7f2f8d6f3dc0) at /root/code/vpp/src/vlib/main.c:1662 #25 vlib_worker_loop (vm=vm@entry=0x7f2f8d6f3dc0) at /root/code/vpp/src/vlib/main.c:1796 #26 0x7f2fba5c4bf8 in vlib_worker_thread_fn (arg=) at /root/code/vpp/src/vlib/threads.c:1872 #27 0x7f2fba529cb0 in clib_calljmp () from /lib/x86_64-linux-gnu/libvppinfra.so.21.06 #28 0x7f2f72896d10 in ?? () #29 0x7f2f76976c5d in eal_thread_loop.cold () from /usr/lib/x86_64-linux-gnu/vpp_plugins/dpdk_plugin.so #30 0x in ?? () In the frame 18 before calling acl_fa_can_add_session, why acl_fa_can_add_session is not failing ? What Is best way to set configuration so that vpp able to use more RAM ? Thanks, Mahamuda -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19442): https://lists.fd.io/g/vpp-dev/message/19442 Mute This Topic: https://lists.fd.io/mt/77321080/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp crash issue with large number of acl session
The entire code path you are shared looks like a heavily modified ACL plugin - so best that you bring it up with the person who did the modifications, or do the tests on the plain ACL plugin code from latest master. --a > On 22 May 2021, at 19:42, NetHappy wrote: > > Hi Andrew, > > Thanks for the reply. I will share the crash back-trace related to > acl_fa_add_session once it is re-occurs. > > Thanks, > Mahamuda > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19439): https://lists.fd.io/g/vpp-dev/message/19439 Mute This Topic: https://lists.fd.io/mt/77321080/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp crash issue with large number of acl session
Hi Andrew, Thanks for the reply. I will share the crash back-trace related to acl_fa_add_session once it is re-occurs. Thanks, Mahamuda -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19438): https://lists.fd.io/g/vpp-dev/message/19438 Mute This Topic: https://lists.fd.io/mt/77321080/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp crash issue with large number of acl session
You still run out of memory. However: #19 is_dest_ip_to_be_monitored (now=139778377404384, len=, fa=0x40, am=) at /root/code/vpp/src/plugins/acl/dataplane_node.c:396 This symbol doesn’t exist in the code i have worked on, so it is a modified VPP code, as such I can not say much about it. --a > On 22 May 2021, at 15:15, NetHappy wrote: > > #19 is_dest_ip_to_be_monitored (now=139778377404384, len=, > fa=0x40, am=) at > /root/code/vpp/src/plugins/acl/dataplane_node.c:396 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19437): https://lists.fd.io/g/vpp-dev/message/19437 Mute This Topic: https://lists.fd.io/mt/77321080/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp crash issue with large number of acl session
Hi,
As per your suggestion i am using vpp v21.06 ( git describe --tags
21.06-rc0-529-ge4db945e1 )
I am still getting crash in vpp with following core dump.
#8 os_out_of_memory () at /root/code/vpp/src/vppinfra/unix-misc.c:221
#9 0x7f20de9b2701 in clib_mem_alloc_aligned_at_offset
(os_out_of_memory_on_failure=1, align_offset=0, align=64, size=98368) at
/root/code/vpp/src/vppinfra/mem.h:243
#10 clib_mem_alloc_aligned (align=64, size=98368) at
/root/code/vpp/src/vppinfra/mem.h:263
#11 alloc_aligned_16_8 (nbytes=6144, h=0x7f209dfea998 ) at
/root/code/vpp/src/vppinfra/bihash_template.c:77
#12 value_alloc_16_8 (h=0x7f209dfea998 , log2_pages=) at /root/code/vpp/src/vppinfra/bihash_template.c:462
#13 0x7f20dee6befd in split_and_rehash_16_8 (h=h@entry=0x7f209dfea998
, old_values=old_values@entry=0x7f20d3232980,
old_log2_pages=old_log2_pages@entry=5, new_log2_pages=new_log2_pages@entry=6)
at /root/code/vpp/src/vppinfra/bihash_template.c:581
#14 0x7f20dee6d96f in clib_bihash_add_del_inline_with_hash_16_8 (arg=0x0,
is_stale_cb=0x0, is_add=, hash=,
add_v=0x7f0b9b91db48,
h=0x7f209dfea998 ) at
/root/code/vpp/src/vppinfra/bihash_template.c:893
#15 clib_bihash_add_del_inline_16_8 (arg=0x0, is_stale_cb=0x0,
is_add=, add_v=0x7f0b9b91db48, h=0x7f209dfea998 )
at /root/code/vpp/src/vppinfra/bihash_template.c:985
#16 clib_bihash_add_del_16_8 (h=0x7f209dfea998 ,
add_v=0x7f0b9b91db48, is_add=) at
/root/code/vpp/src/vppinfra/bihash_template.c:992
#17 0x7f209db51625 in clib_bihash_search_inline_with_hash_8_8
(key_result=0x7f209612ac80, hash=, h=)
at /root/code/vpp/src/vppinfra/bihash_8_8.h:93
#18 clib_bihash_search_inline_8_8 (key_result=0x7f209612ac80, h=) at /root/code/vpp/src/vppinfra/bihash_template.h:465
#19 is_dest_ip_to_be_monitored (now=139778377404384, len=,
fa=0x40, am=) at
/root/code/vpp/src/plugins/acl/dataplane_node.c:396
#20 acl_fa_inner_node_fn (reclassify_sessions=0, node_trace_on=1,
with_stateful_datapath=0, is_l2_path=1, is_input=0, is_ip6=0,
frame=0x7f20b1aa9e80,
node=0x7f20b1a54200, vm=0x7f20a0fdcb80) at
/root/code/vpp/src/plugins/acl/dataplane_node.c:965
#21 acl_fa_outer_node_fn (do_stateful_datapath=0, is_l2_path=1, is_input=0,
is_ip6=0, frame=0x7f20b1aa9e80, node=0x7f20b1a54200, vm=0x7f20a0fdcb80)
at /root/code/vpp/src/plugins/acl/dataplane_node.c:1435
#22 acl_fa_node_fn (is_l2_path=1, is_input=0, is_ip6=0, frame=0x7f20b1aa9e80,
node=0x7f20b1a54200, vm=0x7f20a0fdcb80)
at /root/code/vpp/src/plugins/acl/dataplane_node.c:1459
#23 acl_out_l2_ip4_node_fn_hsw (vm=0x7f20a0fdcb80, node=0x7f20b1a54200,
frame=0x7f20b1aa9e80) at /root/code/vpp/src/plugins/acl/dataplane_node.c:1569
#24 0x7f20de826fa6 in dispatch_node (last_time_stamp=,
frame=0x7f20b1aa9e80, dispatch_state=VLIB_NODE_STATE_POLLING,
type=VLIB_NODE_TYPE_INTERNAL,
node=0x7f20b1a54200, vm=0x7f20b1a54200) at /root/code/vpp/src/vlib/main.c:1039
#25 dispatch_pending_node (vm=vm@entry=0x7f20a0fdcb80,
pending_frame_index=pending_frame_index@entry=7, last_time_stamp=)
at /root/code/vpp/src/vlib/main.c:1198
#26 0x7f20de8288ff in vlib_main_or_worker_loop (is_main=0,
vm=0x7f20a0fdcb80) at /root/code/vpp/src/vlib/main.c:1662
#27 vlib_worker_loop (vm=vm@entry=0x7f20a0fdcb80) at
/root/code/vpp/src/vlib/main.c:1796
#28 0x7f20de855bf8 in vlib_worker_thread_fn (arg=) at
/root/code/vpp/src/vlib/threads.c:1872
#29 0x7f20de7bacb0 in clib_calljmp () from
/lib/x86_64-linux-gnu/libvppinfra.so.21.06
#30 0x7f2096b2bd10 in ?? ()
#31 0x7f209ac0bc5d in eal_thread_loop.cold () from
/usr/lib/x86_64-linux-gnu/vpp_plugins/dpdk_plugin.so
#32 0x in ?? ()
Please note that the i have modified some potion for acl custom requirement.
/etc/vpp/startup.conf contains followings
unix {
nodaemon
log /var/log/vpp.log
full-coredump
coredump-size unlimited
cli-listen /run/vpp/cli.sock
}
api-trace {
on
}
cpu {
main-core 1
corelist-workers 2-3
}
dpdk {
dev :2b:00.0 {
num-rx-queues 1
}
dev :2b:00.1 {
num-rx-queues 1
}
}
acl-plugin {
connection count max 2000
}
Thanks,
Mahamuda
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#19436): https://lists.fd.io/g/vpp-dev/message/19436
Mute This Topic: https://lists.fd.io/mt/77321080/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vpp crash issue with large number of acl session
You will need to increase the ACL plugin heap as well. Or try out latest
master, there the ACL plugin uses the main memory, so the configuration that
yon did should be enough.
--a
> On 5 Oct 2020, at 17:21, NetHappy wrote:
>
>
> Hi
>
> I am trying to use the vpp acl plugin module in a 16 GB system (vpp version
> 19.08). I have increased the session count using
> startup.conf with the following configuration.
>
> acl-plugin {
> connection count max 2000
> }
>
> But I see the system is crashing and its producing the following back trace.
>
>
> #5 0x7f3e2da59428 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> #6 0x7f3e2da5b02a in __GI_abort () at abort.c:89
> #7 0x56097de1ddf3 in os_panic () at
> /home/vagrant/vpp/src/vpp/vnet/main.c:355
> #8 0x7f3e2ee3276a in alloc_aligned_16_8 (nbytes=12288, h=0x7f3deca7d2e8
> ) at /home/vagrant/vpp/src/vppinfra/bihash_template.c:30
> #9 value_alloc_16_8 (h=h@entry=0x7f3deca7d2e8 ,
> log2_pages=log2_pages@entry=7) at
> /home/vagrant/vpp/src/vppinfra/bihash_template.c:290
> #10 0x7f3e2ee333ec in split_and_rehash_16_8 (h=h@entry=0x7f3deca7d2e8
> , old_values=old_values@entry=0x7f2d6eca3800,
> old_log2_pages=old_log2_pages@entry=6,
> new_log2_pages=new_log2_pages@entry=7) at
> /home/vagrant/vpp/src/vppinfra/bihash_template.c:387
> #11 0x7f3e2ee35af1 in clib_bihash_add_del_inline_16_8 (h=0x7f3deca7d2e8
> , add_v=0x7f3dee561d00, is_add=,
> is_stale_cb=,
> arg=) at
> /home/vagrant/vpp/src/vppinfra/bihash_template.c:671
> #12 0x7f3dec6bbe4a in acl_fa_conn_list_add_session
> (now=100206037591960300, sess_id=..., am=)
>
>
> Is adding acl-plugin setting the correct way to increase acl sessions or i
> need to do something else.
>
> Thanks,
> Mahamuda
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#17645): https://lists.fd.io/g/vpp-dev/message/17645
Mute This Topic: https://lists.fd.io/mt/77321080/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
