Did you delete the handle of the shared memory? These two files: /dev/shm/global_vm vpe-api
The mheap header structure is changed, so you have to ask vPP to re-create the shared memory heap. Sorry, I forgot to mention that before. From: 薛欣颖 [mailto:xy...@fiberhome.com] Sent: Monday, April 23, 2018 3:25 PM To: Kingwel Xie <kingwel....@ericsson.com>; Damjan Marion <damar...@cisco.com>; nranns <nra...@cisco.com> Cc: vpp-dev <vpp-dev@lists.fd.io> Subject: Re: Re: [vpp-dev] mheap performance issue and fixup Hi Kingwel, After I merged the three patch ,there is a SIGSEGV when I startup vpp (not every time). And the error didn't appear before . Is there anything I can do to fix it? Program received signal SIGSEGV, Segmentation fault. clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:90 90 cpu = os_get_thread_index (); (gdb) bt #0 clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:90 #1 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #2 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #3 0x00007ffff69322bb in do_percent (_s=0x7fffb6cee348, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cee3e0) at /home/vpp/build-data/../src/vppinfra/format.c:339 #4 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cee3e0) at /home/vpp/build-data/../src/vppinfra/format.c:402 #5 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #6 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cee6b0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #7 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cee6b0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #8 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cee758) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #9 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #10 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #11 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #12 0x00007ffff69322bb in do_percent (_s=0x7fffb6ceea78, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6ceeb10) at /home/vpp/build-data/../src/vppinfra/format.c:339 #13 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6ceeb10) at /home/vpp/build-data/../src/vppinfra/format.c:402 #14 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #15 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6ceede0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #16 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6ceede0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #17 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6ceee88) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #18 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #19 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) ---Type <return> to continue, or q <return> to quit--- at /home/vpp/build-data/../src/vppinfra/vec.c:59 #20 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #21 0x00007ffff69322bb in do_percent (_s=0x7fffb6cef1a8, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cef240) at /home/vpp/build-data/../src/vppinfra/format.c:339 #22 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cef240) at /home/vpp/build-data/../src/vppinfra/format.c:402 #23 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #24 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cef510, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #25 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cef510, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #26 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cef5b8) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #27 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #28 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #29 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #30 0x00007ffff69322bb in do_percent (_s=0x7fffb6cef8d8, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cef970) at /home/vpp/build-data/../src/vppinfra/format.c:339 #31 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cef970) at /home/vpp/build-data/../src/vppinfra/format.c:402 #32 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #33 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cefc40, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #34 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cefc40, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #35 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cefce8) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #36 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #37 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #38 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) ---Type <return> to continue, or q <return> to quit--- at /home/vpp/build-data/../src/vppinfra/vec.h:142 #39 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf0008, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf00a0) at /home/vpp/build-data/../src/vppinfra/format.c:339 #40 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf00a0) at /home/vpp/build-data/../src/vppinfra/format.c:402 #41 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #42 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cf0370, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #43 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf0370, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #44 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cf0418) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #45 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #46 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #47 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #48 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf0738, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf07d0) at /home/vpp/build-data/../src/vppinfra/format.c:339 #49 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf07d0) at /home/vpp/build-data/../src/vppinfra/format.c:402 #50 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #51 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cf0aa0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #52 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf0aa0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #53 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cf0b48) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #54 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #55 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #56 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #57 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf0e68, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf0f00) ---Type <return> to continue, or q <return> to quit--- at /home/vpp/build-data/../src/vppinfra/format.c:339 #58 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf0f00) at /home/vpp/build-data/../src/vppinfra/format.c:402 #59 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #60 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cf11d0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #61 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf11d0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #62 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cf1278) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #63 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #64 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #65 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #66 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf1598, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf1630) at /home/vpp/build-data/../src/vppinfra/format.c:339 #67 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf1630) at /home/vpp/build-data/../src/vppinfra/format.c:402 #68 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #69 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cf1900, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #70 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf1900, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #71 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cf19a8) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #72 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #73 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #74 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #75 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf1cc8, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf1d60) at /home/vpp/build-data/../src/vppinfra/format.c:339 #76 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf1d60) ---Type <return> to continue, or q <return> to quit--- at /home/vpp/build-data/../src/vppinfra/format.c:402 #77 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #78 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cf2030, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #79 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf2030, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #80 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cf20d8) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #81 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #82 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #83 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #84 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf23f8, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf2490) at /home/vpp/build-data/../src/vppinfra/format.c:339 #85 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf2490) at /home/vpp/build-data/../src/vppinfra/format.c:402 #86 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #87 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cf2760, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #88 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf2760, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #89 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cf2808) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #90 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #91 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #92 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #93 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf2b28, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf2bc0) at /home/vpp/build-data/../src/vppinfra/format.c:339 #94 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf2bc0) at /home/vpp/build-data/../src/vppinfra/format.c:402 #95 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") ---Type <return> to continue, or q <return> to quit--- at /home/vpp/build-data/../src/vppinfra/error.c:127 #96 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cf2e90, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #97 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf2e90, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #98 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cf2f38) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #99 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #100 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #101 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #102 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf3258, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf32f0) at /home/vpp/build-data/../src/vppinfra/format.c:339 #103 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf32f0) at /home/vpp/build-data/../src/vppinfra/format.c:402 #104 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #105 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=12, n_user_data_bytes_arg=0x7fffb6cf35c0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #106 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf35c0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #107 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=56, align=4, align_offset=0, offset_return=0x7fffb6cf3668) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #108 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=54, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #109 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=50, data_bytes=54, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #110 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=50, data_bytes=50, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #111 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf3988, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf3a20) at /home/vpp/build-data/../src/vppinfra/format.c:339 #112 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf3a20) at /home/vpp/build-data/../src/vppinfra/format.c:402 #113 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #114 0x00007ffff69496a3 in mheap_get_search_free_bin (v=0x3000a000, bin=14, n_user_data_bytes_arg=0x7fffb6cf3cf0, align=4, align_offset=0) ---Type <return> to continue, or q <return> to quit--- at /home/vpp/build-data/../src/vppinfra/mheap.c:401 #115 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf3cf0, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #116 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=64, align=4, align_offset=0, offset_return=0x7fffb6cf3d98) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #117 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=64, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #118 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=60, data_bytes=64, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #119 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=60, data_bytes=60, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #120 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf40b8, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf4150) at /home/vpp/build-data/../src/vppinfra/format.c:339 #121 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails", va=0x7fffb6cf4150) at /home/vpp/build-data/../src/vppinfra/format.c:402 #122 0x00007ffff692ce4e in _clib_error (how_to_die=2, function_name=0x0, line_number=0, fmt=0x7ffff6995c90 "%s:%d (%s) assertion `%s' fails") at /home/vpp/build-data/../src/vppinfra/error.c:127 #123 0x00007ffff6947841 in mheap_elt_at_uoffset (v=0x3000a000, uo=0) at /home/vpp/build-data/../src/vppinfra/mheap_bootstrap.h:309 #124 0x00007ffff69496c8 in mheap_get_search_free_bin (v=0x3000a000, bin=1, n_user_data_bytes_arg=0x7fffb6cf4470, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:402 #125 0x00007ffff6949e86 in mheap_get_search_free_list (v=0x3000a000, n_user_bytes_arg=0x7fffb6cf4470, align=4, align_offset=0) at /home/vpp/build-data/../src/vppinfra/mheap.c:569 #126 0x00007ffff694a326 in mheap_get_aligned (v=0x3000a000, n_user_data_bytes=12, align=4, align_offset=0, offset_return=0x7fffb6cf4518) at /home/vpp/build-data/../src/vppinfra/mheap.c:700 #127 0x00007ffff697f91e in clib_mem_alloc_aligned_at_offset (size=12, align=4, align_offset=4, os_out_of_memory_on_failure=1) at /home/vpp/build-data/../src/vppinfra/mem.h:92 #128 0x00007ffff697fcde in vec_resize_allocate_memory (v=0x0, length_increment=8, data_bytes=12, header_bytes=4, data_align=4) at /home/vpp/build-data/../src/vppinfra/vec.c:59 #129 0x00007ffff69313c7 in _vec_resize (v=0x0, length_increment=8, data_bytes=8, header_bytes=0, data_align=0) at /home/vpp/build-data/../src/vppinfra/vec.h:142 #130 0x00007ffff69322bb in do_percent (_s=0x7fffb6cf4838, fmt=0x7ffff5f1b18b "%s%c", va=0x7fffb6cf48b0) at /home/vpp/build-data/../src/vppinfra/format.c:339 #131 0x00007ffff6932703 in va_format (s=0x0, fmt=0x7ffff5f1b18b "%s%c", va=0x7fffb6cf48b0) at /home/vpp/build-data/../src/vppinfra/format.c:402 #132 0x00007ffff6932876 in format (s=0x0, fmt=0x7ffff5f1b18b "%s%c") at /home/vpp/build-data/../src/vppinfra/format.c:421 #133 0x00007ffff5f0ce10 in shm_name_from_svm_map_region_args (a=0x7fffb6cf4cf0) at /home/vpp/build-data/../src/svm/svm.c:525 #134 0x00007ffff5f0d60d in svm_map_region (a=0x7fffb6cf4cf0) at /home/vpp/build-data/../src/svm/svm.c:658 #135 0x00007ffff5f0e663 in svm_region_find_or_create (a=0x7fffb6cf4cf0) at /home/vpp/build-data/../src/svm/svm.c:995 #136 0x00007ffff7938e9a in vl_map_shmem (region_name=0x7ffff79554c7 "/vpe-api", is_vlib=1) at /home/vpp/build-data/../src/vlibmemory/memory_shared.c:514 #137 0x00007ffff79413fc in memory_api_init (region_name=0x7ffff79554c7 "/vpe-api") at /home/vpp/build-data/../src/vlibmemory/memory_vlib.c:651 ---Type <return> to continue, or q <return> to quit--- #138 0x00007ffff7942ed0 in memclnt_process (vm=0x7ffff7926840 <vlib_global_main>, node=0x7fffb6cec000, f=0x0) at /home/vpp/build-data/../src/vlibmemory/memory_vlib.c:952 #139 0x00007ffff76a603c in vlib_process_bootstrap (_a=140736237530192) at /home/vpp/build-data/../src/vlib/main.c:1253 #140 0x00007ffff6941570 in clib_calljmp () at /home/vpp/build-data/../src/vppinfra/longjmp.S:128 #141 0x00007fffb571ec20 in ?? () #142 0x00007ffff76a6179 in vlib_process_startup (vm=0x7ffff7926840 <vlib_global_main>, p=0x7fffb6cec000, f=0x0) at /home/vpp/build-data/../src/vlib/main.c:1275 Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) (gdb) Thanks, Xyxue ________________________________ From: Kingwel Xie<mailto:kingwel....@ericsson.com> Date: 2018-04-20 17:29 To: Damjan Marion<mailto:damar...@cisco.com>; Neale Ranns (nranns)<mailto:nra...@cisco.com>; 薛欣颖<mailto:xy...@fiberhome.com> CC: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] mheap performance issue and fixup Hi, Finally I managed to create 3 patches to include all modifications to mheap. Please check below for details. I’ll do some other patches later… https://gerrit.fd.io/r/11950 https://gerrit.fd.io/r/11952 https://gerrit.fd.io/r/11957 Hi Xue, you need at least the first one for your test. Regards, Kingwel From: Kingwel Xie Sent: Thursday, April 19, 2018 9:20 AM To: Damjan Marion <damar...@cisco.com<mailto:damar...@cisco.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: [vpp-dev] mheap performance issue and fixup Hi Damjan, We will do it asap. Actually we are quite new to vPP and even don’t know how to make bug report and code contribution or so. Regards, Kingwel From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> [mailto:vpp-dev@lists.fd.io] On Behalf Of Damjan Marion Sent: Wednesday, April 18, 2018 11:30 PM To: Kingwel Xie <kingwel....@ericsson.com<mailto:kingwel....@ericsson.com>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] mheap performance issue and fixup Dear Kingwel, Thank you for your email. It will be really appreciated if you can submit your changes to gerrit, preferably each point in separate patch. That will be best place to discuss those changes... Thanks in Advance, -- Damjan On 16 Apr 2018, at 10:13, Kingwel Xie <kingwel....@ericsson.com<mailto:kingwel....@ericsson.com>> wrote: Hi all, We recently worked on GTPU tunnel and our target is to create 2M tunnels. It is not as easy as it looks like, and it took us quite some time to figure it out. The biggest problem we found is about mheap, which as you know is the low layer memory management function of vPP. We believe it makes sense to share what we found and what we’ve done to improve the performance of mheap. First of all, mheap is fast. It has well-designed small object cache and multi-level free lists, to speed up the get/put. However, as discussed in the mail list before, it has a performance issue when dealing with align/align_offset allocation. We managed to locate the problem is brought by a pointer ‘rewrite’ in gtp_tunnel_t. This rewrite is a vector and required to be aligned to 64B cache line, therefore with 4 bytes align offset. We realized that it is because that the free list must be very long, meaning so many mheap_elts, but unfortunately it doesn’t have an element which fits to all 3 prerequisites: size, align, and align offset. In this case, each allocation has to traverse all elements till it reaches the end of element. As a result, you might observe each allocation is greater than 100000 clocks/call with ‘show memory verbose’. It indicates the allocation takes too long, while it should be 200~300 clocks/call in general. Also you should have noticed ‘per-attempt’ is quite high, even more than 100. The fix is straight and simple : as discussed int his mail list before, to allocate ‘rewrite’ from a pool, instead of from mheap. Frankly speaking, it looks like a workaround not a real fix, so we spent some time fix the problem thoroughly. The idea is to add a few more bytes to the original required block size so that mheap will always lookup in a bigger free list, then most likely a suitable block can be easily located. Well, now the problem becomes how big is this extra size? It should be at least align+align_offset, not hard to understand. But after careful analysis we think it is better to be like this, see code below: Mheap.c:545 word modifier = (align > MHEAP_USER_DATA_WORD_BYTES ? align + align_offset + sizeof(mheap_elt_t) : 0); bin = user_data_size_to_bin_index (n_user_bytes + modifier); The reason of extra sizeof(mheap_elt_t) is to avoid lo_free_size is too small to hold a complete free element. You will understand it if you really know how mheap_get_search_free_bin is working. I am not going to go through the detail of it. In short, every lookup in free list will always locate a suitable element, in other words, the hit rate of free list will be almost 100%, and the ‘per-attempt’ will be always around 1. The test result looks very promising, please see below after adding 2M gtpu tunnels and 2M routing entries: Thread 0 vpp_main 13689507 objects, 3048367k of 3505932k used, 243663k free, 243656k reclaimed, 106951k overhead, 4194300k capacity alloc. from small object cache: 47325868 hits 65271210 attempts (72.51%) replacements 8266122 alloc. from free-list: 21879233 attempts, 21877898 hits (99.99%), 21882794 considered (per-attempt 1.00) alloc. low splits: 13355414, high splits: 512984, combined: 281968 alloc. from vector-expand: 81907 allocs: 69285673 276.00 clocks/call frees: 55596166 173.09 clocks/call Free list: bin 3: 20(82220170 48) total 1 bin 273: 28340k(80569efc 60) total 1 bin 276: 215323k(8c88df6c 44) total 1 Total count in free bin: 3 You can see, as pointed out before, the hit rate is very high, >99.9%, and per-attempt is ~1. Furthermore, the total elements in free list is only 3. Apart from we discussed above, we also made some other improvements/bug fixes to mheap: 1. Bug fix: macros MHEAP_ELT_OVERHEAD_BYTES & MHEAP_MIN_USER_DATA_BYTES are wrongly defined. In fact MHEAP_ELT_OVERHEAD_BYTES should be (STRUCT_OFFSET_OF (mheap_elt_t, user_data)) 2. mheap_bytes_overhead is wrongly calculating the total overhead – should be number of elements * MHEAP_ELT_OVERHEAD_BYTES 3. Do not make an element if hi_free_size is smaller than 4 times of MHEAP_MIN_USER_DATA_BYTES. This is to avoid memory fragmentation 4. Bug fix: register_node.c:336 is wrongly using vector memory, should be like this: clib_mem_is_heap_object (vec_header (r->name, 0)) 5. Bug fix: dpo_stack_from_node in dpo.c: memory leak, of parent_indices 6. Some fixes and improvements of format_mheap to show more information of heap The code including all fixes is tentatively in our private code base. It can be of course shared if wanted. Really appreciate any comments! Regards, Kingwel
