Copying Neale. He may not respond immediately since he's on PTO until May 13th.
HTH... Dave -----Original Message----- From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Andreas Schultz Sent: Friday, May 10, 2019 5:54 AM To: vpp-dev@lists.fd.io Subject: Re: [vpp-dev] finding a virtual memory leak in VPP Am Do., 9. Mai 2019 um 19:59 Uhr schrieb Dave Barach (dbarach) <dbar...@cisco.com>: > > $ cat /proc/<pid>/maps > > > > You could also start vpp under gdb and set a breakpoint in mmap... uhm, that lead to something unexpected: Breakpoint 1, __GI___mmap64 (addr=0x0, len=33554432, prot=3, flags=34, fd=-1, offset=0) at ../sysdeps/unix/sysv/linux/mmap64.c:44 44 in ../sysdeps/unix/sysv/linux/mmap64.c #0 __GI___mmap64 (addr=0x0, len=33554432, prot=3, flags=34, fd=-1, offset=0) at ../sysdeps/unix/sysv/linux/mmap64.c:44 #1 0x00007ffff74c8cee in clib_mem_vm_alloc (size=33554432) at /usr/src/vpp/src/vppinfra/mem.h:317 #2 0x00007ffff74ca431 in clib_bihash_init_24_8 (h=0x7fffb5ff4a80, name=0x7ffff7759638 "Adjacency Neighbour table", nbuckets=4096, memory_size=33554432) at /usr/src/vpp/src/vppinfra/bihash_template.c:55 #3 0x00007ffff752c97a in adj_nbr_insert (nh_proto=FIB_PROTOCOL_IP4, link_type=VNET_LINK_IP4, nh_addr=0x7ffff76d29e0 <zero_addr>, sw_if_index=7, adj_index=7) at /usr/src/vpp/src/vnet/adj/adj_nbr.c:68 #4 0x00007ffff752ce6f in adj_nbr_alloc (nh_proto=FIB_PROTOCOL_IP4, link_type=VNET_LINK_IP4, nh_addr=0x7ffff76d29e0 <zero_addr>, sw_if_index=7) at /usr/src/vpp/src/vnet/adj/adj_nbr.c:187 #5 0x00007ffff752cf74 in adj_nbr_add_or_lock (nh_proto=FIB_PROTOCOL_IP4, link_type=VNET_LINK_IP4, nh_addr=0x7ffff76d29e0 <zero_addr>, sw_if_index=7) at /usr/src/vpp/src/vnet/adj/adj_nbr.c:233 #6 0x00007ffff7508d3e in fib_path_attached_next_hop_get_adj (path=0x7fffb5f15f34, link=VNET_LINK_IP4) at /usr/src/vpp/src/vnet/fib/fib_path.c:653 #7 0x00007ffff7508dd6 in fib_path_attached_next_hop_set (path=0x7fffb5f15f34) at /usr/src/vpp/src/vnet/fib/fib_path.c:674 #8 0x00007ffff750b6de in fib_path_resolve (path_index=51) at /usr/src/vpp/src/vnet/fib/fib_path.c:1883 #9 0x00007ffff7504c4b in fib_path_list_resolve (path_list=0x7fffb5f14634) at /usr/src/vpp/src/vnet/fib/fib_path_list.c:578 #10 0x00007ffff750515a in fib_path_list_create (flags=FIB_PATH_LIST_FLAG_SHARED, rpaths=0x7fffb5ff495c) at /usr/src/vpp/src/vnet/fib/fib_path_list.c:736 #11 0x00007ffff74f94bc in fib_entry_src_api_path_swap (src=0x7fffb5ff49bc, entry=0x7fffb5ff0cb4, pl_flags=FIB_PATH_LIST_FLAG_NONE, rpaths=0x7fffb5ff495c) at /usr/src/vpp/src/vnet/fib/fib_entry_src_api.c:47 #12 0x00007ffff74f5fd8 in fib_entry_src_action_path_swap (fib_entry=0x7fffb5ff0cb4, source=FIB_SOURCE_PLUGIN_HI, flags=FIB_ENTRY_FLAG_ATTACHED, rpaths=0x7fffb5ff495c) at /usr/src/vpp/src/vnet/fib/fib_entry_src.c:1658 #13 0x00007ffff74ea527 in fib_entry_create (fib_index=2, prefix=0x7fffb4bd9870, source=FIB_SOURCE_PLUGIN_HI, flags=FIB_ENTRY_FLAG_ATTACHED, paths=0x7fffb5ff495c) at /usr/src/vpp/src/vnet/fib/fib_entry.c:736 #14 0x00007ffff74d4bfb in fib_table_entry_path_add2 (fib_index=2, prefix=0x7fffb4bd9870, source=FIB_SOURCE_PLUGIN_HI, flags=FIB_ENTRY_FLAG_ATTACHED, rpath=0x7fffb5ff495c) at /usr/src/vpp/src/vnet/fib/fib_table.c:576 #15 0x00007ffff74d4aeb in fib_table_entry_path_add (fib_index=2, prefix=0x7fffb4bd9870, source=FIB_SOURCE_PLUGIN_HI, flags=FIB_ENTRY_FLAG_ATTACHED, next_hop_proto=DPO_PROTO_IP4, next_hop=0x0, next_hop_sw_if_index=7, next_hop_fib_index=4294967295, next_hop_weight=1, next_hop_labels=0x0, path_flags=FIB_ROUTE_PATH_FLAG_NONE) at /usr/src/vpp/src/vnet/fib/fib_table.c:548 My plugin uses a interface per session (like other tunnel implementation do) and the adds a route per session pointing into that interface (the fib_table_entry_path_add). At the end this result in new pool instance for the adjacency. And a pool allocates some space by default. So, every session end up instantiating a new pool through the fib entry. It is no surprise that this doesn't scale beyond a few thousand session. I know can I can rewrite my tunnel code to use a mid chain adjacency. But is that the only choice or do I do something wrong with the interface and fib entry? How do the other tunnel implementations scale that use fib entries? Many thanks Andreas Andreas > > > > D. > > > > > > > > From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Andreas > Schultz > Sent: Thursday, May 9, 2019 12:18 PM > To: vpp-dev@lists.fd.io > Subject: [vpp-dev] finding a virtual memory leak in VPP > > > > Hi, > > > > Something in VPP (most likely my UPF/GTP plugin) is causing an abnormal high > virtual memory consumption. `show memory` without and with memory-trace > enable can not explain it. > > > > With some 10k sessions I see as much as 1TB virtual memory usage, but only > about 200MB or so residential memory. `show memory` also only reports about > 200MB. > > > > My best guess is that something requests virtual memory pages from the OS and > does not return them. > > Any hint on how to track this down? What in VPP could request that much > virtual memory without actually using it? > > > > Many thanks > > Andreas > > > > -- > > -- > Dipl.-Inform. Andreas Schultz > > ----------------------- enabling your networks ---------------------- > Travelping GmbH Phone: +49-391-81 90 99 0 > Roentgenstr. 13 Fax: +49-391-81 90 99 299 > 39108 Magdeburg Email: i...@travelping.com > GERMANY Web: http://www.travelping.com > > Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 > > Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 > --------------------------------------------------------------------- -- -- Dipl.-Inform. Andreas Schultz ----------------------- enabling your networks ---------------------- Travelping GmbH Phone: +49-391-81 90 99 0 Roentgenstr. 13 Fax: +49-391-81 90 99 299 39108 Magdeburg Email: i...@travelping.com GERMANY Web: http://www.travelping.com Company Registration: Amtsgericht Stendal Reg No.: HRB 10578 Geschaeftsfuehrer: Holger Winkelmann VAT ID No.: DE236673780 ---------------------------------------------------------------------
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12977): https://lists.fd.io/g/vpp-dev/message/12977 Mute This Topic: https://lists.fd.io/mt/31557024/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
