Hi all, When we only use one CPU core, the cmd "show trace max 5000" works well. But it will crash when we use four CPU cores because of "out of memory". Below are some information, any guides?
root@vBRAS:~# cat /proc/meminfo MemTotal: 4028788 kB MemFree: 585636 kB MemAvailable: 949116 kB Buffers: 22696 kB Cached: 592600 kB SwapCached: 0 kB Active: 1773520 kB Inactive: 118616 kB Active(anon): 1295912 kB Inactive(anon): 45640 kB Active(file): 477608 kB Inactive(file): 72976 kB Unevictable: 3656 kB Mlocked: 3656 kB SwapTotal: 976380 kB SwapFree: 976380 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 1280520 kB Mapped: 112324 kB Shmem: 62296 kB Slab: 84456 kB SReclaimable: 35976 kB SUnreclaim: 48480 kB KernelStack: 5968 kB PageTables: 267268 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 2466484 kB Committed_AS: 5368769328 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 348160 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 512 HugePages_Free: 384 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 96064 kB DirectMap2M: 3049472 kB DirectMap1G: 3145728 kB 0: load_one_plugin:63: Loaded plugin: /usr/lib/vpp_api_test_plugins/ioam_vxlan_gpe_test_plugin.so 0: vlib_pci_bind_to_uio: Skipping PCI device 0000:02:0e.0 as host interface ens46 is up EAL: Detected 4 lcore(s) EAL: No free hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... [New Thread 0x7b0019efa700 (LWP 5207)] [New Thread 0x7b00196f9700 (LWP 5208)] [New Thread 0x7b0018ef8700 (LWP 5209)] EAL: PCI device 0000:02:01.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:06.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:07.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:08.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:09.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:0a.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:0b.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:0c.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:0d.0 on NUMA socket -1 EAL: probe driver: 8086:100f net_e1000_em EAL: PCI device 0000:02:0e.0 on NUMA socket -1 EAL: Device is blacklisted, not initializing DPDK physical memory layout: Segment 0: phys:0x7d400000, len:2097152, virt:0x7b0015000000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 Segment 1: phys:0x7d800000, len:266338304, virt:0x7affe4600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0 [New Thread 0x7b00186f7700 (LWP 5210)] /usr/bin/vpp[5202]: dpdk_ipsec_process:241: DPDK Cryptodev support is disabled, default to OpenSSL IPsec /usr/bin/vpp[5202]: dpdk_lib_init:1084: 16384 mbufs allocated but total rx/tx ring size is 18432 /usr/bin/vpp[5202]: svm_client_scan_this_region_nolock:1139: /vpe-api: cleanup ghost pid 4719 /usr/bin/vpp[5202]: svm_client_scan_this_region_nolock:1139: /global_vm: cleanup ghost pid 4719 Thread 1 "vpp_main" received signal SIGABRT, Aborted. 0x00007fffef655428 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) (gdb) (gdb) (gdb) p errno /*there are only 81 opened fd belong to progress VPP*/ $1 = 9 (gdb) bt #0 0x00007fffef655428 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fffef65702a in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x000000000040724e in os_panic () at /home/vbras/new_trunk/VBRASV100R001_new_trunk/vpp1704/build-data/../src/vpp/vnet/main.c:290 #3 0x00007fffefe6b49b in clib_mem_alloc_aligned_at_offset (os_out_of_memory_on_failure=1, align_offset=<optimized out>, align=4, size=18768606) /*mmap*/ at /home/vbras/new_trunk/VBRASV100R001_new_trunk/vpp1704/build-data/../src/vppinfra/mem.h:102 #4 vec_resize_allocate_memory (v=<optimized out>, length_increment=length_increment@entry=1, data_bytes=<optimized out>, header_bytes=<optimized out>, header_bytes@entry=0, data_align=data_align@entry=4) at /home/vbras/new_trunk/VBRASV100R001_new_trunk/vpp1704/build-data/../src/vppinfra/vec.c:84 #5 0x0000000000420f04 in _vec_resize (data_align=<optimized out>, header_bytes=<optimized out>, data_bytes=<optimized out>, length_increment=<optimized out>, v=<optimized out>) at /home/vbras/new_trunk/VBRASV100R001_new_trunk/vpp1704/build-data/../src/vppinfra/vec.h:142 #6 vl_api_cli_request_t_handler (mp=<optimized out>) at /home/vbras/new_trunk/VBRASV100R001_new_trunk/vpp1704/build-data/../src/vpp/api/api.c:1132 #7 0x00007ffff7bce2e3 in vl_msg_api_handler_with_vm_node (am=0x7ffff7dd6160 <api_main>, the_msg=0x30521e4c, vm=0x7ffff79aa2a0 <vlib_global_main>, node=<optimized out>) at /home/vbras/new_trunk/VBRASV100R001_new_trunk/vpp1704/build-data/../src/vlibapi/api_shared.c:502 #8 0x00007ffff79b619f in memclnt_process (vm=<optimized out>, node=0x7fffaebbe000, f=<optimized out>) at /home/vbras/new_trunk/VBRASV100R001_new_trunk/vpp1704/build-data/../src/vlibmemory/memory_vlib.c:543 #9 0x00007ffff7755fa6 in vlib_process_bootstrap (_a=<optimized out>) at /home/vbras/new_trunk/VBRASV100R001_new_trunk/vpp1704/build-data/../src/vlib