Hi Maintainers, I have two supermicro x86 servers with unusual numa configs. The first server has two physical CPUs, but only one numa node. The other server has two physical CPUs, two numa nodes, but only numa node 0 has attached memories. The detailed CPU info are dumped as below.
VPP crashed on both server with below call-stack. VPP gets numa nodes ‘0-1’ from ‘/sys/devices/system/node/possible’, and initialize buffers for all of numa nodes, while numa node 1 has no memories, so below function failed with numa node. If ‘/sys/devices/system/node/has_memories’ is used to detect the numa nodes, on which VPP initialize buffers, there will be no such crash issue. Could you please evaluate this issue, and evaluate if this is a reasonable change requirement, or should we just skip the issue because of the abnormal numa configs in my servers? #0 vlib_physmem_shared_map_create (vm=0x7ffff6853240 <vlib_global_main>, name=0x7fffb786621c "buffers-numa-0", size=1073741824, log2_page_sz=30, numa_node=0, map_index=0x7fffb51ffd84) at /home/lijian/tasks/origin/src/vlib/physmem.c:45 #1 0x00007ffff657b279 in vlib_buffer_main_init_numa_node (vm=0x7ffff6853240 <vlib_global_main>, numa_node=0) at /home/lijian/tasks/origin/src/vlib/buffer.c:672 #2 0x00007ffff657b8f9 in vlib_buffer_main_init (vm=0x7ffff6853240 <vlib_global_main>) at /home/lijian/tasks/origin/src/vlib/buffer.c:801 #3 0x00007ffff65c7bb2 in vlib_main (vm=0x7ffff6853240 <vlib_global_main>, input=0x7fffb51fffb0) at /home/lijian/tasks/origin/src/vlib/main.c:1988 #4 0x00007ffff662570a in thread0 (arg=140737329312320) at /home/lijian/tasks/origin/src/vlib/unix/main.c:606 #5 0x00007ffff6079544 in clib_calljmp () from /home/lijian/tasks/origin/build-root/build-vpp_debug-native/vpp/lib/libvppinfra.so.19.04 #6 0x00007fffffffd2b0 in ?? () #7 0x00007ffff6625bd7 in vlib_unix_main (argc=36, argv=0x55555586f3b0) at /home/lijian/tasks/origin/src/vlib/unix/main.c:675 #8 0x000055555555bedc in main (argc=36, argv=0x55555586f3b0) at /home/lijian/tasks/origin/src/vpp/vnet/main.c:274 lijian@net-x86-supermicro-01:~/tasks/origin$ sudo numactl -H available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 node 0 size: 64383 MB node 0 free: 33048 MB node distances: node 0 0: 10 lijian@net-x86-supermicro-01:~/tasks/origin$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Thread(s) per core: 1 Core(s) per socket: 10 Socket(s): 2 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Stepping: 2 CPU MHz: 2300.179 CPU max MHz: 3000.0000 CPU min MHz: 1200.0000 BogoMIPS: 4601.05 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 25600K NUMA node0 CPU(s): 0-19 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts lijian@net-x86-supermicro-02:~/tasks/origin$ sudo numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 node 0 size: 64311 MB node 0 free: 30865 MB node 1 cpus: 10 11 12 13 14 15 16 17 18 19 node 1 size: 0 MB node 1 free: 0 MB node distances: node 0 1 0: 10 20 1: 20 10 lijian@net-x86-supermicro-02:~/tasks/origin$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 20 On-line CPU(s) list: 0-19 Thread(s) per core: 1 Core(s) per socket: 10 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz Stepping: 2 CPU MHz: 1200.176 CPU max MHz: 2300.0000 CPU min MHz: 1200.0000 BogoMIPS: 4599.71 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 25600K NUMA node0 CPU(s): 0-9 NUMA node1 CPU(s): 10-19 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts flush_l1d lijian@net-x86-supermicro-02:~/tasks/origin$ Thanks. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12381): https://lists.fd.io/g/vpp-dev/message/12381 Mute This Topic: https://lists.fd.io/mt/30160902/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
