From: vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> On Behalf Of Benoit Ganne (bganne) via lists.fd.io Sent: Monday, November 7, 2022 3:56 AM To: Jieqiang Wang <jieqiang.w...@arm.com> Cc: Lijian Zhang <lijian.zh...@arm.com>; Tianyu Li <tianyu...@arm.com>; nd <n...@arm.com>; vpp-dev@lists.fd.io; moham...@hawari.fr Subject: Re: [vpp-dev] Issues about RDMA native plugin
Hi Jieqiang, Thanks a lot for your report! CC’ing vpp-dev because that may be of interest to others, and also you can get better support. My comments/questions inline prefixed with BG: in red Best ben From: Jieqiang Wang <jieqiang.w...@arm.com<mailto:jieqiang.w...@arm.com>> Sent: Monday, November 7, 2022 9:25 To: Benoit Ganne (bganne) <bga...@cisco.com<mailto:bga...@cisco.com>>; moham...@hawari.fr<mailto:moham...@hawari.fr> Cc: Lijian Zhang <lijian.zh...@arm.com<mailto:lijian.zh...@arm.com>>; Tianyu Li <tianyu...@arm.com<mailto:tianyu...@arm.com>>; nd <n...@arm.com<mailto:n...@arm.com>> Subject: Issues about RDMA native plugin Hi Ben/Mohammed, I am Jieqiang Wang from Arm Open Source Software team focusing on open source network software such as VPP. Recently our team did some investigation on RDMA native plugin on both Arm and X86 platform and met some crash and performance issues. I’m writing this email to share what are the issues and hope to get some input from you. Before going through the issues, I would like to share my local test environment for using RDMA native plugin. VPP version: vpp v23.02-rc0~89-ge7adafeaf Compiler to build VPP binary: Clang 13.0.0 Mellanox NIC: MCX516A-CDA_Ax ConnectX-5 100GbE dual-port QSFP28; PCIe4.0 x16 Server CPU: Arm: Neoverse-N1(Ampere Altra 1P) X86: Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz(Dell PowerEdge R820) Here are the issues we found using RDMA native plugin. Issue #1: VPP crashes when scales to multiple worker cores/threads. For example, running VPP(two worker threads) with startup.conf and CLI command file L3-rdma.exec in the attachment works fine. However, when injecting packets into VPP, VPP would just crash with output messages like the following shows. We saw similar segmentation faults for different test cases like L2 cross-connect/L2 mac learning. But we didn’t see crash issues for VPP debug version. We guess VPP buffer metadata has been corrupted in the case of multiple worker threads using VPP release version but are not sure how to debug this issue. Any suggestion to find the root cause of this issue? Thread 2 "vpp_wk_0" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7efb7649b700 (LWP 3355470)] 0x00007efd7842e8f6 in ip4_lookup_inline (vm=<optimized out>, frame=<optimized out>, node=<optimized out>) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.h:338 338 if (PREDICT_FALSE (lb0->lb_n_buckets > 1)) (gdb) bt #0 0x00007efd7842e8f6 in ip4_lookup_inline (vm=<optimized out>, frame=<optimized out>, node=<optimized out>) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.h:338 #1 ip4_lookup_node_fn_skx (vm=<optimized out>, node=0x7efc7c455b00, frame=0x7efc7c7e16c0) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.c:101 #2 0x00007efd7711effb in dispatch_node (vm=0x7efc7c43bd40, node=0x7efc7c455b00, type=VLIB_NODE_TYPE_INTERNAL, dispatch_state=VLIB_NODE_STATE_POLLING, frame=<optimized out>, last_time_stamp=2494501287259978) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:960 #3 dispatch_pending_node (vm=0x7efc7c43bd40, pending_frame_index=<optimized out>, last_time_stamp=2494501287259978) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1119 #4 vlib_main_or_worker_loop (vm=<optimized out>, is_main=0) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1588 #5 vlib_worker_loop (vm=<optimized out>, vm@entry=0x7efc7c43bd40) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1722 #6 0x00007efd77171dda in vlib_worker_thread_fn (arg=0x7efc78814d00) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/threads.c:1598 #7 0x00007efd7716c8f1 in vlib_worker_thread_bootstrap_fn (arg=0x7efc78814d00) at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/threads.c:418 #8 0x00007efd7709b609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #9 0x00007efd76dd8163 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 BG: do you reproduce the issue on both x86 and ARM? Or only on ARM? I’m especially thinking about difference in memory coherency that could bite us here… [Honnappa] On Arm, the barriers for working with MMIO are different than the ones used for normal shared memory. Currently, in VPP, this distinction does not exist (please correct me if I am wrong). For ex: update_nic_descriptors_in_memory(); store_io_barrier(); /*** Currently, this API abstraction does not exist in VPP ***/ ring_door_bell_on_nic(); So, we need new APIs for the IO barriers and those need to be incorporated in the native drivers. For DPDK drivers, this is addressed in DPDK. I cannot think of any other synchronization/memory model issue. Issue #2: Huge performance gap between Striding RQ mode and Non-Striding RQ mode. We saw a huge performance gap when using RDMA interfaces created with Striding RQ enabled VS RDMA interfaces with Striding RQ disabled. On both Arm and X86 server mentioned above, throughput numbers for L3 IPv4 routing test case are about 12 Mpps with Striding RQ mode enabled. Surprisingly, it drops to ~ 30000 pps when Striding RQ mode is disabled. Is this behavior expected? BG: I’ve seen that before. Usually, striding RQ should be more performant but I’m not sure how bad non-striding should be… Did you see the same drop in performance with DPDK mlx5 driver when disabling striding RQ? See mprq_en parameter https://doc.dpdk.org/guides/nics/mlx5.html#driver-options Hope my email looks not messy to you. Look forward to your reply! Thanks, Jieqiang Wang IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22147): https://lists.fd.io/g/vpp-dev/message/22147 Mute This Topic: https://lists.fd.io/mt/94862787/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
