[vpp-dev] Gerrit Review Id 37579

[lihuawei]

Hi everyone,
 
I have added a fix as gerrit review id 37579.


gerrit link: https://gerrit.fd.io/r/c/vpp/+/37579


short description: cli show policer's help info is not consistent with it's 
arguments.


Please have a look and review.


Best regards,
Huawei LI
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22142): https://lists.fd.io/g/vpp-dev/message/22142
Mute This Topic: https://lists.fd.io/mt/94861307/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[vpp-dev] Gerrit Review Id 37585

[lihuawei]

Hi everyone,
 
I have added a fix as gerrit review id 37585.


gerrit link: https://gerrit.fd.io/r/c/vpp/+/37585


short description: fix crash when delete classify table used by interface.
when classify table is used by interface, the classify table can be deleted 
without any check and hint. And vpp will raise crash when process policer 
classify node on the interface, becaeuse of the classify table used by the 
interface is not existed now.


Please have a look and review.


Best regards,
Huawei LI
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22143): https://lists.fd.io/g/vpp-dev/message/22143
Mute This Topic: https://lists.fd.io/mt/94861156/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Gerrit Review Id 37579

[filvarga]

Hi,

That was already done. Please check your status.

Best regards,
Filip Varga


po 7. 11. 2022 o 9:01 lihuawei  napísal(a):

> Hi everyone,
>
> I have added a fix as gerrit review id 37579.
>
> *gerrit link: https://gerrit.fd.io/r/c/vpp/+/37579
> *
>
> *short description: *cli show policer's help info is not consistent with
> it's arguments.
>
> Please have a look and review.
>
> Best regards,
> Huawei LI
>
> 
>
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22144): https://lists.fd.io/g/vpp-dev/message/22144
Mute This Topic: https://lists.fd.io/mt/94861307/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Issues about RDMA native plugin

[Benoit Ganne (bganne) via lists.fd.io]

Hi Jieqiang,

Thanks a lot for your report! CC’ing vpp-dev because that may be of interest to 
others, and also you can get better support.
My comments/questions inline prefixed with BG: in red

Best
ben

From: Jieqiang Wang 
Sent: Monday, November 7, 2022 9:25
To: Benoit Ganne (bganne) ; moham...@hawari.fr
Cc: Lijian Zhang ; Tianyu Li ; nd 

Subject: Issues about RDMA native plugin

Hi Ben/Mohammed,

I am Jieqiang Wang from Arm Open Source Software team focusing on open source 
network software such as VPP.
Recently our team did some investigation on RDMA native plugin on both Arm and 
X86 platform and met some crash and performance issues.
I’m writing this email to share what are the issues and hope to get some input 
from you.

Before going through the issues, I would like to share my local test 
environment for using RDMA native plugin.
VPP version: vpp v23.02-rc0~89-ge7adafeaf
Compiler to build VPP binary: Clang 13.0.0
Mellanox NIC: MCX516A-CDA_Ax ConnectX-5 100GbE dual-port QSFP28; PCIe4.0 x16
Server CPU:
Arm: Neoverse-N1(Ampere Altra 1P)
X86: Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz(Dell PowerEdge R820)

Here are the issues we found using RDMA native plugin.
Issue #1: VPP crashes when scales to multiple worker cores/threads.
For example, running VPP(two worker threads) with startup.conf and CLI command 
file L3-rdma.exec in the attachment works fine. However, when injecting packets 
into VPP, VPP would just crash with output messages like the following shows.
We saw similar segmentation faults for different test cases like L2 
cross-connect/L2 mac learning. But we didn’t see crash issues for VPP debug 
version. We guess VPP buffer metadata has been corrupted in the case of 
multiple worker threads using VPP release version but are not sure how to debug 
this issue.
Any suggestion to find the root cause of this issue?

Thread 2 "vpp_wk_0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7efb7649b700 (LWP 3355470)]
0x7efd7842e8f6 in ip4_lookup_inline (vm=, frame=, node=) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.h:338
338   if (PREDICT_FALSE (lb0->lb_n_buckets > 1))
(gdb) bt
#0  0x7efd7842e8f6 in ip4_lookup_inline (vm=, 
frame=, node=) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.h:338
#1  ip4_lookup_node_fn_skx (vm=, node=0x7efc7c455b00, 
frame=0x7efc7c7e16c0) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.c:101
#2  0x7efd7711effb in dispatch_node (vm=0x7efc7c43bd40, 
node=0x7efc7c455b00, type=VLIB_NODE_TYPE_INTERNAL, 
dispatch_state=VLIB_NODE_STATE_POLLING, frame=,
last_time_stamp=2494501287259978) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:960
#3  dispatch_pending_node (vm=0x7efc7c43bd40, pending_frame_index=, last_time_stamp=2494501287259978) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1119
#4  vlib_main_or_worker_loop (vm=, is_main=0) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1588
#5  vlib_worker_loop (vm=, vm@entry=0x7efc7c43bd40) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1722
#6  0x7efd77171dda in vlib_worker_thread_fn (arg=0x7efc78814d00) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/threads.c:1598
#7  0x7efd7716c8f1 in vlib_worker_thread_bootstrap_fn (arg=0x7efc78814d00) 
at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/threads.c:418
#8  0x7efd7709b609 in start_thread (arg=) at 
pthread_create.c:477
#9  0x7efd76dd8163 in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

BG: do you reproduce the issue on both x86 and ARM? Or only on ARM? I’m 
especially thinking about difference in memory coherency that could bite us 
here…

Issue #2: Huge performance gap between Striding RQ mode and Non-Striding RQ 
mode.
We saw a huge performance gap when using RDMA interfaces created with Striding 
RQ enabled VS RDMA interfaces with Striding RQ disabled.
On both Arm and X86 server mentioned above, throughput numbers for L3 IPv4 
routing test case are about 12 Mpps with Striding RQ mode enabled. 
Surprisingly, it drops to ~ 3 pps when Striding RQ mode is disabled.
Is this behavior expected?

BG: I’ve seen that before. Usually, striding RQ should be more performant but 
I’m not sure how bad non-striding should be… Did you see the same drop in 
performance with DPDK mlx5 driver when disabling striding RQ? See mprq_en 
parameter https://doc.dpdk.org/guides/nics/mlx5.html#driver-options

Hope my email looks not messy to you.
Look forward to your reply!

Thanks,
Jieqiang Wang

IMPORTANT NOTICE: The contents of this email and any attachments are 
confidential and may also be privileged. If you are not the intended recipient, 
please notify the sender immediately and do not disclose the contents to any 
other person, use it for any purpose, or store or copy the information in any 
medium. Thank you.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply

Re: [vpp-dev] Gerrit Review Id 37579

[lihuawei]

Sorry about dup mails, something goes wrong with my email client. It's ok Now.

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22146): https://lists.fd.io/g/vpp-dev/message/22146
Mute This Topic: https://lists.fd.io/mt/94861307/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Issues about RDMA native plugin

[Honnappa Nagarahalli]

From: vpp-dev@lists.fd.io  On Behalf Of Benoit Ganne 
(bganne) via lists.fd.io
Sent: Monday, November 7, 2022 3:56 AM
To: Jieqiang Wang 
Cc: Lijian Zhang ; Tianyu Li ; nd 
; vpp-dev@lists.fd.io; moham...@hawari.fr
Subject: Re: [vpp-dev] Issues about RDMA native plugin

Hi Jieqiang,

Thanks a lot for your report! CC’ing vpp-dev because that may be of interest to 
others, and also you can get better support.
My comments/questions inline prefixed with BG: in red

Best
ben

From: Jieqiang Wang mailto:jieqiang.w...@arm.com>>
Sent: Monday, November 7, 2022 9:25
To: Benoit Ganne (bganne) mailto:bga...@cisco.com>>; 
moham...@hawari.fr
Cc: Lijian Zhang mailto:lijian.zh...@arm.com>>; Tianyu Li 
mailto:tianyu...@arm.com>>; nd 
mailto:n...@arm.com>>
Subject: Issues about RDMA native plugin

Hi Ben/Mohammed,

I am Jieqiang Wang from Arm Open Source Software team focusing on open source 
network software such as VPP.
Recently our team did some investigation on RDMA native plugin on both Arm and 
X86 platform and met some crash and performance issues.
I’m writing this email to share what are the issues and hope to get some input 
from you.

Before going through the issues, I would like to share my local test 
environment for using RDMA native plugin.
VPP version: vpp v23.02-rc0~89-ge7adafeaf
Compiler to build VPP binary: Clang 13.0.0
Mellanox NIC: MCX516A-CDA_Ax ConnectX-5 100GbE dual-port QSFP28; PCIe4.0 x16
Server CPU:
Arm: Neoverse-N1(Ampere Altra 1P)
X86: Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz(Dell PowerEdge R820)

Here are the issues we found using RDMA native plugin.
Issue #1: VPP crashes when scales to multiple worker cores/threads.
For example, running VPP(two worker threads) with startup.conf and CLI command 
file L3-rdma.exec in the attachment works fine. However, when injecting packets 
into VPP, VPP would just crash with output messages like the following shows.
We saw similar segmentation faults for different test cases like L2 
cross-connect/L2 mac learning. But we didn’t see crash issues for VPP debug 
version. We guess VPP buffer metadata has been corrupted in the case of 
multiple worker threads using VPP release version but are not sure how to debug 
this issue.
Any suggestion to find the root cause of this issue?

Thread 2 "vpp_wk_0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7efb7649b700 (LWP 3355470)]
0x7efd7842e8f6 in ip4_lookup_inline (vm=, frame=, node=) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.h:338
338   if (PREDICT_FALSE (lb0->lb_n_buckets > 1))
(gdb) bt
#0  0x7efd7842e8f6 in ip4_lookup_inline (vm=, 
frame=, node=) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.h:338
#1  ip4_lookup_node_fn_skx (vm=, node=0x7efc7c455b00, 
frame=0x7efc7c7e16c0) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vnet/ip/ip4_forward.c:101
#2  0x7efd7711effb in dispatch_node (vm=0x7efc7c43bd40, 
node=0x7efc7c455b00, type=VLIB_NODE_TYPE_INTERNAL, 
dispatch_state=VLIB_NODE_STATE_POLLING, frame=,
last_time_stamp=2494501287259978) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:960
#3  dispatch_pending_node (vm=0x7efc7c43bd40, pending_frame_index=, last_time_stamp=2494501287259978) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1119
#4  vlib_main_or_worker_loop (vm=, is_main=0) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1588
#5  vlib_worker_loop (vm=, vm@entry=0x7efc7c43bd40) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/main.c:1722
#6  0x7efd77171dda in vlib_worker_thread_fn (arg=0x7efc78814d00) at 
/home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/threads.c:1598
#7  0x7efd7716c8f1 in vlib_worker_thread_bootstrap_fn (arg=0x7efc78814d00) 
at /home/jiewan01/tasks/vpp_dpdk_mlx/latest/src/vlib/threads.c:418
#8  0x7efd7709b609 in start_thread (arg=) at 
pthread_create.c:477
#9  0x7efd76dd8163 in clone () at 
../sysdeps/unix/sysv/linux/x86_64/clone.S:95

BG: do you reproduce the issue on both x86 and ARM? Or only on ARM? I’m 
especially thinking about difference in memory coherency that could bite us 
here…
[Honnappa]  On Arm, the barriers for working with MMIO are different than the 
ones used for normal shared memory. Currently, in VPP, this distinction does 
not exist (please correct me if I am wrong).

For ex:

update_nic_descriptors_in_memory();
store_io_barrier(); /*** Currently, this API abstraction does not exist in VPP 
***/
ring_door_bell_on_nic();

So, we need new APIs for the IO barriers and those need to be incorporated in 
the native drivers. For DPDK drivers, this is addressed in DPDK.
I cannot think of any other synchronization/memory model issue.

Issue #2: Huge performance gap between Striding RQ mode and Non-Striding RQ 
mode.
We saw a huge performance gap when using RDMA interfaces created with Striding 
RQ enabled VS RDMA interfaces with Striding RQ disabled.
On both Arm and X86 server mentioned above, thro

Re: [vpp-dev] Issues about RDMA native plugin

[Benoit Ganne (bganne) via lists.fd.io]

>>> Issue #1: VPP crashes when scales to multiple worker cores/threads.
[...]
>>> We guess VPP buffer metadata has been corrupted in the case of multiple
>>> worker threads using VPP release version

>> BG: do you reproduce the issue on both x86 and ARM? Or only on ARM? I’m
>> especially thinking about difference in memory coherency that could bite
>> us here…

> [Honnappa]  On Arm, the barriers for working with MMIO are different than
> the ones used for normal shared memory. Currently, in VPP, this
> distinction does not exist (please correct me if I am wrong).

Yes, I was thinking about something along that line. Currently we use 
CLIB_MEMORY_STORE_BARRIER() defined as __builtin_ia32_sfence() on x86 and 
__sync_synchronize() on ARM.
__sync_synchronize() should be a full memory barrier but is it enough for MMIO 
on ARM?

Best
ben

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22148): https://lists.fd.io/g/vpp-dev/message/22148
Mute This Topic: https://lists.fd.io/mt/94862787/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [vpp-dev] Issues about RDMA native plugin

[Honnappa Nagarahalli]

> 
> >>> Issue #1: VPP crashes when scales to multiple worker cores/threads.
> [...]
> >>> We guess VPP buffer metadata has been corrupted in the case of
> >>> multiple worker threads using VPP release version
> 
> >> BG: do you reproduce the issue on both x86 and ARM? Or only on ARM?
> >> I’m especially thinking about difference in memory coherency that
> >> could bite us here…
> 
> > [Honnappa]  On Arm, the barriers for working with MMIO are different
> > than the ones used for normal shared memory. Currently, in VPP, this
> > distinction does not exist (please correct me if I am wrong).
> 
> Yes, I was thinking about something along that line. Currently we use
> CLIB_MEMORY_STORE_BARRIER() defined as __builtin_ia32_sfence() on x86
> and __sync_synchronize() on ARM.
> __sync_synchronize() should be a full memory barrier but is it enough for
> MMIO on ARM?
Yes, this is the code I am referring to. That is not enough. It generates 'dmb 
ish'.

We need to use 'dmb oshst' for store-store barrier and 'dmb oshld' for 
load-load barrier.

I would suggest defining 3 new APIs:

xxx_io_mb() - Full barrier (on ARM, 'dmb osh')
xxx_io_rmb() - load-load barrier ('dmb oshld')
xxx_io_wmb() - store-store barrier ('dmd oshst')

I do not think there are any compiler intrinsics that generate these 
instructions.

> 
> Best
> ben

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22149): https://lists.fd.io/g/vpp-dev/message/22149
Mute This Topic: https://lists.fd.io/mt/94862787/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-