Hi All,
After I run below two cmds manually, ping traffic recovered ,
but why tc mirred not working after sometime
tc qdisc add dev eth1 handle ffff: ingress
tc filter add dev eth1 parent ffff: u32 match u32 0 0 action mirred egress
redirect dev dtap0
Best Regards,
Kevin
From: [email protected] <[email protected]> On Behalf Of Kevin Yan via
lists.fd.io
Sent: Friday, December 2, 2022 4:39 PM
To: Peter Morrow <[email protected]>; Stephen Hemminger
<[email protected]>
Cc: [email protected]; Long Li <[email protected]>
Subject: COMMERCIAL BULK: Re: [E] COMMERCIAL BULK: Re: [vpp-dev] Issues with
failsafe dpdk pmd in Azure
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the
sender and know the content is safe.
Hi Peter, Stephen and Long,
I am facing some issues when running VPP on Azure VM, can you
please help and give some suggestion if possible.
I'm running CentOS 7.9 with kernel version 3.10.0 on Azure VM,
VPP version is 20.09 and DPDK version is 20.11, below is the snapshot of vpp
startup.conf related to netvsc device
dpdk {
socket-mem 0
no-multi-seg
vdev net_vdev_netvsc0,iface=eth1
vdev net_vdev_netvsc1,iface=eth2
netvsc_dev eth1 {
vpp_interface_name fpeth1
num-rx-queues 1
num-tx-queues 1
num-rx-desc 1024
num-tx-desc 1024
}
netvsc_dev eth2 {
vpp_interface_name fpeth2
num-rx-queues 1
num-tx-queues 1
num-rx-desc 1024
num-tx-desc 1024
}
}
Forget the netvsc_dev section, this is added by me in order to
change the failsafe interface name ,otherwise it will always use the default
name: FailsafeEthernet1, FailsafeEthernet2, etc.
Btw, for my kernel version(3.10.0) , it can only run failsafe
PMD in VPP/DDPK, right?
Basically, we are using two netvsc interfaces and vpp can come
up without any issue, show hard/show log output looks good and is listed as
below:
vpp# sh hard
Name Idx Link Hardware
fpeth1 1 up fpeth1
Link speed: 50 Gbps
Ethernet address 00:0d:3a:57:cc:aa
FailsafeEthernet
carrier up full duplex mtu 1504
flags: admin-up pmd rx-ip4-cksum
Devargs: fd(30),dev(net_tap_vsc0,remote=eth1)
rx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
tx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
max rx packet len: 1522
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: ipv4-cksum udp-cksum tcp-cksum scatter
rx offload active: ipv4-cksum
tx offload avail: ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs
tx offload active: none
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
ipv6-ex ipv6
rss active: none
tx burst function: failsafe_tx_burst_fast
rx burst function: failsafe_rx_burst_fast
tx frames ok 1507
tx bytes ok 95370
rx frames ok 322
rx bytes ok 33127
extended stats:
rx_good_packets 322
tx_good_packets 1507
rx_good_bytes 33127
tx_good_bytes 95370
rx_q0_packets 322
rx_q0_bytes 33127
tx_q0_packets 1507
tx_q0_bytes 95370
tx_sub0_good_packets 1507
tx_sub0_good_bytes 95370
tx_sub0_q0_packets 1507
tx_sub0_q0_bytes 95370
tx_sub0_unicast_packets 322
tx_sub0_unicast_bytes 30910
tx_sub0_multicast_packets 29
tx_sub0_multicast_bytes 3066
tx_sub0_broadcast_packets 1209
tx_sub0_broadcast_bytes 80718
rx_sub1_good_packets 322
rx_sub1_good_bytes 33127
rx_sub1_q0_packets 322
rx_sub1_q0_bytes 33127
fpeth2 2 up fpeth2
Link speed: 50 Gbps
Ethernet address 00:0d:3a:57:cf:f0
FailsafeEthernet
carrier up full duplex mtu 1504
flags: admin-up pmd rx-ip4-cksum
Devargs: fd(45),dev(net_tap_vsc1,remote=eth2)
rx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
tx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
max rx packet len: 1522
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: ipv4-cksum udp-cksum tcp-cksum scatter
rx offload active: ipv4-cksum
tx offload avail: ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs
tx offload active: none
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
ipv6-ex ipv6
rss active: none
tx burst function: failsafe_tx_burst_fast
rx burst function: failsafe_rx_burst_fast
tx frames ok 829
tx bytes ok 49814
rx frames ok 13
rx bytes ok 2860
extended stats:
rx_good_packets 13
tx_good_packets 829
rx_good_bytes 2860
tx_good_bytes 49814
rx_q0_packets 13
rx_q0_bytes 2860
tx_q0_packets 829
tx_q0_bytes 49814
tx_sub0_good_packets 829
tx_sub0_good_bytes 49814
tx_sub0_q0_packets 829
tx_sub0_q0_bytes 49814
tx_sub0_multicast_packets 18
tx_sub0_multicast_bytes 1776
tx_sub0_broadcast_packets 853
tx_sub0_broadcast_bytes 59358
rx_sub1_good_packets 13
rx_sub1_good_bytes 2860
rx_sub1_q0_packets 13
rx_sub1_q0_bytes 2860
local0 0 down local0
Link speed: unknown
local
2022/12/02 08:01:50:364 warn dpdk EAL init args: -c 3 -n 4
--in-memory --vdev net_vdev_netvsc0,iface=eth1 --vdev
net_vdev_netvsc1,iface=eth2 --file-prefix vpp --master
-lcore 0
2022/12/02 08:01:50:778 warn dpdk net_failsafe devargs:
fd(30),dev(net_tap_vsc0,remote=eth1)
2022/12/02 08:01:50:778 warn dpdk net_failsafe iface: eth1, vpp
name: fpeth1
2022/12/02 08:01:50:784 warn dpdk net_failsafe devargs:
fd(45),dev(net_tap_vsc1,remote=eth2)
2022/12/02 08:01:50:784 warn dpdk net_failsafe iface: eth2, vpp
name: fpeth2
2022/12/02 08:01:50:794 notice dpdk EAL: Detected 8 lcore(s)
2022/12/02 08:01:50:794 notice dpdk EAL: Detected 1 NUMA nodes
2022/12/02 08:01:50:794 notice dpdk EAL: Selected IOVA mode 'PA'
2022/12/02 08:01:50:794 notice dpdk EAL: No available hugepages
reported in hugepages-1048576kB
2022/12/02 08:01:50:794 notice dpdk EAL: No free hugepages
reported in hugepages-1048576kB
2022/12/02 08:01:50:794 notice dpdk EAL: No available hugepages
reported in hugepages-1048576kB
2022/12/02 08:01:50:794 notice dpdk EAL: Probing VFIO support...
2022/12/02 08:01:50:794 notice dpdk EAL: WARNING! Base virtual
address hint (0xa80001000 != 0x7fc440000000) not respected!
2022/12/02 08:01:50:794 notice dpdk EAL: This may cause issues
with mapping memory into secondary processes
2022/12/02 08:01:50:794 notice dpdk EAL: WARNING! Base virtual
address hint (0xec0c61000 != 0x7fc03fe00000) not respected!
2022/12/02 08:01:50:794 notice dpdk EAL: This may cause issues
with mapping memory into secondary processes
2022/12/02 08:01:50:794 notice dpdk EAL: WARNING! Base virtual
address hint (0xec18c2000 != 0x7fbc3fc00000) not respected!
2022/12/02 08:01:50:794 notice dpdk EAL: This may cause issues
with mapping memory into secondary processes
2022/12/02 08:01:50:794 notice dpdk EAL: WARNING! Base virtual
address hint (0xec2523000 != 0x7fb83fa00000) not respected!
2022/12/02 08:01:50:794 notice dpdk EAL: This may cause issues
with mapping memory into secondary processes
2022/12/02 08:01:50:794 notice dpdk EAL: Using memfd is not
supported, falling back to anonymous hugepages
2022/12/02 08:01:50:794 notice dpdk EAL: Probe PCI driver:
mlx5_pci (15b3:1016) device: 96ac:00:02.0 (socket 0)
2022/12/02 08:01:50:794 notice dpdk common_mlx5: RTE_MEM is
selected.
2022/12/02 08:01:50:794 notice dpdk mlx5_pci: No available
register for Sampler.
2022/12/02 08:01:50:794 notice dpdk EAL: Probe PCI driver:
mlx5_pci (15b3:1016) device: a18f:00:02.0 (socket 0)
2022/12/02 08:01:50:794 notice dpdk mlx5_pci: No available
register for Sampler.
2022/12/02 08:01:50:794 notice dpdk EAL: VFIO support not
initialized
2022/12/02 08:01:50:794 notice dpdk EAL: Couldn't map new region
for DMA
2022/12/02 08:01:53:862 notice dpdk common_mlx5: Unable to find
virtually contiguous chunk for address (0x1000000000). rte_memseg_contig_walk()
failed.
2022/12/02 08:01:53:862 notice dpdk common_mlx5: Unable to find
virtually contiguous chunk for address (0x1000000000). rte_memseg_contig_walk()
failed.
Ping from VPP to outside working fine after vpp started up and ip configured on
vpp interface, but after sometimes ping stopped working and vpp can not
receive any pkts
I learned that:
Failsafe ends up creating three kernel interfaces per device.
1. Synthetic slow path device (vmbus)
2. SRIOV VF device (Mellanox)
3. TAP device
Slow path packets with failsafe are received on the vmbus device, then a TC
mirred
action runs BPF to do flow management and it goes to TAP device. From the TAP
device
it goes to DPDK, and on to VPP...
On my setup , vmbus interface is eth1, tap device is dtap0, when ping works, I
can capture echo response both on eth1 and dtap0 interface using tcpdump, when
ping not working, echo response is only been captured on eth1 but not on dtap0,
so looks the path between vmbus(eth1) and tap device(dtap0) broken, but from
vpp perspective, the interface status is still okay and no any error logs. I
have no idea how to debug this issue.
appreciate if anybody can give me some suggestions
Best Regards,
Kevin
From: [email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>> On Behalf Of Peter Morrow
Sent: Monday, April 11, 2022 4:49 PM
To: Stephen Hemminger
<[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>; Long Li
<[email protected]<mailto:[email protected]>>
Subject: [E] COMMERCIAL BULK: Re: [vpp-dev] Issues with failsafe dpdk pmd in
Azure
[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the
sender and know the content is safe.
Thanks Stephen,
Very happy to stick with netvsc in that case.
Peter.
________________________________
From: Stephen Hemminger
<[email protected]<mailto:[email protected]>>
Sent: 10 April 2022 18:59
To: Peter Morrow <[email protected]<mailto:[email protected]>>
Cc: [email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>>; Long Li
<[email protected]<mailto:[email protected]>>
Subject: Re: Issues with failsafe dpdk pmd in Azure
On Sat, 9 Apr 2022 09:20:43 +0000
Peter Morrow <[email protected]<mailto:[email protected]>> wrote:
> Hello,
>
> I currently have vpp working nicely in Azure via the netsvc pmd, however
> after reading
> https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk and
> https://fd.io/docs/vpp/v2101/usecases/vppinazure.html it sounds like I should
> be using the failsafe pmd instead. So, I gave this a try but ran into some
> issues, some of which I've seen discussed on this email group but none of the
> solutions have worked for me thus far. I was able to make the failsafe pmd
> work via dpdk-testpmd with dpdk standalone from debian bullseye (dpdk 20.11).
You have it backwards. Failsafe is the older driver which was developed to be
generic.
Faisafe is slower because it has to go through the kernel for the slow path.
Would like to deprecate use of failsafe but there are some use cases such as
rte_flow which
netvsc PMD does not support. Supporting rte_flow in a software driver would
require a significant
amount of work (same problem as virtio).
>
> I'm running vpp 22.06 and an external dpdk at version 21.11, though also see
> the same thing when downgrading to 20.11. We have 3 interfaces, the first is
> a non accelerated networking interface which is not to be controlled by vpp
> (eth0) the 2nd are data interfaces which are vpp owned. My dpdk section of
> the vpp startup config looks like this:
>
> dpdk {
>
> dev 0ed6:00:02.0
> vdev net_vdev_netvsc0,iface=eth1
>
> dev 6fa1:00:02.0
> vdev net_vdev_netvsc1,iface=eth2
>
> base-virtaddr 0x200000000
> }
>
> When vpp starts up the 2 interfaces are shown:
>
> me@azure:~$ sudo vppctl sh hard
> Name Idx Link Hardware
> FailsafeEthernet2 1 up FailsafeEthernet2
> Link speed: 50 Gbps
> RX Queues:
> queue thread mode
> 0 vpp_wk_0 (1) polling
> Ethernet address 00:22:48:4c:c0:e5
> FailsafeEthernet
> carrier up full duplex max-frame-size 1518
> flags: maybe-multiseg tx-offload rx-ip4-cksum
> Devargs: fd(17),dev(net_tap_vsc0,remote=eth1)
> rx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
> tx: queues 8 (max 16), desc 1024 (min 0 max 65535 align 1)
> max rx packet len: 1522
> promiscuous: unicast off all-multicast off
> vlan offload: strip off filter off qinq off
> rx offload avail: ipv4-cksum udp-cksum tcp-cksum scatter
> rx offload active: ipv4-cksum scatter
> tx offload avail: ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs
> tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
> rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
> ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
> ipv6-ex ipv6
> rss active: none
> tx burst function: (not available)
> rx burst function: (not available)
>
> FailsafeEthernet4 2 up FailsafeEthernet4
> Link speed: 50 Gbps
> RX Queues:
> queue thread mode
> 0 vpp_wk_1 (2) polling
> Ethernet address 00:22:48:4c:c6:4a
> FailsafeEthernet
> carrier up full duplex max-frame-size 1518
> flags: maybe-multiseg tx-offload rx-ip4-cksum
> Devargs: fd(33),dev(net_tap_vsc1,remote=eth2)
> rx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
> tx: queues 8 (max 16), desc 1024 (min 0 max 65535 align 1)
> max rx packet len: 1522
> promiscuous: unicast off all-multicast off
> vlan offload: strip off filter off qinq off
> rx offload avail: ipv4-cksum udp-cksum tcp-cksum scatter
> rx offload active: ipv4-cksum scatter
> tx offload avail: ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs
> tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
> rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
> ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
> ipv6-ex ipv6
> rss active: none
> tx burst function: (not available)
> rx burst function: (not available)
>
> local0 0 down local0
> Link speed: unknown
> local
> me@azure:~$
>
> However on enabling our application we see the following messages in the the
> vpp log & the vpp interfaces are unusable:
>
> 2022/04/09 09:11:16:424 notice dpdk net_failsafe: Link update
> failed for sub_device 0 with error 1
> 2022/04/09 09:11:16:424 notice dpdk net_failsafe: Link update
> failed for sub_device 0 with error 1
> 2022/04/09 09:12:32:380 notice dpdk common_mlx5: Unable to find
> virtually contiguous chunk for address (0x1000000000).
> rte_memseg_contig_walk() failed.
>
> 2022/04/09 09:12:36:144 notice ip6/link enable: FailsafeEthernet2
> 2022/04/09 09:12:36:144 error interface hw_add_del_mac_address:
> dpdk_add_del_mac_address: mac address add/del failed: -28
> 2022/04/09 09:12:36:145 error interface hw_add_del_mac_address:
> dpdk_add_del_mac_address: mac address add/del failed: -28
> 2022/04/09 09:12:36:145 error interface hw_add_del_mac_address:
> dpdk_add_del_mac_address: mac address add/del failed: -28
> 2022/04/09 09:12:36:145 error interface hw_add_del_mac_address:
> dpdk_add_del_mac_address: mac address add/del failed: -28
> 2022/04/09 09:12:36:146 notice dpdk Port 2: MAC address array
> full
> 2022/04/09 09:12:36:146 notice dpdk Port 2: MAC address array
> full
> 2022/04/09 09:12:36:146 notice dpdk Port 2: MAC address array
> full
> 2022/04/09 09:12:36:146 notice dpdk Port 2: MAC address array
> full
>
> Earlier snippets from dmesg:
>
> [ 6.189856] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [ 8.644980] hv_vmbus: registering driver uio_hv_generic
> [ 12.586115] mlx5_core 0ed6:00:02.0 enP3798s2: Link up
> [ 12.586914] hv_netvsc 0022484c-c0e5-0022-484c-c0e50022484c eth1: Data path
> switched to VF: enP3798s2
> [ 12.846211] mlx5_core 6fa1:00:02.0 enP28577s3: Link up
> [ 12.847014] hv_netvsc 0022484c-c64a-0022-484c-c64a0022484c eth2: Data path
> switched to VF: enP28577s3
> [ 13.032549] tun: Universal TUN/TAP device driver, 1.6
> [ 13.149802] Mirror/redirect action on
> [ 13.414199] hv_netvsc 0022484c-c0e5-0022-484c-c0e50022484c eth1: VF slot 2
> added
> [ 13.549360] hv_netvsc 0022484c-c0e5-0022-484c-c0e50022484c eth1: VF slot 2
> added
> [ 13.716361] hv_netvsc 0022484c-c64a-0022-484c-c64a0022484c eth2: VF slot 3
> added
> [ 13.848514] hv_netvsc 0022484c-c64a-0022-484c-c64a0022484c eth2: VF slot 3
> added
> [ 15.123859] tc mirred to Houston: device dtap1 is down
> [ 16.986207] tc mirred to Houston: device dtap0 is down
> [ 20.129536] tc mirred to Houston: device dtap1 is down
> [ 21.996293] tc mirred to Houston: device dtap0 is down
> [ 25.129538] tc mirred to Houston: device dtap1 is down
> [ 27.003053] tc mirred to Houston: device dtap0 is down
> [ 30.143527] tc mirred to Houston: device dtap1 is down
> [ 32.006497] tc mirred to Houston: device dtap0 is down
> [ 35.149622] tc mirred to Houston: device dtap1 is down
> [ 37.014221] tc mirred to Houston: device dtap0 is down
> [ 40.158724] tc mirred to Houston: device dtap1 is down
> [ 42.024193] tc mirred to Houston: device dtap0 is down
> [ 45.166728] tc mirred to Houston: device dtap1 is down
> [ 45.747723] tc mirred to Houston: device dtap0 is down
> [ 47.030589] tc mirred to Houston: device dtap0 is down
> [ 50.172306] tc mirred to Houston: device dtap1 is down
> [ 52.039827] tc mirred to Houston: device dtap0 is down
> [ 52.137147] hv_balloon: Max. dynamic memory size: 32768 MB
>
> The dtap0/1 is down messages are continuing.
>
> Can anyone suggest a problem with the configuration I've tried in Azure?
Failsafe ends up creating three kernel interfaces per device.
1. Synthetic slow path device (vmbus)
2. SRIOV VF device (Mellanox)
3. TAP device
Slow path packets with failsafe are received on the vmbus device, then a TC
mirred
action runs BPF to do flow management and it goes to TAP device. From the TAP
device
it goes to DPDK, and on to VPP...
Looks like a bug in how tap devices are setup by either VPP or hv_netvsc (if
using
that dummy setup device).
Use netvsc PMD if possible.
Long Li is the person doing most of the work on DPDK for Azure now.
________________________________
This e-mail message may contain confidential or proprietary information of
Mavenir Systems, Inc. or its affiliates and is intended solely for the use of
the intended recipient(s). If you are not the intended recipient of this
message, you are hereby notified that any review, use or distribution of this
information is absolutely prohibited and we request that you delete all copies
in your control and contact us by e-mailing to
[email protected]<mailto:[email protected]>. This message contains the
views of its author and may not necessarily reflect the views of Mavenir
Systems, Inc. or its affiliates, who employ systems to monitor email messages,
but make no representation that such messages are authorized, secure,
uncompromised, or free from computer viruses, malware, or other defects. Thank
You
________________________________
This e-mail message may contain confidential or proprietary information of
Mavenir Systems, Inc. or its affiliates and is intended solely for the use of
the intended recipient(s). If you are not the intended recipient of this
message, you are hereby notified that any review, use or distribution of this
information is absolutely prohibited and we request that you delete all copies
in your control and contact us by e-mailing to [email protected]. This
message contains the views of its author and may not necessarily reflect the
views of Mavenir Systems, Inc. or its affiliates, who employ systems to monitor
email messages, but make no representation that such messages are authorized,
secure, uncompromised, or free from computer viruses, malware, or other
defects. Thank You
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22269): https://lists.fd.io/g/vpp-dev/message/22269
Mute This Topic: https://lists.fd.io/mt/95404520/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
