[vpp-dev] Issues with failsafe dpdk pmd in Azure
Hello,
I currently have vpp working nicely in Azure via the netsvc pmd, however after
reading https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk and
https://fd.io/docs/vpp/v2101/usecases/vppinazure.html it sounds like I should
be using the failsafe pmd instead. So, I gave this a try but ran into some
issues, some of which I've seen discussed on this email group but none of the
solutions have worked for me thus far. I was able to make the failsafe pmd work
via dpdk-testpmd with dpdk standalone from debian bullseye (dpdk 20.11).
I'm running vpp 22.06 and an external dpdk at version 21.11, though also see
the same thing when downgrading to 20.11. We have 3 interfaces, the first is a
non accelerated networking interface which is not to be controlled by vpp
(eth0) the 2nd are data interfaces which are vpp owned. My dpdk section of the
vpp startup config looks like this:
dpdk {
dev 0ed6:00:02.0
vdev net_vdev_netvsc0,iface=eth1
dev 6fa1:00:02.0
vdev net_vdev_netvsc1,iface=eth2
base-virtaddr 0x2
}
When vpp starts up the 2 interfaces are shown:
me@azure:~$ sudo vppctl sh hard
NameIdx Link Hardware
FailsafeEthernet2 1 up FailsafeEthernet2
Link speed: 50 Gbps
RX Queues:
queue thread mode
0 vpp_wk_0 (1) polling
Ethernet address 00:22:48:4c:c0:e5
FailsafeEthernet
carrier up full duplex max-frame-size 1518
flags: maybe-multiseg tx-offload rx-ip4-cksum
Devargs: fd(17),dev(net_tap_vsc0,remote=eth1)
rx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
tx: queues 8 (max 16), desc 1024 (min 0 max 65535 align 1)
max rx packet len: 1522
promiscuous: unicast off all-multicast off
vlan offload: strip off filter off qinq off
rx offload avail: ipv4-cksum udp-cksum tcp-cksum scatter
rx offload active: ipv4-cksum scatter
tx offload avail: ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
ipv6-ex ipv6
rss active:none
tx burst function: (not available)
rx burst function: (not available)
FailsafeEthernet4 2 up FailsafeEthernet4
Link speed: 50 Gbps
RX Queues:
queue thread mode
0 vpp_wk_1 (2) polling
Ethernet address 00:22:48:4c:c6:4a
FailsafeEthernet
carrier up full duplex max-frame-size 1518
flags: maybe-multiseg tx-offload rx-ip4-cksum
Devargs: fd(33),dev(net_tap_vsc1,remote=eth2)
rx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
tx: queues 8 (max 16), desc 1024 (min 0 max 65535 align 1)
max rx packet len: 1522
promiscuous: unicast off all-multicast off
vlan offload: strip off filter off qinq off
rx offload avail: ipv4-cksum udp-cksum tcp-cksum scatter
rx offload active: ipv4-cksum scatter
tx offload avail: ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
ipv6-ex ipv6
rss active:none
tx burst function: (not available)
rx burst function: (not available)
local0 0down local0
Link speed: unknown
local
me@azure:~$
However on enabling our application we see the following messages in the the
vpp log & the vpp interfaces are unusable:
2022/04/09 09:11:16:424 notice dpdk net_failsafe: Link update
failed for sub_device 0 with error 1
2022/04/09 09:11:16:424 notice dpdk net_failsafe: Link update
failed for sub_device 0 with error 1
2022/04/09 09:12:32:380 notice dpdk common_mlx5: Unable to find
virtually contiguous chunk for address (0x10). rte_memseg_contig_walk()
failed.
2022/04/09 09:12:36:144 notice ip6/link enable: FailsafeEthernet2
2022/04/09 09:12:36:144 error interface hw_add_del_mac_address:
dpdk_add_del_mac_address: mac address add/del failed: -28
2022/04/09 09:12:36:145 error interface hw_add_del_mac_address:
dpdk_add_del_mac_address: mac address add/del failed: -28
2022/04/09 09:12:36:145 error interface hw_add_del_mac_address:
dpdk_add_del_mac_address: mac address add/del failed: -28
2022/04/09 09:12:36:145 error interface hw_add_del_mac_address:
dpdk_add_del_mac_address: mac address add/del failed: -28
2022/04/09 09:12:36:146 notice dpdk Port 2: MAC address array full
2022/04/09 09:12:36:146 notice dpdk Port 2: MAC address array full
2022/04/09 09:12:36:146 notice dpdk Port 2: MAC address array full
2022/04/09 09:12:36:146 notic
Re: [vpp-dev] Issues with failsafe dpdk pmd in Azure
Thanks Stephen,
Very happy to stick with netvsc in that case.
Peter.
From: Stephen Hemminger
Sent: 10 April 2022 18:59
To: Peter Morrow
Cc: vpp-dev@lists.fd.io ; Long Li
Subject: Re: Issues with failsafe dpdk pmd in Azure
On Sat, 9 Apr 2022 09:20:43 +
Peter Morrow wrote:
> Hello,
>
> I currently have vpp working nicely in Azure via the netsvc pmd, however
> after reading
> https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk and
> https://fd.io/docs/vpp/v2101/usecases/vppinazure.html it sounds like I should
> be using the failsafe pmd instead. So, I gave this a try but ran into some
> issues, some of which I've seen discussed on this email group but none of the
> solutions have worked for me thus far. I was able to make the failsafe pmd
> work via dpdk-testpmd with dpdk standalone from debian bullseye (dpdk 20.11).
You have it backwards. Failsafe is the older driver which was developed to be
generic.
Faisafe is slower because it has to go through the kernel for the slow path.
Would like to deprecate use of failsafe but there are some use cases such as
rte_flow which
netvsc PMD does not support. Supporting rte_flow in a software driver would
require a significant
amount of work (same problem as virtio).
>
> I'm running vpp 22.06 and an external dpdk at version 21.11, though also see
> the same thing when downgrading to 20.11. We have 3 interfaces, the first is
> a non accelerated networking interface which is not to be controlled by vpp
> (eth0) the 2nd are data interfaces which are vpp owned. My dpdk section of
> the vpp startup config looks like this:
>
> dpdk {
>
> dev 0ed6:00:02.0
> vdev net_vdev_netvsc0,iface=eth1
>
> dev 6fa1:00:02.0
> vdev net_vdev_netvsc1,iface=eth2
>
> base-virtaddr 0x2
> }
>
> When vpp starts up the 2 interfaces are shown:
>
> me@azure:~$ sudo vppctl sh hard
> NameIdx Link Hardware
> FailsafeEthernet2 1 up FailsafeEthernet2
> Link speed: 50 Gbps
> RX Queues:
> queue thread mode
> 0 vpp_wk_0 (1) polling
> Ethernet address 00:22:48:4c:c0:e5
> FailsafeEthernet
> carrier up full duplex max-frame-size 1518
> flags: maybe-multiseg tx-offload rx-ip4-cksum
> Devargs: fd(17),dev(net_tap_vsc0,remote=eth1)
> rx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
> tx: queues 8 (max 16), desc 1024 (min 0 max 65535 align 1)
> max rx packet len: 1522
> promiscuous: unicast off all-multicast off
> vlan offload: strip off filter off qinq off
> rx offload avail: ipv4-cksum udp-cksum tcp-cksum scatter
> rx offload active: ipv4-cksum scatter
> tx offload avail: ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs
> tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
> rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
>ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
>ipv6-ex ipv6
> rss active:none
> tx burst function: (not available)
> rx burst function: (not available)
>
> FailsafeEthernet4 2 up FailsafeEthernet4
> Link speed: 50 Gbps
> RX Queues:
> queue thread mode
> 0 vpp_wk_1 (2) polling
> Ethernet address 00:22:48:4c:c6:4a
> FailsafeEthernet
> carrier up full duplex max-frame-size 1518
> flags: maybe-multiseg tx-offload rx-ip4-cksum
> Devargs: fd(33),dev(net_tap_vsc1,remote=eth2)
> rx: queues 1 (max 16), desc 1024 (min 0 max 65535 align 1)
> tx: queues 8 (max 16), desc 1024 (min 0 max 65535 align 1)
> max rx packet len: 1522
> promiscuous: unicast off all-multicast off
> vlan offload: strip off filter off qinq off
> rx offload avail: ipv4-cksum udp-cksum tcp-cksum scatter
> rx offload active: ipv4-cksum scatter
> tx offload avail: ipv4-cksum udp-cksum tcp-cksum tcp-tso multi-segs
> tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
> rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
>ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
>ipv6-ex ipv6
> rss active:none
> tx burst function: (not available)
> rx burst function: (not available)
>
> local0 0down local0
> Link speed: unknown
> local
> me@azure:~$
>
> However on enabling our application we see the following messages in the the
> vpp log & the vpp interfaces are unusable:
>
> 2022/04/09 09:11:16:424 notice dpdk net_failsafe: Link update
Re: [vpp-dev] Issues with failsafe dpdk pmd in Azure
Thanks Long,
We are good now that the guidance is to stick with the netvsc pmd. Replying
here so others might see this message on the vpp mailing list.
Peter.
From: Long Li
Sent: 11 April 2022 22:39
To: Peter Morrow ; Stephen Hemminger
Cc: vpp-dev@lists.fd.io
Subject: RE: Issues with failsafe dpdk pmd in Azure
Hi Peter,
The failure you see on Failsafe/MLX5:
> 2022/04/09 09:11:16:424 notice dpdk net_failsafe: Link update
> failed for sub_device 0 with error 1
> 2022/04/09 09:12:32:380 notice dpdk common_mlx5: Unable to find
> virtually contiguous chunk for address (0x10).
> rte_memseg_contig_walk() failed.
This is a known issue with MLX5 registering a buffer not allocated through DPDK
API. This can happen if other driver or other code pass a buffer to MLX5 but
MLX5 can’t register it with the hardware.
The error detail is at:
https://github.com/DPDK/dpdk/blob/7cf3d07c3adcb015c303e4cdf2ef9712a65ce46d/drivers/common/mlx5/mlx5_common_mr.c#L626<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FDPDK%2Fdpdk%2Fblob%2F7cf3d07c3adcb015c303e4cdf2ef9712a65ce46d%2Fdrivers%2Fcommon%2Fmlx5%2Fmlx5_common_mr.c%23L626&data=04%7C01%7Clongli%40microsoft.com%7C35fdd512f3284030374908d876c4b96e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637389934630918596%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FztzwFazZZ9Nk1KEc7S9HFFioQe2r5hTYBI5y6lWYQo%3D&reserved=0>
If this is troubling on your setup, can you set “mr_ext_memseg_en=0” for mlx5?
This should get rid of this error.
Long
From: Peter Morrow
Sent: Monday, April 11, 2022 1:49 AM
To: Stephen Hemminger
Cc: vpp-dev@lists.fd.io; Long Li
Subject: Re: Issues with failsafe dpdk pmd in Azure
You don't often get email from pe...@graphiant.com<mailto:pe...@graphiant.com>.
Learn why this is important<http://aka.ms/LearnAboutSenderIdentification>
Thanks Stephen,
Very happy to stick with netvsc in that case.
Peter.
From: Stephen Hemminger
mailto:step...@networkplumber.org>>
Sent: 10 April 2022 18:59
To: Peter Morrow mailto:pe...@graphiant.com>>
Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>
mailto:vpp-dev@lists.fd.io>>; Long Li
mailto:lon...@microsoft.com>>
Subject: Re: Issues with failsafe dpdk pmd in Azure
On Sat, 9 Apr 2022 09:20:43 +
Peter Morrow mailto:pe...@graphiant.com>> wrote:
> Hello,
>
> I currently have vpp working nicely in Azure via the netsvc pmd, however
> after reading
> https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fvirtual-network%2Fsetup-dpdk&data=05%7C01%7Clongli%40microsoft.com%7Cd7df8d748c7a4f209f0208da1b982cd7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637852638113192348%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FFePUbwfVxYZwbDhVa19I97v5m9z7JjhQ76ZylDhm%2FU%3D&reserved=0>
> and
> https://fd.io/docs/vpp/v2101/usecases/vppinazure.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffd.io%2Fdocs%2Fvpp%2Fv2101%2Fusecases%2Fvppinazure.html&data=05%7C01%7Clongli%40microsoft.com%7Cd7df8d748c7a4f209f0208da1b982cd7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637852638113192348%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KmU7iQ9YyOfIiOENM5oqUdr%2BwOzxaHffJ0H9qLjxmsQ%3D&reserved=0>
> it sounds like I should be using the failsafe pmd instead. So, I gave this a
> try but ran into some issues, some of which I've seen discussed on this email
> group but none of the solutions have worked for me thus far. I was able to
> make the failsafe pmd work via dpdk-testpmd with dpdk standalone from debian
> bullseye (dpdk 20.11).
You have it backwards. Failsafe is the older driver which was developed to be
generic.
Faisafe is slower because it has to go through the kernel for the slow path.
Would like to deprecate use of failsafe but there are some use cases such as
rte_flow which
netvsc PMD does not support. Supporting rte_flow in a software driver would
require a significant
amount of work (same problem as virtio).
>
> I'm running vpp 22.06 and an external dpdk at version 21.11, though also see
> the same thing when downgrading to 20.11. We have 3 interfaces, the first is
> a non accelerated networking interface which is not to be controlled by vpp
> (eth0) the 2nd are data interfaces which are vpp owned. My dpdk section of
> the vpp startup config looks like this:
>
> dpdk {
>
> dev 0ed6:00:02.0
> vdev net_vdev_netvsc
[vpp-dev] optimal buffer configuration on Azure
Hello,
In this example i've got a 4 vCPU Azure VM with 16G of RAM, 2G of that is given
to 1024 2MB huge pages:
$ cat /proc/meminfo | grep -i huge
AnonHugePages: 71680 kB
ShmemHugePages:0 kB
FileHugePages: 0 kB
HugePages_Total:1024
HugePages_Free:1
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize: 2048 kB
Hugetlb: 2097152 kB
$
There are 2 interfaces which are vpp owned and which are both using the netvsc
pmd:
$ sudo vppctl sh hard
NameIdx Link Hardware
GigabitEthernet1 1 up GigabitEthernet1
Link speed: 50 Gbps
RX Queues:
queue thread mode
0 vpp_wk_0 (1) polling
1 vpp_wk_1 (2) polling
Ethernet address 60:45:bd:85:22:97
Microsoft Hyper-V Netvsc
carrier up full duplex max-frame-size 0
flags: tx-offload rx-ip4-cksum
Devargs:
rx: queues 2 (max 64), desc 1024 (min 0 max 65535 align 1)
tx: queues 2 (max 64), desc 1024 (min 1 max 4096 align 1)
max rx packet len: 65536
promiscuous: unicast off all-multicast off
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum rss-hash
rx offload active: ipv4-cksum
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
multi-segs
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp ipv6
rss active:ipv4-tcp ipv4 ipv6-tcp ipv6
tx burst function: (not available)
rx burst function: (not available)
GigabitEthernet2 2 up GigabitEthernet2
Link speed: 50 Gbps
RX Queues:
queue thread mode
0 vpp_wk_2 (3) polling
1 vpp_wk_0 (1) polling
Ethernet address 60:45:bd:85:23:94
Microsoft Hyper-V Netvsc
carrier up full duplex max-frame-size 0
flags: tx-offload rx-ip4-cksum
Devargs:
rx: queues 2 (max 64), desc 1024 (min 0 max 65535 align 1)
tx: queues 2 (max 64), desc 1024 (min 1 max 4096 align 1)
max rx packet len: 65536
promiscuous: unicast off all-multicast off
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum rss-hash
rx offload active: ipv4-cksum
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
multi-segs
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp ipv6
rss active:ipv4-tcp ipv4 ipv6-tcp ipv6
tx burst function: (not available)
rx burst function: (not available)
local0 0down local0
Link speed: unknown
local
$
Config file looks like this:
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
gid vpp
}
api-trace {
on
}
api-segment {
gid vpp
}
socksvr {
socket-name /run/vpp/api.sock
}
plugins {
# Common plugins.
plugin default { disable }
plugin dpdk_plugin.so { enable }
plugin linux_cp_plugin.so { enable }
plugin crypto_native_plugin.so { enable }
< -- snip lots of plugins -- >
}
dpdk {
# VMBUS UUID.
dev 6045bd85-2297-6045-bd85-22976045bd85 {
num-rx-queues 4
num-tx-queues 4
name GigabitEthernet1
}
# VMBUS UUID.
dev 6045bd85-2394-6045-bd85-23946045bd85 {
num-rx-queues 4
num-tx-queues 4
name GigabitEthernet2
}
}
cpu {
skip-cores 0
main-core 0
corelist-workers 1-3
}
buffers {
# Max buffers based on data size & huge page configuration.
buffers-per-numa 853440
default data-size 2048
page-size default-hugepage
}
statseg {
size 128M
}
My issue is that I start to see errors from the mlnx5 driver when using a large
number of buffers:
2022/06/29 12:44:11:427 notice dpdk common_mlx5: Unable to find
virtually contiguous chunk for address (0x10). rte_memseg_contig_walk()
failed.
2022/06/29 12:44:11:427 notice dpdk common_mlx5: Unable to find
virtually contiguous chunk for address (0x103fe0). rte_memseg_contig_walk()
failed.
2022/06/29 12:44:11:427 notice dpdk common_mlx5: Unable to find
virtually contiguous chunk for address (0x104000). rte_memseg_contig_walk()
failed.
2022/06/29 12:44:11:427 notice dpdk common_mlx5: Unable to find
virtually contiguous chunk for address (0x104020). rte_memseg_contig_walk()
failed.
The spew continues.
With a smaller number of buffers I don't see this problem and there are no
issues with the packet forwarding side of things. I'm not sure what the buffer
limit is before things
go bad.
I read the excellent description of how buffer sizes are calculated here:
https://lists.fd.io/g/vpp-dev/topic/buffer_occupancy_calculation/76605334?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,20,7660533
Re: [vpp-dev] VPP on Azure
Hi Siddarth,
The linked document is quite old and I don't think it can be relied on in it's
current state. I am using vpp (22.06-20220307) in Azure with external dpdk from
debian 11 (20.11) and it's working well except for an issue with larger buffer
sizes.
What sort of errors are you seeing in the vpp log? Are you sure that the
interfaces that you want vpp to control are DOWN when vpp starts? Depending on
how you've deployed the VM dhcp may be being run for the AN interfaces which
you want to be vpp owned, which would block them from being used by vpp.
Another point is that I've had more success binding devices using their VMBUS
id instead of the PCI ID, for example:
dpdk {
# VMBUS UUID.
dev 6045bd81-248e-6045-bd81-248e6045bd81 {
num-rx-queues 4
num-tx-queues 4
name GigabitEthernet1
}
# VMBUS UUID.
dev 6045bd81-2600-6045-bd81-26006045bd81 {
num-rx-queues 4
num-tx-queues 4
name GigabitEthernet2
}
}
VPP is up and running:
$ sudo vppctl sh hard
NameIdx Link Hardware
GigabitEthernet1 1 up GigabitEthernet1
Link speed: 50 Gbps
RX Queues:
queue thread mode
0 main (0) polling
1 main (0) polling
2 main (0) polling
3 main (0) polling
Ethernet address 60:45:bd:81:24:8e
Microsoft Hyper-V Netvsc
carrier up full duplex max-frame-size 0
flags: admin-up tx-offload rx-ip4-cksum
Devargs:
rx: queues 4 (max 64), desc 1024 (min 0 max 65535 align 1)
tx: queues 4 (max 64), desc 1024 (min 1 max 4096 align 1)
max rx packet len: 65536
promiscuous: unicast off all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum rss-hash
rx offload active: ipv4-cksum
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
multi-segs
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp ipv6
rss active:ipv4-tcp ipv4 ipv6-tcp ipv6
tx burst function: (not available)
rx burst function: (not available)
tx frames ok 263350
tx bytes ok 41871764
rx frames ok 253901
rx bytes ok 33154932
extended stats:
rx_good_packets 253901
tx_good_packets 263350
rx_good_bytes 33154932
tx_good_bytes 41871764
rx_q0_packets 4849
rx_q0_bytes 298233
rx_q1_packets 1345
rx_q1_bytes 147229
rx_q2_packets32967
rx_q2_bytes3625057
rx_q3_packets 1330
rx_q3_bytes 145796
rx_q0_good_packets4849
rx_q0_good_bytes298233
rx_q0_undersize_packets 3462
rx_q0_size_65_127_packets 1066
rx_q0_size_128_255_packets 318
rx_q0_size_256_511_packets 3
rx_q1_good_packets1345
rx_q1_good_bytes147229
rx_q1_undersize_packets 2
rx_q1_size_65_127_packets 1035
rx_q1_size_128_255_packets 307
rx_q1_size_1024_1518_packets 1
rx_q2_good_packets 32967
rx_q2_good_bytes 3625057
rx_q2_multicast_packets 31662
rx_q2_undersize_packets 1
rx_q2_size_65_127_packets32663
rx_q2_size_128_255_packets 303
rx_q3_good_packets1330
rx_q3_good_bytes145796
rx_q3_size_65_127_packets 1016
rx_q3_size_128_255_packets 313
rx_q3_size_256_511_packets 1
vf_rx_good_packets 213410
vf_tx_good_packets 263350
vf_rx_good_bytes 28938617
vf_tx_good_bytes 41871764
vf_rx_q0_packets16
vf_rx_q0_bytes
