> On Dec 7, 2020, at 5:16 PM, Damjan Marion <damjan.mar...@gmail.com> wrote: > >> On 07.12.2020., at 22:55, Christian Hopps <cho...@chopps.org> wrote: > > I just bumped it to 2.13.10 and added REMAKE_INITRD=“yes” into debian/dkms.in > so that should be fixed: > > $ lsinitramfs /boot/initrd.img | grep i40e > usr/lib/modules/5.8.0-31-generic/updates/dkms/i40e.ko
Updated, and works.
>
>>
>>> also, I asked for your crete interface config….
>>
>> create interface avf 0000:65:0a.0
I switched this to also include "rx-queue-size 2048" to see if it helped with
my other problem, but it didn't.
>>
>> As mentioned above, when I rmmod/modprobe the new driver I don't hit the lut
>> error anymore.
>>
>> Now I need to figure out why I see such bad performance (tons of rx
>> discards) using these interfaces (when using 2 of them), but not when using
>> 1 avf VF and the other interface is a 10G i520 nic (dpdk driver).
>>
>> Bad Side:
>>
>> $ docker-compose exec p1 vppctl show hard
>> Name Idx Link Hardware
>> avf-0/65/2/0 2 up avf-0/65/2/0
>> Link speed: 10 Gbps
>> Ethernet address 02:41:0d:0d:0d:0b
>> flags: initialized admin-up vaddr-dma link-up rx-interrupts
>> offload features: l2 adv-link-speed vlan rx-polling rss-pf
>> num-queue-pairs 6 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
>> speed
>> stats:
>> rx bytes 1520728906
>> rx unicast 184910147
>> rx discards 129083610
>> tx bytes 4226915088
>> tx unicast 184349288
>> avf-0/65/a/0 1 up avf-0/65/a/0
>> Link speed: 10 Gbps
>> Ethernet address 02:41:0b:0b:0b:0b
>> flags: initialized admin-up vaddr-dma link-up rx-interrupts
>> offload features: l2 adv-link-speed vlan rx-polling rss-pf
>> num-queue-pairs 6 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
>> speed
>> stats:
>> rx bytes 3781507000
>> rx unicast 93533516
>> rx broadcast 3
>> rx discards 73223358
>> tx bytes 2212324998
>> tx unicast 13714424
>> ipsec0 3 up ipsec0
>> Link speed: unknown
>> IPSec
>> local0 0 down local0
>> Link speed: unknown
>> local
>> loop0 4 up loop0
>> Link speed: unknown
>> Ethernet address de:ad:00:00:00:00
>>
>> Good Side:
>>
>> $ docker-compose exec p2 vppctl show hard
>> Name Idx Link Hardware
>> TenGigabitEthernet17/0/0 1 up TenGigabitEthernet17/0/0
>> Link speed: 10 Gbps
>> Ethernet address f8:f2:1e:3c:15:ec
>> Intel 82599
>> carrier up full duplex mtu 9206
>> flags: admin-up pmd maybe-multiseg tx-offload intel-phdr-cksum
>> rx-ip4-cksum
>> Devargs:
>> rx: queues 1 (max 128), desc 1024 (min 32 max 4096 align 8)
>> tx: queues 6 (max 64), desc 1024 (min 32 max 4096 align 8)
>> pci: device 8086:10fb subsystem 8086:0003 address 0000:17:00.00 numa 0
>> max rx packet len: 15872
>> promiscuous: unicast off all-multicast on
>> vlan offload: strip off filter off qinq off
>> rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum tcp-lro
>> macsec-strip vlan-filter vlan-extend jumbo-frame
>> scatter
>> security keep-crc rss-hash
>> rx offload active: ipv4-cksum jumbo-frame scatter
>> tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum sctp-cksum
>> tcp-tso macsec-insert multi-segs security
>> tx offload active: udp-cksum tcp-cksum multi-segs
>> rss avail: ipv4-tcp ipv4-udp ipv4 ipv6-tcp-ex ipv6-udp-ex ipv6-tcp
>> ipv6-udp ipv6-ex ipv6
>> rss active: none
>> tx burst function: ixgbe_xmit_pkts
>> rx burst function: ixgbe_recv_scattered_pkts_vec
>>
>> tx frames ok 183386223
>> tx bytes ok 277646741622
>> rx frames ok 182834391
>> rx bytes ok 276811244995
>> extended stats:
>> rx_good_packets 182834488
>> tx_good_packets 183386223
>> rx_good_bytes 276811391853
>> tx_good_bytes 277646741622
>> rx_q0packets 182834488
>> rx_q0bytes 276811391853
>> tx_q0packets 183386223
>> tx_q0bytes 277646741622
>> rx_size_65_to_127_packets 15
>> rx_size_1024_to_max_packets 182834425
>> rx_multicast_packets 15
>> rx_total_packets 182834439
>> rx_total_bytes 276811319181
>> tx_total_packets 183386223
>> tx_size_1024_to_max_packets 183386223
>> out_pkts_untagged 183386223
>> avf-0/65/e/0 2 up avf-0/65/e/0
>> Link speed: 10 Gbps
>> Ethernet address 02:42:0c:0c:0c:0c
>> flags: initialized admin-up vaddr-dma link-up rx-interrupts
>> offload features: l2 adv-link-speed vlan rx-polling rss-pf
>> num-queue-pairs 6 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
>> speed
>> stats:
>> rx bytes 2520527812
>> rx unicast 92644250
>> rx broadcast 3
>> tx bytes 2677106258
>> tx unicast 20118042
>> ipsec0 3 up ipsec0
>> Link speed: unknown
>> IPSec
>> local0 0 down local0
>> Link speed: unknown
>> local
>> loop0 4 up loop0
>> Link speed: unknown
>> Ethernet address de:ad:00:00:00:00
>>
>
> Weird, no idea based on this outputs… What show run says?
I've slowed the t-rex input to 1% of the tunnel rate (5G) and still saw drops.
I raise it back to 80% of a 7G tunnel to try and get more realistic numbers on
the input routine (otherwise they were showing 0.00 vectors per call)
The input on the second interface from t-rex is the same, the input on the
tunnel interface (the first interface) is the same rate. Notice the difference
on the processing times though on the bad system (the one with 2 avf receive
interfaces) vs the good system (a single avf and then a dpdk 10G intel nic).
Is there some sort of locking issue with the AVF? The bad system avf-input is
running 3 times slower than on the good system.
Bad system:
Thread 1 vpp_wk_0 (lcore 1)
Time 5.1, 10 sec internal node vector rate 14.59 loops/sec 3665191.01
vector rates in 3.0972e5, out 1.6779e5, drop 2.5475e4, punt 0.0000e0
Name State Calls Vectors
Suspends Packet-Clocks Vectors/Call
avf-input polling 5353214 863814
0 4.55e3 .16
drop active 999 131154
0 1.72e1 131.29
error-drop active 999 131154
0 1.24e1 131.29
ethernet-input active 70791 863814
0 1.71e2 12.20
ip4-input-no-checksum active 70791 863814
0 2.34e2 12.20
ip4-lookup active 70791 863814
0 2.10e2 12.20
ip4-midchain active 70791 863814
0 3.24e2 12.20
iptfs-encap4-tun active 70791 863814
0 2.19e3 12.20
iptfs-pacer polling 5353214 730698
0 7.59e3 .14
---------------
Thread 2 vpp_wk_1 (lcore 2)
Time 5.1, 10 sec internal node vector rate 7.19 loops/sec 3472087.93
vector rates in 2.6609e5, out 2.6609e5, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors
Suspends Packet-Clocks Vectors/Call
avf-input polling 2692944 1369942
0 3.09e3 .51
esp4-decrypt-tun active 186680 1369942
0 3.99e3 7.34
ethernet-input active 186680 1369942
0 2.22e2 7.34
ip4-input-no-checksum active 186680 1369942
0 1.29e2 7.34
ip4-local active 186680 1369942
0 3.73e2 7.34
ip4-lookup active 186680 1369942
0 2.12e2 7.34
ipsec4-tun-input active 186680 1369942
0 4.26e2 7.34
iptfs-decap-reorder active 186680 1369942
0 5.39e2 7.34
Good System:
Thread 1 vpp_wk_0 (lcore 7)
Time 5.0, 10 sec internal node vector rate 2.18 loops/sec 3615274.68
vector rates in 8.6359e5, out 4.3199e5, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors
Suspends Packet-Clocks Vectors/Call
avf-input polling 17632667 2175512
0 1.64e3 .12
ethernet-input active 1001924 2175512
0 1.85e2 2.17
ip4-input-no-checksum active 1001924 2175512
0 1.40e2 2.17
ip4-lookup active 1001924 2175512
0 1.42e2 2.17
ip4-midchain active 1001924 2175512
0 1.52e2 2.17
iptfs-encap4-tun active 1001924 2175512
0 1.60e3 2.17
iptfs-pacer polling 17632667 2173458
0 1.85e3 .12
---------------
Thread 2 vpp_wk_1 (lcore 8)
Time 5.0, 10 sec internal node vector rate 1.03 loops/sec 593885.57
vector rates in 5.6564e5, out 5.6564e5, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors
Suspends Packet-Clocks Vectors/Call
dpdk-input polling 3450363 2848554
0 6.81e2 .83
esp4-decrypt-tun active 2758406 2848554
0 1.83e3 1.03
ethernet-input active 2758406 2848554
0 3.49e2 1.03
ip4-input-no-checksum active 2758406 2848554
0 2.10e2 1.03
ip4-local active 2758406 2848554
0 2.21e2 1.03
ip4-lookup active 2758406 2848554
0 2.25e2 1.03
ipsec4-tun-input active 2758406 2848554
0 3.21e2 1.03
iptfs-decap-reorder active 2758406 2848554
0 1.35e3 1.03
Thanks,
Chris.
>
>> Thanks,
>> Chris.
>>
>>>
>>>>
>>>>>
>>>>>>
>>>>>>> What kernel and driver version do you use?
>>>>>>
>>>>>> Host Config:
>>>>>>
>>>>>> $ cat /etc/lsb-release
>>>>>> DISTRIB_ID=Ubuntu
>>>>>> DISTRIB_RELEASE=20.10
>>>>>> DISTRIB_CODENAME=groovy
>>>>>> DISTRIB_DESCRIPTION="Ubuntu 20.10"
>>>>>> $ uname -a
>>>>>> Linux labnh 5.8.0-31-generic #33-Ubuntu SMP Mon Nov 23 18:44:54 UTC 2020
>>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>>>
>>>>>> Docker Config (compiled and run inside):
>>>>>>
>>>>>> root@p1:/# cat /etc/lsb-release
>>>>>> DISTRIB_ID=Ubuntu
>>>>>> DISTRIB_RELEASE=18.04
>>>>>> DISTRIB_CODENAME=bionic
>>>>>> DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"
>>>>>>
>>>>>>>
>>>>>>> Have you tried latest PF driver from intel?
>>>>>>
>>>>>> No; however, I am running such a new ubuntu (5.8 kernel) so I was hoping
>>>>>> that was sufficient.
>>>>>
>>>>> Agree, still, please, do me a favour and try with latest, so I know i’m
>>>>> looking at the same thing.
>>>>
>>>> I did the build and installed the deb, I also updated the firmware on the
>>>> NIC; however,
>>>>
>>>> $ dpkg -l | grep i40
>>>> ii i40e-dkms 2.12.6
>>>> all Intel i40e adapter driver
>>>>
>>>> $ sudo ethtool -i enp101s0f0
>>>> driver: i40e
>>>> version: 2.8.20-k
>>>> firmware-version: 8.15 0x80009621 1.2829.0
>>>> expansion-rom-version:
>>>> bus-info: 0000:65:00.0
>>>> supports-statistics: yes
>>>> supports-test: yes
>>>> supports-eeprom-access: yes
>>>> supports-register-dump: yes
>>>> supports-priv-flags: yes
>>>>
>>>> It doesn't seem to have used it. Is there something else I need to do to
>>>> have it use the dkms driver?
>>>
>>> rmmod i40e; modprobe i40e
>>> or reboot…
signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18279): https://lists.fd.io/g/vpp-dev/message/18279 Mute This Topic: https://lists.fd.io/mt/78779141/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
