> On 2 Jul 2022, at 08:44, Hrvoje Popovski <[email protected]> wrote:
>
> On 28.6.2022. 15:11, Jonathan Matthew wrote:
>> This adds the (not quite) final bits to em(4) to enable multiple rx/tx
>> queues.
>> Note that desktop/laptop models (I218, I219 etc.) do not support multiple
>> queues,
>> so this only really applies to servers and network appliances (including
>> APU2).
>>
>> It also removes the 'em_enable_msix' variable, in favour of using MSI-X on
>> devices
>> that support multiple queues and MSI or INTX everywhere else.
>>
>> I've tested this with an I350 on amd64 and arm64, where it works as
>> expected, and
>> with the I218-LM in my laptop where it does nothing (as expected).
>> More testing is welcome, especially in forwarding environments.
>
>
> Hi,
>
> I'm testing this diff in forwarding setup where source is 10.113.0/24
> connected to em2 and destination is 10.114.0/24 connected to em3. I'm
> doing random source and destination per ip.
>
> dmesg:
> em2 at pci6 dev 0 function 2 "Intel I350" rev 0x01, msix, 8 queues
> em3 at pci6 dev 0 function 3 "Intel I350" rev 0x01, msix, 8 queues
>
> netstat:
> 10.113.0/24 192.168.113.11 UGS 0 0 - 8 em2
> 10.114.0/24 192.168.114.11 UGS 0 404056853 - 8 em3
>
>
> ifconfig:
> em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> lladdr 40:f2:e9:ec:b4:14
> index 5 priority 0 llprio 3
> media: Ethernet autoselect (1000baseT full-duplex,master)
> status: active
> inet 192.168.113.1 netmask 0xffffff00 broadcast 192.168.113.255
> em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> lladdr 40:f2:e9:ec:b4:15
> index 6 priority 0 llprio 3
> media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
> status: active
> inet 192.168.114.1 netmask 0xffffff00 broadcast 192.168.114.255
>
>
> with vmstat -i
> irq160/em2:0 4740972 3538
> irq161/em2:1 4740979 3538
> irq162/em2:2 4740977 3538
> irq163/em2:3 4740978 3538
> irq164/em2:4 4740965 3538
> irq165/em2:5 4740972 3538
> irq166/em2:6 4740971 3538
> irq167/em2:7 4740965 3538
> irq168/em2 2 0
> irq169/em3:0 4741258 3538
> irq177/em3 2 0
>
>
> should I see 8 queues on em3 as on em2 ?
em(4) isn't populating the mbuf flowid field with the rss hash value the chip
calculates when it receives packets, so there's no flow identifier for the
network stack to use to assign packets to output queues on the way out. this
means everything lands on the default (0th) queue.
cheers,
dlg
>
> x3550m4# tcpdump -ni em3
> tcpdump: listening on em3, link-type EN10MB
> 00:39:26.663617 10.113.0.230.9 > 10.114.0.154.9: udp 18
> 00:39:26.663618 10.113.0.176.9 > 10.114.0.3.9: udp 18
> 00:39:26.663619 10.113.0.37.9 > 10.114.0.7.9: udp 18
> 00:39:26.663620 10.113.0.200.9 > 10.114.0.197.9: udp 18
> 00:39:26.663620 10.113.0.37.9 > 10.114.0.230.9: udp 18
> 00:39:26.663621 10.113.0.95.9 > 10.114.0.216.9: udp 18
> 00:39:26.663622 10.113.0.8.9 > 10.114.0.187.9: udp 18
> 00:39:26.663623 10.113.0.56.9 > 10.114.0.107.9: udp 18
> 00:39:26.663624 10.113.0.4.9 > 10.114.0.39.9: udp 18
> 00:39:26.663624 10.113.0.244.9 > 10.114.0.188.9: udp 18
> 00:39:26.663625 10.113.0.166.9 > 10.114.0.15.9: udp 18
> 00:39:26.663626 10.113.0.7.9 > 10.114.0.78.9: udp 18
> 00:39:26.663627 10.113.0.147.9 > 10.114.0.202.9: udp 18
> 00:39:26.663628 10.113.0.144.9 > 10.114.0.184.9: udp 18
> 00:39:26.663628 10.113.0.221.9 > 10.114.0.100.9: udp 18
> 00:39:26.663630 10.113.0.69.9 > 10.114.0.231.9: udp 18
> 00:39:26.663648 10.113.0.71.9 > 10.114.0.64.9: udp 18
>
>
> vmstat -iz
> irq160/em2:0 4740972 3501
> irq161/em2:1 4740979 3501
> irq162/em2:2 4740977 3501
> irq163/em2:3 4740978 3501
> irq164/em2:4 4740965 3501
> irq165/em2:5 4740972 3501
> irq166/em2:6 4740971 3501
> irq167/em2:7 4740965 3501
> irq168/em2 2 0
> irq169/em3:0 4741258 3501
> irq170/em3:1 0 0
> irq171/em3:2 0 0
> irq172/em3:3 0 0
> irq173/em3:4 0 0
> irq174/em3:5 0 0
> irq175/em3:6 0 0
> irq176/em3:7 0 0
> irq177/em3 2 0
>
