Hi, Thanks for getting back to me.
There was some traffic running. The guy who ran the tests says that there were IRQs coming in on the "iavf-0000:b5:02.7:mbx" interrupt at roughly 1 per second without any traffic, while the interrupt rate on the "iavf-net1-TxRx-<X>" interrupts roughly correlated with the rate of traffic.
Yes, vfs were created. (Which makes sense, since we saw interrupt counts going up on the "iavf-net1-TxRx-<X>" interrupts which only exist for the vfs.)
I just did another test myself in a different lab, and I reproduced the problem for TxRx IRQs belonging to two devices which both use the "igb" driver, version 5.6.0-k. The "i40e" TxRx queues on this system didn't show the problem, they were all taking interrupts on CPU0. I didn't try creating any VFs on this system. The results are below.
controller-0:~$ cat /proc/interrupts |cut -c -50,710-|grep TxRx|grep ens2;sleep 10;cat /proc/interrupts |cut -c -50,710-|grep TxRx|grep ens2 590: 0 0 8484 0 IR-PCI-MSI-edge ens2f0-TxRx-0 591: 0 0 4087 0 IR-PCI-MSI-edge ens2f0-TxRx-1 592: 0 0 4087 0 IR-PCI-MSI-edge ens2f0-TxRx-2 593: 0 0 4090 0 IR-PCI-MSI-edge ens2f0-TxRx-3 594: 0 0 4091 0 IR-PCI-MSI-edge ens2f0-TxRx-4 595: 0 0 4087 0 IR-PCI-MSI-edge ens2f0-TxRx-5 596: 0 0 4361 0 IR-PCI-MSI-edge ens2f0-TxRx-6 597: 0 0 4087 0 IR-PCI-MSI-edge ens2f0-TxRx-7 599: 0 0 8481 0 IR-PCI-MSI-edge ens2f1-TxRx-0 600: 0 0 4085 0 IR-PCI-MSI-edge ens2f1-TxRx-1 601: 0 0 4085 0 IR-PCI-MSI-edge ens2f1-TxRx-2 602: 0 0 4093 0 IR-PCI-MSI-edge ens2f1-TxRx-3 603: 0 0 4085 0 IR-PCI-MSI-edge ens2f1-TxRx-4 604: 0 0 4086 0 IR-PCI-MSI-edge ens2f1-TxRx-5 605: 0 0 4358 0 IR-PCI-MSI-edge ens2f1-TxRx-6 606: 0 0 4085 0 IR-PCI-MSI-edge ens2f1-TxRx-7
590: 0 0 8494 0 IR-PCI-MSI-edge ens2f0-TxRx-0 591: 0 0 4092 0 IR-PCI-MSI-edge ens2f0-TxRx-1 592: 0 0 4092 0 IR-PCI-MSI-edge ens2f0-TxRx-2 593: 0 0 4095 0 IR-PCI-MSI-edge ens2f0-TxRx-3 594: 0 0 4096 0 IR-PCI-MSI-edge ens2f0-TxRx-4 595: 0 0 4092 0 IR-PCI-MSI-edge ens2f0-TxRx-5 596: 0 0 4366 0 IR-PCI-MSI-edge ens2f0-TxRx-6 597: 0 0 4092 0 IR-PCI-MSI-edge ens2f0-TxRx-7 599: 0 0 8492 0 IR-PCI-MSI-edge ens2f1-TxRx-0 600: 0 0 4090 0 IR-PCI-MSI-edge ens2f1-TxRx-1 601: 0 0 4090 0 IR-PCI-MSI-edge ens2f1-TxRx-2 602: 0 0 4098 0 IR-PCI-MSI-edge ens2f1-TxRx-3 603: 0 0 4090 0 IR-PCI-MSI-edge ens2f1-TxRx-4 604: 0 0 4091 0 IR-PCI-MSI-edge ens2f1-TxRx-5 605: 0 0 4363 0 IR-PCI-MSI-edge ens2f1-TxRx-6 606: 0 0 4090 0 IR-PCI-MSI-edge ens2f1-TxRx-7
Even though we can see that the interrupts are coming in on CPU2, the affinity for IRQ 590 indicates that there should only be interrupts on CPUs 0 and 1.
controller-0:~$ sudo cat /proc/irq/590/smp_affinity Password: 00000003,00000003 Chris On 1/20/2021 9:52 PM, Fujinaka, Todd wrote:
I hope the following makes sense. The following comes from the developers. questions: 1 .did you run any traffic or NIC was in idle state? 2. did you create VFs? please check the following repro steps to see if they match yours 1. on diskless the boot script was changed to: intel_iommu=on irqaffinity=0-1,24-25 /proc/irq/default_smp_affinity is "000...000,03000003" 2. compiled, installed, and reloaded both the i40e and iavf drivers 3. /proc/interrupts shows: CPU0 CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 CPU32 CPU33 CPU34 CPU35 CPU36 CPU37 CPU38 CPU39 CPU40 CPU41 CPU42 CPU43 CPU44 CPU45 CPU46 CPU47 CPU48 CPU49 CPU50 CPU51 CPU52 CPU53 CPU54 CPU55 CPU56 CPU57 CPU58 CPU59 CPU60 CPU61 CPU62 CPU63 0: 82 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 8: 58 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi 10: 232 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge ipmi_si 16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi i801_smbus 24: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME, pciehp @ 4. created 2 VFs on first interface of NIC. 5. /proc/interrupts shows: CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 CPU32 CPU33 CPU34 CPU35 CPU36 CPU37 CPU38 CPU39 CPU40 CPU41 CPU42 CPU43 CPU44 CPU45 CPU46 CPU47 CPU48 CPU49 CPU50 CPU51 CPU52 CPU53 CPU54 CPU55 CPU56 CPU57 CPU58 CPU59 CPU60 CPU61 CPU62 CPU63 0: 82 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 8: 58 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi 10: 232 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge ipmi_si 16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi i801_smbus 24: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME, pciehp 25: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME @ Todd Fujinaka Software Application Engineer Data Center Group Intel Corporation todd.fujin...@intel.com -----Original Message----- From: Chris Friesen <chris.frie...@windriver.com> Sent: Friday, January 15, 2021 12:20 PM To: Fujinaka, Todd <todd.fujin...@intel.com>; e1000-devel@lists.sourceforge.net Subject: Re: [E1000-devel] IRQ affinity not working for iavf devices? The irqbalance daemon is not running. At the time of the analysis, "cat /proc/irq/<NUM>/smp_affinity_list" showed "0-1,24-25" but cat /proc/interrupts showed the interrupt counts increasing on CPU 4. If the interrupt affinity was modified by a userspace tool like irqbalance, I believe that /proc/irq/<NUM>/smp_affinity_list would show the updated affinity. This is the thing that I find really weird, the mismatch between the configured affinity and the observed interrupt counts. Chris On 1/15/2021 11:03 AM, Fujinaka, Todd wrote:I've been reminded that there are other daemons that could be moving around the CPU affinities, such as irqbalance. Make sure that daemon isn't started. Todd Fujinaka Software Application Engineer Data Center Group Intel Corporation todd.fujin...@intel.com -----Original Message----- From: Fujinaka, Todd <todd.fujin...@intel.com> Sent: Friday, January 15, 2021 8:44 AM To: Chris Friesen <chris.frie...@windriver.com>; e1000-devel@lists.sourceforge.net Subject: Re: [E1000-devel] IRQ affinity not working for iavf devices? Sorry for the slow response. I'm asking around internally as I'm not that familiar with iavf. I'll let you know when I hear back. Todd Fujinaka Software Application Engineer Data Center Group Intel Corporation todd.fujin...@intel.com -----Original Message----- From: Chris Friesen <chris.frie...@windriver.com> Sent: Tuesday, January 12, 2021 3:53 PM To: e1000-devel@lists.sourceforge.net Subject: [E1000-devel] IRQ affinity not working for iavf devices? Hi, I have a CentOS 7 linux system with 48 logical CPUs and a number of Intel NICs running the i40e driver. It was booted with irqaffinity=0-1,24-25 in the kernel boot args, resulting in /proc/irq/default_smp_affinity showing "0000,03000003". CPUs 2-11 are set as "isolated" in the kernel boot args. The iavf driver is 3.7.61.20 and the i40e driver is 2.10.19.82 The problem I'm seeing is that /proc/interrupts shows iavf interrupts on other CPUs. For example, here are some on CPU 4 where I would not expect to see any interrupts given that "cat /proc/irq/<NUM>/smp_affinity_list" reports "0-1,24-25". cat /proc/interrupts | grep -e CPU -e 941: -e 942: -e 943: -e 944: -e 945: -e 961: -e 962: -e 963: -e 964: -e 965: CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 941: 0 0 0 0 28490 0 IR-PCI-MSI-edge iavf-0000:b5:03.6:mbx 942: 0 0 0 0 333832 0 IR-PCI-MSI-edge iavf-net1-TxRx-0 943: 0 0 0 0 300842 0 IR-PCI-MSI-edge iavf-net1-TxRx-1 944: 0 0 0 0 333845 0 IR-PCI-MSI-edge iavf-net1-TxRx-2 945: 0 0 0 0 333822 0 IR-PCI-MSI-edge iavf-net1-TxRx-3 961: 0 0 0 0 28492 0 IR-PCI-MSI-edge iavf-0000:b5:02.7:mbx 962: 0 0 0 0 435608 0 IR-PCI-MSI-edge iavf-net1-TxRx-0 963: 0 0 0 0 394832 0 IR-PCI-MSI-edge iavf-net1-TxRx-1 964: 0 0 0 0 398414 0 IR-PCI-MSI-edge iavf-net1-TxRx-2 965: 0 0 0 0 192847 0 IR-PCI-MSI-edge iavf-net1-TxRx-3 Is this expected? It seems like the iavf and/or the i40e aren't respecting the configured SMP affinity for the interrupt in question. Thanks, Chris _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
_______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel Ethernet, visit https://forums.intel.com/s/topic/0TO0P00000018NbWAI/intel-ethernet
