MSI/MSI-X implementation and interrupt handling on i386/amd64
Hi, brief background: on an amd64 VM (1 CPU on VMWare ESXi) I had a network interface (vmx) failing because it could not get an interrupt slot. The vmx wants 3 interrupts per interface (tx/rx/link-state). I had a few on an admin machine and one started failing when ahcisata was changed to use MSI (not ahcisata's fault, obviously). On i386/amd64 each CPU has a 32 bitmask for interrupts (1 bit per) - but 16 of the 32 are reserved for legacy IRQs (on the first CPU). MSI-X allows for 2048 interrupts. On a physical machine with many CPUs the MSI interrupts are farmed out across the different CPUs so would not be apparent to most. (and no problem for those 65+ core machines). For my personal use, I've hacked around by ignoring the reserved legacy IRQ region because it's not relevant to me in my VM with MSI/MSI-X. Other people using single CPU VMs may start bumping into this issue so just making people aware. I haven't looked into changing how interrupts are handled or if there would be significant performance penalty. Regards, Geoff FYI (pin17 is mpt0): % intrctl list interrupt id CPU0 device name(s) ioapic0 pin 90* acpi SCI ioapic0 pin 10* pckbc1 kbd ioapic0 pin 12 0* pckbc2 aux ioapic0 pin 14 0* piixide0 primary ioapic0 pin 15 0* piixide0 secondary ioapic0 pin 17 3481843* unknown ioapic0 pin 18 54* uhci0 ioapic0 pin 19 0* ehci0 msi0 vec 0 0* ahcisata0 msix1 vec 0 16215* vmx0: tx 0 msix1 vec 1 406335* vmx0: rx 0 msix1 vec 2 0* vmx0: link msix2 vec 0 100571* vmx1: tx 0 msix2 vec 1 178436* vmx1: rx 0 msix2 vec 2 0* vmx1: link msix3 vec 0 327583* vmx2: tx 0 msix3 vec 13141480* vmx2: rx 0 msix3 vec 2 0* vmx2: link
Re: MSI/MSI-X implementation and interrupt handling on i386/amd64
Hi Geoff, Saitoh-san pointed me at this email. I've been looking at MSI briefly - should have some work in place to sort out this situation. About your specific situation: Geoff Wing writes: > Hi, > brief background: on an amd64 VM (1 CPU on VMWare ESXi) I had a network > interface (vmx) failing because it could not get an interrupt slot. The > vmx wants 3 interrupts per interface (tx/rx/link-state). I had a few > on an admin machine and one started failing when ahcisata was changed to > use MSI (not ahcisata's fault, obviously). > > On i386/amd64 each CPU has a 32 bitmask for interrupts (1 bit per) - but > 16 of the 32 are reserved for legacy IRQs (on the first CPU). MSI-X allows > for 2048 interrupts. On a physical machine with many CPUs the MSI interrupts > are farmed out across the different CPUs so would not be apparent to most. > (and no problem for those 65+ core machines). > > For my personal use, I've hacked around by ignoring the reserved legacy IRQ > region because it's not relevant to me in my VM with MSI/MSI-X. Other > people using single CPU VMs may start bumping into this issue so just > making people aware. I haven't looked into changing how interrupts are > handled or if there would be significant performance penalty. > You could have a stopgap fix by just using a 64 bit mask and equivalent supporting data structures instead of the 32bit one. You'll probably need to also look at spl.S assembler primitives that access the pending bitmask via assembler instructions and teach them how to do this on a 64bit pending string. The right thing to do is to stop using a bit mask entirely, and using a bit more scalable Data structure for this. This isn't trivial though - the assembler stuff will be harder to maintain correctness than a straightup buslocked bitscan/compare etc. In any case, I'll report back when I get around to this. Many Thanks, -- ~cherry
Re: MSI/MSI-X implementation and interrupt handling on i386/amd64
Le jeu. 6 déc. 2018 à 16:05, Cherry G.Mathew a écrit : > The right thing to do is to stop using a bit mask entirely, and using > a bit more scalable Data structure for this. This isn't trivial though - > the assembler stuff will be harder to maintain correctness than a > straightup buslocked bitscan/compare etc. What about just bumping this to 64 on amd64, where we have the 64-bit atomic ops? While keeping i386 still on 32. We seem to have already i386 and amd64 variants of the interrupt assembler, so maybe not so bad that they would diverge further. It would be nice to do something to bump the limit. If we have general consensus is that this is worth doing, I can try to write something and see how ugly/difficult it would become with 64bit bitmasks. I don't feel like delving into rewriting this to use completely different structure ... Jaromir
Re: MSI/MSI-X implementation and interrupt handling on i386/amd64
Hi, On 2018/12/11 6:49, Jaromír Doleček wrote: > Le jeu. 6 déc. 2018 à 16:05, Cherry G.Mathew a écrit : >> The right thing to do is to stop using a bit mask entirely, and using >> a bit more scalable Data structure for this. This isn't trivial though - >> the assembler stuff will be harder to maintain correctness than a >> straightup buslocked bitscan/compare etc. > > What about just bumping this to 64 on amd64, where we have the 64-bit > atomic ops? While keeping i386 still on 32. > > We seem to have already i386 and amd64 variants of the interrupt > assembler, so maybe not so bad that they would diverge further. > > It would be nice to do something to bump the limit. If we have general > consensus is that this is worth doing, I can try to write something > and see how ugly/difficult it would become with 64bit bitmasks. I > don't feel like delving into rewriting this to use completely > different structure ... I agree it. I mention some old Athlon 64 series (before socket AM2) do not support cmpxchg16b instruction. That would affect rewriting spllower to support 64 bit interrupt bitmask. Thanks, -- // Internet Initiative Japan Inc. Device Engineering Section, IoT Platform Development Department, Network Division, Technology Unit Kengo NAKAHARA
Re: MSI/MSI-X implementation and interrupt handling on i386/amd64
Moving this to port-amd64 (bcced current-users@ for reference) Le mar. 11 déc. 2018 à 04:34, Kengo NAKAHARA a écrit : > I mention some old Athlon 64 series (before socket AM2) do not support > cmpxchg16b instruction. That would affect rewriting spllower to support > 64 bit interrupt bitmask. Indeed, need to do runtime check for cmpxchg16b support. I'm investigating an initial solution which will keep the ipending+ilevel still as 64-bit quantity suitable for cmpxchg8b, using 4 bits for the ilevel and remaining 60 for ipending. The 60/4 split with cmpxchg8b should work for i386 too. Besides spl.S/vector.S, so far I only found one place using atomic write on the ipending - clearing interrupt from ipending in intr disestablish. This one can easily change to just block all interrupts during the clearing, since this is not performance critical path. Jaromir
