On Mon, 26 Oct 2020 22:50:45 +0100 Thomas Gleixner wrote: > On Mon, Oct 26 2020 at 14:11, Jacob Keller wrote: > > On 10/26/2020 1:11 PM, Thomas Gleixner wrote: > >> On Mon, Oct 26 2020 at 12:21, Jacob Keller wrote: > >>> Are there drivers which use more than one interrupt per queue? I know > >>> drivers have multiple management interrupts.. and I guess some drivers > >>> do combined 1 interrupt per pair of Tx/Rx.. It's also plausible to to > >>> have multiple queues for one interrupt .. I'm not sure how a single > >>> queue with multiple interrupts would work though. > >> > >> For block there is always one interrupt per queue. Some Network drivers > >> seem to have seperate RX and TX interrupts per queue. > > That's true when thinking of Tx and Rx as a single queue. Another way to > > think about it is "one rx queue" and "one tx queue" each with their own > > interrupt... > > > > Even if there are devices which force there to be exactly queue pairs, > > you could still think of them as separate entities? > > Interesting thought. > > But as Jakub explained networking queues are fundamentally different > from block queues on the RX side. For block the request issued on queue > X will raise the complete interrupt on queue X. > > For networking the TX side will raise the TX interrupt on the queue on > which the packet was queued obviously or should I say hopefully. :) > > But incoming packets will be directed to some receive queue based on a > hash or whatever crystallball logic the firmware decided to implement. > > Which makes this not really suitable for the managed interrupt and > spreading approach which is used by block-mq. Hrm... > > But I still think that for curing that isolation stuff we want at least > some information from the driver. Alternative solution would be to grant > the allocation of interrupts and queues and have some sysfs knob to shut > down queues at runtime. If that shutdown results in releasing the queue > interrupt (via free_irq()) then the vector exhaustion problem goes away. > > Needs more thought and information (for network oblivious folks like > me).
One piece of information that may be useful is that even tho the RX packets may be spread semi-randomly the user space can still control which queues are included in the mechanism. There is an indirection table in the HW which allows to weigh queues differently, or exclude selected queues from the spreading. Other mechanisms exist to filter flows onto specific queues. IOW just because a core has an queue/interrupt does not mean that interrupt will ever fire, provided its excluded from RSS. Another piece is that by default we suggest drivers allocate 8 RX queues, and online_cpus TX queues. The number of queues can be independently controlled via ethtool -L. Drivers which can't support separate queues will default to online_cpus queue pairs, and let ethtool -L only set the "combined" parameter. There are drivers which always allocate online_cpus interrupts, and then some of them will go unused if #qs < #cpus. My unpopular opinion is that for networking devices all the heuristics we may come up with are going to be a dead end. We need an explicit API to allow users placing queues on cores, and use managed IRQs for data queues. (I'm assuming that managed IRQs will let us reliably map a MSI-X vector to a core :))
