On 2009/09/15 11:48, Eitan Eliahu wrote:
Thanks Lucy for the detailed explanation.
The problem I have is for 4 separate devices (pci functions). Only the
first instance is able to allocate MSI-X vectors and the other once fail
to a level interrupt (x86 on b111). My understanding that even w/o IRM
the limitation is per device rather than system wide.
There are limitations on both system level and device level. Currently,
there are only 224 vectors available for I/O devices in the whole x86
system. And 32 of them are for devices interrupting at priority 5. That
is to say, devices with interrupt priority 5 share 32 vectors.
I assume you have 4 separate network devices as you are working on
networking driver. :) So the default interrupt priority of your devices
is 5 and there are at most 32 vectors available for all your devices.
How many vectors are granted to your first instance? If the number is
around 30, it's not strange that there is no MSI-X vector available to
other devices. Please do not request MSI-X vectors too aggressively.
Thanks,
Lucy
Have I missed anything?
Eitan
-----Original Message-----
From: xiuyan.w...@sun.com [mailto:xiuyan.w...@sun.com]
Sent: Monday, September 14, 2009 8:23 PM
To: Eitan Eliahu
Cc: driver-discuss@opensolaris.org
Subject: Re: [driver-discuss] MSI-X vector shortage
Hi Eitan,
Generally, MSI/X interrupts are limited by the root complex
implementation. So the OS needs some relatively fair allocation
algorithm to assign such limited resources.
In Solaris, without IRM, the default maximum number of MSI/X interrupt
can be allocated to a pci function is 8 for SPARC and the number is 2
for x86. Tt can be tuned by increasing ddi_msix_alloc_limit through
/etc/system. But on x86, the total number of interrupt vectors for
interrupt priority 5 is 32 (5 is the default interrupt priority for
network devices), so you can not allocate more interrupts than
32 even with tuning.
With IRM, the above limitation is removed and the allocation algorithm
is more flexible. But for x86, the driver can only allocate more
interrupts for a pci function when the fix of the below CR is available:
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6669984
In my opinion, it's quite challenging for a IRM-capable driver to handle
the situation of interrupt reclaim when the system is short of interrupt
resources.
Having said the above, ddi_intr_dup_handler() might be able to address
your issue if you do want to use large number of interrupts, which lets
more MSI-X "slot" to share the same interrupt vector if the driver can
not get as more interrupts as desired. But it only works on SPARC.
Please see the man page for the details on the usage.
Thanks,
Lucy
On 2009/09/15 06:26, Eitan Eliahu wrote:
It seems that the driver fails to allocate MSI-X interrupts after
30-32
vectors had been allocated. Setting higher number in
ddi_msix_alloc_limit does not resolve the issue.
Should the driver use new dynamic Interrupt Resource Manager
interface?
What is the reason for the above limitation?
Please advice.
Eitan
------------------------------------------------------------------------
_______________________________________________
driver-discuss mailing list
driver-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/driver-discuss
_______________________________________________
driver-discuss mailing list
driver-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/driver-discuss
