> This case also includes the contract for Crossbow framework to use these
> interrupt affinity interfaces in place of existing PCITool ioctl
> interfaces.
>
If I look at the this case in isolation from its expected consumers, and
with pcitool as the
only consumer of the CPU affinity APIs, I have no trouble sending a +1.
However, when considering the overall architecture that includes both
this case's deliverables as well as the changes
expected imminently from its external consumers, I am unclear on how
the system will behave when
we use both pcitool and those consumers' interrupt settings.
I'll use the interaction with Crossbow as an example. The point is
similar for the interaction with other tools
(intd(1m), etc). Say the system has a physical NIC nxge0, whose
interrupts are bound the cpu's 1,2,3,4
using pcitool modified by this case. Later one creates vnic1 over nxge0
which
used a couple of hardware rings out of nxge0's, and uses dladm
set-linkprop cpus=5,6 vnic1.
With the changes imported from this case, the implementation of that
call of dladm will attempt
to have the MSI/X interrupts assigned to the rings (thus the vnic1)
bound to CPUs 5 and 6.
Will such setting of vnic1's interrupts, fail because of a conflict
with a previous binding by pcitool?
will it succeed silently? what will the call to pcitool querying about
the interrupt binding for
nxge0 return then? CPUs 1,2,3,4 only, as set ? or will it surprisingly
show 1,2,3,4,5,6?
how about the other way around? will dladm get-linkprop vnic1 see the
settings that were previously done
by pcitool?
Kais.
> Constraints:
> a) Set affinity limitations for certain interrupt types
> Fixed or INTx interrupts could be either exclusive or sharable
> depending
> on hardware. Because there is no good way to detect that, the current
> implementation will refuse any set affinity requests for INTx
> interrupts.
>
> On x86 platforms, multiple MSI interrupts of a single PCI function need
> to be rerouted together since all MSI interrupts share the same MSI
> address, which in turn includes same CPU number. Hence the current x86
> implementation will refuse any set affinity requests for MSI
> interrupts.
> The future phase of this project may support MSI group retarget,
> similar
> to PCITool method.
>
> b) CPU offline considerations
> CPUs may be online/offlined through administrative interfaces. When
> a CPU is offlined, all of the interrupts targeting it are re-targeted.
> The OS will pick any set of the surviving CPUs for re-targeting. The
> OS is under no obligation to maintain drivers' interrupt affinity
> preferences.
>
> The first phase of this project will not provide any callback on CPU
> online/offline events. Such callback events need to be defined in the
> future. If a driver or framework is interested in maintaining optimal
> CPU targeting, it should monitor its interrupt CPU bindings on a
> regular
> basis using ddi_intr_get_affinity(9f) or register a callback to receive
> various CPU specific events using register_cpu_setup_func(). Where as,
> the userland entities should subscribe to CPU DR specific sysevents.
>
> 4.5.2 PCITool Enhancements
>
> Current syntax:
> pcitool pci@<unit-address> -i ino=ino
> [ -r [ -c ] | -w cpu=CPU [ -g ] ] [ -v ] [ -q ]
>
> Proposed syntax:
> pcitool pci@<unit-address> -i <ino#> | all
> [ -r [ -c ] | -w <cpu#> [ -g ] ] [ -v ] [ -q ]
>
> pcitool pci@<unit-address> -m <msi#> | all
> [ -r [ -c ] | -w <cpu#> [ -g ] ] [ -v ] [ -q ]
>
> The PCItool is a low-level tool which provides a facility for getting and
> setting interrupt routing information. This project is making some minor
> syntax changes to PCITool since the current syntax is not compliant with
> existing userland guidelines.
>
> In addition, this project is adding a new "-m" option to retrieve and
> reroute the interrupt target CPU for MSI/Xs on SPARC platforms.
>
> On SPARC platforms, the INO is mapped to an interrupt mondo, and where as
> one or more MSI/Xs are mapped to an INO. So, INO and MSI/Xs are
> individually
> retargetable. Use "-i " option to retrieve or reroute a given INO, and
> where as use "-m" option for MSI/Xs.
>
> On x86 platforms, both INOs and MSI/Xs are mapped to the same interrupt
> vectors. Use "-i" option to retrieve and reroute any interrupt vectors
> (both INO and MSI/Xs). So, "-m" option is not required on x86 platforms.
> Hence it is not supported.
>
> 4.6 Interfaces
>
> 4.6.1 Exported Interfaces
>
> Interface Stability Comments
> ----------------------------+---------------+--------------------------
> ddi_intr_target_t Project Interrupt target CPU
> Private
> ddi_intr_get_affinity Project Get interrupt target CPU
> Private
> ddi_intr_set_affinity Project Set interrupt target CPU
> Private
> -----------------------------------------------------------------------
>
> 4.6.2 Imported Interfaces
>
> Interface Stability Comments
> ----------------------------+---------------+--------------------------
> DDI_INTR_FLAG_RETARGETABLE Project Return this new flag
> (RO) to
> Private ddi_intr_get_cap() callers if
> current interrupt type in use
> is retargetable
>
> pcitool Project Minor syntax changes. Added
> Private new -m option for MSI/Xs.
> -----------------------------------------------------------------------
>
> 5. References
> [1] Solaris Interrupt Project Webpage
> http://pciexpress.sfbay/intr
>
> [2] Advanced DDI Interrupt Functions - PSARC/2004/253
> http://sac.sfbay.sun.com/PSARC/2004/253
>
> [3] Interrupt Resource Management - PSARC/2008/628
> http://sac.sfbay.sun.com/PSARC/2008/628
>
> [4] PCITool and its nexus ioctl support - PSARC/2005/232
> http://sac.sfbay.sun.com/PSARC/2005/232
>
> [5] PCITool Public Interrupts - PSARC/2009/215
> http://sac.sfbay.sun.com/PSARC/2009/215
>
> 6. Resources and Schedule
> 6.4 Steering Committee requested information
> 6.4.1 Consolidation C-team Name:
> ON
> 6.5 ARC review type: FastTrack
> 6.6 ARC Exposure: open
>
>
> 6. Resources and Schedule
> 6.4. Steering Committee requested information
> 6.4.1. Consolidation C-team Name:
> ON
> 6.5. ARC review type: FastTrack
> 6.6. ARC Exposure: open
>
>
>
>
