I'm sponsoring this case for Zhijun Robin Fu with a requested patch binding
and volatile commitment.
Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
This information is Copyright 2009 Sun Microsystems
1. Introduction
1.1. Project/Component Working Name:
Pcitool Extensions
1.2. Name of Document Author/Supplier:
Author: Zhijun Fu
1.3 Date of This Document:
07 December, 2009
4. Technical Description
4.1. Introduction
Pcitool is a low-level tool which provides a facility for getting and
setting interrupt routing information. It is first introduced by
PSARC/2005/232 and then made public by PSARC/2009/215. Currently on
x86 it refers to an interrupt by using the interrupt vector, which
isn't unique on a multi-CPU x86 system.
On x86 platforms, the interrupt vectors are associated with the local
APIC (Advanced Programmable Interrupt Controller) which is a component
of CPU. So the number of total interrupt vectors in the system scales
with the number of CPUs present in the system. And an interrupt vector
number is only unique on the CPU which owns the vector, it isn't global
unique in a system which has multiple CPUs.
To uniquely identify an interrupt on x86, both interrupt vector number
and the CPU which owns this vector are needed. This project will define
the necessary changes to pcitool to uniquely identify an interrupt on
x86, and also it will propose some output changes to be more user friendly.
4.2. Details
- Pcitool syntax changes
Current syntax:
pcitool pci@<unit-address> -i <ino#> | all
[ -r [ -c ] | -w <cpu#> [ -g ] ] [ -v ] [ -q ]
pcitool pci@<unit-address> -m <msi#> | all
[ -r [ -c ] | -w <cpu#> [ -g ] ] [ -v ] [ -q ]
Proposed syntax:
pcitool pci@<unit-address> -i <cpu#,ino#> | all
[ -r [ -c ] | -w <cpu#> [ -g ] ] [ -v ] [ -q ]
pcitool pci@<unit-address> -m <cpu#,msi#> | all
[ -r [ -c ] | -w <cpu#> [ -g ] ] [ -v ] [ -q ]
On SPARC platforms, use "-i" option to retrieve or reroute a given INO,
and where as use "-m" option for MSI/Xs. On x86 platforms, use "-i" option
to retrieve and reroute any interrupt vectors (both INO and MSI/Xs).
"-m" option is not required on x86 platforms. Hence it is not supported.
This project adds a new "cpu#" field in addition to "ino#" to uniquely
represent an interrupt on x86 platform. The new field is not required
for SPARC platforms, thus is ingored in these cases.
- Pcitool output changes
This project also makes some changes to simplify the pcitool output, making
it more user friendly.
Current output:
Show ino 0x19 on /pci at 1e,600000
# pcitool /pci at 1e,600000 -i 19
ino 19 mapped to cpu 0
Device: /pci at 1e,600000/pci at 0/pci at 9/pci at 0/scsi at 1
Driver: mpt, instance 0
Reroute ino 0x19 from cpu 0 to cpu 1
# pcitool /pci at 1e,600000 -i 19 -w 1
Interrupts on ino 19 reassigned: Old cpu: 0, New cpu: 1
Reroute a group of INOs starting at 60 from cpu 0 to cpu 1
# pcitool /pci at 0,0 -i 60 -w 1 -g
Interrupts on ino group starting at ino 60 reassigned: Old
cpu: 0, New cpu: 1
Proposed output:
Show ino 0x19 on /pci at 1e,600000
# pcitool /pci at 1e,600000 -i 19
0x0,0x19: mpt0 /pci at 1e,600000/pci at 0/pci at 9/pci at 0/scsi at 1
Reroute ino 0x19 from cpu 0 to cpu 1
# pcitool /pci at 1e,600000 -i 19 -w 1
0x0,0x19 -> 0x1,0x19
Reroute a group of INOs starting at 60 from cpu 0 to cpu 1
# pcitool /pci at 0,0 -i 60 -w 1 -g
0x0,0x60 => 0x1,0x60
Note that on x86 platforms cpu must be specified along with ino, and the
ino might change after being rerouted to another cpu. In this case reroute
ino 0x19 from cpu 0 to cpu 1 could be:
# pcitool /pci at 1e,600000 -i 0,19 -w 1
0x0,0x19 -> 0x1,0x21
Show ino 0x19 on cpu 0:
# pcitool /pci at 1e,600000 -i 0,19
0x0,0x19: mpt0 /pci at 1e,600000/pci at 0/pci at 9/pci at 0/scsi at 1
- Data structure changes
pcitool_intr_set_t
This is the argument data structure for PCITOOL_DEVICE_SET_INTR ioctl,
which maps a given interrupt or a group of interrupt to a given CPU.
This project adds a new field "old_cpu" to identify the CPU which
owns the interrupt vector, this together with interrupt vector number
uniquely represents the interrupt to be rerouted.
pcitool_intr_info_t
This is the argument data structure for PCITOOL_SYSTEM_INTR_INFO ioctl,
which retrieves system interrupt information. This project adds a new
field "num_cpu" to reflect how many CPUs are present in the system,
since the number of total interrupt vectors in the system scales with
the number of CPUs.
4.3. Interfaces
Imported Interfaces
Interface Stability Comments
----------------------------+---------------+--------------------------
pcitool Project Some syntax changes. Added
Private new cpu# field, changed out-
put format
PCITOOL_DEVICE_SET_INTR Consolidation Ioctl to remap CPU of intr
Private or group of intr
PCITOOL_SYSTEM_INTR_INFO Consolidation Ioctl to retrieve system
Private interrupt info
pcitool_intr_set_t Consolidation PCITOOL_DEVICE_SET_INTR arg
Private data structure
pcitool_intr_info_t Consolidation PCITOOL_SYSTEM_INTR_INFO arg
Private data structure
-----------------------------------------------------------------------
4.4. Dependencies
The changes described in this case will be done as a follow on work to the
following RFE:
CR 6669984 Solaris x86 need to provide large number of interrupt vectors
for MSI/MSI-X
The reason for this dependency is the inherent limitations on the current
implementation of pcplusmp module (APIC module), where the total number
of vectors available in the system is limited to only the number of vectors
of a single APIC. So having these changes on the current implementation
adds no value to the system.
NOTE: The current plan is to integrate the pcitool changes along with
the APIC code changes for the above RFE.
5. References
[1] PCITool and its nexus ioctl support - PSARC/2005/232
http://sac.sfbay.sun.com/PSARC/2005/232
[2] PCITool Public Interrupts - PSARC/2009/215
http://sac.sfbay.sun.com/PSARC/2009/215
6. Resources and Schedule
6.4. Steering Committee requested information
6.4.1. Consolidation C-team Name:
ON
6.5. ARC review type: FastTrack
6.6. ARC Exposure: open
