Re: MSI-X support
On 4.5.2016. 16:32, Mark Kettenis wrote: >>> This is great, thanks for doing this! I'm a bit surprised that >>> we don't need to the same suspend/resume dance in ppb(4) as with >>> MSI. >>> >> >> That is an excellent point I overlooked. Kettenis, do we? > > Almost certainly. I committed the diff, but left the bits out that > start using it in em(4) and xhci(4). > Hi all, is it a good time to enable msi-x on em(4) ?
Re: MSI-X support
> Date: Wed, 4 May 2016 07:14:15 -0700 > From: Mike Larkin > > On Wed, May 04, 2016 at 04:01:54PM +0200, Mike Belopuhov wrote: > > On Tue, May 03, 2016 at 21:40 +0200, Mark Kettenis wrote: > > > Today mpi@ reminded me that I had written support for MSI-X some time > > > ago. Since he is interested in using multiple vectors, I extended the > > > code I had a bit to support that feature as well. This introduces a > > > new function: > > > > > > int pci_intr_map_msix(struct pci_attach_args *, int, pci_intr_handle_t *); > > > > > > You use it like pci_intr_map_msi(9) and pci_intr_map(9), buthave to > > > pass it a vector number as the 2nd argument. Typically you'll pass 0, > > > wich will map the 1st vector, which is always there on hardware with > > > MSI-X support. Some hardware supports more than 1 vector. Typical > > > examples are network cards that support multiple rings. Please be > > > aware that on architectures like i386 and amd64, the number of > > > interrupt vectors at each level is limited. If you consume too many > > > of them, devices may fail to attach. > > > > > > As for the implementation. This only adds code for amd64. I'll > > > probably add support for sparc64 as well. I'm hesitant about i386 > > > because it has even less available interrupt vectors than amd64. > > > > > > Notice that the amd64 implementation uses _bus_space_map() and > > > _bus_space_unmap(). That is deliberate. The MSI-X registers miht > > > share a BAR with other registers that our drivers already map. The > > > underscored versions of the mapping routines make sure that we don't > > > fail to map things in that case. > > > > > > ok? > > > > > > > This is great, thanks for doing this! I'm a bit surprised that > > we don't need to the same suspend/resume dance in ppb(4) as with > > MSI. > > > > That is an excellent point I overlooked. Kettenis, do we? Almost certainly. I committed the diff, but left the bits out that start using it in em(4) and xhci(4).
Re: MSI-X support
On Tue, May 03, 2016 at 21:40 +0200, Mark Kettenis wrote: > Today mpi@ reminded me that I had written support for MSI-X some time > ago. Since he is interested in using multiple vectors, I extended the > code I had a bit to support that feature as well. This introduces a > new function: > > int pci_intr_map_msix(struct pci_attach_args *, int, pci_intr_handle_t *); > > You use it like pci_intr_map_msi(9) and pci_intr_map(9), buthave to > pass it a vector number as the 2nd argument. Typically you'll pass 0, > wich will map the 1st vector, which is always there on hardware with > MSI-X support. Some hardware supports more than 1 vector. Typical > examples are network cards that support multiple rings. Please be > aware that on architectures like i386 and amd64, the number of > interrupt vectors at each level is limited. If you consume too many > of them, devices may fail to attach. > > As for the implementation. This only adds code for amd64. I'll > probably add support for sparc64 as well. I'm hesitant about i386 > because it has even less available interrupt vectors than amd64. > > Notice that the amd64 implementation uses _bus_space_map() and > _bus_space_unmap(). That is deliberate. The MSI-X registers miht > share a BAR with other registers that our drivers already map. The > underscored versions of the mapping routines make sure that we don't > fail to map things in that case. > > ok? > This is great, thanks for doing this! I'm a bit surprised that we don't need to the same suspend/resume dance in ppb(4) as with MSI.
Re: MSI-X support
On Wed, May 04, 2016 at 04:01:54PM +0200, Mike Belopuhov wrote: > On Tue, May 03, 2016 at 21:40 +0200, Mark Kettenis wrote: > > Today mpi@ reminded me that I had written support for MSI-X some time > > ago. Since he is interested in using multiple vectors, I extended the > > code I had a bit to support that feature as well. This introduces a > > new function: > > > > int pci_intr_map_msix(struct pci_attach_args *, int, pci_intr_handle_t *); > > > > You use it like pci_intr_map_msi(9) and pci_intr_map(9), buthave to > > pass it a vector number as the 2nd argument. Typically you'll pass 0, > > wich will map the 1st vector, which is always there on hardware with > > MSI-X support. Some hardware supports more than 1 vector. Typical > > examples are network cards that support multiple rings. Please be > > aware that on architectures like i386 and amd64, the number of > > interrupt vectors at each level is limited. If you consume too many > > of them, devices may fail to attach. > > > > As for the implementation. This only adds code for amd64. I'll > > probably add support for sparc64 as well. I'm hesitant about i386 > > because it has even less available interrupt vectors than amd64. > > > > Notice that the amd64 implementation uses _bus_space_map() and > > _bus_space_unmap(). That is deliberate. The MSI-X registers miht > > share a BAR with other registers that our drivers already map. The > > underscored versions of the mapping routines make sure that we don't > > fail to map things in that case. > > > > ok? > > > > This is great, thanks for doing this! I'm a bit surprised that > we don't need to the same suspend/resume dance in ppb(4) as with > MSI. > That is an excellent point I overlooked. Kettenis, do we?
Re: MSI-X support
On Wed, May 04, 2016 at 08:50:53AM +0200, Mark Kettenis wrote:
> > Date: Tue, 3 May 2016 15:23:07 -0700
> > From: Mike Larkin
> >
> > On Tue, May 03, 2016 at 09:40:28PM +0200, Mark Kettenis wrote:
> > > Today mpi@ reminded me that I had written support for MSI-X some time
> > > ago. Since he is interested in using multiple vectors, I extended the
> > > code I had a bit to support that feature as well. This introduces a
> > > new function:
> > >
> > > int pci_intr_map_msix(struct pci_attach_args *, int, pci_intr_handle_t *);
> > >
> > > You use it like pci_intr_map_msi(9) and pci_intr_map(9), buthave to
> > > pass it a vector number as the 2nd argument. Typically you'll pass 0,
> > > wich will map the 1st vector, which is always there on hardware with
> > > MSI-X support. Some hardware supports more than 1 vector. Typical
> > > examples are network cards that support multiple rings. Please be
> > > aware that on architectures like i386 and amd64, the number of
> > > interrupt vectors at each level is limited. If you consume too many
> > > of them, devices may fail to attach.
> > >
> > > As for the implementation. This only adds code for amd64. I'll
> > > probably add support for sparc64 as well. I'm hesitant about i386
> > > because it has even less available interrupt vectors than amd64.
> > >
> > > Notice that the amd64 implementation uses _bus_space_map() and
> > > _bus_space_unmap(). That is deliberate. The MSI-X registers miht
> > > share a BAR with other registers that our drivers already map. The
> > > underscored versions of the mapping routines make sure that we don't
> > > fail to map things in that case.
> > >
> > > ok?
> > >
> >
> > Did you want to include the if_em and xhci bits too?
>
> Yes. They seem to work. But I can leave them out if people deem this
> too risky. They'll be separate commits anyway.
>
Just checking. ok mlarkin@ if you didn't move on this already.
> > > Index: arch/alpha/pci/pci_machdep.h
> > > ===
> > > RCS file: /cvs/src/sys/arch/alpha/pci/pci_machdep.h,v
> > > retrieving revision 1.29
> > > diff -u -p -r1.29 pci_machdep.h
> > > --- arch/alpha/pci/pci_machdep.h 26 Jul 2015 05:09:44 - 1.29
> > > +++ arch/alpha/pci/pci_machdep.h 3 May 2016 19:26:30 -
> > > @@ -104,6 +104,7 @@ int alpha_sysctl_chipset(int *, u_int, c
> > > #define pci_conf_write(c, t, r, v)
> > > \
> > > (*(c)->pc_conf_write)((c)->pc_conf_v, (t), (r), (v))
> > > #define pci_intr_map_msi(pa, ihp) (-1)
> > > +#define pci_intr_map_msix(pa, vec, ihp) (-1)
> > > #define pci_intr_string(c, ih)
> > > \
> > > (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
> > > #define pci_intr_line(c, ih)
> > > \
> > > Index: arch/amd64/include/i82093var.h
> > > ===
> > > RCS file: /cvs/src/sys/arch/amd64/include/i82093var.h,v
> > > retrieving revision 1.4
> > > diff -u -p -r1.4 i82093var.h
> > > --- arch/amd64/include/i82093var.h21 May 2011 15:58:27 -
> > > 1.4
> > > +++ arch/amd64/include/i82093var.h3 May 2016 19:26:30 -
> > > @@ -70,6 +70,7 @@ struct ioapic_softc {
> > >
> > > #define APIC_INT_VIA_APIC0x1000
> > > #define APIC_INT_VIA_MSG 0x2000
> > > +#define APIC_INT_VIA_MSGX0x4000
> > > #define APIC_INT_APIC_MASK 0x00ff
> > > #define APIC_INT_APIC_SHIFT 16
> > > #define APIC_INT_PIN_MASK0xff00
> > > Index: arch/amd64/include/pci_machdep.h
> > > ===
> > > RCS file: /cvs/src/sys/arch/amd64/include/pci_machdep.h,v
> > > retrieving revision 1.24
> > > diff -u -p -r1.24 pci_machdep.h
> > > --- arch/amd64/include/pci_machdep.h 29 Oct 2015 23:08:45 -
> > > 1.24
> > > +++ arch/amd64/include/pci_machdep.h 3 May 2016 19:26:30 -
> > > @@ -82,7 +82,10 @@ intpci_conf_size(pci_chipset_tag_t, pc
> > > pcireg_t pci_conf_read(pci_chipset_tag_t, pcit
Re: MSI-X support
On 4.5.2016. 8:50, Mark Kettenis wrote: >> Date: Tue, 3 May 2016 15:23:07 -0700 >> From: Mike Larkin >> >> On Tue, May 03, 2016 at 09:40:28PM +0200, Mark Kettenis wrote: >>> Today mpi@ reminded me that I had written support for MSI-X some time >>> ago. Since he is interested in using multiple vectors, I extended the >>> code I had a bit to support that feature as well. This introduces a >>> new function: >>> >>> int pci_intr_map_msix(struct pci_attach_args *, int, pci_intr_handle_t *); >>> >>> You use it like pci_intr_map_msi(9) and pci_intr_map(9), buthave to >>> pass it a vector number as the 2nd argument. Typically you'll pass 0, >>> wich will map the 1st vector, which is always there on hardware with >>> MSI-X support. Some hardware supports more than 1 vector. Typical >>> examples are network cards that support multiple rings. Please be >>> aware that on architectures like i386 and amd64, the number of >>> interrupt vectors at each level is limited. If you consume too many >>> of them, devices may fail to attach. >>> >>> As for the implementation. This only adds code for amd64. I'll >>> probably add support for sparc64 as well. I'm hesitant about i386 >>> because it has even less available interrupt vectors than amd64. >>> >>> Notice that the amd64 implementation uses _bus_space_map() and >>> _bus_space_unmap(). That is deliberate. The MSI-X registers miht >>> share a BAR with other registers that our drivers already map. The >>> underscored versions of the mapping routines make sure that we don't >>> fail to map things in that case. >>> >>> ok? >>> >> >> Did you want to include the if_em and xhci bits too? > > Yes. They seem to work. But I can leave them out if people deem this > too risky. They'll be separate commits anyway. Hi, it seems to work fine here ... em0 at pci9 dev 0 function 0 "Intel I350" rev 0x01: msix, address 40:f2:e9:10:56:1c em1 at pci9 dev 0 function 1 "Intel I350" rev 0x01: msix, address 40:f2:e9:10:56:1d em2 at pci9 dev 0 function 2 "Intel I350" rev 0x01: msix, address 40:f2:e9:10:56:1e em3 at pci9 dev 0 function 3 "Intel I350" rev 0x01: msix, address 40:f2:e9:10:56:1f dmesg OpenBSD 5.9-current (GENERIC.MP) #1: Tue May 3 23:41:02 CEST 2016 r...@x3550m4.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP RTC BIOS diagnostic error 80 real mem = 34314878976 (32725MB) avail mem = 33270431744 (31729MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0x7e67c000 (84 entries) bios0: vendor IBM version "-[D7E146CUS-1.82]-" date 04/09/2015 bios0: IBM IBM System x3550 M4 Server -[7914T91]- acpi0 at bios0: rev 2 acpi0: sleep states S0 S5 acpi0: tables DSDT FACP TCPA ERST HEST HPET APIC MCFG OEM0 OEM1 SLIT SRAT SLIC SSDT SSDT SSDT SSDT SSDT SSDT DMAR acpi0: wakeup devices MRP1(S4) DCC0(S4) MRP3(S4) MRP5(S4) EHC2(S5) PEX0(S5) PEX7(S5) EHC1(S5) IP2P(S3) MRPB(S4) MRPC(S4) MRPD(S4) MRPF(S4) MRPG(S4) MRPH(S4) MRPI(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 2400.40 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 2400.00 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,SENSOR,ARAT cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) cpu2: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz, 2400.00 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,
Re: MSI-X support
> Date: Tue, 3 May 2016 15:23:07 -0700
> From: Mike Larkin
>
> On Tue, May 03, 2016 at 09:40:28PM +0200, Mark Kettenis wrote:
> > Today mpi@ reminded me that I had written support for MSI-X some time
> > ago. Since he is interested in using multiple vectors, I extended the
> > code I had a bit to support that feature as well. This introduces a
> > new function:
> >
> > int pci_intr_map_msix(struct pci_attach_args *, int, pci_intr_handle_t *);
> >
> > You use it like pci_intr_map_msi(9) and pci_intr_map(9), buthave to
> > pass it a vector number as the 2nd argument. Typically you'll pass 0,
> > wich will map the 1st vector, which is always there on hardware with
> > MSI-X support. Some hardware supports more than 1 vector. Typical
> > examples are network cards that support multiple rings. Please be
> > aware that on architectures like i386 and amd64, the number of
> > interrupt vectors at each level is limited. If you consume too many
> > of them, devices may fail to attach.
> >
> > As for the implementation. This only adds code for amd64. I'll
> > probably add support for sparc64 as well. I'm hesitant about i386
> > because it has even less available interrupt vectors than amd64.
> >
> > Notice that the amd64 implementation uses _bus_space_map() and
> > _bus_space_unmap(). That is deliberate. The MSI-X registers miht
> > share a BAR with other registers that our drivers already map. The
> > underscored versions of the mapping routines make sure that we don't
> > fail to map things in that case.
> >
> > ok?
> >
>
> Did you want to include the if_em and xhci bits too?
Yes. They seem to work. But I can leave them out if people deem this
too risky. They'll be separate commits anyway.
> > Index: arch/alpha/pci/pci_machdep.h
> > ===
> > RCS file: /cvs/src/sys/arch/alpha/pci/pci_machdep.h,v
> > retrieving revision 1.29
> > diff -u -p -r1.29 pci_machdep.h
> > --- arch/alpha/pci/pci_machdep.h26 Jul 2015 05:09:44 - 1.29
> > +++ arch/alpha/pci/pci_machdep.h3 May 2016 19:26:30 -
> > @@ -104,6 +104,7 @@ int alpha_sysctl_chipset(int *, u_int, c
> > #definepci_conf_write(c, t, r, v)
> > \
> > (*(c)->pc_conf_write)((c)->pc_conf_v, (t), (r), (v))
> > #definepci_intr_map_msi(pa, ihp) (-1)
> > +#definepci_intr_map_msix(pa, vec, ihp) (-1)
> > #definepci_intr_string(c, ih)
> > \
> > (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
> > #definepci_intr_line(c, ih)
> > \
> > Index: arch/amd64/include/i82093var.h
> > ===
> > RCS file: /cvs/src/sys/arch/amd64/include/i82093var.h,v
> > retrieving revision 1.4
> > diff -u -p -r1.4 i82093var.h
> > --- arch/amd64/include/i82093var.h 21 May 2011 15:58:27 - 1.4
> > +++ arch/amd64/include/i82093var.h 3 May 2016 19:26:30 -
> > @@ -70,6 +70,7 @@ struct ioapic_softc {
> >
> > #define APIC_INT_VIA_APIC 0x1000
> > #define APIC_INT_VIA_MSG 0x2000
> > +#define APIC_INT_VIA_MSGX 0x4000
> > #define APIC_INT_APIC_MASK 0x00ff
> > #define APIC_INT_APIC_SHIFT16
> > #define APIC_INT_PIN_MASK 0xff00
> > Index: arch/amd64/include/pci_machdep.h
> > ===
> > RCS file: /cvs/src/sys/arch/amd64/include/pci_machdep.h,v
> > retrieving revision 1.24
> > diff -u -p -r1.24 pci_machdep.h
> > --- arch/amd64/include/pci_machdep.h29 Oct 2015 23:08:45 -
> > 1.24
> > +++ arch/amd64/include/pci_machdep.h3 May 2016 19:26:30 -
> > @@ -82,7 +82,10 @@ int pci_conf_size(pci_chipset_tag_t, pc
> > pcireg_t pci_conf_read(pci_chipset_tag_t, pcitag_t, int);
> > void pci_conf_write(pci_chipset_tag_t, pcitag_t, int,
> > pcireg_t);
> > -intpci_intr_map_msi(struct pci_attach_args *,
> > pci_intr_handle_t *);
> > +intpci_intr_map_msi(struct pci_attach_args *,
> > + pci_intr_handle_t *);
> > +intpci_intr_map_msix(struct pci_attach_args *,
> > + int, pci_intr_handle_t *);
> > intpci_intr_map(struct pci_attach_args *,
> > pci_intr_handle_t *);
> > const char *pci_intr_string(pci_ch
Re: MSI-X support
On Tue, May 03, 2016 at 09:40:28PM +0200, Mark Kettenis wrote:
> Today mpi@ reminded me that I had written support for MSI-X some time
> ago. Since he is interested in using multiple vectors, I extended the
> code I had a bit to support that feature as well. This introduces a
> new function:
>
> int pci_intr_map_msix(struct pci_attach_args *, int, pci_intr_handle_t *);
>
> You use it like pci_intr_map_msi(9) and pci_intr_map(9), buthave to
> pass it a vector number as the 2nd argument. Typically you'll pass 0,
> wich will map the 1st vector, which is always there on hardware with
> MSI-X support. Some hardware supports more than 1 vector. Typical
> examples are network cards that support multiple rings. Please be
> aware that on architectures like i386 and amd64, the number of
> interrupt vectors at each level is limited. If you consume too many
> of them, devices may fail to attach.
>
> As for the implementation. This only adds code for amd64. I'll
> probably add support for sparc64 as well. I'm hesitant about i386
> because it has even less available interrupt vectors than amd64.
>
> Notice that the amd64 implementation uses _bus_space_map() and
> _bus_space_unmap(). That is deliberate. The MSI-X registers miht
> share a BAR with other registers that our drivers already map. The
> underscored versions of the mapping routines make sure that we don't
> fail to map things in that case.
>
> ok?
>
Did you want to include the if_em and xhci bits too?
-ml
>
> Index: arch/alpha/pci/pci_machdep.h
> ===
> RCS file: /cvs/src/sys/arch/alpha/pci/pci_machdep.h,v
> retrieving revision 1.29
> diff -u -p -r1.29 pci_machdep.h
> --- arch/alpha/pci/pci_machdep.h 26 Jul 2015 05:09:44 - 1.29
> +++ arch/alpha/pci/pci_machdep.h 3 May 2016 19:26:30 -
> @@ -104,6 +104,7 @@ int alpha_sysctl_chipset(int *, u_int, c
> #define pci_conf_write(c, t, r, v)
> \
> (*(c)->pc_conf_write)((c)->pc_conf_v, (t), (r), (v))
> #define pci_intr_map_msi(pa, ihp) (-1)
> +#define pci_intr_map_msix(pa, vec, ihp) (-1)
> #define pci_intr_string(c, ih)
> \
> (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
> #define pci_intr_line(c, ih)
> \
> Index: arch/amd64/include/i82093var.h
> ===
> RCS file: /cvs/src/sys/arch/amd64/include/i82093var.h,v
> retrieving revision 1.4
> diff -u -p -r1.4 i82093var.h
> --- arch/amd64/include/i82093var.h21 May 2011 15:58:27 - 1.4
> +++ arch/amd64/include/i82093var.h3 May 2016 19:26:30 -
> @@ -70,6 +70,7 @@ struct ioapic_softc {
>
> #define APIC_INT_VIA_APIC0x1000
> #define APIC_INT_VIA_MSG 0x2000
> +#define APIC_INT_VIA_MSGX0x4000
> #define APIC_INT_APIC_MASK 0x00ff
> #define APIC_INT_APIC_SHIFT 16
> #define APIC_INT_PIN_MASK0xff00
> Index: arch/amd64/include/pci_machdep.h
> ===
> RCS file: /cvs/src/sys/arch/amd64/include/pci_machdep.h,v
> retrieving revision 1.24
> diff -u -p -r1.24 pci_machdep.h
> --- arch/amd64/include/pci_machdep.h 29 Oct 2015 23:08:45 - 1.24
> +++ arch/amd64/include/pci_machdep.h 3 May 2016 19:26:30 -
> @@ -82,7 +82,10 @@ intpci_conf_size(pci_chipset_tag_t, pc
> pcireg_t pci_conf_read(pci_chipset_tag_t, pcitag_t, int);
> void pci_conf_write(pci_chipset_tag_t, pcitag_t, int,
> pcireg_t);
> -int pci_intr_map_msi(struct pci_attach_args *, pci_intr_handle_t *);
> +int pci_intr_map_msi(struct pci_attach_args *,
> + pci_intr_handle_t *);
> +int pci_intr_map_msix(struct pci_attach_args *,
> + int, pci_intr_handle_t *);
> int pci_intr_map(struct pci_attach_args *, pci_intr_handle_t *);
> const char *pci_intr_string(pci_chipset_tag_t, pci_intr_handle_t);
> void *pci_intr_establish(pci_chipset_tag_t, pci_intr_handle_t,
> Index: arch/amd64/pci/pci_machdep.c
> ===
> RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
> retrieving revision 1.62
> diff -u -p -r1.62 pci_machdep.c
> --- arch/amd64/pci/pci_machdep.c 14 Mar 2015 03:38:46 - 1.62
> +++ arch/amd64/pci/pci_machdep.c 3 May 2016 19:26:30 -
> @@ -395,6 +395,143 @@ pci_intr_map_msi(struct pci_attach_args
> return 0;
> }
>
> +void msix_hwmask(struct pic *, i
MSI-X support
Today mpi@ reminded me that I had written support for MSI-X some time
ago. Since he is interested in using multiple vectors, I extended the
code I had a bit to support that feature as well. This introduces a
new function:
int pci_intr_map_msix(struct pci_attach_args *, int, pci_intr_handle_t *);
You use it like pci_intr_map_msi(9) and pci_intr_map(9), buthave to
pass it a vector number as the 2nd argument. Typically you'll pass 0,
wich will map the 1st vector, which is always there on hardware with
MSI-X support. Some hardware supports more than 1 vector. Typical
examples are network cards that support multiple rings. Please be
aware that on architectures like i386 and amd64, the number of
interrupt vectors at each level is limited. If you consume too many
of them, devices may fail to attach.
As for the implementation. This only adds code for amd64. I'll
probably add support for sparc64 as well. I'm hesitant about i386
because it has even less available interrupt vectors than amd64.
Notice that the amd64 implementation uses _bus_space_map() and
_bus_space_unmap(). That is deliberate. The MSI-X registers miht
share a BAR with other registers that our drivers already map. The
underscored versions of the mapping routines make sure that we don't
fail to map things in that case.
ok?
Index: arch/alpha/pci/pci_machdep.h
===
RCS file: /cvs/src/sys/arch/alpha/pci/pci_machdep.h,v
retrieving revision 1.29
diff -u -p -r1.29 pci_machdep.h
--- arch/alpha/pci/pci_machdep.h26 Jul 2015 05:09:44 - 1.29
+++ arch/alpha/pci/pci_machdep.h3 May 2016 19:26:30 -
@@ -104,6 +104,7 @@ int alpha_sysctl_chipset(int *, u_int, c
#definepci_conf_write(c, t, r, v)
\
(*(c)->pc_conf_write)((c)->pc_conf_v, (t), (r), (v))
#definepci_intr_map_msi(pa, ihp) (-1)
+#definepci_intr_map_msix(pa, vec, ihp) (-1)
#definepci_intr_string(c, ih)
\
(*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
#definepci_intr_line(c, ih)
\
Index: arch/amd64/include/i82093var.h
===
RCS file: /cvs/src/sys/arch/amd64/include/i82093var.h,v
retrieving revision 1.4
diff -u -p -r1.4 i82093var.h
--- arch/amd64/include/i82093var.h 21 May 2011 15:58:27 - 1.4
+++ arch/amd64/include/i82093var.h 3 May 2016 19:26:30 -
@@ -70,6 +70,7 @@ struct ioapic_softc {
#define APIC_INT_VIA_APIC 0x1000
#define APIC_INT_VIA_MSG 0x2000
+#define APIC_INT_VIA_MSGX 0x4000
#define APIC_INT_APIC_MASK 0x00ff
#define APIC_INT_APIC_SHIFT16
#define APIC_INT_PIN_MASK 0xff00
Index: arch/amd64/include/pci_machdep.h
===
RCS file: /cvs/src/sys/arch/amd64/include/pci_machdep.h,v
retrieving revision 1.24
diff -u -p -r1.24 pci_machdep.h
--- arch/amd64/include/pci_machdep.h29 Oct 2015 23:08:45 - 1.24
+++ arch/amd64/include/pci_machdep.h3 May 2016 19:26:30 -
@@ -82,7 +82,10 @@ int pci_conf_size(pci_chipset_tag_t, pc
pcireg_t pci_conf_read(pci_chipset_tag_t, pcitag_t, int);
void pci_conf_write(pci_chipset_tag_t, pcitag_t, int,
pcireg_t);
-intpci_intr_map_msi(struct pci_attach_args *, pci_intr_handle_t *);
+intpci_intr_map_msi(struct pci_attach_args *,
+ pci_intr_handle_t *);
+intpci_intr_map_msix(struct pci_attach_args *,
+ int, pci_intr_handle_t *);
intpci_intr_map(struct pci_attach_args *, pci_intr_handle_t *);
const char *pci_intr_string(pci_chipset_tag_t, pci_intr_handle_t);
void *pci_intr_establish(pci_chipset_tag_t, pci_intr_handle_t,
Index: arch/amd64/pci/pci_machdep.c
===
RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
retrieving revision 1.62
diff -u -p -r1.62 pci_machdep.c
--- arch/amd64/pci/pci_machdep.c14 Mar 2015 03:38:46 - 1.62
+++ arch/amd64/pci/pci_machdep.c3 May 2016 19:26:30 -
@@ -395,6 +395,143 @@ pci_intr_map_msi(struct pci_attach_args
return 0;
}
+void msix_hwmask(struct pic *, int);
+void msix_hwunmask(struct pic *, int);
+void msix_addroute(struct pic *, struct cpu_info *, int, int, int);
+void msix_delroute(struct pic *, struct cpu_info *, int, int, int);
+
+struct pic msix_pic = {
+ {0, {NULL}, NULL, 0, "msix", NULL, 0, 0},
+ PIC_MSI,
+#ifdef MULTIPROCESSOR
+ {},
+#endif
+ msix_hwmask,
+ msix_hwunmask,
+ msix_addroute,
+ msix_delroute,
+ NULL,
+ ioapic_edge_stubs
+};
+
+/*
+ * We pack the MSI-X vector number into the lower 8 bits of
