Re: MSI-X & Interrupting CPU > 0

Fri, 24 Jan 2020 16:58:23 -0800 

David Gwynne schreef op 2020-01-25 01:28:
On 23 Jan 2020, at 10:38 pm, Mark Kettenis <mark.kette...@xs4all.nl> wrote: 

Martin Pieuchot schreef op 2020-01-23 11:28:

I'd like to make progress towards interrupting multiple CPUs in order 
to
one day make use of multiple queues in some network drivers.  The 
road
towards that goal is consequent and I'd like to proceed in steps to 
make
it easier to squash bugs.  I'm currently thinking of the following 
steps:

1. Is my interrupt handler safe to be executed on CPU != CPU0?



Except for things that are inherently tied to a specific CPU (clock 
interrupts,
performance counters, etc) I think the answer here should always be 
"yes".


Agreed.


It probably only makes sense for mpsafe handlers to run on secondary 
CPUs though.


Only because keeping !mpsafe handlers on one CPU means they're less
likely to need to spin against other !mpsafe interrupts on other CPUs
waiting for the kernel lock before they can execute. Otherwise this
shouldn't matter.




2. Is it safe to execute this handler on two or more CPUs at the same
   time?



I think that is never safe.  Unless you run execute the handler on 
different "data".
Running multiple rx interrupt handlers on different CPUs should be 
fine.


Agreed.




3. How does interrupting multiple CPUs influence packet processing in
   the softnet thread?  Is any knowledge required (CPU affinity?) to
   have an optimum processing when multiple softnet threads are used?


I think this is my question to answer.

Packet sources (ie, rx rings) are supposed to be tied to a specific
nettq. Part of this is to avoid packet reordering where multiple
nettqs for one ring could overlap processing of packets for a single
TCP stream. The other part is so a busy nettq can apply backpressure
when it is overloaded to the rings that are feeding it.

Experience from other systems is that affinity does matter, but
running stuff in parallel matters more. Affinity between rings and
nettqs is something that can be worked on later.


4. How to split traffic in one incoming NIC between multiple 
processing

   units?



You'll need to have some sort of hardware filter that uses a hash of 
the
packet header to assign an rx queue such that all packets from a 
single "flow"
end up on the same queue and therefore will be processed by the same 
interrupt

handler.


Yep.




This new journey comes with the requirement of being able to 
interrupt

an arbitrary CPU.  For that we need a new API.  Patrick gave me the
diff below during u2k20 and I'd like to use it to start a discussion.

We currently have 6 drivers using pci_intr_map_msix().  Since we want 
to
be able to specify a CPU should we introduce a new function like in 
the
diff below or do we prefer to add a new argument (cpuid_t?) to this 
one?
This change in itself should already allow us to proceed with the 
first

item of the above list.



I'm not sure you want to have the driver pick the CPU to which to 
assign the

interrupt.  In fact I think that doesn't make sense at all.  The CPU

should be picked by more generic code instead.  But perhaps we do need 
to

pass a hint from the driver to that code.


Letting the driver pick the CPU is Good Enough(tm) today. It may limit
us to 70 or 80 percent of some theoretical maximum, but we don't have
the machinery to make a better decision on behalf of the driver at
this point. It is much better to start with something simple today
(ie, letting the driver pick the CPU) and improve on it after we hit
the limits with the simple thing.

I also look at how far dfly has got, and from what I can tell their
MSI-X stuff let's the driver pick the CPU. So it can't be too bad.




Then we need a way to read the MSI-X control table size using the 
define

PCI_MSIX_CTL_TBLSIZE() below.  This can be done in MI, we might also

want to print that information in dmesg, some maybe cache it in 
pci(4)?



There are already defines for MSIX in pcireg.h, some of which are 
duplicated
by the defines in this diff.  Don't think caching makes all that much 
sense.
Don't think we need to print the table size in dmesg; pcidump(8) 
already

prints it.  Might make sense to print the vector number though.


I'm ok with with using pcidump(8) to see what a particular device
offers rather than having it in dmesg. I'd avoid putting vectors in
dmesg output, cos if have a lot of rings there's going to be a lot of
dmesg output. Probably better to make vmstat -i more useful, or systat
mb.




Does somebody has a better/stronger/magic way to achieve this goal?



I playes a little bit with assigning interrupts to different CPUs in 
the
past, but at that point this didn't really result in a performance 
boost.
That was quite a while ago though.  I don't think there are 
fundamental problems

in getting this going.


Well, packet processing still goes through a single nettq, and that's
the limit I hit on my firewalls. I have a lot of CARP, LACP and VLAN
stuff though, so my cost per packet is probably higher than most.
However, unless your workload is tpmr(4) without any filtering, I'd be
surprised if ISR handling was the limit you're hitting.


What do you think?


I think it needs to go in so we can talk about something else.



I think using a "cpu ID" is the wrong thing though.  These IDs are 
somewhat

confusing and the lookup can be expensive.  I think having a pointer to

the struct cpuinfo * is better.  Unless maybe we're serious about 
wanting

hotplug CPU support in OpenBSD in the near future.



Index: arch/alpha/pci/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/alpha/pci/pci_machdep.h,v
retrieving revision 1.30
diff -u -p -r1.30 pci_machdep.h
--- arch/alpha/pci/pci_machdep.h        4 May 2016 14:30:00 -0000       1.30
+++ arch/alpha/pci/pci_machdep.h        23 Jan 2020 09:54:50 -0000
@@ -105,6 +105,8 @@ int alpha_sysctl_chipset(int *, u_int, c
    (*(c)->pc_conf_write)((c)->pc_conf_v, (t), (r), (v))
#define pci_intr_map_msi(pa, ihp)       (-1)
#define pci_intr_map_msix(pa, vec, ihp) (-1)
+#define        pci_intr_map_msix_cpuid(pa, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
#define pci_intr_string(c, ih)                                          \
    (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
#define pci_intr_line(c, ih)                                            \
Index: arch/amd64/amd64/acpi_machdep.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/amd64/acpi_machdep.c,v
retrieving revision 1.89
diff -u -p -r1.89 acpi_machdep.c
--- arch/amd64/amd64/acpi_machdep.c     20 Dec 2019 07:49:31 -0000      1.89
+++ arch/amd64/amd64/acpi_machdep.c     23 Jan 2020 09:54:50 -0000
@@ -194,7 +194,7 @@ acpi_intr_establish(int irq, int flags,
        type = (flags & LR_EXTIRQ_MODE) ? IST_EDGE : IST_LEVEL;
        return (intr_establish(-1, (struct pic *)apic, map->ioapic_pin,
-           type, level, handler, arg, what));
+           type, level, NULL, handler, arg, what));
#else
        return NULL;
#endif
Index: arch/amd64/amd64/intr.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/amd64/intr.c,v
retrieving revision 1.52
diff -u -p -r1.52 intr.c
--- arch/amd64/amd64/intr.c     25 Mar 2019 18:45:27 -0000      1.52
+++ arch/amd64/amd64/intr.c     23 Jan 2020 09:54:50 -0000
@@ -282,13 +282,20 @@ duplicate:
        } else {
other:
                /*
-                * Otherwise, look for a free slot elsewhere. Do the primary
-                * CPU first.
+                * Otherwise, look for a free slot elsewhere. If cip is null, it
+                * means try primary cpu but accept secondary, otherwise we need
+                * a slot on the requested cpu.
                 */
-               ci = &cpu_info_primary;
+               if (*cip == NULL)
+                       ci = &cpu_info_primary;
+               else
+                       ci = *cip;
                error = intr_allocate_slot_cpu(ci, pic, pin, &slot);
                if (error == 0)
                        goto found;
+               /* Can't alloc on the requested cpu, fail. */
+               if (*cip != NULL)
+                       return EBUSY;
                /*
                 * ..now try the others.
@@ -323,10 +330,9 @@ int        intr_shared_edge;
void *

intr_establish(int legacy_irq, struct pic *pic, int pin, int type, 
int level,

-    int (*handler)(void *), void *arg, const char *what)

+    struct cpu_info *ci, int (*handler)(void *), void *arg, const 
char *what)

{
        struct intrhand **p, *q, *ih;
-       struct cpu_info *ci;
        int slot, error, idt_vec;
        struct intrsource *source;
        struct intrstub *stubp;
Index: arch/amd64/include/intr.h
===================================================================
RCS file: /cvs/src/sys/arch/amd64/include/intr.h,v
retrieving revision 1.31
diff -u -p -r1.31 intr.h
--- arch/amd64/include/intr.h   21 Dec 2018 01:51:07 -0000      1.31
+++ arch/amd64/include/intr.h   23 Jan 2020 09:54:50 -0000
@@ -201,8 +201,8 @@ void intr_calculatemasks(struct cpu_info

int intr_allocate_slot_cpu(struct cpu_info *, struct pic *, int, int 
*);
int intr_allocate_slot(struct pic *, int, int, int, struct cpu_info 
**, int *,

            int *);

-void *intr_establish(int, struct pic *, int, int, int, int (*)(void 
*),

-           void *, const char *);
+void *intr_establish(int, struct pic *, int, int, int,
+           struct cpu_info *, int (*)(void *), void *, const char *);
void intr_disestablish(struct intrhand *);
int intr_handler(struct intrframe *, struct intrhand *);
void cpu_intr_init(struct cpu_info *);
Index: arch/amd64/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/amd64/include/pci_machdep.h,v
retrieving revision 1.28
diff -u -p -r1.28 pci_machdep.h
--- arch/amd64/include/pci_machdep.h    25 Jun 2019 16:46:32 -0000      1.28
+++ arch/amd64/include/pci_machdep.h    23 Jan 2020 09:54:50 -0000
@@ -47,10 +47,12 @@ extern struct bus_dma_tag pci_bus_dma_ta
 */
typedef void *pci_chipset_tag_t;
typedef u_int32_t pcitag_t;
+struct cpu_info;
typedef struct {
        pcitag_t tag;
        int line, pin;
+       struct cpu_info *cpu;
} pci_intr_handle_t;
#define pci_intr_line(pc,ih)    ((ih.line) & 0xff)
@@ -83,6 +85,9 @@ int           pci_intr_map_msi(struct pci_attach_
                    pci_intr_handle_t *);
int             pci_intr_map_msix(struct pci_attach_args *,
                    int, pci_intr_handle_t *);
+int            pci_intr_map_msix_cpuid(struct pci_attach_args *,
+                   int, pci_intr_handle_t *, int);
+int            pci_intr_msix_count(pci_chipset_tag_t, pcitag_t);
int             pci_intr_map(struct pci_attach_args *, pci_intr_handle_t *);
const char      *pci_intr_string(pci_chipset_tag_t, pci_intr_handle_t);
void            *pci_intr_establish(pci_chipset_tag_t, pci_intr_handle_t,
Index: arch/amd64/isa/isa_machdep.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/isa/isa_machdep.c,v
retrieving revision 1.29
diff -u -p -r1.29 isa_machdep.c
--- arch/amd64/isa/isa_machdep.c        14 Oct 2017 04:44:43 -0000      1.29
+++ arch/amd64/isa/isa_machdep.c        23 Jan 2020 09:54:50 -0000
@@ -312,7 +312,7 @@ isa_intr_establish(isa_chipset_tag_t ic,
        KASSERT(pic);
-       return intr_establish(irq, pic, pin, type, level, ih_fun,
+       return intr_establish(irq, pic, pin, type, level, NULL, ih_fun,
            ih_arg, ih_what);
}
Index: arch/amd64/pci/pci_machdep.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
retrieving revision 1.73
diff -u -p -r1.73 pci_machdep.c
--- arch/amd64/pci/pci_machdep.c        7 Sep 2019 13:46:19 -0000       1.73
+++ arch/amd64/pci/pci_machdep.c        23 Jan 2020 09:54:50 -0000
@@ -444,6 +444,7 @@ pci_intr_map_msi(struct pci_attach_args
        ihp->tag = tag;
        ihp->line = APIC_INT_VIA_MSG;
        ihp->pin = 0;
+       ihp->cpu = NULL;
        return 0;
}
@@ -549,11 +550,28 @@ msix_delroute(struct pic *pic, struct cp
}
int

-pci_intr_map_msix(struct pci_attach_args *pa, int vec, 
pci_intr_handle_t *ihp)

+pci_intr_msix_count(pci_chipset_tag_t pc, pcitag_t tag)
+{
+       pcireg_t reg;
+       int off;
+
+       if (pci_get_capability(pc, tag, PCI_CAP_MSIX, &off, NULL) == 0)
+               return (0);
+
+       reg = pci_conf_read(pc, tag, off + PCI_MSIX_CTL);
+
+       return (PCI_MSIX_CTL_TBLSIZE(reg));
+}
+
+int
+pci_intr_map_msix_cpuid(struct pci_attach_args *pa, int vec,
+    pci_intr_handle_t *ihp, int cpuid)
{
        pci_chipset_tag_t pc = pa->pa_pc;
        pcitag_t tag = pa->pa_tag;
        pcireg_t reg;
+       struct cpu_info *ci = NULL;
+       CPU_INFO_ITERATOR cii;
        KASSERT(PCI_MSIX_VEC(vec) == vec);
@@ -564,13 +582,29 @@ pci_intr_map_msix(struct pci_attach_args
        if (vec > PCI_MSIX_MC_TBLSZ(reg))
                return 1;
+       if (cpuid != -1) {
+               CPU_INFO_FOREACH(cii, ci) {
+                       if (ci->ci_cpuid == cpuid)
+                               break;
+               }
+               if (ci == NULL)
+                       return (ENOENT);
+       }
+
        ihp->tag = PCI_MSIX_PIN(tag, vec);
        ihp->line = APIC_INT_VIA_MSGX;
        ihp->pin = 0;
+       ihp->cpu = ci;
        return 0;
}
int

+pci_intr_map_msix(struct pci_attach_args *pa, int vec, 
pci_intr_handle_t *ihp)

+{
+       return (pci_intr_map_msix_cpuid(pa, vec, ihp, -1));
+}
+
+int
pci_intr_map(struct pci_attach_args *pa, pci_intr_handle_t *ihp)
{
        int pin = pa->pa_rawintrpin;
@@ -593,6 +627,7 @@ pci_intr_map(struct pci_attach_args *pa,
        ihp->tag = pa->pa_tag;
        ihp->line = line;
        ihp->pin = pin;
+       ihp->cpu = NULL;
#if NIOAPIC > 0
        pci_decompose_tag(pa->pa_pc, pa->pa_tag, &bus, &dev, &func);
@@ -734,11 +769,11 @@ pci_intr_establish(pci_chipset_tag_t pc,
        if (ih.line & APIC_INT_VIA_MSG) {
                return intr_establish(-1, &msi_pic, tag, IST_PULSE, level,
-                   func, arg, what);
+                   ih.cpu, func, arg, what);
        }
        if (ih.line & APIC_INT_VIA_MSGX) {
                return intr_establish(-1, &msix_pic, tag, IST_PULSE, level,
-                   func, arg, what);
+                   ih.cpu, func, arg, what);
        }
        pci_decompose_tag(pc, ih.tag, &bus, &dev, NULL);
@@ -764,7 +799,8 @@ pci_intr_establish(pci_chipset_tag_t pc,
        }
#endif

-	return intr_establish(irq, pic, pin, IST_LEVEL, level, func, arg, 
what);

+       return intr_establish(irq, pic, pin, IST_LEVEL, level, NULL,
+           func, arg, what);
}
void
Index: arch/arm/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/arm/include/pci_machdep.h,v
retrieving revision 1.15
diff -u -p -r1.15 pci_machdep.h
--- arch/arm/include/pci_machdep.h      9 Jul 2018 09:53:06 -0000       1.15
+++ arch/arm/include/pci_machdep.h      23 Jan 2020 10:16:58 -0000
@@ -84,6 +84,8 @@ struct arm32_pci_chipset {
    (*(c)->pa_pc->pc_intr_map_msi)((c), (ihp))
#define pci_intr_map_msix(c, vec, ihp)                                  \
    (*(c)->pa_pc->pc_intr_map_msix)((c), (vec), (ihp))
+#define        pci_intr_map_msix_cpuid(pa, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
#define pci_intr_string(c, ih)                                          \
    (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
#define pci_intr_establish(c, ih, l, h, a, nm)                          \
Index: arch/arm64/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/arm64/include/pci_machdep.h,v
retrieving revision 1.6
diff -u -p -r1.6 pci_machdep.h
--- arch/arm64/include/pci_machdep.h    25 Jun 2019 16:46:32 -0000      1.6
+++ arch/arm64/include/pci_machdep.h    23 Jan 2020 09:54:50 -0000
@@ -96,6 +96,8 @@ struct arm64_pci_chipset {
    (*(c)->pa_pc->pc_intr_map_msi)((c), (ihp))
#define pci_intr_map_msix(c, vec, ihp)                                  \
    (*(c)->pa_pc->pc_intr_map_msix)((c), (vec), (ihp))
+#define        pci_intr_map_msix_cpuid(c, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
#define pci_intr_string(c, ih)                                          \
    (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
#define pci_intr_establish(c, ih, l, h, a, nm)                          \
Index: arch/hppa/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/hppa/include/pci_machdep.h,v
retrieving revision 1.12
diff -u -p -r1.12 pci_machdep.h
--- arch/hppa/include/pci_machdep.h     4 May 2016 14:30:00 -0000       1.12
+++ arch/hppa/include/pci_machdep.h     23 Jan 2020 09:54:50 -0000
@@ -82,6 +82,8 @@ struct hppa_pci_chipset_tag {
    (*(p)->pa_pc->pc_intr_map)((p), (ihp))
#define pci_intr_map_msi(p, ihp)        (-1)
#define pci_intr_map_msix(p, vec, ihp)  (-1)
+#define        pci_intr_map_msix_cpuid(p, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
#define pci_intr_line(c, ih)    (ih)
#define pci_intr_string(c, ih)                                          \
    (*(c)->pc_intr_string)((c)->_cookie, (ih))
Index: arch/i386/pci/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/i386/pci/pci_machdep.h,v
retrieving revision 1.30
diff -u -p -r1.30 pci_machdep.h
--- arch/i386/pci/pci_machdep.h 19 Aug 2018 08:23:47 -0000      1.30
+++ arch/i386/pci/pci_machdep.h 23 Jan 2020 09:54:50 -0000
@@ -97,6 +97,8 @@ struct pci_attach_args;
int             pci_intr_map_msi(struct pci_attach_args *, pci_intr_handle_t *);
int             pci_intr_map(struct pci_attach_args *, pci_intr_handle_t *);
#define         pci_intr_map_msix(p, vec, ihp)  (-1)
+#define                pci_intr_map_msix_cpuid(p, vec, ihp, cpu) (-1)
+#define                pci_intr_msix_count(c, t)       (0)
#define         pci_intr_line(c, ih)    ((ih).line)
const char      *pci_intr_string(pci_chipset_tag_t, pci_intr_handle_t);
void            *pci_intr_establish(pci_chipset_tag_t, pci_intr_handle_t,
Index: arch/landisk/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/landisk/include/pci_machdep.h,v
retrieving revision 1.10
diff -u -p -r1.10 pci_machdep.h
--- arch/landisk/include/pci_machdep.h  4 May 2016 14:30:01 -0000       1.10
+++ arch/landisk/include/pci_machdep.h  23 Jan 2020 09:54:50 -0000
@@ -77,6 +77,8 @@ void landisk_pci_conf_interrupt(void *v,
        landisk_pci_intr_map(pa, ihp)
#define pci_intr_map_msi(pa, ihp)       (-1)
#define pci_intr_map_msix(pa, vec, ihp) (-1)
+#define        pci_intr_map_msix_cpuid(pa, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
#define pci_intr_string(v, ih) \
        landisk_pci_intr_string(v, ih)
#define pci_intr_establish(v, ih, level, ih_fun, ih_arg, ih_name) \
Index: arch/loongson/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/loongson/include/pci_machdep.h,v
retrieving revision 1.10
diff -u -p -r1.10 pci_machdep.h

--- arch/loongson/include/pci_machdep.h	4 May 2016 14:30:01 
-0000	1.10

+++ arch/loongson/include/pci_machdep.h 23 Jan 2020 09:54:50 -0000
@@ -77,6 +77,8 @@ struct mips_pci_chipset {
    (*(c)->pa_pc->pc_intr_map)((c), (ihp))
#define pci_intr_map_msi(c, ihp)        (-1)
#define pci_intr_map_msix(c, vec, ihp)  (-1)
+#define        pci_intr_map_msix_cpuid(c, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
#define pci_intr_string(c, ih)                                          \
    (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
#define pci_intr_establish(c, ih, l, h, a, nm)                          \
Index: arch/macppc/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/macppc/include/pci_machdep.h,v
retrieving revision 1.4
diff -u -p -r1.4 pci_machdep.h
--- arch/macppc/include/pci_machdep.h   5 Dec 2019 12:46:54 -0000       1.4
+++ arch/macppc/include/pci_machdep.h   23 Jan 2020 09:54:50 -0000
@@ -84,6 +84,8 @@ void          pci_conf_write(pci_chipset_tag_t,
int             pci_intr_map(struct pci_attach_args *, pci_intr_handle_t *);
int             pci_intr_map_msi(struct pci_attach_args *, pci_intr_handle_t *);
#define         pci_intr_map_msix(p, vec, ihp)  (-1)
+#define                pci_intr_map_msix_cpuid(p, vec, ihp, cpu) (-1)
+#define                pci_intr_msix_count(c, t)       (0)
int             pci_intr_line(pci_chipset_tag_t, pci_intr_handle_t);
const char      *pci_intr_string(pci_chipset_tag_t, pci_intr_handle_t);
void            *pci_intr_establish(pci_chipset_tag_t, pci_intr_handle_t,
Index: arch/octeon/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/octeon/include/pci_machdep.h,v
retrieving revision 1.10
diff -u -p -r1.10 pci_machdep.h
--- arch/octeon/include/pci_machdep.h   18 Jun 2018 13:54:03 -0000      1.10
+++ arch/octeon/include/pci_machdep.h   23 Jan 2020 09:54:50 -0000
@@ -115,6 +115,8 @@ pci_conf_write_db(void *cookie, pcitag_t
    (*(c)->pa_pc->pc_intr_map)((c), (ihp))
#define pci_intr_map_msi(c, ihp)        (-1)
#define pci_intr_map_msix(c, vec, ihp)  (-1)
+#define        pci_intr_map_msix_cpuid(c, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
#define pci_intr_string(c, ih)                                          \
    (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
#define pci_intr_establish(c, ih, l, h, a, nm)                          \
Index: arch/sgi/pci/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/sgi/pci/pci_machdep.h,v
retrieving revision 1.17
diff -u -p -r1.17 pci_machdep.h
--- arch/sgi/pci/pci_machdep.h  4 May 2016 14:30:01 -0000       1.17
+++ arch/sgi/pci/pci_machdep.h  23 Jan 2020 09:54:50 -0000
@@ -88,6 +88,8 @@ struct mips_pci_chipset {
    (*(c)->pa_pc->pc_intr_map)((c), (ihp))
#define pci_intr_map_msi(c, ihp)        (-1)
#define pci_intr_map_msix(c, vec, ihp)  (-1)
+#define        pci_intr_map_msix_cpuid(c, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
#define pci_intr_string(c, ih)                                          \
    (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
#define pci_intr_establish(c, ih, l, h, a, nm)                          \
Index: arch/sparc64/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/sparc64/include/pci_machdep.h,v
retrieving revision 1.36
diff -u -p -r1.36 pci_machdep.h
--- arch/sparc64/include/pci_machdep.h  5 Dec 2019 12:46:54 -0000       1.36
+++ arch/sparc64/include/pci_machdep.h  23 Jan 2020 10:17:46 -0000
@@ -107,6 +107,9 @@ void                *pci_intr_establish(pci_chipset_ta
                                 int, int (*)(void *), void *, const char *);
void            pci_intr_disestablish(pci_chipset_tag_t, void *);
+#define                pci_intr_map_msix_cpuid(pa, vec, ihp, cpu)      (-1)
+#define                pci_intr_msix_count(c, t)                       (0)
+
void            pci_msi_enable(pci_chipset_tag_t, pcitag_t, bus_addr_t, int);
void            pci_msix_enable(pci_chipset_tag_t, pcitag_t, bus_space_tag_t,
                    int, bus_addr_t, uint32_t);
Index: dev/pci/pcireg.h
===================================================================
RCS file: /cvs/src/sys/dev/pci/pcireg.h,v
retrieving revision 1.59
diff -u -p -r1.59 pcireg.h
--- dev/pci/pcireg.h    2 Nov 2019 10:14:57 -0000       1.59
+++ dev/pci/pcireg.h    23 Jan 2020 09:55:15 -0000
@@ -624,6 +624,19 @@ typedef u_int8_t pci_revision_t;
#define PCI_PCIE_LCSR2_TLS_32   0x00000005
/*
+ * Capability ID: 0x11
+ * MSIX
+ */
+
+#define        PCI_MSIX_CTL                    0x00
+#define        PCI_MSIX_CTL_ENABLE             0x80000000
+#define        PCI_MSIX_CTL_FUNCMASK           0x40000000
+#define        PCI_MSIX_CTL_TBLSIZE_MASK       0x07ff0000
+#define        PCI_MSIX_CTL_TBLSIZE_SHIFT      16
+#define        PCI_MSIX_CTL_TBLSIZE(ofs)       \

+	((((ofs) & PCI_MSIX_CTL_TBLSIZE_MASK) >> 
PCI_MSIX_CTL_TBLSIZE_SHIFT) + 1)

+
+/*
 * PCI Express; enhanced capabilities
 */
#define PCI_PCIE_ECAP           0x100
























					Reply via email to