Re: MSI-X & Interrupting CPU > 0

Thu, 23 Jan 2020 05:16:28 -0800 

Martin Pieuchot schreef op 2020-01-23 11:28:
I'd like to make progress towards interrupting multiple CPUs in order to 
one day make use of multiple queues in some network drivers.  The road
towards that goal is consequent and I'd like to proceed in steps to make it easier to squash bugs. I'm currently thinking of the following steps: 

 1. Is my interrupt handler safe to be executed on CPU != CPU0?



Except for things that are inherently tied to a specific CPU (clock 
interrupts,
performance counters, etc) I think the answer here should always be 
"yes".
It probably only makes sense for mpsafe handlers to run on secondary 
CPUs though.



 2. Is it safe to execute this handler on two or more CPUs at the same
    time?



I think that is never safe.  Unless you run execute the handler on 
different "data".

Running multiple rx interrupt handlers on different CPUs should be fine.


 3. How does interrupting multiple CPUs influence packet processing in
    the softnet thread?  Is any knowledge required (CPU affinity?) to
    have an optimum processing when multiple softnet threads are used?


 4. How to split traffic in one incoming NIC between multiple 
processing

    units?


You'll need to have some sort of hardware filter that uses a hash of the

packet header to assign an rx queue such that all packets from a single 
"flow"
end up on the same queue and therefore will be processed by the same 
interrupt

handler.


This new journey comes with the requirement of being able to interrupt
an arbitrary CPU.  For that we need a new API.  Patrick gave me the
diff below during u2k20 and I'd like to use it to start a discussion.


We currently have 6 drivers using pci_intr_map_msix().  Since we want 
to

be able to specify a CPU should we introduce a new function like in the

diff below or do we prefer to add a new argument (cpuid_t?) to this 
one?

This change in itself should already allow us to proceed with the first
item of the above list.



I'm not sure you want to have the driver pick the CPU to which to assign 
the

interrupt.  In fact I think that doesn't make sense at all.  The CPU

should be picked by more generic code instead.  But perhaps we do need 
to

pass a hint from the driver to that code.


Then we need a way to read the MSI-X control table size using the 
define

PCI_MSIX_CTL_TBLSIZE() below.  This can be done in MI, we might also
want to print that information in dmesg, some maybe cache it in pci(4)?



There are already defines for MSIX in pcireg.h, some of which are 
duplicated
by the defines in this diff.  Don't think caching makes all that much 
sense.

Don't think we need to print the table size in dmesg; pcidump(8) already
prints it.  Might make sense to print the vector number though.


Does somebody has a better/stronger/magic way to achieve this goal?


I playes a little bit with assigning interrupts to different CPUs in the

past, but at that point this didn't really result in a performance 
boost.
That was quite a while ago though.  I don't think there are fundamental 
problems

in getting this going.


What do you think?

Index: arch/alpha/pci/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/alpha/pci/pci_machdep.h,v
retrieving revision 1.30
diff -u -p -r1.30 pci_machdep.h
--- arch/alpha/pci/pci_machdep.h        4 May 2016 14:30:00 -0000       1.30
+++ arch/alpha/pci/pci_machdep.h        23 Jan 2020 09:54:50 -0000
@@ -105,6 +105,8 @@ int alpha_sysctl_chipset(int *, u_int, c
     (*(c)->pc_conf_write)((c)->pc_conf_v, (t), (r), (v))
 #define        pci_intr_map_msi(pa, ihp)       (-1)
 #define        pci_intr_map_msix(pa, vec, ihp) (-1)
+#define        pci_intr_map_msix_cpuid(pa, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
 #define        pci_intr_string(c, ih)                                          
\
     (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
 #define        pci_intr_line(c, ih)                                            
\
Index: arch/amd64/amd64/acpi_machdep.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/amd64/acpi_machdep.c,v
retrieving revision 1.89
diff -u -p -r1.89 acpi_machdep.c
--- arch/amd64/amd64/acpi_machdep.c     20 Dec 2019 07:49:31 -0000      1.89
+++ arch/amd64/amd64/acpi_machdep.c     23 Jan 2020 09:54:50 -0000
@@ -194,7 +194,7 @@ acpi_intr_establish(int irq, int flags,

        type = (flags & LR_EXTIRQ_MODE) ? IST_EDGE : IST_LEVEL;
        return (intr_establish(-1, (struct pic *)apic, map->ioapic_pin,
-           type, level, handler, arg, what));
+           type, level, NULL, handler, arg, what));
 #else
        return NULL;
 #endif
Index: arch/amd64/amd64/intr.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/amd64/intr.c,v
retrieving revision 1.52
diff -u -p -r1.52 intr.c
--- arch/amd64/amd64/intr.c     25 Mar 2019 18:45:27 -0000      1.52
+++ arch/amd64/amd64/intr.c     23 Jan 2020 09:54:50 -0000
@@ -282,13 +282,20 @@ duplicate:
        } else {
 other:
                /*
-                * Otherwise, look for a free slot elsewhere. Do the primary
-                * CPU first.
+                * Otherwise, look for a free slot elsewhere. If cip is null, it
+                * means try primary cpu but accept secondary, otherwise we need
+                * a slot on the requested cpu.
                 */
-               ci = &cpu_info_primary;
+               if (*cip == NULL)
+                       ci = &cpu_info_primary;
+               else
+                       ci = *cip;
                error = intr_allocate_slot_cpu(ci, pic, pin, &slot);
                if (error == 0)
                        goto found;
+               /* Can't alloc on the requested cpu, fail. */
+               if (*cip != NULL)
+                       return EBUSY;

                /*
                 * ..now try the others.
@@ -323,10 +330,9 @@ int        intr_shared_edge;

 void *

 intr_establish(int legacy_irq, struct pic *pic, int pin, int type, int 
level,

-    int (*handler)(void *), void *arg, const char *what)

+    struct cpu_info *ci, int (*handler)(void *), void *arg, const char 
*what)

 {
        struct intrhand **p, *q, *ih;
-       struct cpu_info *ci;
        int slot, error, idt_vec;
        struct intrsource *source;
        struct intrstub *stubp;
Index: arch/amd64/include/intr.h
===================================================================
RCS file: /cvs/src/sys/arch/amd64/include/intr.h,v
retrieving revision 1.31
diff -u -p -r1.31 intr.h
--- arch/amd64/include/intr.h   21 Dec 2018 01:51:07 -0000      1.31
+++ arch/amd64/include/intr.h   23 Jan 2020 09:54:50 -0000
@@ -201,8 +201,8 @@ void intr_calculatemasks(struct cpu_info

 int intr_allocate_slot_cpu(struct cpu_info *, struct pic *, int, int 
*);
 int intr_allocate_slot(struct pic *, int, int, int, struct cpu_info 
**, int *,

            int *);

-void *intr_establish(int, struct pic *, int, int, int, int (*)(void 
*),

-           void *, const char *);
+void *intr_establish(int, struct pic *, int, int, int,
+           struct cpu_info *, int (*)(void *), void *, const char *);
 void intr_disestablish(struct intrhand *);
 int intr_handler(struct intrframe *, struct intrhand *);
 void cpu_intr_init(struct cpu_info *);
Index: arch/amd64/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/amd64/include/pci_machdep.h,v
retrieving revision 1.28
diff -u -p -r1.28 pci_machdep.h
--- arch/amd64/include/pci_machdep.h    25 Jun 2019 16:46:32 -0000      1.28
+++ arch/amd64/include/pci_machdep.h    23 Jan 2020 09:54:50 -0000
@@ -47,10 +47,12 @@ extern struct bus_dma_tag pci_bus_dma_ta
  */
 typedef void *pci_chipset_tag_t;
 typedef u_int32_t pcitag_t;
+struct cpu_info;

 typedef struct {
        pcitag_t tag;
        int line, pin;
+       struct cpu_info *cpu;
 } pci_intr_handle_t;

 #define        pci_intr_line(pc,ih)    ((ih.line) & 0xff)
@@ -83,6 +85,9 @@ int           pci_intr_map_msi(struct pci_attach_
                    pci_intr_handle_t *);
 int            pci_intr_map_msix(struct pci_attach_args *,
                    int, pci_intr_handle_t *);
+int            pci_intr_map_msix_cpuid(struct pci_attach_args *,
+                   int, pci_intr_handle_t *, int);
+int            pci_intr_msix_count(pci_chipset_tag_t, pcitag_t);
 int            pci_intr_map(struct pci_attach_args *, pci_intr_handle_t *);
 const char     *pci_intr_string(pci_chipset_tag_t, pci_intr_handle_t);
 void           *pci_intr_establish(pci_chipset_tag_t, pci_intr_handle_t,
Index: arch/amd64/isa/isa_machdep.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/isa/isa_machdep.c,v
retrieving revision 1.29
diff -u -p -r1.29 isa_machdep.c
--- arch/amd64/isa/isa_machdep.c        14 Oct 2017 04:44:43 -0000      1.29
+++ arch/amd64/isa/isa_machdep.c        23 Jan 2020 09:54:50 -0000
@@ -312,7 +312,7 @@ isa_intr_establish(isa_chipset_tag_t ic,

        KASSERT(pic);

-       return intr_establish(irq, pic, pin, type, level, ih_fun,
+       return intr_establish(irq, pic, pin, type, level, NULL, ih_fun,
            ih_arg, ih_what);
 }

Index: arch/amd64/pci/pci_machdep.c
===================================================================
RCS file: /cvs/src/sys/arch/amd64/pci/pci_machdep.c,v
retrieving revision 1.73
diff -u -p -r1.73 pci_machdep.c
--- arch/amd64/pci/pci_machdep.c        7 Sep 2019 13:46:19 -0000       1.73
+++ arch/amd64/pci/pci_machdep.c        23 Jan 2020 09:54:50 -0000
@@ -444,6 +444,7 @@ pci_intr_map_msi(struct pci_attach_args
        ihp->tag = tag;
        ihp->line = APIC_INT_VIA_MSG;
        ihp->pin = 0;
+       ihp->cpu = NULL;
        return 0;
 }

@@ -549,11 +550,28 @@ msix_delroute(struct pic *pic, struct cp
 }

 int

-pci_intr_map_msix(struct pci_attach_args *pa, int vec, 
pci_intr_handle_t *ihp)

+pci_intr_msix_count(pci_chipset_tag_t pc, pcitag_t tag)
+{
+       pcireg_t reg;
+       int off;
+
+       if (pci_get_capability(pc, tag, PCI_CAP_MSIX, &off, NULL) == 0)
+               return (0);
+
+       reg = pci_conf_read(pc, tag, off + PCI_MSIX_CTL);
+
+       return (PCI_MSIX_CTL_TBLSIZE(reg));
+}
+
+int
+pci_intr_map_msix_cpuid(struct pci_attach_args *pa, int vec,
+    pci_intr_handle_t *ihp, int cpuid)
 {
        pci_chipset_tag_t pc = pa->pa_pc;
        pcitag_t tag = pa->pa_tag;
        pcireg_t reg;
+       struct cpu_info *ci = NULL;
+       CPU_INFO_ITERATOR cii;

        KASSERT(PCI_MSIX_VEC(vec) == vec);

@@ -564,13 +582,29 @@ pci_intr_map_msix(struct pci_attach_args
        if (vec > PCI_MSIX_MC_TBLSZ(reg))
                return 1;

+       if (cpuid != -1) {
+               CPU_INFO_FOREACH(cii, ci) {
+                       if (ci->ci_cpuid == cpuid)
+                               break;
+               }
+               if (ci == NULL)
+                       return (ENOENT);
+       }
+
        ihp->tag = PCI_MSIX_PIN(tag, vec);
        ihp->line = APIC_INT_VIA_MSGX;
        ihp->pin = 0;
+       ihp->cpu = ci;
        return 0;
 }

 int

+pci_intr_map_msix(struct pci_attach_args *pa, int vec, 
pci_intr_handle_t *ihp)

+{
+       return (pci_intr_map_msix_cpuid(pa, vec, ihp, -1));
+}
+
+int
 pci_intr_map(struct pci_attach_args *pa, pci_intr_handle_t *ihp)
 {
        int pin = pa->pa_rawintrpin;
@@ -593,6 +627,7 @@ pci_intr_map(struct pci_attach_args *pa,
        ihp->tag = pa->pa_tag;
        ihp->line = line;
        ihp->pin = pin;
+       ihp->cpu = NULL;

 #if NIOAPIC > 0
        pci_decompose_tag(pa->pa_pc, pa->pa_tag, &bus, &dev, &func);
@@ -734,11 +769,11 @@ pci_intr_establish(pci_chipset_tag_t pc,

        if (ih.line & APIC_INT_VIA_MSG) {
                return intr_establish(-1, &msi_pic, tag, IST_PULSE, level,
-                   func, arg, what);
+                   ih.cpu, func, arg, what);
        }
        if (ih.line & APIC_INT_VIA_MSGX) {
                return intr_establish(-1, &msix_pic, tag, IST_PULSE, level,
-                   func, arg, what);
+                   ih.cpu, func, arg, what);
        }

        pci_decompose_tag(pc, ih.tag, &bus, &dev, NULL);
@@ -764,7 +799,8 @@ pci_intr_establish(pci_chipset_tag_t pc,
        }
 #endif


-	return intr_establish(irq, pic, pin, IST_LEVEL, level, func, arg, 
what);

+       return intr_establish(irq, pic, pin, IST_LEVEL, level, NULL,
+           func, arg, what);
 }

 void
Index: arch/arm/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/arm/include/pci_machdep.h,v
retrieving revision 1.15
diff -u -p -r1.15 pci_machdep.h
--- arch/arm/include/pci_machdep.h      9 Jul 2018 09:53:06 -0000       1.15
+++ arch/arm/include/pci_machdep.h      23 Jan 2020 10:16:58 -0000
@@ -84,6 +84,8 @@ struct arm32_pci_chipset {
     (*(c)->pa_pc->pc_intr_map_msi)((c), (ihp))
 #define        pci_intr_map_msix(c, vec, ihp)                                  
\
     (*(c)->pa_pc->pc_intr_map_msix)((c), (vec), (ihp))
+#define        pci_intr_map_msix_cpuid(pa, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
 #define        pci_intr_string(c, ih)                                          
\
     (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
 #define        pci_intr_establish(c, ih, l, h, a, nm)                          
\
Index: arch/arm64/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/arm64/include/pci_machdep.h,v
retrieving revision 1.6
diff -u -p -r1.6 pci_machdep.h
--- arch/arm64/include/pci_machdep.h    25 Jun 2019 16:46:32 -0000      1.6
+++ arch/arm64/include/pci_machdep.h    23 Jan 2020 09:54:50 -0000
@@ -96,6 +96,8 @@ struct arm64_pci_chipset {
     (*(c)->pa_pc->pc_intr_map_msi)((c), (ihp))
 #define        pci_intr_map_msix(c, vec, ihp)                                  
\
     (*(c)->pa_pc->pc_intr_map_msix)((c), (vec), (ihp))
+#define        pci_intr_map_msix_cpuid(c, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
 #define        pci_intr_string(c, ih)                                          
\
     (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
 #define        pci_intr_establish(c, ih, l, h, a, nm)                          
\
Index: arch/hppa/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/hppa/include/pci_machdep.h,v
retrieving revision 1.12
diff -u -p -r1.12 pci_machdep.h
--- arch/hppa/include/pci_machdep.h     4 May 2016 14:30:00 -0000       1.12
+++ arch/hppa/include/pci_machdep.h     23 Jan 2020 09:54:50 -0000
@@ -82,6 +82,8 @@ struct hppa_pci_chipset_tag {
     (*(p)->pa_pc->pc_intr_map)((p), (ihp))
 #define        pci_intr_map_msi(p, ihp)        (-1)
 #define        pci_intr_map_msix(p, vec, ihp)  (-1)
+#define        pci_intr_map_msix_cpuid(p, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
 #define        pci_intr_line(c, ih)    (ih)
 #define        pci_intr_string(c, ih)                                          
\
     (*(c)->pc_intr_string)((c)->_cookie, (ih))
Index: arch/i386/pci/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/i386/pci/pci_machdep.h,v
retrieving revision 1.30
diff -u -p -r1.30 pci_machdep.h
--- arch/i386/pci/pci_machdep.h 19 Aug 2018 08:23:47 -0000      1.30
+++ arch/i386/pci/pci_machdep.h 23 Jan 2020 09:54:50 -0000
@@ -97,6 +97,8 @@ struct pci_attach_args;
 int            pci_intr_map_msi(struct pci_attach_args *, pci_intr_handle_t *);
 int            pci_intr_map(struct pci_attach_args *, pci_intr_handle_t *);
 #define                pci_intr_map_msix(p, vec, ihp)  (-1)
+#define                pci_intr_map_msix_cpuid(p, vec, ihp, cpu) (-1)
+#define                pci_intr_msix_count(c, t)       (0)
 #define                pci_intr_line(c, ih)    ((ih).line)
 const char     *pci_intr_string(pci_chipset_tag_t, pci_intr_handle_t);
 void           *pci_intr_establish(pci_chipset_tag_t, pci_intr_handle_t,
Index: arch/landisk/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/landisk/include/pci_machdep.h,v
retrieving revision 1.10
diff -u -p -r1.10 pci_machdep.h
--- arch/landisk/include/pci_machdep.h  4 May 2016 14:30:01 -0000       1.10
+++ arch/landisk/include/pci_machdep.h  23 Jan 2020 09:54:50 -0000
@@ -77,6 +77,8 @@ void landisk_pci_conf_interrupt(void *v,
        landisk_pci_intr_map(pa, ihp)
 #define        pci_intr_map_msi(pa, ihp)       (-1)
 #define        pci_intr_map_msix(pa, vec, ihp) (-1)
+#define        pci_intr_map_msix_cpuid(pa, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
 #define        pci_intr_string(v, ih) \
        landisk_pci_intr_string(v, ih)
 #define        pci_intr_establish(v, ih, level, ih_fun, ih_arg, ih_name) \
Index: arch/loongson/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/loongson/include/pci_machdep.h,v
retrieving revision 1.10
diff -u -p -r1.10 pci_machdep.h
--- arch/loongson/include/pci_machdep.h 4 May 2016 14:30:01 -0000       1.10
+++ arch/loongson/include/pci_machdep.h 23 Jan 2020 09:54:50 -0000
@@ -77,6 +77,8 @@ struct mips_pci_chipset {
     (*(c)->pa_pc->pc_intr_map)((c), (ihp))
 #define        pci_intr_map_msi(c, ihp)        (-1)
 #define        pci_intr_map_msix(c, vec, ihp)  (-1)
+#define        pci_intr_map_msix_cpuid(c, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
 #define        pci_intr_string(c, ih)                                          
\
     (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
 #define        pci_intr_establish(c, ih, l, h, a, nm)                          
\
Index: arch/macppc/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/macppc/include/pci_machdep.h,v
retrieving revision 1.4
diff -u -p -r1.4 pci_machdep.h
--- arch/macppc/include/pci_machdep.h   5 Dec 2019 12:46:54 -0000       1.4
+++ arch/macppc/include/pci_machdep.h   23 Jan 2020 09:54:50 -0000
@@ -84,6 +84,8 @@ void          pci_conf_write(pci_chipset_tag_t,
 int            pci_intr_map(struct pci_attach_args *, pci_intr_handle_t *);
 int            pci_intr_map_msi(struct pci_attach_args *, pci_intr_handle_t *);
 #define                pci_intr_map_msix(p, vec, ihp)  (-1)
+#define                pci_intr_map_msix_cpuid(p, vec, ihp, cpu) (-1)
+#define                pci_intr_msix_count(c, t)       (0)
 int            pci_intr_line(pci_chipset_tag_t, pci_intr_handle_t);
 const char     *pci_intr_string(pci_chipset_tag_t, pci_intr_handle_t);
 void           *pci_intr_establish(pci_chipset_tag_t, pci_intr_handle_t,
Index: arch/octeon/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/octeon/include/pci_machdep.h,v
retrieving revision 1.10
diff -u -p -r1.10 pci_machdep.h
--- arch/octeon/include/pci_machdep.h   18 Jun 2018 13:54:03 -0000      1.10
+++ arch/octeon/include/pci_machdep.h   23 Jan 2020 09:54:50 -0000
@@ -115,6 +115,8 @@ pci_conf_write_db(void *cookie, pcitag_t
     (*(c)->pa_pc->pc_intr_map)((c), (ihp))
 #define        pci_intr_map_msi(c, ihp)        (-1)
 #define        pci_intr_map_msix(c, vec, ihp)  (-1)
+#define        pci_intr_map_msix_cpuid(c, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
 #define        pci_intr_string(c, ih)                                          
\
     (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
 #define        pci_intr_establish(c, ih, l, h, a, nm)                          
\
Index: arch/sgi/pci/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/sgi/pci/pci_machdep.h,v
retrieving revision 1.17
diff -u -p -r1.17 pci_machdep.h
--- arch/sgi/pci/pci_machdep.h  4 May 2016 14:30:01 -0000       1.17
+++ arch/sgi/pci/pci_machdep.h  23 Jan 2020 09:54:50 -0000
@@ -88,6 +88,8 @@ struct mips_pci_chipset {
     (*(c)->pa_pc->pc_intr_map)((c), (ihp))
 #define        pci_intr_map_msi(c, ihp)        (-1)
 #define        pci_intr_map_msix(c, vec, ihp)  (-1)
+#define        pci_intr_map_msix_cpuid(c, vec, ihp, cpu) (-1)
+#define        pci_intr_msix_count(c, t)       (0)
 #define        pci_intr_string(c, ih)                                          
\
     (*(c)->pc_intr_string)((c)->pc_intr_v, (ih))
 #define        pci_intr_establish(c, ih, l, h, a, nm)                          
\
Index: arch/sparc64/include/pci_machdep.h
===================================================================
RCS file: /cvs/src/sys/arch/sparc64/include/pci_machdep.h,v
retrieving revision 1.36
diff -u -p -r1.36 pci_machdep.h
--- arch/sparc64/include/pci_machdep.h  5 Dec 2019 12:46:54 -0000       1.36
+++ arch/sparc64/include/pci_machdep.h  23 Jan 2020 10:17:46 -0000
@@ -107,6 +107,9 @@ void                *pci_intr_establish(pci_chipset_ta
                                 int, int (*)(void *), void *, const char *);
 void           pci_intr_disestablish(pci_chipset_tag_t, void *);

+#define                pci_intr_map_msix_cpuid(pa, vec, ihp, cpu)      (-1)
+#define                pci_intr_msix_count(c, t)                       (0)
+
 void           pci_msi_enable(pci_chipset_tag_t, pcitag_t, bus_addr_t, int);
 void           pci_msix_enable(pci_chipset_tag_t, pcitag_t, bus_space_tag_t,
                    int, bus_addr_t, uint32_t);
Index: dev/pci/pcireg.h
===================================================================
RCS file: /cvs/src/sys/dev/pci/pcireg.h,v
retrieving revision 1.59
diff -u -p -r1.59 pcireg.h
--- dev/pci/pcireg.h    2 Nov 2019 10:14:57 -0000       1.59
+++ dev/pci/pcireg.h    23 Jan 2020 09:55:15 -0000
@@ -624,6 +624,19 @@ typedef u_int8_t pci_revision_t;
 #define PCI_PCIE_LCSR2_TLS_32  0x00000005

 /*
+ * Capability ID: 0x11
+ * MSIX
+ */
+
+#define        PCI_MSIX_CTL                    0x00
+#define        PCI_MSIX_CTL_ENABLE             0x80000000
+#define        PCI_MSIX_CTL_FUNCMASK           0x40000000
+#define        PCI_MSIX_CTL_TBLSIZE_MASK       0x07ff0000
+#define        PCI_MSIX_CTL_TBLSIZE_SHIFT      16
+#define        PCI_MSIX_CTL_TBLSIZE(ofs)       \

+	((((ofs) & PCI_MSIX_CTL_TBLSIZE_MASK) >> PCI_MSIX_CTL_TBLSIZE_SHIFT) 
+ 1)

+
+/*
  * PCI Express; enhanced capabilities
  */
 #define PCI_PCIE_ECAP          0x100
























					Reply via email to