tl;dr: pcie access = latency it's interesting to take a look at irq latency for a number of devices on different machines. it looks like the experimental device --- the interrupt does nothing --- gives us a lower bound for irq latency, which works out to be 36ns (!). clearly this doesn't count dispatch, or the overhead of trap().
m# aux/cpuid -i Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz vector.mach bus irq count sum(cycles) cycles/call type name m# /usr/quanstro/bin/rc/irqlatency 66.0 11 24605 3143914 127.78 msi-x experimental 50.0 50 348487 802964226 2304.14 lapic APIC timer 50.6 50 324950 841831354 2590.65 lapic APIC timer 50.4 50 319746 832771958 2604.48 lapic APIC timer 50.2 50 323061 846873676 2621.40 lapic APIC timer 50.3 50 323080 867020068 2683.61 lapic APIC timer 50.5 50 323056 886385231 2743.75 lapic APIC timer 50.7 50 324165 905699976 2793.95 lapic APIC timer 50.1 50 323519 997341922 3082.79 lapic APIC timer 65.0 11 4383 45452712 10370.23 msi-x ether0 66.5 5 66222 777537556 11741.38 ioapic usbehci 65.7 11 9 247991 27554.56 msi sdF (ahci) 65.1 4 1084 38400923 35425.21 ioapic COM1 ; aux/cpuid -i AMD Phenom(tm) II X4 965 Processor ; irqlatency 50.3 50 83106879 109878207918 1322.13 lapic APIC timer 50.2 50 83070335 109844976915 1322.31 lapic APIC timer 50.1 50 83228230 110448926024 1327.06 lapic APIC timer 50.0 50 90772899 169367434384 1865.84 lapic APIC timer 65.2 10 1179147 11639594649 9871.20 msi-x ether0 65.0 1 17657 178037258 10083.10 ioapic i8042 68.2 12 312539 3476219928 11122.52 ioapic kbdaux 66.0 4 911 21026234 23080.39 ioapic COM1 71.1 11 58 1780132 30691.93 msi sdE (ahci) 68.1 10 1 7934988 7934988.00 ioapic usbohci given the efficency of locks, and the same ethernet devices (i211), we can reason that since the latency is prop. to the frequency, 10370*3.4/3.5 = 10073.71428571 the latency must largely be pcie register reads and writes. sadly, the msi-x interrupt doesn't handle single causes, otherwise no register access would be necessary. - erik ps. interesting question: why do the lapic timers take longer on the faster machine? that's double the latency. is this an intel thing? (note this is a different ethernet chipset, and a different driver so the comparison with the other two is bogus.) lilly; aux/cpuid -i Intel(R) Atom(TM) CPU D525 @ 1.80GHz lilly; irqlatency 50.3 50 802180313 1705456340928 2126.03 lapic APIC timer 50.1 50 803988180 1811997635679 2253.76 lapic APIC timer 50.2 50 832505288 1914666440067 2299.89 lapic APIC timer 50.0 50 830861969 2509217805168 3020.02 lapic APIC timer 70.2 15 2 16587 8293.50 ioapic usbehci 66.2 11 36660721 362749665600 9894.78 msi ether0 66.1 10 395700 4099132008 10359.19 msi ether1 65.0 1 1 10404 10404.00 ioapic i8042 68.1 14 35 379197 10834.20 ioapic usbuhci 67.3 10 95 1065114 11211.73 ioapic usbehci 65.3 4 603 8503128 14101.37 ioapic COM1