Hi Keith,
On Tue, Jun 14, 2016 at 3:47 PM, Wiles, Keith <keith.wiles at intel.com> wrote:
>>> Normally the limitation is in the hardware, basically how the PCI bus is
>>> connected to the CPUs (or sockets). How the PCI buses are connected to the
>>> system depends on the Mother board design. I normally see the buses
>>> attached to socket 0, but you could have some of the buses attached to the
>>> other sockets or all on one socket via a PCI bridge device.
>>>
>>> No easy way around the problem if some of your PCI buses are split or all
>>> on a single socket. Need to look at your system docs or look at lspci it
>>> has an option to dump the PCI bus as an ASCII tree, at least on Ubuntu.
>>
>>This is the motherboard we use on our system:
>>
>>http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRX.cfm
>>
>>I need to swap some NICs around (as now we moved everything on socket
>>1) before I can share the lspci output.
>
> FYI: the option for lspci is ?lspci ?tv?, but maybe more options too.
>
I retested with two 10G X710 ports connected back to back:
port 0: 0000:01:00.3 - socket 0
port 1: 0000:81:00.3 - socket 1
I ran the following scenarios:
- assign 16 threads from CPU 0 on socket 0 to port 0 and 16 threads
from CPU 1 to port 1 => setup rate of 1.6M sess/s
- assign only the 16 threads from CPU0 for both ports (so 8 threads on
socket 0 for port 0 and 8 threads on socket 0 for port 1) => setup
rate of 3M sess/s
- assign only the 16 threads from CPU1 for both ports (so 8 threads on
socket 1 for port 0 and 8 threads on socket 1 for port 1) => setup
rate of 3M sess/s
I also tried a scenario with two machines connected back to back each
of which had a NIC on socket 1. I assigned 16 threads from socket 1 on
each machine to the port and performance scaled to 6M sess/s as
expected.
I double checked all our memory allocations and, at least in the
tested scenario, we never use memory that's not on the same socket as
the core.
I pasted below the output of lspci -tv. I see that 0000:01:00.3 and
0000:81:00.3 are connected to different PCI bridges but on each of
those bridges there are also "Intel Corporation Xeon E7 v3/Xeon E5
v3/Core i7 DMA Channel <X>" devices.
It would be great if you could also take a look in case I
missed/misunderstood something.
Thanks,
Dumitru
# lspci -tv
-+-[0000:ff]-+-08.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0
| +-08.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0
| +-08.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0
| +-09.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1
| +-09.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1
| +-09.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1
| +-0b.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
R3 QPI Link 0 & 1 Monitoring
| +-0b.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
R3 QPI Link 0 & 1 Monitoring
| +-0b.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
R3 QPI Link 0 & 1 Monitoring
| +-0c.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0d.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0d.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0f.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Buffered Ring Agent
| +-0f.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Buffered Ring Agent
| +-0f.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Buffered Ring Agent
| +-0f.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Buffered Ring Agent
| +-0f.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
System Address Decoder & Broadcast Registers
| +-0f.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
System Address Decoder & Broadcast Registers
| +-0f.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
System Address Decoder & Broadcast Registers
| +-10.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
PCIe Ring Interface
| +-10.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
PCIe Ring Interface
| +-10.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Scratchpad & Semaphore Registers
| +-10.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Scratchpad & Semaphore Registers
| +-10.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Scratchpad & Semaphore Registers
| +-12.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Home Agent 0
| +-12.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Home Agent 0
| +-12.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Home Agent 1
| +-12.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Home Agent 1
| +-13.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Target Address, Thermal & RAS Registers
| +-13.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Target Address, Thermal & RAS Registers
| +-13.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel Target Address Decoder
| +-13.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel Target Address Decoder
| +-13.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO Channel 0/1 Broadcast
| +-13.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO Global Broadcast
| +-14.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel 0 Thermal Control
| +-14.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel 1 Thermal Control
| +-14.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel 0 ERROR Registers
| +-14.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel 1 ERROR Registers
| +-14.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 0 & 1
| +-14.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 0 & 1
| +-14.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 0 & 1
| +-14.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 0 & 1
| +-16.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Target Address, Thermal & RAS Registers
| +-16.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Target Address, Thermal & RAS Registers
| +-16.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel Target Address Decoder
| +-16.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel Target Address Decoder
| +-16.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO Channel 2/3 Broadcast
| +-16.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO Global Broadcast
| +-17.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel 0 Thermal Control
| +-17.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel 1 Thermal Control
| +-17.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel 0 ERROR Registers
| +-17.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel 1 ERROR Registers
| +-17.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 2 & 3
| +-17.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 2 & 3
| +-17.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 2 & 3
| +-17.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 2 & 3
| +-1e.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1e.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1e.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1e.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1e.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1f.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU
| \-1f.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU
+-[0000:80]-+-02.0-[81]--+-00.0 Intel Corporation Ethernet
Controller X710 for 10GbE SFP+
| | +-00.1 Intel Corporation Ethernet
Controller X710 for 10GbE SFP+
| | +-00.2 Intel Corporation Ethernet
Controller X710 for 10GbE SFP+
| | \-00.3 Intel Corporation Ethernet
Controller X710 for 10GbE SFP+
| +-03.0-[82]----00.0 Intel Corporation Ethernet
Controller XL710 for 40GbE QSFP+
| +-03.2-[83]----00.0 Intel Corporation Ethernet
Controller XL710 for 40GbE QSFP+
| +-04.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 0
| +-04.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 1
| +-04.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 2
| +-04.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 3
| +-04.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 4
| +-04.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 5
| +-04.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 6
| +-04.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 7
| +-05.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Address Map, VTd_Misc, System Management
| +-05.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug
| +-05.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
RAS, Control Status and Global Errors
| \-05.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC
+-[0000:7f]-+-08.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0
| +-08.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0
| +-08.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0
| +-09.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1
| +-09.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1
| +-09.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1
| +-0b.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
R3 QPI Link 0 & 1 Monitoring
| +-0b.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
R3 QPI Link 0 & 1 Monitoring
| +-0b.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
R3 QPI Link 0 & 1 Monitoring
| +-0c.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0c.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0d.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0d.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Unicast Registers
| +-0f.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Buffered Ring Agent
| +-0f.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Buffered Ring Agent
| +-0f.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Buffered Ring Agent
| +-0f.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Buffered Ring Agent
| +-0f.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
System Address Decoder & Broadcast Registers
| +-0f.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
System Address Decoder & Broadcast Registers
| +-0f.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
System Address Decoder & Broadcast Registers
| +-10.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
PCIe Ring Interface
| +-10.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
PCIe Ring Interface
| +-10.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Scratchpad & Semaphore Registers
| +-10.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Scratchpad & Semaphore Registers
| +-10.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Scratchpad & Semaphore Registers
| +-12.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Home Agent 0
| +-12.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Home Agent 0
| +-12.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Home Agent 1
| +-12.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Home Agent 1
| +-13.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Target Address, Thermal & RAS Registers
| +-13.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Target Address, Thermal & RAS Registers
| +-13.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel Target Address Decoder
| +-13.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel Target Address Decoder
| +-13.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO Channel 0/1 Broadcast
| +-13.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO Global Broadcast
| +-14.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel 0 Thermal Control
| +-14.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel 1 Thermal Control
| +-14.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel 0 ERROR Registers
| +-14.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 0 Channel 1 ERROR Registers
| +-14.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 0 & 1
| +-14.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 0 & 1
| +-14.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 0 & 1
| +-14.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 0 & 1
| +-16.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Target Address, Thermal & RAS Registers
| +-16.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Target Address, Thermal & RAS Registers
| +-16.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel Target Address Decoder
| +-16.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel Target Address Decoder
| +-16.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO Channel 2/3 Broadcast
| +-16.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO Global Broadcast
| +-17.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel 0 Thermal Control
| +-17.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel 1 Thermal Control
| +-17.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel 0 ERROR Registers
| +-17.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Integrated Memory Controller 1 Channel 1 ERROR Registers
| +-17.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 2 & 3
| +-17.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 2 & 3
| +-17.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 2 & 3
| +-17.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DDRIO (VMSE) 2 & 3
| +-1e.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1e.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1e.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1e.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1e.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Power Control Unit
| +-1f.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU
| \-1f.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU
\-[0000:00]-+-00.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2
+-01.0-[01]--+-00.0 Intel Corporation Ethernet
Controller X710 for 10GbE SFP+
| +-00.1 Intel Corporation Ethernet
Controller X710 for 10GbE SFP+
| +-00.2 Intel Corporation Ethernet
Controller X710 for 10GbE SFP+
| \-00.3 Intel Corporation Ethernet
Controller X710 for 10GbE SFP+
+-02.0-[02]--+-00.0 Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection
| \-00.1 Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection
+-02.2-[03]--+-00.0 Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection
| \-00.1 Intel Corporation 82599ES 10-Gigabit
SFI/SFP+ Network Connection
+-03.0-[04]--
+-03.2-[05]--
+-04.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 0
+-04.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 1
+-04.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 2
+-04.3 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 3
+-04.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 4
+-04.5 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 5
+-04.6 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 6
+-04.7 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
DMA Channel 7
+-05.0 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
Address Map, VTd_Misc, System Management
+-05.1 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug
+-05.2 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7
RAS, Control Status and Global Errors
+-05.4 Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC
+-11.0 Intel Corporation C610/X99 series chipset SPSR
+-11.4 Intel Corporation C610/X99 series chipset sSATA
Controller [AHCI mode]
+-14.0 Intel Corporation C610/X99 series chipset USB
xHCI Host Controller
+-16.0 Intel Corporation C610/X99 series chipset MEI Controller #1
+-16.1 Intel Corporation C610/X99 series chipset MEI Controller #2
+-1a.0 Intel Corporation C610/X99 series chipset USB
Enhanced Host Controller #2
+-1c.0-[06]--
+-1c.3-[07-08]----00.0-[08]----00.0 ASPEED Technology,
Inc. ASPEED Graphics Family
+-1c.4-[09]--+-00.0 Intel Corporation I350 Gigabit
Network Connection
| \-00.1 Intel Corporation I350 Gigabit
Network Connection
+-1d.0 Intel Corporation C610/X99 series chipset USB
Enhanced Host Controller #1
+-1f.0 Intel Corporation C610/X99 series chipset LPC Controller
+-1f.2 Intel Corporation C610/X99 series chipset 6-Port
SATA Controller [AHCI mode]
+-1f.3 Intel Corporation C610/X99 series chipset SMBus Controller
\-1f.6 Intel Corporation C610/X99 series chipset Thermal Subsystem
