Hi,
I am trying to use an Intel 82576 NIC on an AMD IOMMU system with
SR-IOV. When I load the igb module with max_vfs=1 to enable a virtual
function I get IO_PAGE_FAULTS from the virtual functions. The relevant
part of dmesg is:
[ 45.788134] igb: Intel(R) Gigabit Ethernet Network Driver - version 3.4.7-k
[ 45.795090] igb: Copyright (c) 2007-2012 Intel Corporation.
[ 45.801049] igb 0000:02:00.0: irq 80 for MSI/MSI-X
[ 45.801056] igb 0000:02:00.0: irq 81 for MSI/MSI-X
[ 45.801061] igb 0000:02:00.0: irq 82 for MSI/MSI-X
[ 45.801067] igb 0000:02:00.0: irq 83 for MSI/MSI-X
[ 45.901445] pci 0000:02:10.0: [8086:10ca] type 00 class 0x020000
[ 45.901585] AMD-Vi: New device 0000:02:10.0
[ 45.906486] igb 0000:02:00.0: 1 VFs allocated
[ 45.937918] igbvf: Intel(R) Gigabit Virtual Function Network Driver -
version 2.0.1-k
[ 45.945751] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.
[ 46.071749] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 46.078605] igb 0000:02:00.0: eth5: (PCIe:2.5Gb/s:Width x4) 00:1b:21:49:2e:cc
[ 46.085804] igb 0000:02:00.0: eth5: PBA No: E43709-003
[ 46.090946] igb 0000:02:00.0: Using MSI-X interrupts. 2 rx queue(s), 1 tx
queue(s)
[ 46.098870] igb 0000:02:00.1: irq 84 for MSI/MSI-X
[ 46.098876] igb 0000:02:00.1: irq 85 for MSI/MSI-X
[ 46.098881] igb 0000:02:00.1: irq 86 for MSI/MSI-X
[ 46.098886] igb 0000:02:00.1: irq 87 for MSI/MSI-X
[ 46.104262] AMD-Vi: Using protection domain 23 for device 0000:02:00.0
[ 46.172988] IPv6: ADDRCONF(NETDEV_UP): eth5: link is not ready
[ 46.202875] pci 0000:02:10.1: [8086:10ca] type 00 class 0x020000
[ 46.203013] AMD-Vi: New device 0000:02:10.1
[ 46.207935] igb 0000:02:00.1: 1 VFs allocated
[ 46.373149] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 46.380019] igb 0000:02:00.1: eth1: (PCIe:2.5Gb/s:Width x4) 00:1b:21:49:2e:cd
[ 46.387213] igb 0000:02:00.1: eth6: PBA No: E43709-003
[ 46.392347] igb 0000:02:00.1: Using MSI-X interrupts. 2 rx queue(s), 1 tx
queue(s)
[ 46.400072] igbvf 0000:02:10.0: enabling device (0000 -> 0002)
[ 46.405977] igbvf 0000:02:10.0: irq 88 for MSI/MSI-X
[ 46.405983] igbvf 0000:02:10.0: irq 89 for MSI/MSI-X
[ 46.405988] igbvf 0000:02:10.0: irq 90 for MSI/MSI-X
[ 46.411492] AMD-Vi: Using protection domain 24 for device 0000:02:00.1
[ 46.480625] IPv6: ADDRCONF(NETDEV_UP): eth6: link is not ready
[ 46.486980] igbvf 0000:02:10.0: Intel(R) 82576 Virtual Function
[ 46.492895] igbvf 0000:02:10.0: Address: ce:5e:41:2f:36:ce
[ 46.498510] igbvf 0000:02:10.1: enabling device (0000 -> 0002)
[ 46.504394] igbvf 0000:02:10.1: irq 91 for MSI/MSI-X
[ 46.504400] igbvf 0000:02:10.1: irq 92 for MSI/MSI-X
[ 46.504405] igbvf 0000:02:10.1: irq 93 for MSI/MSI-X
[ 46.527012] igbvf 0000:02:10.1: Intel(R) 82576 Virtual Function
[ 46.532931] igbvf 0000:02:10.1: Address: 52:3e:8f:47:60:da
[ 46.573209] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000
address=0x000000021e170000 flags=0x0050]
[ 46.575620] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[ 46.589607] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000
address=0x000000021e170040 flags=0x0050]
[ 46.600186] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000
address=0x000000021e170080 flags=0x0050]
[ 46.610763] AMD-Vi: Event logged [IO_PAGE_FAULT device=02:10.0 domain=0x0000
address=0x000000021e1700c0 flags=0x0050]
[ 46.669940] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
The devices (physical and virtual) are not operational after this. I
think this is a problem in the igb or igbvf driver. The addresses
reported in the IO_PAGE_FAULTS are system-ram addresses are were not
handed out by the AMD IOMMU driver (the driver only hands out DMA
handles below 4GB). Also the reported domain is 0 which means that the
driver for that device has not yet issued _any_ call to the DMA-API. But
the device is doing an DMA write request (as seen in the flags). Any
ideas? Also let me know if you need any additional information.
Thanks,
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
